Senior Software Engineer - API & Data Connectors - MarTech/AdTech
On-siteSan Francisco, California, United States
Job Summary
Architect Connector Abstractions by designing and maintaining core Abstract Base Classes that establish clean, typed SDK-like interfaces for all data ingestion. Build High-Performance Native Connectors for local filesystems, Amazon S3, and Apache Iceberg using DuckDB relations and Apache Arrow primitives. Own the Flask Runtime Service to expose standardized operational contracts for catalog, check, discover, and read operations. Integrate Broad Source Catalogs by maintaining the optional PyAirbyte integration to bridge hundreds of enterprise data sources. Optimize Data Transfer Semantics to ensure low latency and high memory efficiency when materializing data references through pipeline execution layers.
Required Qualifications
- 5+ years writing production software
- deep mastery of Python 3.10+ (async/await, strict type hints, ABCs, Pydantic, and clean OOP abstraction patterns)
- Experience building and operating microservices/APIs using Flask or similar lightweight web frameworks
- a focus on high-throughput JSON/stream serialization
- Proven track record of building extensible SDKs, abstract interfaces, or pluggable framework architectures used by other engineers
- Deep experience with modern embedded analytical engines and columnar formats: DuckDB relations, Apache Arrow, and Apache Iceberg
- Hands-on experience with cloud storage ingestion patterns (Amazon S3, event-driven S3 file watchers, local storage abstractions)
- Familiarity with standard ELT/ETL lifecycle concepts (catalog, schema discovery, health check, read/stream)
- Experience integrating or extending data connector frameworks like PyAirbyte (airbyte>=0.20.0), Singer, or Meltano
- Experience handling complex Python dependency management and packaging (Poetry, pip-tools) for modular framework plugins
- Daily comfort with Linux, Docker, Git, automated testing (pytest), and CI/CD pipelines
Desired Qualifications
- Familiarity with custom event-driven file watching and streaming architectures
- Performance optimization experience dropping into Rust or C++ for bottleneck operations
- Experience working with low-overhead data reference passing engines (e.g., Flowbee DataRef)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.