Lead Data Engineer ID71008
On-siteSão Paulo, São Paulo, Brazil
Job Summary
Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale. Make architectural decisions on partitioning, file formats, schema strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake. Own the design of scheduled batch workflows on Airflow, defining pipeline structure, dependencies, and triggering strategy while driving architectural conversations. Drive use of the AWS data stack including S3-backed data lakes, Athena, and EKS, partnering directly with DevOps to clarify requirements. Review PRs to enforce code quality standards and guide senior developers to ensure alignment with established engineering practices. This role requires a high degree of autonomy to define problems, gather context, and shape technical approaches for ad hoc challenges.
Required Qualifications
- 7+ years of engineering experience
- proven track record designing and implementing ETL pipelines
- making architectural decisions for large-volume data systems
- Hands-on experience with OLAP-style analytical data architecture
- comparable experience with Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar
- OLAP depth
- hands-on experience designing against a data lake sitting on object storage (S3 or equivalent)
- partitioning strategy
- file formats (Parquet/ORC)
- cost/performance tradeoffs
- Deep familiarity with DAG-style workflow definition and triggering
- substantial prior Airflow experience
- enough depth in a comparable orchestrator (Dagster, Prefect, Luigi, Step Functions)
- practical comfort across the AWS data stack
- S3-backed data lake
- serverless query engines (Athena or equivalent)
- EKS/Kubernetes
- ability to drive infrastructure conversations with DevOps
- Backend proficiency in Python (FastAPI or Flask)
- Comfortable with REST and GraphQL
- Docker
- PostgreSQL
- Highly comfortable in Mac/Linux terminal-centric environments
- Practical, hands-on use of AI-assisted development tools (e.g., Claude Code)
- critical judgment to challenge AI output when it compromises long-term maintainability
- leadership presence to set the standard for how the team uses AI tooling responsibly
- ability to hold and defend a technical opinion
- challenging a stakeholder's or a tool's proposed quick fix with sound reasoning
- pursuit of a solution that scales and is maintainable long-term
- pragmatic enough to ship
- Comfort with ambiguity
- work is frequently ad hoc and underspecified
- defining the problem
- gathering context
- identifying constraints
- framing the work
- Upper-Intermediate English level
Desired Qualifications
- Direct production experience with Athena specifically
- Working knowledge of TypeScript/React
- Production AI features using AWS Bedrock, LangChain, Pydantic AI, or similar
- Monorepo tooling (Nx)
- modern package managers (Poetry, UV, Yarn)
- Redis/caching layers
- SageMaker
- Experience with marketing data structures
- campaign management APIs
- digital advertising metrics
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.