Lead Data Engineer ID71008
On-siteMedellín, Antioquia, Colombia
Job Summary
Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale. Make architectural decisions on partitioning, file formats, schema strategy, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. Own the design of scheduled batch workflows on Airflow, defining pipeline structure, dependencies, and triggering strategy while driving architectural conversations. Drive use of the AWS data stack including Athena and EKS, partnering with DevOps to clarify requirements. Review PRs and enforce code quality standards while guiding senior developers to ensure alignment with established engineering practices. Define problem scope and technical approach for ad hoc or underspecified challenges before implementation.
Required Qualifications
- 7+ years of engineering experience
- proven track record designing and implementing ETL pipelines
- making architectural decisions for large-volume data systems
- Hands-on experience with OLAP-style analytical data architecture
- comparable experience with Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar
- OLAP depth
- hands-on experience designing against a data lake sitting on object storage
- S3 or equivalent
- serverless engine
- partitioning strategy
- file formats (Parquet/ORC)
- cost/performance tradeoffs
- Deep familiarity with DAG-style workflow definition and triggering
- substantial prior Airflow experience
- or enough depth in a comparable orchestrator (Dagster, Prefect, Luigi, Step Functions)
- practical comfort across the AWS data stack
- S3-backed data lake
- serverless query engines (Athena or equivalent)
- EKS/Kubernetes
- ability to drive infrastructure conversations with DevOps
- Backend proficiency in Python
- FastAPI or Flask
- Comfortable with REST and GraphQL
- Docker
- PostgreSQL
- Mac/Linux terminal-centric environments
- Practical, hands-on use of AI-assisted development tools
- critical judgment to challenge AI output
- leadership presence to set the standard for how the team uses AI tooling
- ability to hold and defend a technical opinion
- challenging a stakeholder's or a tool's proposed quick fix
- pursuit of a solution that scales and is maintainable long-term
- pragmatic enough to ship
- Comfort with ambiguity
- Upper-Intermediate English level
Desired Qualifications
- Direct production experience with Athena specifically
- Working knowledge of TypeScript/React
- Production AI features using AWS Bedrock, LangChain, Pydantic AI, or similar
- Monorepo tooling (Nx)
- modern package managers (Poetry, UV, Yarn)
- Redis/caching layers
- SageMaker
- Experience with marketing data structures
- campaign management APIs
- digital advertising metrics
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.