Lead / Principal Data Engineer - DBT & Databricks
RemoteColombia or Costa Rica
Job Summary
Design, build, and maintain DBT models on Azure Databricks using Delta Lake, Unity Catalog, and Medallion Architecture principles to deliver curated Gold-layer data domains. Leverage AI-assisted development tools for model generation, refactoring, and documentation while establishing comprehensive testing strategies including schema validation and automated quality controls. Define data contracts, enforce governance standards, and implement observability practices across data platforms. Drive modernization initiatives and contribute to engineering playbooks for DBT, Databricks, and AI-powered development. Measure code adoption and delivery outcomes to continuously improve productivity.
Required Qualifications
- 5+ years of experience in Data Engineering
- At least 2 years of hands-on experience building and operating production-grade DBT projects, including models, tests, macros, snapshots, and large-scale refactoring
- Strong SQL skills
- Solid Python programming experience
- Hands-on experience with Azure Databricks, including Delta Lake, Unity Catalog, and Medallion Architecture
- Strong knowledge of DBT testing frameworks, data validation, and quality assurance best practices
- Familiarity with tools such as dbt-expectations, Great Expectations, or Delta Live Tables expectations
- Experience working with Git-based pull request workflows, code reviews, and CI/CD deployment pipelines
- Experience using AI-powered development tools such as Claude Code, GitHub Copilot, Cursor, or similar solutions
- Ability to independently own data domains and collaborate directly with business stakeholders
- Strong communication, analytical, and problem-solving skills
Desired Qualifications
- Experience curating Databricks Genie environments or building semantic layers for natural-language analytics
- Experience within Insurance or Financial Services domains
- Experience working with policy, claims, exposure, or regulatory reporting data
- Hands-on experience with PySpark for large-scale data transformations
- Experience with Databricks Workflows, Airflow, or similar orchestration platforms
- DBT and/or Databricks certifications
- Experience implementing AI-driven modernization and migration programs
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.