Senior Databricks Engineer
$140,000–$160,000 year
RemoteUnited States
Job Summary
Design and deploy scalable data infrastructure using the Databricks platform and Medallion Architecture to support the Department of State's "4Ps" data lake. Architect highly available pipelines with PySpark and SQL, while integrating third-party systems via RESTful APIs and implementing infrastructure-as-code with Databricks Asset Bundles. Establish data quality governance through monitoring dashboards and optimize cluster performance for cost-effectiveness. Mentor team members, conduct code reviews, and develop reusable Python libraries to standardize frameworks across projects. Work closely with project managers and business owners to translate requirements into production-ready ETL/ELT processes.
Required Qualifications
- Secret security clearance
- 8+ years of dedicated Data Engineering experience
- at least 3+ years functioning in a senior capacity focused on the Databricks ecosystem
- Expert-level proficiency in PySpark and complex SQL
- Proven ability to create, package, and manage custom Python libraries (e.g., building Wheel files) for scalable reuse across multiple data pipelines and projects
- Deep understanding of HTTP REST methods (GET, POST, PUT, etc.) with hands-on experience extracting, paginating, and processing data from external API sources
- Deep theoretical and practical understanding of modern Data Warehousing concepts and hands-on experience building out the Medallion Architecture
- Proven experience automating Databricks deployments
- hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps (creating YAML pipelines, managing service principals, and handling multi-workspace deployments)
- Strong familiarity with the broader cloud ecosystem surrounding Databricks (e.g., Azure Data Lake Storage Gen2, Azure Key Vault)
- Proficiency with Git-based version control workflows
- Concise and comprehensive written and oral communication skills for both technically understanding and customer usage support
- Bachelor's Degree
Desired Qualifications
- Experience with a wide range of data products, BI tools, and visualization libraries such as Tableau, PowerBI, Jupyter Notebooks, d3js, and Analytics Search Tools such as Elasticsearch, SOLR, Lucene
- Experience working with HR application (PeopleSoft) data for reporting purposes
- 3+ year(s) .NET, Java or similar OOP experience
- Experience writing Oracle PL/SQL queries, packages, and procedures
- Experience architecting Data Vault 2.0 solutions
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.