Senior Data Engineer - Databricks
$140,000–$180,000 year
On-siteMcLean, Virginia, United States
Job Summary
Lead and architect data migrations using Databricks with a focus on performance, reliability, and scalability. Assess ETL jobs, workflows, data marts, and BI tools while addressing technical inquiries regarding customization, integration, and enterprise architecture. Inspect existing pipelines to discern their purpose and re-implement them efficiently in Databricks, manipulating structured and unstructured data within a Delta Lake architecture. Support an Agile software development lifecycle for teams building web-based interfaces and analytics models. This role contributes to the growth of the AI & Data Exploitation Practice at Steampunk, a change agent in federal contracting.
Required Qualifications
- Ability to hold a position of public trust with the US government
- 5-7 years industry experience coding commercial software
- 5-7 years direct experience in Data Engineering
- Experience working with database/data warehouse/data mart solutions in cloud
- Key must have skill sets – Databricks
- Key must have skill sets – SQL
- Key must have skill sets – PySpark/Python
- Key must have skill sets – AWS
- Experience working with database/data warehouse/data mart solutions in cloud (Preferably AWS. Alternatively Azure, GCP)
- Experience working with Data Lakehouse architecture and Delta Lake/Apache Iceberg
- Advanced working SQL knowledge and experience working with relational databases, query authoring and optimization (SQL) as well as working familiarity with a variety of databases
- Experience manipulating, processing, and extracting value from large, disconnected datasets
- Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in Databricks
- Experience manipulating structured and unstructured data
- Experience architecting data systems (transactional and warehouses)
- Experience with the SDLC, CI/CD, and operating in dev/test/prod environments
- Experience working in an Agile environment
- Experience supporting project teams of developers and data scientists who build web-based interfaces, dashboards, reports, and analytics/machine learning models
Desired Qualifications
- Excellent communication and customer service skills
- Passion for data and problem solving
- Experience working with database/data warehouse/data mart solutions in cloud (Preferably AWS. Alternatively Azure, GCP)
- Relational SQL (Preferably T-SQL. Alternatively pgSQL, MySQL)
- Data pipeline and workflow management tools: Databricks Workflows, Airflow, Step Functions, etc
- AWS cloud services: Databricks on AWS, S3, EC2, RDS (or Azure equivalents)
- Object-oriented/object function scripting languages: PySpark/Python, Java, C++, Scala, etc
- Experience with data cataloging tools such as Informatica EDC, Unity Catalog, Collibra, Alation, Purview, or DataZone
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.