Senior Data Engineer
On-siteArlington, Virginia, United States
Job Summary
Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities for mission-critical technology modernization. Translate complex requirements into secure, scalable solutions using Apache Spark, Delta Lake, and Python to support batch, streaming, and AI/ML workloads. Define engineering standards, mentor engineers, and establish data quality, governance, and observability practices across enterprise-scale teams. Partner with cybersecurity and mission stakeholders to ensure compliance and reliability. Requires U.S. citizenship, active DoW Secret clearance, and 9+ years of experience leading Databricks and Spark implementations.
Required Qualifications
- U.S. Citizens
- active DoW Secret (or higher) clearance
- Bachelor's degree in Computer Science, Engineering, or a related technical field
- 9+ years of relevant data engineering or software engineering experience
- Experience leading enterprise-scale data platform and pipeline implementations using Databricks
- Advanced proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake
- Experience architecting large-scale ETL/ELT, batch, and streaming pipelines
- Experience designing lakehouse architectures, data models, schemas, and data contracts
- Experience managing and optimizing Databricks jobs, workflows, compute resources, and Spark workloads
- Experience implementing data quality, monitoring, lineage, metadata management, and governance capabilities
- Experience with Unity Catalog or similar data-governance and access-control solutions
- Experience operating Databricks within AWS, Azure, or Google Cloud
- Experience with CI/CD, infrastructure as code, automated testing, and source control
- Strong understanding of data security, privacy, governance, and access-control principles
- Experience leading technical reviews, mentoring engineers, and communicating architecture decisions
Desired Qualifications
- Relevant Databricks certification
- Experience supporting DoW, federal, Advana, or other mission data environments
- Experience building data platforms in classified, regulated, or mission-critical environments
- Experience with Terraform, Databricks Asset Bundles, Airflow, or similar automation and orchestration tools
- Experience architecting streaming solutions using Kafka, Kinesis, Pulsar, or Spark Structured Streaming
- Experience supporting AI/ML pipelines, feature platforms, or MLflow
- Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.