Lead Software Engineer - Databricks, Spark, AWS
On-site · Plano, Texas, United States
Job Summary
Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Spark; establish lakehouse patterns with Delta Lake; own Databricks cluster strategy; orchestrate jobs with Databricks Workflows; design secure data ingestion and transformation frameworks leveraging AWS services (S3, Glue, IAM, Secrets Manager, CloudWatch, Lambda, Kinesis/MSK); enforce data quality, lineage, and governance with Unity Catalog/Glue Catalog; drive Spark performance engineering; build reusable libraries in Python/Java; implement CI/CD for data projects; collaborate with platform security; lead incident response and reliability improvements. Includes extensive experience with Databricks, Spark, AWS, data engineering, data governance, and leadership in engineering best practices.
Required Qualifications
- Formal training or certification on software engineering concepts and 5+ years applied experience
- 10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR
- Strong proficiency in Python and/or Java for data processing, platform tooling, and automation
- Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses)
- Solid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB
- Proven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering
- Deep skills in Spark performance tuning and Databricks cluster setup/optimization
- Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices)
- Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls
- Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.