Software Engineer II - Databricks
On-siteBournemouth, England, United Kingdom
Job Summary
Design and implement batch and streaming data pipelines using Databricks, Spark, Delta Lake, and orchestrators like Airflow. Develop and optimize PySpark and SQL jobs for performance, reliability, and cost efficiency while building curated data models and automated testing. Manage cluster configurations, monitor production workloads, and resolve incidents. Integrate diverse data sources and apply governance best practices including access controls and lineage. Implement CI/CD for notebooks and code, leveraging enterprise-authorized AI coding tools to improve delivery speed and validate outputs through peer review. Create documentation for pipelines and operational runbooks.
Required Qualifications
- Experience with building data pipelines on Databricks and/or Apache Spark in production
- Strong coding skills in Python (PySpark)
- Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM)
- Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data Factory, etc.)
- Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS)
- Solid understanding of data engineering fundamentals: data modeling, ETL/ELT patterns, reliability, observability, and performance tuning
- Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, testing, troubleshooting, or documentation) with demonstrated ability to critically evaluate and validate AI-generated outputs
- Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations
Desired Qualifications
- Experience with streaming (Structured Streaming, Kafka/Event Hubs/Kinesis)
- Experience implementing data quality frameworks (Great Expectations, DQ) and data testing in CI
- Exposure to Unity Catalog (or similar) for governance and fine-grained permissions
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.