Data Engineer III – Databricks & Python
On-site · Glasgow, Scotland, United Kingdom
Job Summary
Data Engineer III on the External Regulatory Financial Control (ERFC) Technology team who designs, develops, and maintains scalable data pipelines using Python and PySpark on Databricks platform on AWS. Responsibilities include processing and transforming large-scale financial datasets, optimizing queries and workflows for performance at scale, implementing data quality checks and monitoring, participating in the full SDLC, and providing Production Support and SRE functions. The role collaborates with Product Owners and end users to support business use cases and leverages AI tools to rapidly build and test new data pipelines (e.g., CoPilot, Claude Code).
Required Qualifications
- Strong hands-on experience in data engineering or related roles
- Strong proficiency in Python and PySpark for large-scale data processing
- Demonstrated experience with Databricks platform and Apache Spark ecosystem
- Proven track record of building and optimizing data pipelines for big data workloads
- Strong SQL skills with experience in query optimization and performance tuning
- Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda)
- Strong analytical skills with ability to investigate data issues, identify root causes, and implement solutions
- Experience with the complete SDLC, Jenkins, Spinnaker, Sonar and Agile methodologies
- Bachelor's degree in Computer Science, Engineering, Mathematics, or related technical field
- Experience with data quality checks, monitoring, and alerting mechanisms
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.