Lead Software Engineer - Python, Databricks, AWS
On-siteGlasgow, Scotland, United Kingdom
Job Summary
Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, and Databricks on AWS to transform large-scale financial datasets. Execute creative software solutions involving design, development, and technical troubleshooting, while writing secure, high-quality production code and reviewing work from others. Lead communities of practice, mentor Associate-level engineers, and document data flows to ensure consistency and knowledge sharing. Support data migration initiatives transitioning legacy systems to cloud-based warehouses and identify automated remediation for recurring issues to improve operational stability. Collaborate with stakeholders to develop data management strategies and participate fully in the SDLC from requirements gathering through maintenance.
Required Qualifications
- Proven experience in data management, ETL/ELT pipeline development, and large-scale data processing
- Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning
- Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar)
- Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.)
- Strong understanding of data quality, security, and lineage best practices
- Experience with cloud-based data warehouse migration and modernization
- Proficient in CI/CD, continuous delivery methods (Jules/Jenkins, Spinnaker, Sonar), the full Software Development Life Cycle, and Agile methodologies
- In-depth knowledge of the financial services industry and their IT systems
- Excellent problem-solving, troubleshooting, and analytical skills with ability to investigate data issues, identify root causes, and implement solutions
- Strong communication and documentation abilities, with the ability to collaborate effectively with business and technical stakeholders
Desired Qualifications
- Knowledge of regulatory reporting and financial data aggregation techniques
- Experience with data orchestration tools (Airflow, Step Functions, etc.)
- Databricks or AWS certifications
- Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning)
- Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.