Platform Architect
$384–$384 year
HybridLondon, England, United Kingdom
Job Summary
Assess existing workloads to select appropriate Azure Databricks compute models, then configure and implement Serverless capabilities across notebooks, jobs, SQL workloads, and data pipelines. Develop workload placement standards, establish mandatory tagging standards, and integrate automated validation into CI/CD pipelines to enforce governance. Monitor platform performance and costs, delivering granular cost attribution and reporting while identifying oversized or underutilised resources. Design, build, and optimise scalable data ingestion and transformation solutions using Python, PySpark, and Delta Lake to support analytics, BI, and machine learning. Enhance the Databricks Discovery Zone to support migration and modernisation of analytics and data science workloads currently running on POSIT/RStudio. Collaborate effectively with data engineers, data scientists, architects, and security teams to drive continuous optimisation across architecture, engineering, and operational support.
Required Qualifications
- BPSS
- Possess deep hands-on experience implementing, administering, and troubleshooting enterprise-scale Azure Databricks platforms in production
- Have strong expertise in Databricks Serverless architecture, workload placement, compute optimisation, Databricks SQL, Delta Lake, and query and workload performance optimisation
- Have strong knowledge of Azure networking, identity, security, monitoring, secrets management, and private connectivity
- Possess advanced Python, PySpark, and SQL development skills within enterprise-scale data engineering environments
- Have experience with jobs, workflows, orchestration, incremental processing, CDC, data quality, reconciliation, and operational monitoring
- Have experience implementing FinOps practices, tagging strategies, cost attribution, monitoring, budget management, and cost optimisation
- Have experience integrating REST APIs and external data sources within enterprise data platforms
- Demonstrate strong understanding of platform governance, security, compliance, operational support, and controlled delivery within complex or regulated organisations
- Collaborate effectively with data engineers, data scientists, architects, security teams, platform teams, and business stakeholders
- Demonstrate the ability to work across architecture, engineering, implementation, optimisation, and operational support without dependence on specialist teams
- Communicate effectively and produce clear technical documentation, standards, operational procedures, and architectural decisions
- Candidates must be legally authorized to live and work in the country where the position is based, without requiring employer sponsorship
Desired Qualifications
- Have experience migrating analytics and data science workloads from POSIT/RStudio to Databricks; highly desirable
- Have experience converting R-based workloads and libraries to modern data platform solutions; highly desirable
- Possess AI/ML experience, including LLM integration, RAG/vector retrieval, model serving, and model lifecycle management; highly desirable
- Have experience delivering large-scale platform transformation programmes; highly desirable
- Have experience operating within highly regulated or complex enterprise environments; highly desirable
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.