Databricks Platform Architect
RemoteUnited States
Job Summary
Lead architectural design and migration strategies to transition 150+ complex SQL Server stored procedures into PySpark and structured declarative pipelines on the Databricks Lakehouse. Define control flows, implement performance tuning for sub-30-second reporting SLAs, and deploy enterprise orchestration using Apache Airflow or Databricks Workflows. Establish data governance models, schema evolution patterns, and AI-assisted code quality frameworks while leading design reviews and pair-programming sessions to accelerate team knowledge transfer. This role requires 10+ years of experience in data architecture and cloud migrations, with a focus on modernizing legacy financial and tax allocation engines.
Required Qualifications
- Deep expert-level knowledge of Databricks (Lakehouse architecture, Delta Lake, Unity Catalog) and Apache Spark / PySpark
- Strong background in relational databases, with advanced proficiency in SQL Server, T-SQL, and Stored Procedures
- Ability to reverse-engineer and refactor legacy database logic into distributed paradigms
- Hands-on experience with Apache Airflow or similar modern workflow orchestrators
- Proven track record in cost optimization (FinOps), cluster tuning, autoscaling configurations, and handling skewed data profiles
- Experience with Infrastructure as Code (Terraform), data build tool (dbt), testing frameworks (PyTest), and automated Git-based workflows
- 10+ years of experience in Data Engineering/Architecture
- At least 3+ years specifically leading large-scale cloud data migrations
- Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field
Desired Qualifications
- Databricks Certified Data Engineer Associate / Professional
- Databricks Certified Solutions Architect
- AWS Certified Database Specialist or equivalent Cloud Certifications
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.