Databricks SME (Subject Matter Expert)
RemoteUnited States
Job Summary
Lead the design, development, and optimization of Databricks-based data platforms and pipelines while building scalable ETL/ELT workflows using Apache Spark. Provide technical expertise on Databricks architecture, performance tuning, and best practices to support enterprise-level data engineering and analytics initiatives. Collaborate with cross-functional teams to gather requirements, implement Delta Lake and data governance frameworks, and support migration of legacy data platforms. Optimize data processing workflows for performance and cost efficiency, mentor development teams on cloud-native solutions, and troubleshoot production issues. Ensure adherence to security, compliance, and coding standards. Work remotely with global clients and teams, leveraging expertise in Python, PySpark, SQL, and Scala within modern lakehouse architectures.
Required Qualifications
- 10+ years of experience in Data Engineering and Big Data technologies
- Strong hands-on expertise in Databricks and Apache Spark
- Experience with Python, PySpark, SQL, and Scala
- Expertise in Delta Lake, Unity Catalog, and Databricks Workflows
- Strong understanding of data warehousing and modern lakehouse architecture
- Experience with cloud platforms such as Microsoft Azure, Amazon AWS, or Google GCP
- Experience integrating Databricks with BI and reporting tools
- Knowledge of CI/CD pipelines and DevOps practices
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management abilities
Desired Qualifications
- Databricks Certified Professional certification
- Experience in real-time streaming using Kafka or Structured Streaming
- Exposure to AI/ML workflows within Databricks
- Prior experience working with global remote teams
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.