Data Quality Engineer (Databricks)
On-site · Mumbai, Maharashtra, India
Job Summary
Data Quality Engineer (Databricks) responsible for designing, implementing, and operating ADC's enterprise data quality framework within the Databricks platform. Deliver automated profiling, quality rule execution, cleansing, monitoring, remediation support, and quality reporting across 170 datasets and 1,346 prioritised Critical Data Elements (CDEs). Collaborates with Data Modellers, Data Catalogue Specialists, business data owners, and platform engineers to establish scalable, reusable quality controls that improve trust, accuracy, completeness, consistency, timeliness, validity, and uniqueness across the data estate. Key duties include Databricks platform configuration/administration (clusters, PySpark notebooks, Delta Lake structures, Unity Catalog), data profiling and quality assessments across six dimensions (completeness, accuracy, consistency, validity, timeliness, uniqueness), developing a reusable Data Quality Rule Factory to automate thousands of rules, embedding quality gates within Bronze/Silver/Gold pipelines, automated remediation and cleansing with PySpark, MLflow-enabled anomaly detection and deduplication, exception handling with a Failed Record Register, and delivering governance artefacts, dashboards, and audit trails.
Required Qualifications
- Strong experience designing and implementing enterprise Data Quality frameworks
- Advanced Databricks engineering experience
- Strong PySpark development skills
- Experience with Delta Lake
- Unity Catalog
- Databricks Workflows and Jobs
- Databricks SQL
- Experience building scalable data validation and quality rule frameworks
- Knowledge of machine learning techniques for anomaly detection and data quality monitoring
- Experience using MLflow for model management and deployment
- Strong understanding of data governance, metadata management, and data lifecycle processes
- Experience integrating data quality metrics into reporting platforms such as Power BI
- Knowledge of cloud-based data engineering and modern lakehouse architectures
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.