Azure Databricks DevOps Engineer
On-siteCenturion, Gauteng, South Africa
Job Summary
Deploy, configure, and maintain Azure Databricks environments while implementing infrastructure automation using Terraform, ARM templates, or Bicep. Develop and manage CI/CD pipelines with Azure DevOps, GitHub Actions, or Jenkins to automate data pipeline deployments. Set up monitoring solutions using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution, and troubleshoot Databricks, AKS, and networking incidents. Implement security best practices including RBAC, encryption, and network policies, then collaborate with Data Scientists and Engineers to optimize ETL pipelines and establish technical documentation.
Required Qualifications
- Bachelors or Masters degree in Computer Science, IT, or a related field
- 5+ years of experience in DevOps, Cloud Engineering, or Data Engineering roles
- Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics
- Proficiency in Terraform, ARM Templates, or Bicep for infrastructure automation
- Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins
- Knowledge of Python, Scala, or PowerShell for automation and scripting
- Familiarity with containerization (Docker, Kubernetes, AKS)
- Solid understanding of Azure networking, security, and identity management (IAM, RBAC, AAD)
- Experience with monitoring and logging tools (Azure Monitor, Log Analytics, Prometheus, Grafana)
- Strong problem-solving skills and ability to work in an Agile/Scrum environment
Desired Qualifications
- Azure Certifications (e.g., AZ-400, AZ-104, DP-203, DP-900)
- Experience with Big Data processing frameworks (Apache Spark, Delta Lake)
- Understanding of ML Ops and working with machine learning models in Databricks
- Knowledge of Event-driven architectures (Azure Event Hubs, Kafka, Service Bus)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.