Senior Data Engineer
On-siteSydney, New South Wales, Australia
Job Summary
Design and own scalable batch and streaming data architectures, selecting optimal storage, compute, and processing patterns to ensure reliability and cost efficiency. Develop and maintain high-quality ETL/ELT pipelines at scale, handling incremental loads, CDC, and failure recovery while implementing robust data quality checks and monitoring. Lead technical collaboration by reviewing code and data models, translating business requirements into technical solutions, and communicating trade-offs to stakeholders and leadership. Mentor junior and mid-level engineers while driving engineering best practices across the organization. This role combines architecture ownership with business impact, focusing on platforms that power analytics, reporting, and advanced use cases like machine learning and real-time insights.
Required Qualifications
- Strong hands‐on experience with Azure Data Lake Gen2, Synapse, Databricks
- Solid experience in PySpark / Spark SQL, Databricks performance tuning & job optimization
- Proficient in Complex SQL (window functions, CTEs, execution plans)
- Strong understanding in Mentoring junior and mid‐level engineers, Communicating trade‐offs to architects and product teams
- Experience working with Azure DevOps or GitHub Actions
- 6 years
Desired Qualifications
- Experience with Streaming & Real‐Time Data Processing on Designing low‐latency pipelines
- Exposure to Data Governance & Metadata Management in Data classification & sensitivity labels
- Familiarity with Advanced Python Engineering Practices on Performance‐aware Python design
- Experience working in Analytics & BI Awareness
- Knowledge of Cloud Cost Optimization & FinOps
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.