Phyton with Spark Developer
On-site · Chennai, Tamil Nadu, India
Job Summary
Design and develop robust Python and PySpark data pipelines (ingestion, transformation, enrichment) using SQL, Spark SQL, and UDFs; write and optimize complex queries for analytics and reporting. Collaborate with data architects, data scientists, and analysts to translate requirements into technical specs. Unit-test, integrate-test, and review code; maintain CI/CD pipelines (Git, Jenkins, Docker) for automated builds and deployments; monitor production workloads and troubleshoot performance, memory, and job failures. Document data lineage, pipeline design, and operational run-books; stay current with technologies and contribute innovations (e.g., AI/ML). Participate in Agile ceremonies, mentor junior engineers, and promote best practices in Python, Spark optimization, and data-engineering patterns.
Required Qualifications
- Bachelor's degree or equivalent in Computer Science, Engineering, or a related field
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.