Staff Engineer, Big Data
On-siteGurugram, Haryana, India
EXPIREDGurugram, Haryana, IndiaOn-siteFull TimeSenior LevelEnterprise
Full TimeSenior LevelEnterprise
Job Summary
Design, implement, and maintain scalable Big Data and cloud solutions using GCP, BigQuery, PySpark, Python, and Airflow. Develop and optimize data workflows, complex SQL queries, and serverless applications while containerizing workloads on Kubernetes. Implement Infrastructure as Code with Terraform, manage CI/CD pipelines, and configure security policies including IAM and VPC networking. Troubleshoot production issues, write comprehensive unit tests, and collaborate with cross-functional teams to improve operational efficiency.
Required Qualifications
- 5.5+ years
- Strong experience in Data Engineering, Cloud Engineering, or Big Data Engineering
- Must-have expertise in Google BigQuery, Python, SQL, PySpark, GCP fundamentals, and Kubernetes
- Strong hands-on experience with BigQuery and SQL, including writing and optimizing complex queries for large-scale data processing
- Strong programming experience in Python, with hands-on experience in developing scalable applications and APIs
- Strong experience with PySpark/Spark and familiarity with Big Data technologies such as Hive
- Hands-on experience with Apache Airflow or similar orchestration services for building and managing data workflows
- Experience with GCP serverless services, particularly Cloud Functions and Cloud Run
- Good experience with Docker and Kubernetes for containerization, deployment, orchestration, scaling, and troubleshooting
- Experience with Python API frameworks such as FastAPI, Flask, or Django
- Experience with CI/CD pipelines, preferably using GitLab CI/CD and Octopus Deploy, along with Git-based version control
- Good understanding of Terraform and Infrastructure as Code (IaC) for provisioning and managing cloud resources
- Good understanding of GCP networking, VPC, load balancing, IAM, API security, monitoring, logging, and alerting
- Strong troubleshooting, analytical, problem-solving, communication, and collaboration skills
- Bachelor's or master's degree in computer science, Information Technology, or a related field
Desired Qualifications
- FastAPI is preferred
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.