Senior AWS Data Engineer
On-siteSaidapet, State of Tamil Nādu, Republic of India or Saidapet, State of Tamil Nādu, Republic of India
Job Summary
Design, build, and optimize scalable data pipelines using PySpark, Databricks, and SQL on AWS cloud platforms. Implement batch and streaming ingestion frameworks for structured, semi-structured, and unstructured data while developing reusable ETL/ELT components. Perform data transformation, cleansing, validation, and enrichment with Python, then build and maintain data models and marts supporting BI, analytics, and AI initiatives. Ensure pipelines are well-tested, monitored, and robust with proper logging, alerting, and performance optimization for large datasets. Collaborate with analysts, scientists, and business users to understand requirements and drive continuous improvement in engineering practices. Work in a hybrid environment within Agilisium, a Life Sciences AI services company, applying best practices in software engineering, version control, and agile development.
Required Qualifications
- 4 to 6 years of professional experience in Data Engineering or a related field
- Strong programming experience with Python
- experience using Python for data wrangling, pipeline automation, and scripting
- Deep expertise in writing complex and optimized SQL queries on large-scale datasets
- Solid hands-on experience with PySpark
- distributed data processing frameworks
- Expertise working with Databricks
- Experience with AWS cloud services such as S3, Glue, EMR, Athena, Redshift, and Lambda
- Practical understanding of ETL/ELT development patterns
- data modeling principles (Star/Snowflake schemas)
- Experience with job orchestration tools like Airflow, Databricks Jobs, or AWS Step Functions
- Understanding of data lake, lakehouse, and data warehouse architectures
- Familiarity with DevOps and CI/CD tools for code deployment (e.g., Git, Jenkins, GitHub Actions)
- Strong troubleshooting and performance optimization skills in large-scale data processing environments
- Excellent communication and collaboration skills
- ability to work in cross-functional agile teams
Desired Qualifications
- AWS or Databricks certifications (e.g., AWS Certified Data Analytics, Databricks Data Engineer Associate/Professional)
- Exposure to data observability, monitoring, and alerting frameworks (e.g., Monte Carlo, Datadog, CloudWatch)
- Experience working in healthcare, life sciences, finance, or another regulated industry
- Familiarity with data governance and compliance standards (GDPR, HIPAA, etc.)
- Knowledge of modern data architectures (Data Mesh, Data Fabric)
- Exposure to streaming data tools like Kafka, Kinesis, or Spark Structured Streaming
- Experience with data visualization tools such as Power BI, Tableau, or QuickSight
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.