Data Engineer in Test
On-siteIslamabad, Islamabad, Pakistan or Lahore, Punjab, Pakistan
Job Summary
Write PySpark and SQL scripts to validate data pipelines, transformations, and integrations using Azure Data Lake, Synapse, and Data Factory. Design and run automated tests for data validation, storage, and retrieval while continuously enhancing test strategies including smoke, performance, functional, and regression testing. Collaborate with development teams to assess workflow changes, participate in data reconciliation, and maintain documentation of testing processes and findings. Communicate testing progress to stakeholders, highlight blockers, and ensure alignment with business objectives. Identify and resolve issues in complex Big Data environments, manage multiple projects simultaneously, and confidently challenge existing assumptions to enhance data engineering practices.
Required Qualifications
- Bachelor's degree in computer science or related discipline
- 3-6 years of hands-on experience in Data Engineering particularly in Data Lake environments on Azure's cloud platform
- Proficient in Azure Data Factory, Azure Synapse Analytics and Databricks for big data processing and scaled data quality checks
- Proficiency in SQL, capable of writing and optimizing both simple and complex queries for data validation and testing purposes
- Proficient in PySpark, with experience in data manipulation and transformation, and a demonstrated ability to write and execute test scripts for data processing and validation
- Hands-on experience with Functional & system integration testing in big data environments, ensuring seamless data flow and accuracy across multiple systems
- Knowledge and ability to design and execute test cases in a behaviour-driven development environment
- Fluency in Agile methodologies, with active participation in Scrum ceremonies and a strong understanding of Agile principles
- Familiarity with tools like Jira, including experience with X-Ray or Jira Zephyr for defect management and test case management
- Proven experience working on high-traffic and large-scale software products, ensuring data quality, reliability, and performance under demanding conditions
- Identify and resolve issues in complex Big Data environments, ensuring data accuracy and system efficiency
- Initiate and manage tasks, delivering high-quality results within deadlines in a dynamic setting
- Confidently challenge existing assumptions and processes to enhance data engineering and QA practices
- Maintain organized testing schedules and documentation, ensuring thorough coverage of all scenarios
- Exhibit strong interpersonal skills, working effectively in cross-functional, multicultural teams with minimal supervision
- Comfortable in Agile and Scrum environments, proficient with tools like Jira, and adaptable to changing priorities
- Manage multiple projects simultaneously, demonstrating strong planning and organizational skills to ensure timely task completion
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.