Data Test Engineer (SC Cleared)
RemoteIndia
Job Summary
Define and manage the overall test strategy across cloud-native data platforms, leading end-to-end testing for data pipelines, APIs, and distributed systems. Design scalable automated test frameworks integrated into GitLab CI/CD pipelines and oversee PySpark-based validation on AWS EMR. Lead quality assurance activities for large-scale data lakes containing structured and unstructured datasets, ensuring data integrity and transformation accuracy. Provide leadership and mentoring to QA teams while identifying and communicating quality risks to stakeholders. Report on test execution metrics and drive root cause analysis for production defects. This fully remote role supports UK operations within a UK-based consultancy, requiring an active SC clearance.
Required Qualifications
- Proven experience as a Test Manager, Lead Test Engineer, or QA Lead within cloud and data engineering environments
- Strong background in automated testing for cloud-native and data-centric platforms
- Experience testing large-scale data pipelines, transformation processes, and distributed systems
- Experience managing or leading QA/testing teams in Agile delivery environments
- Strong stakeholder management and communication skills
- Experience establishing test governance, reporting, and quality metrics
- Strong problem-solving and decision-making capability
- Experience working within regulated or security-conscious delivery environments
- SFIA Level 5 capability — provides leadership, influences strategy, and drives quality improvements
- Active SC clearance (mandatory)
- Strong proficiency in Python, SQL, Java, and scripting languages such as Bash
- Hands-on experience testing Big Data platforms and Data Lake solutions
- Strong experience with AWS EMR, Apache Airflow, and Jupyter Notebooks
- Experience performing PySpark-based data interrogation, validation, and testing
- Strong knowledge of AWS services including S3, Glue, Athena, EMR, CloudWatch, IAM, DynamoDB, EC2, and ECR
- Experience designing, implementing, and managing enterprise-scale automated test frameworks
- Strong understanding of CI/CD practices using GitLab pipelines
- Experience with infrastructure-as-code tools such as Terraform
- Understanding of dimensional data models and slowly changing dimensions
- Experience validating structured and unstructured data workflows
- Strong analytical and root cause analysis skills
- Experience leading and mentoring QA/Test Engineering teams
- Must be comfortable operating within UK regulatory, governance, and delivery frameworks
- Must be available for weekend shifts
- Must be able to lift 50 lbs
Desired Qualifications
- Experience in Big Data and Data Lake environments, including testing both structured and unstructured datasets
- Comfortable operating in highly automated CI/CD-driven delivery environments
- Experience with AWS Glue, Amazon Athena, AWS S3, AWS EMR, DynamoDB, and related cloud services
- Experience with Apache Airflow
- Experience with Jupyter Notebooks
- Experience with Docker and containerised testing environments
- Experience validating structured and unstructured data workflows
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.