Data & Software Engineer
On-siteChantilly, Virginia, United States
Job Summary
Design data flows for a custom application by working with stakeholders to assess feasibility and build production pipelines and ETL workflows at scale. Debug data quality issues and performance bottlenecks while leveraging advanced Python, PySpark, and orchestration tools like Airflow. Containerize and deploy solutions in AWS environments using Terraform, and manage complex analytical workloads with SQL, NoSQL, and geospatial tools. Contribute to documentation, design patterns, and data lineage tracking within Unity Catalog and Apache Polaris. Support end-to-end pipeline development, including integration of AI/ML services and Bash scripting for automation.
Required Qualifications
- TS/SCI FSP Clearance
- Advanced Python programming skills
- familiarity with Java
- an understanding of data security, privacy, governance and compliance principles
- a demonstrated history of building production data pipelines and ETL workflows at scale
- Minimum of 5 years' experience with Apache Spark & PySpark
- Advanced Python skills (including Pandas & NumPy)
- Docker, Podman
- AWS S3, Lambda & Step functions
- Apache Iceberg, Airflow, etc.
- SQL (with Trino)
- NoSQL, DynamoDB
- Unity Catalog OSS, Apache Polaris
- Apache Superset
- Terraform or CloudFormation
- OpenLineage
- H3, PostGIS
- Building end-to-end data pipelines leveraging Python
- Using orchestration tools to deploy data pipelines, including configuring and updating Spark Jobs
- Containerizing and deploying applications in cloud environments like AWS
- Working with MySQL and PostgreSQL including performance tuning, schema design, and query optimization for complex, analytical workloads
- Leveraging industry standard tools for code control (Git, IaaC control, etc.)
- Working with data catalogs, tracking data lineage and handling a variety of data formats, including Geospatial
- Using Bash scripting for automation and data processing tasks
- Integrating Al/ML services and models
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.