Senior Data Engineer - Remote (USA)
$89,649–$152,404 year
On-siteReston, Virginia, United States
Job Summary
Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow, while developing and deploying workflows on the Databricks platform. Create API services for data integration, build interactive visualizations with AWS QuickSight, and construct infrastructure for optimal extraction, transformation, and loading from various sources. Monitor infrastructure performance, develop data quality and validation jobs, and assemble complex datasets meeting business requirements. Write unit and integration tests, collaborate with DevOps engineers on CI, CD, and IaC, perform code reviews, and improve data availability through frequent refreshes and tiered storage. Maintain security and privacy for data at rest and in transit. Requires Public Trust clearance and US residency. ICF's Health Engineering Systems team partners with customers to articulate and execute visions for success.
Required Qualifications
- Bachelor's degree
- 7+ years of hands-on software development experience
- 4+ years of experience building data pipelines using Python, Java, and cloud technologies
- hands-on experience leveraging Spark and Hive for large-scale data processing
- Candidate must be able to obtain and maintain a Public Trust clearance
- Candidate must reside in the US
- be authorized to work in the US
- work must be performed in the US
- Must have lived in the US 3 full years out of the last 5 years
Desired Qualifications
- Experience building job workflows with the Databricks platform
- Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, and Amazon SNS
- Familiar with work to build processes that support data transformation, workload management, data structures, dependency and metadata
- Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users
- Experienced in data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up
- Demonstrated understanding using software and tools including relational NoSQL and SQL databases including Cassandra and Postgres
- workflow management and pipeline tools such as Airflow, Luigi and Azkaban
- stream-processing systems like Spark-Streaming and Storm
- object function/object-oriented scripting languages including Scala, C++, Java and Python
- Familiar with DevOps methodologies, including CI/CD pipelines (Github Actions) and IaC (Terraform)
- Ability to obtain and maintain a Public Trust
- residing in the United States
- Experience with Agile methodology, using test-driven development
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.