Data Engineer - Multiple Positions
On-siteChesterfield, Missouri, United States
Chesterfield, Missouri, United StatesOn-siteContractBachelors DegreeSmall
ContractBachelors DegreeSmall
Job Summary
Design and deploy robust data pipelines integrating structured and unstructured data from diverse sources to support client business decisions. Architect complex solutions by developing scalable streaming and batch processes using Databricks, Spark Scala, and Python across AWS, Azure, and GCP. Implement DataOps principles, CI/CD workflows, and best practices to optimize development and deployment while managing risks within customer proposals. Facilitate knowledge transfer through training and reusable documentation to enhance team and client engagement effectiveness.
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience)
- 3+ years of experience as a Data Engineer
- Proficiency in at least two major cloud platforms (AWS, Azure, GCP)
- Proven experience in designing, developing, and implementing comprehensive data engineering solutions using Databricks, specifically for large-scale data processing and integration projects
- Experience in applying DataOps principles and implementing CI/CD and DevOps practices within data environments to optimize development and deployment workflows
- Expert-level proficiency in Spark Scala, Python, and PySpark
- In-depth knowledge of data architecture, including Spark Streaming, Spark Core, Spark SQL, and data modeling
- Hands-on experience with various data management technologies and tools, such as Kafka, StreamSets, and MapReduce
- Proficient in using advanced analytics and machine learning frameworks, including Apache Spark MLlib, TensorFlow, and PyTorch, to drive data insights and solutions
- Extensive experience in data migration from on-premises to cloud environments and in implementing data solutions on Databricks across cloud platforms (AWS, Azure, GCP)
- Skilled in designing and executing end-to-end data engineering solutions using Databricks, focusing on large-scale data processing and integration
- Proven hands-on experience with Databricks administration and operations, including notebooks, clusters, jobs, and data pipelines
- Experience integrating Databricks with other data tools and platforms to enhance overall data management and analytics capabilities
Desired Qualifications
- Certification in Databricks Engineering (Professional)
- Microsoft Certified: Azure Data Engineer Associate
- GCP Certified: Professional Google Cloud Certified
- AWS Certified Solutions Architect Professional
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.