Big Data Engineer
$110,000–$125,000 year
HybridLansing, Michigan, United States
Job Summary
Design, develop, and optimize large-scale data processing pipelines using Spark, Scala, and Hive. Build and maintain data ingestion workflows with Apache NiFi while developing real-time streaming solutions via Apache Kafka. Monitor application logs and system health using Kibana, Elasticsearch, and Logstash, and implement CI/CD pipelines with Git, Jenkins, and Nexus. Ensure data quality, integrity, and governance across platforms, participate in architecture discussions, and troubleshoot complex production issues. Collaborate with cross-functional teams in an Agile environment, mentor junior engineers, and contribute to technical best practices. Requires 8+ years of Big Data experience with hands-on expertise in the Hadoop ecosystem. Located in Chicago, IL with hybrid onsite options.
Required Qualifications
- 8+ years of experience in Big Data technologies
- Strong hands-on experience with Spark, Scala, Hive (HQL), and the Hadoop ecosystem
- Experience implementing Apache NiFi and Apache Kafka
- Proficiency in Shell Scripting or Python
- Experience with PostgreSQL
- Familiarity with scheduling tools such as Oozie and Control-M
- Working knowledge of Git, Jenkins, Nexus, Maven, Ansible, and SonarQube
- Strong understanding of CI/CD practices
- Experience in client-facing roles, including requirement gathering workshops
- Minimum of two end-to-end Big Data implementation projects
- Ability to work independently and contribute effectively within Agile teams
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.