Data Engineer, DevOps
On-siteSouth San Francisco, California, United States
Job Summary
Design, implement, and deploy cloud infrastructure including managed databases, application servers, and data warehouses to power the biological data factory's robots, instruments, and machine learning platform. Collaborate with scientists and bioengineers to develop data architectures on high-throughput platforms, identify architectural performance improvements, and join an on-call rotation to ensure maximum productivity. Manage medium-sized datasets in object storage systems like S3, define infrastructure following compliance standards such as GDPR and HIPAA, and deploy monitoring and alerting for scalable services. Requires 2-3 years of AWS experience, proficiency in Linux, Python, and Terraform, with a passion for making a difference in the world.
Required Qualifications
- 2-3 years of experience with provisioning AWS cloud services
- Experience with cloud configuration and resource management tools such as Terraform
- Experience architecting reliable infrastructure platforms including monitoring and alerting, load balancing, scalable services, multi-region
- Experience with at least one high-end distributed data processing environment (Hadoop, Spark, etc)
- Experience with batch computing systems such as AWS Batch, SLURM
- Experience with container build and deployment systems like Docker, Kubernetes, or ECS
- Ability to communicate effectively and collaborate with people of diverse backgrounds and job functions
- Proficiency in Linux environment (including shell scripting and Python programming)
- Experience with database languages (e.g., SQL, No-SQL)
- Experience with version control practices and tools (Git, Mercurial, etc.)
- Experience with biological data
- Experience with managing medium-sized data sets (100TB+) in object storage systems like S3
- Experience with defining infrastructure following compliance (GDPR, HIPAA, etc.)
- Experience with data processing pipelines
- Experience with deploying and monitoring machine learning models in a production environment
Desired Qualifications
- Experience with GCP and Azure
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.