Google Cloud Platform Data Engineer
On-siteBristol, England, United Kingdom
Job Summary
Develop robust data processing jobs using Google Cloud Dataflow, Dataproc, and BigQuery, while designing and delivering automated pipelines with Cloud Composer orchestration. Lead solution delivery efforts or define end-to-end software development lifecycles, shaping team behavior for specifications, sprint planning, and documentation. Own the development process by establishing data engineering standards, influencing technical discussions with client stakeholders, and coaching team members on expertise. Ensure production-ready solutions meet business requirements through CI/CD pipelines, data quality alerting, and governance tools like Dataplex. This Principal GCP Data Engineer role supports a consultancy delivering data-driven solutions for FTSE 100 and Fortune 500 clients, requiring full sponsorship and offering a benefits package including private medical insurance and annual performance bonuses.
Required Qualifications
- Experience delivering and deploying production-ready data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc
- Experience developing end-to-end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam
- Expert understanding of when to use a range of data storage technologies including relational/non-relational, document, row-based/columnar data stores, data warehousing and data lakes
- Expert understanding of data pipeline patterns and approaches such as event-driven architectures, ETL/ELT, stream processing and data visualisation
- Experience working with business owners to translate business requirements into technical specifications and solution designs that satisfies the data requirements of the business
- Experience working with metadata management products such as Cloud Data Catalog and Collibra and Data Governance tools like Dataplex
- Experience in developing solutions on GCP using cloud-native principles and patterns
- Experience building data quality alerting and data quarantine solutions to ensure downstream datasets can be trusted
- Experience implementing CI/CD pipelines using techniques including as git code control/branching, automated tests and automated deployments
- Comfortable working in an Agile team using Scrum or Kanban methodologies
Desired Qualifications
- Experience of working on migrations of enterprise scale data platforms including Hadoop and traditional data warehouses
- An understanding of machine learning model development lifecycle, feature engineering, training and testing
- Good understanding or hands-on experience of Kafka
- Experience as a DBA or developer on RDBMS such as PostgreSQL, MySQL, Oracle or SQL Server
- Experience designing data applications to meet non-functional requirements such as performance and availability
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.