Data Engineer (Remote - California Residents Only)
RemoteCalifornia, United States
California, United StatesRemoteFull Time
Full Time
Job Summary
Manage and refine AWS cloud infrastructure including EC2, VPC, RDS, and Lambda while leading ETL/ELT processes with Snowflake, Airflow, dbt, and Python. Review, redesign, and expand analytics architecture to optimize data pipelines for terabyte-scale intake, automating manual processes and improving data delivery. Maintain comprehensive documentation of data architecture from macro diagrams to micro-level scripts and lead meetings to analyze information and collaborate with stakeholders. Continuously improve pipelines by applying industry best practices and developing expert knowledge of the higher education planning platform serving K-12 students, parents, and educators.
Required Qualifications
- SQL: 2 years
- Cloud infrastructure: 2 years
- US work authorization
- Remote/Virtual - All candidates must live in California
- Extensive experience building out complex data architectures, including the testing, maintenance, and refinement of all data pipelines
- Experience successfully scaling up an organization's data intake to terabytes and petabytes of data
- Optimizing data delivery and automating manual processes
- Comfortable working with structured and unstructured data
- Troubleshoot data loading and processing tools via SQL, Python, shell scripting, AWS
- Take on a leadership role in special projects when needed
- Extensive experience developing robust data documentation and data governance protocols
- Proactively and successfully partner and collaborate with other subject matter experts on a project management basis
- Comfortable leading and project managing work with many unknowns and ambiguous solutions
- Deep knowledge and/or curiosity about the needs and behaviors of students, educators, and parents
- Passion for educational equity
- Manage, refine, and enhance AWS cloud services infrastructure
- Lead ETL/ELT processes which include the development, refinement, and implementation of data loading and processing tools, including Snowflake, Airflow, Python, SQL, dbt, and shell scripts
- Review, redesign, and expand existing analytics and data processing architecture to create an optimal data pipeline(s)
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, designing infrastructure for greater scalability and automation
- Maintain and update documentation of data architecture from both the macro view (i.e. architectural diagrams) and the micro view (i.e. script level tasks in Airflow)
- Lead meetings, research processes, collect data, analyze information
- Collaborate with the key stakeholders of your projects
- Develop and maintain expert knowledge of our platform and organization
- Continuously improve data pipelines and architecture by staying updated on industry trends and best practices
- Build processes supporting data transformation, data structures, metadata, dependency, and workload management
- Advanced proficiency with cloud analytical tools: Snowflake, Redshift, Hadoop, Spark, Kafka
- Advanced proficiency with ETL tools like Matillion, dbt, Talend
- Advanced proficiency with data pipeline and workflow management tools: Airflow, Azkaban, Luigi
- Advanced proficiency building and optimizing data pipelines, cloud architectures, and data sets
- Advanced proficiency with scripting languages like Python, R, Scala
- Advanced proficiency developing cloud infrastructure in AWS or currently has or is in the process of obtaining AWS Certification as a Solutions Architect - Associate
- Expert SQL knowledge (SQL Server, PostgreSQL, MySQL, etc.) and understanding of relational databases, query authoring and optimization, as well as working familiarity with a variety of databases
- A successful history of manipulating, processing and extracting value from large structured and unstructured datasets
Desired Qualifications
- Strong decision-making skills and collaborative spirit with the ability to take abstract brainstorming and generate concrete proposals for action
- Advance projects without detailed supervision, balancing multiple responsibilities, and providing colleagues with actionable proposals for advancing collective efforts
- Thrive in a fast-paced environment with changing priorities and deadlines
- Juggle multiple projects of various scopes with ease and grace
- Meticulous attention to detail
- Excellent verbal communication skills; ability to communicate with various levels of professionals
- Strong organizational, project, and time management skills
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.