Lead Data Engineer
$190,000–$220,000 year
HybridSan Francisco, California, United States
Job Summary
Lead a cross-functional scrum team of onshore and offshore data engineers, setting priorities and removing blockers to deliver reliable data products. Own the architecture, maintenance, and continuous improvement of critical financial data pipelines supporting GL automation, investor reporting, tax documents, and collections workflows. Partner with finance, accounting, and operations stakeholders to translate business requirements into scalable technical solutions while driving adoption of modern data platform capabilities like Databricks, dbt, and Dagster. Establish coding standards and data quality frameworks that scale across the team, leveraging AI tools for QA automation, code review, and intelligent runbooks to empower L1 support. Build and improve monitoring, alerting, and observability practices to ensure pipeline reliability and rapid incident response. Requires 7+ years in data engineering with 2+ years leading teams, strong SQL and AWS expertise, and in-person attendance on Tuesdays, Wednesdays, and Thursdays in San Francisco.
Required Qualifications
- 7+ years of experience in data engineering
- 2+ years leading teams or projects in a technical lead capacity
- bachelor's degree in a related field
- equivalent work experience
- hands-on experience using AI tools to accelerate your work and improve output quality
- comfortable using AI tools yourself and showing colleagues how
- thoughtful about limitations and where human judgment matters most
- Strong expertise in SQL
- data modeling
- data warehouse concepts
- building production data pipelines at scale
- Experience with orchestration tools such as Airflow, Oozie, or Dagster
- distributed processing frameworks like Spark or PySpark
- Working knowledge of AWS services (EMR, S3, Redshift)
- modern data platforms such as Snowflake or Databricks
- You take ownership of outcomes
- proactively identifying risks and workflow improvements before they become blockers
- Strong communication and organizational skills
- experience collaborating across business, product, and engineering teams
- Experience leading or mentoring offshore engineering teams
- managing work across time zones
- balance technical depth with business context
- understanding how data pipelines support financial processes and regulatory requirements
- In-person attendance is essential for this role's success
- remote placement will not be considered
- flexible working across time zones when necessary
- As needed travel to Happen Bank offices and/or other locations
Desired Qualifications
- Experience with dbt for data transformation
- Elementary for data validation
- Hands-on use of AI tools such as Claude, ChatGPT, or GitHub Copilot to improve development workflows
- Background in financial services, accounting systems, or investor reporting processes
- Experience building self-service analytics tools using Streamlit or similar frameworks
- Familiarity with data quality frameworks
- alerting systems
- SRE practices for data infrastructure
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.