Director, Data Engineering
$135,375–$135,375 year
On-siteNew York City, New York, United States or New York, United States
Job Summary
Analyze complex business requirements and partner with product, analytics, and engineering teams to design scalable data solutions. Design, develop, and optimize complex SQL transformations and data pipelines in Snowflake and Databricks using Delta Lake and PySpark. Collaborate with frontend engineers to design or modify schema for optimal software performance. Advanced database administration and development including stored procedures, user define functions, triggers and ELT/ETL packages using PySpark, SQL, and Python for large-scale data processing. Develop advanced Snowflake database objects including stored procedures, user define functions, views, tasks, streams, and dynamic tables. Uphold quality standards and create monitoring tools to ensure high fidelity data. Troubleshoot database issues, identify root causes, and implement optimal solutions. Extract, transform, and load data from multiple data sources using languages and tools such as Python and AWS Glue. Design, build, test, debug, monitor, and troubleshoot ETL and software processes in Python. Serve as an escalation point for any issues that may arise. Ability to follow implementation standards, develop documentation and transfer of knowledge.
Required Qualifications
- 7+ years of experience in Data Engineering with advanced SQL development writing complex queries, CTEs, window functions, stored procedures, and performance tuning.
- Strong experience with Snowflake as an enterprise data warehouse.
- Able to work independently and balance multiple tasks.
- High competency level in MS Office with specific strengths in Excel (knowledge of VBA a plus) and PowerPoint
- Advanced SQL programming skills, Redshift preferred (creating views, stored procedures, functions, etc.)
- Experience with Databricks, Snowflake (creating views, stored procedures, functions, etc.)
- Experience building scalable data pipelines using Databricks and Apache Spark (PySpark).
- Experience with Delta Lake, Unity Catalog, and modern lakehouse architectures.
- Experience with Business Intelligence tools such as Tableau or Power BI.
- Experience integrating REST APIs and third-party data sources.
- Experience supporting analytics and data science teams.
- Experience with CI/CD pipelines and Git-based development workflows.
- Experience with cloud platforms (AWS, Azure, or GCP) is preferred, with AWS experience considered a plus rather than a primary requirement.
- Excellent data modelling skills and experience working with large data sets.
- Experience with digital media measurement and analytics
- Experience with machine learning principles and applications a plus
- Possession of excellent written, oral communication and presentation skills
- Proven experience managing or mentoring a team.
Desired Qualifications
- Media Agency experience preferred
- knowledge of VBA a plus
- Redshift preferred (creating views, stored procedures, functions, etc.)
- AWS experience considered a plus rather than a primary requirement
- Experience with machine learning principles and applications a plus
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.