Head of Experimentation
$271,000–$373,000 year
RemoteUnited States
Job Summary
Own the Experimentation pillar by directing Product leadership, partnering with Engineering and Design in a triad model, and making the investment case across in-product experimentation, warehouse-native analysis, and scalable infrastructure. Make experimentation the measurement layer of the AI SDLC by productizing AI-native primitives, building a closed loop from offline evaluation through automatic promotion and rollback, and establishing a self-improving feedback loop for agents. Win high-maturity buyers by earning technical confidence from senior data scientists and deciding on non-negotiable statistical depth and workflow capabilities to ship pivotal reference deals. Expand warehouse-native coverage across major data layers to deliver parity on analysis modes, variance reduction, and arbitrary-window metrics. Operate a high-performing function by running a disciplined roadmap, driving AI-assisted engineering productivity, and hiring where gaps exist. Represent the category externally to translate strategy to the field and equip sales to win head-to-head against sophisticated data organizations.
Required Qualifications
- Senior product leader (GM, VP, or equivalent) with a track record of owning a product line that competes on statistical rigor and data infrastructure
- Deep, operator-level fluency in experimentation methodology: causal inference, variance reduction, ratio metrics, sequential testing, exposure design, multi-armed bandits, and composite/multi-objective metrics
- The realities of running these at scale against production data warehouses and against non-deterministic systems where output variance, not just user variance, drives sample-size and significance decisions
- Earned credibility with data science leaders and experimentation specialists at sophisticated organizations — and can recruit them
- Led a function that includes engineering, design, and data science
- Comfortable setting a multi-quarter roadmap, championing investment allocation, and reporting results to an executive team and board
- Clear, direct communicator
- Decides fast with incomplete information
- Prefers shipping and learning to requirements documents
- Opinionated about where experimentation is going in an AI-native world — and specifically, how agents and autonomous systems will use experimentation infrastructure differently than human teams do
Desired Qualifications
- Built or scaled experimentation at an organization where it was core infrastructure, not a secondary analytics capability
- Personally won competitive evaluations where a sophisticated data-science organization was the deciding voice
- Shipped warehouse-native data products and understand the operational realities of running experiments directly against customer data infrastructure
- Sees experimentation as how software teams prove that any change — whether built by a person or an AI agent — actually worked. That evidence layer is core infrastructure, not a reporting afterthought
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.