Applied AI Data Analyst
$180,000–$240,000 year
On-siteNew York City, New York, United States
Job Summary
Analyze edge cases, model predictions, and extraction accuracy for LLM-driven document processing systems and agentic workflows. Dig deep into complex datasets to determine why extraction or time-series models succeeded or failed. Define, curate, and maintain gold-standard evaluation datasets for automated backtesting of AI agents and demand forecasting models. Build dashboards and evaluation harnesses to monitor system health, model drift, and AI performance metrics across release cycles. Partner directly with applied AI engineers, product managers, and business operations managers to translate qualitative problem definitions into quantitative product insights.
Required Qualifications
- Graduate degree (M.S. or Ph.D.) in Computer Science, Data Science, Statistics, Mathematics, Physics, or a related STEM field
- 2–4 years of experience in a quantitative data analyst or AI evaluation role, working closely alongside AI/ML R&D teams
- Proficiency in Python
- Hands-on works with AI/ML models
- Strong capabilities with SQL
- Enterprise data infrastructure (PostgreSQL, Snowflake, dbt, AWS)
Desired Qualifications
- Experience with LLM observability, tracing, and eval frameworks (LangSmith, Arize AI, or TruLens)
- Proficiency with agentic workflows or multi-agent orchestration
- Familiarity with demand forecasting, time-series models, or statistical modeling
- Comfort in fast-moving, high-growth NYC startup environments
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.