Data Scientist
On-siteFort Meade, Maryland, United States
Job Summary
Design, build, and maintain benchmarks for model and agentic workflow testing using InspectAI, including dataset curation, scoring frameworks, and visualization tools. Clean, transform, and validate supporting data to generate metrics that measure model performance in consistent, auditable ways. Partner with Researchers to ground benchmark design in sound methodology and with Developers to operationalize evaluation workflows into stable tooling. Document dataset lineage and scoring rationale to ensure results are reproducible in airgapped or constrained environments. This role supports mission-critical government organizations and businesses facing complex technology challenges through custom software development and premium solutions tailored for governmental needs.
Required Qualifications
- Active TS/SCI w/FS Poly (NSA) Clearance
- Experience with AI Evaluation & Frameworks such as InspectAI
- Experience with LLM Benchmarking
- Experience with Agentic Workflow Evaluation (tool-use, multi-step task performance)
- Experience with Statistical Analysis
- Experience with Data Cleaning & Validation
- Experience with Data Visualization Tools
- Experience with Jupyter Notebooks
- Experience with MongoDB
- Experience with Cloud/Containerized Environments
- Experience with Airgapped & Constrained Operations
- Experience with Git Version Control
- Experience with Experimental Design
- Experience with Dataset Lineage/Traceability
- Experience with Jira
- Experience with Confluence
- BS in Computer Science or similar technical field
- 4 years of relevant experience
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.