AI Evaluation Subject Matter Expert
Hybrid · Charleston, South Carolina, United States
Job Summary
AI Evaluation Subject Matter Expert to help plan, design, and execute evaluation of AI-enabled capabilities for Navy CANES modernization. Responsibilities include developing evaluation strategies, test plans, and metrics; assessing AI/ML tools for operational relevance, reliability, security, and explainability; designing test scenarios reflecting Navy afloat conditions; defining data requirements and ground-truth methods; supporting AI red teaming and safety reviews; producing technical reports and executive summaries for Navy and leadership; collaborating with systems engineers, cyber, data science, software, test teams, and government stakeholders; providing SME input on AI governance, lifecycle management, and acquisition support. The role requires strong analytical and communication skills, experience in defense/airg systems, and an active DoD Secret clearance, with hybrid work arrangements in Charleston, SC.
Required Qualifications
- Bachelor’s degree in computer science, data science, artificial intelligence, engineering, mathematics, statistics, cybersecurity, operations research, information systems, or a related technical discipline preferred. Advanced degree preferred.
- Active DoD Secret clearance.
- Demonstrated experience evaluating AI, machine learning, data analytics, automation, or decision-support systems in defense, intelligence, cybersecurity, network operations, enterprise IT, or mission system environments.
- Strong understanding of AI / ML evaluation methods, test design, performance metrics, validation approaches, model limitations, and operational risk assessment.
- Experience developing test plans, evaluation frameworks, measures of effectiveness, measures of performance, data collection plans, and technical reports.
- Familiarity with cybersecurity, enterprise networks, tactical networks, system monitoring, anomaly detection, log analytics, or network operations use cases.
- Ability to assess AI-enabled systems in operationally constrained environments, including limited bandwidth, degraded connectivity, edge computing, and mission-critical infrastructure.
- Understanding of responsible AI concepts, including transparency, explainability, human oversight, robustness, security, bias, accountability, and lifecycle monitoring.
- Experience working with cross-functional engineering, cyber, data science, software, test, and government stakeholder teams.
- Strong written and verbal communication skills, including the ability to brief complex AI evaluation findings to technical and non-technical audiences.
- Active DoD Secret clearance.
Desired Qualifications
- Bachelor’s degree preferred
- Active DoD Secret clearance
- Experience evaluating AI/ML systems in defense/cyber
- Strong written and verbal communication
- Ability to brief complex AI evaluation findings to technical and non-technical audiences
- Experience developing test plans, evaluation frameworks, and metrics
- Familiarity with CANES, Navy afloat networks, RMF, cyber survivability testing
- Experience with red teaming or adversarial ML
- Cross-functional collaboration with engineering, cyber, data science, and government stakeholders
- Knowledge of responsible AI concepts (explainability, bias, oversight)
- Operational experience in constrained environments (limited bandwidth, edge computing)
- Ability to produce technical reports and executive summaries
- Background in data science, statistics, engineering, or related field
- Experience with data collection plans and ground-truth methods
- Understanding of AI governance and model lifecycle management
- Familiarity with DoD acquisition decision support
- Strong briefing and stakeholder management skills
- Experience with AI evaluation metrics (accuracy, precision, recall, latency, robustness)
- Knowledge of Navy CANES program context
- Bachelor’s degree in computer science, data science, AI, engineering, math, statistics, cybersecurity, or related field
- Advanced degree is a plus
- Top Secret clearance eligibility may be considered
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.