Senior AI Agent & Evaluations Engineer
Hybrid · Portland, Oregon, United States
Portland, Oregon, United StatesHybridFull TimeSenior LevelNot SpecifiedStartup
Type
Full Time
Level
Senior Level
Education
Not Specified
Company size
Startup
Job Summary
Senior AI Agent & Evaluations Engineer at Vacatia responsible for designing agent behavior, building evaluation frameworks, creating guardrails, and continuously improving agent performance across Vacatia's AI agent ecosystem. Leads prompt design, routing logic, evaluation pipelines, and integration with business workflows; collaborates with engineers and stakeholders to translate policies and domain expertise into measurable agent behavior and robust, observable AI systems.
Required Qualifications
- Proven experience shipping and owning production AI agents or LLM-powered systems beyond proof-of-concept environments
- Deep expertise in prompt engineering (system prompts, tool usage, context management, output constraints, and agent behavior design)
- Hands-on experience building evaluation frameworks using golden datasets, scoring rubrics, LLM-as-judge methodologies, and regression testing
- Strong familiarity with modern AI development tools such as Claude Code, Codex, or similar coding agents
- Experience with agent observability and evaluation platforms such as LangSmith, Langfuse, Arize, Galileo, or comparable solutions
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.