Remote AI Quality Analyst - Chinese
$31,200–$31,200 year
On-siteSeattle, Washington, United States
Job Summary
Design and execute multi-turn conversational prompts (1-5 turns) requiring the AI to utilize personal information and experiences. Analyze responses for Grounding issues, ensuring claims are supported by evidence rather than flawed inferences or hallucinations. Rigorously evaluate and stack-rank two model responses side-by-side to determine which is most helpful, easy to use, and enjoyable. Write clear, defensible rationales for comparisons, explicitly referencing specific turn numbers where issues or positive aspects occurred. Extract and verify "Debug Info" from the model to confirm chat summaries and data sources were properly utilized. Maintain strict data hygiene by deleting evaluation conversations to prevent them from polluting future chat history.
Required Qualifications
- Chinese Proficiency: Ability to read and write in Chinese with a high degree of comp
- BS/BA degree or equivalent experience in a relevant field (e.g., Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field)
- Commitments : Required: at least 4 hours per day and up to 40 hours per week with 4 hours of overlap with PST
- Engagement type: Contractor
- Engagement Length: 3 months
Desired Qualifications
- Creative Prompt Engineering: Experience in designing creative, multi-turn starting prompts based on personal context to thoroughly test the model's capabilities
- Strong Evaluation Acumen: Understanding of personalization concepts, including the ability to identify incorrect personalization, poor inferences, and forced connections
- Meticulous Attention to Detail: The ability to review Side-by-Side (SxS) model responses and spot subtle differences in naturalness and overnarrating
- Excellent Written Communication: Superior ability to write clear, concise, and structured rationales for model rankings, explicitly referencing specific turn numbers
- Feedback: Ability to provide constructive feedback and detailed annotations
- Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.