Data Annotation Specialist, Generalist
RemoteCanada
Job Summary
Evaluate and rank model outputs by assessing accuracy, helpfulness, tone, and safety while writing clear justifications. Stress-test models adversarially to surface failure modes, unsafe behavior, and capability gaps, then document reproducible cases for engineering teams. Create high-quality training and evaluation datasets by authoring prompts, responses, and exemplars, and build rubrics to apply consistent grading criteria across text, image, and structured formats. Calibrate standards through inter-annotator agreement checks and flag ambiguous edge cases rather than guessing. Adapt to evolving experimental tasks and report on model performance trends to guide cross-functional partners. This part-time independent contractor role requires 16 hours per week at CAD $30/hour, is remote with BYOD, and involves exposure to not safe for work content.
Required Qualifications
- 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance, and/or preference ranking
- Experience applying detailed guidelines to complex and often ambiguous content, with strong contextual and sociocultural judgment, sensitivity to nuance, tone, and register, and the ability to reason well in cases where there is no single correct answer
- Comfort with ambiguity: a willingness to flag unclear or uncovered edge cases rather than guess, and to work productively on novel, experimental tasks whose definitions are still evolving
- A sharp, curious eye for inconsistencies, subtle errors, and model failure modes, including the instinct to probe models adversarially and surface where they break
- Familiarity with how large language models behave, such as hallucination, sycophancy, and instruction-following gaps
- Excellent command of written English and strong reading comprehension, with the ability to clearly justify your evaluations, including why an output is correct or incorrect, high-quality or low-quality, and to write clear prompts and exemplars that explain your reasoning
- Strong attention to detail and commitment to accuracy, with the ability to maintain consistency across high-volume and monotonous tasks
- Comfort working with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML
- Strong execution in a remote environment, including good time management, comfort using new tools, and the ability to work independently in a global, asynchronous team
- Ability to work independently in a global, asynchronous team
- Good time management
- Comfort using new tools
- 16 hours per week
- CAD $30/hour contract rate
- BYOD (Bring Your Own Device, laptop)
- Remote
- Canada only
- Part-time independent contractor position
Desired Qualifications
- preference ranking
- quality assurance
- fluency in another language
- Familiarity with how large language models behave, such as hallucination, sycophancy, and instruction-following gaps
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.