Senior AI Data Annotation Jobs in Toronto (Remote)

Rex.zone is hiring senior contributors to lead AI data annotation and QA evaluation workstreams that improve training data quality for modern AI/ML systems. This is a remote, full-time role aligned to real-world LLM training pipelines across NLP, computer vision, and content safety domains. About The Role You will translate product and research goals into labeling instructions, refine guidelines for edge cases, adjudicate ambiguity, and drive measurable quality improvements through audits, calibration, and error analysis. Work includes RLHF preference ranking, prompt evaluation, named entity recognition, computer vision annotation (bounding boxes/segmentation), and content safety labeling. What You Will Do Create, iterate, and maintain annotation guidelines and SOPs for complex datasets Perform and audit data labeling with measurable annotation guidelines compliance Execute RLHF comparison tasks and curate preference data with consistent rubrics Run QA evaluation programs: sampling plans, golden sets, inter-annotator agreement, defect taxonomy tracking Connect annotation outcomes to model performance improvement and large language model evaluation outcomes Document dataset lineage, versioning, and evaluation criteria Required Qualifications Senior-level experience in AI data annotation / data labeling and rubric-based review Ability to interpret ambiguous examples, adjudicate edge cases, and write clear instructions Understanding of how labeled datasets impact RLHF outcomes and downstream model behavior Strong communication and cross-functional collaboration (Engineering, Product, Research) Preferred Qualifications Hands-on RLHF workflows and prompt evaluation for conversational AI Content safety labeling and policy-driven evaluation experience Multimodal computer vision annotation exposure (detection/segmentation) Experience scaling annotation programs with vendors or distributed teams How To Apply Apply via Rex.zone with a resume highlighting AI data annotation scope, QA evaluation experience, and any RLHF or prompt evaluation work. Include examples of guideline writing, calibration, adjudication, and training data quality improvements.

Lead AI data annotation and QA workstreams to improve training data quality for LLM pipelines. Responsibilities include creating labeling guidelines, performing audits, and executing RLHF preference ranking tasks.

Requires senior-level experience in AI data labeling and the ability to adjudicate complex edge cases. Candidates should understand how labeled datasets impact RLHF outcomes and downstream model behavior.

Relevées dans la description du poste. Confirmez les exigences importantes ci-dessus.

• AI Data Annotation
• QA Evaluation
• RLHF
• Prompt Evaluation
• Named Entity Recognition
• Computer Vision Annotation
• Content Safety Labeling
• Guideline Writing
• Error Analysis
• Inter-annotator Agreement
• Dataset Lineage
• Cross-functional Collaboration

• Data & Analytics
• Technology
• Software
• Engineering
• Science & Research

Expérience minimale

5+ ans

Postuler avant le

29 août 2026

Langue de l’offre

anglais

Heures de travail

40 heures par semaine

Exigences de lieu

Country, Toronto, Ontario, Canada

Niveau d’expérience

Mid-Senior level