Senior Data Annotation Jobs in Montreal

Overview Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role. Key Responsibilities Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling Design, maintain, and enforce annotation guidelines compliance and taxonomy standards Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits Conduct prompt evaluation and rubric-based scoring aligned to model requirements Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines Document edge cases and escalation paths to reduce guideline drift Support onboarding and calibration using examples, gold sets, and retraining plans Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy Required Qualifications Senior data annotation or data labeling operations experience with QA evaluation ownership Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows Familiarity with NLP tasks such as named entity recognition and text classification Comfortable working remotely with cross-functional stakeholders Nice to Have Content safety labeling or policy-driven evaluation experience Inter-annotator agreement metrics and quality sampling methodologies Computer vision annotation exposure (bounding boxes, polygons, segmentation) Multilingual evaluation or locale-specific guidelines Work Model Remote, Full-time. Apply with relevant annotation experience and examples of QA evaluation or guideline development work.

The role involves creating and evaluating high-quality training data for AI/ML systems through labeling, RLHF, and prompt evaluation. It also requires leading QA workflows, maintaining annotation guidelines, and partnering with engineering stakeholders to improve model performance.

Candidates must have senior-level experience in data annotation operations with a strong grasp of taxonomy design and LLM evaluation workflows. Familiarity with NLP tasks and the ability to work remotely with cross-functional teams is required.

Relevées dans la description du poste. Confirmez les exigences importantes ci-dessus.

• Data Labeling
• RLHF
• LLM Evaluation
• Prompt Evaluation
• QA Review
• NLP
• Computer Vision Annotation
• Content Safety Labeling
• Taxonomy Design
• Inter-annotator Agreement
• Dataset Audits
• Named Entity Recognition
• Text Classification
• Guideline Development
• Sampling Plans
• Error Analysis

• Data & Analytics
• Technology
• Software
• Engineering
• Science & Research

Expérience minimale

5+ ans

Postuler avant le

29 août 2026

Langue de l’offre

anglais

Heures de travail

40 heures par semaine

Exigences de lieu

Country, Montreal, Quebec, Canada

Niveau d’expérience

Mid-Senior level