[Hiring] Labeler / Annotator – AI Response Evaluation @Blueprint Technologies

Role Description

We are seeking a detail-oriented Labeler / Annotator to evaluate responses generated by AI systems in German . This role focuses on side-by-side (SBS) evaluation of outputs from different AI models across real-world scenarios. You will play a key role in improving how AI systems understand and communicate in German. This is not a translation role; it is an evaluation and analysis role requiring strong judgment and attention to detail.

What You’ll Work On:

• Evaluate AI responses across scenarios such as:
• General-purpose Q&A
• Web search results
• File-based and image-based responses
• Image and file generation tasks
• Single-turn and multi-turn conversations

Responsibilities:

• Perform side-by-side (SBS) comparisons of AI-generated responses
• Evaluate outputs based on:
• Accuracy
• Relevance
• Clarity
• Instruction-following
• Identify nuances in tone, meaning, and cultural context across German
• Apply detailed, scenario-specific annotation guidelines
• Maintain consistency and high-quality evaluations

Qualifications

• Native or professional fluency in German
• Strong English reading comprehension (required for guidelines)
• Prior experience with side-by-side (SBS) labeling or annotation
• Excellent analytical thinking and attention to detail
• Ability to quickly understand and apply structured evaluation frameworks

Requirements

• Background in linguistics, translation, or localization (preferred)
• Experience in:
• AI evaluation
• Data annotation
• Search relevance or content quality analysis

Benefits

• Competitive compensation aligned with local markets and experience
• Estimated hourly range: R$75 – R$85 BRL
• Estimated monthly equivalent (full-time): R$12,000 – R$13,600 BRL
• Actual compensation will be determined based on experience, skills, and internal equity
• All new hires will complete a structured onboarding program, including:
• Training sessions and practice exercises
• Calibration against quality benchmarks
• Qualification review before moving into production work
• This role is hired through an Employer of Record (EOR) partner, ensuring full compliance with local labor laws and payroll requirements
• Employees receive locally compliant benefits, which may include:
• Statutory benefits (e.g., paid leave, 13th salary where applicable)
• Social security and healthcare contributions
• Additional benefits aligned with local market practices