[Hiring] AI Evaluation Engineer @Bask Health

Role Description

We're looking for an AI Evaluation Engineer to help ensure that AI-powered features across Bask Health behave accurately, reliably, and consistently before and after they reach production.

• Build and maintain evaluation workflows that test large language models, AI assistants, retrieval systems, and automated features against clearly defined product requirements.
• This is a hands-on engineering role that combines AI evaluation, software testing, automation, and production quality assurance.
• Develop test scenarios, evaluation datasets, automated checks, and tools that help engineering teams identify problems before they affect users.
• Work may include testing AI-generated responses, validating structured outputs, evaluating tool-calling behavior, checking retrieval accuracy, and identifying regressions caused by changes to prompts, models, or application code.
• Collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and quality-focused engineering teams to define expected behavior and translate those expectations into repeatable evaluations.
• Help establish processes for reviewing AI failures, documenting recurring issues, and determining whether a feature meets its acceptance criteria.
• Pay particular attention to privacy, permissions, unsupported medical claims, and AI behavior that could interfere with approved clinical or pharmacy workflows.
• Contribute to evaluation standards while independently implementing and maintaining the tools and tests that support AI quality.
• Enjoy finding difficult edge cases and understand how AI behavior differs from traditional deterministic software.

Qualifications

• 3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
• Experience developing automated tests for production software applications.
• Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
• Familiarity with large language models and modern AI-powered applications.
• Understanding of software testing methodologies and test case design.
• Experience building automated testing scripts, frameworks, or validation tools.
• Familiarity with API testing and backend application integrations.
• Understanding of prompt engineering and common AI model failure modes.
• Ability to evaluate AI-generated responses against expected requirements.
• Experience working with structured data and validating application outputs.
• Familiarity with regression testing and software release processes.
• Experience debugging application failures and reproducing technical issues.
• Understanding of authentication, authorization, and secure application behavior.
• Familiarity with databases and application data models.
• Experience working with Git and collaborative software development workflows.
• Understanding of continuous integration and automated testing pipelines.
• Familiarity with logging, monitoring, and production troubleshooting.
• Ability to create clear, reproducible test cases.
• Strong analytical thinking and attention to detail.
• Ability to distinguish application defects from model behavior limitations.
• Experience documenting technical issues and communicating findings.
• Familiarity with cloud-based application environments.
• Ability to collaborate effectively with engineers, product managers, and other stakeholders.
• Strong problem-solving skills and a quality-focused engineering mindset.
• Ability to independently own evaluation and test automation projects.

Nice to Have

• Experience evaluating large language models or generative AI applications.
• Familiarity with OpenAI, Anthropic, or comparable model APIs.
• Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
• Familiarity with LLM-as-a-judge evaluation techniques.
• Experience developing AI evaluation datasets and test benchmarks.
• Understanding of human evaluation and response labeling workflows.
• Experience testing retrieval-augmented generation systems.
• Familiarity with embeddings, semantic search, and vector databases.
• Experience evaluating AI agents and tool-calling behavior.
• Familiarity with prompt injection testing and AI security evaluations.
• Experience implementing AI regression tests in CI/CD pipelines.
• Experience building automated evaluation reports or dashboards.
• Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.
• Experience testing TypeScript or Node.js applications.
• Familiarity with React or Next.js application testing.
• Experience working with AWS or serverless infrastructure.
• Experience evaluating AI features in healthcare or other sensitive environments.
• Understanding of HIPAA and healthcare data privacy considerations.
• Experience developing human-in-the-loop quality review processes.
• Experience using AI-assisted development tools such as Cursor or Claude Code.