[Hiring] Senior Data Scientist @Ellipsis Health

Role Description

As a Data Scientist specializing in AI Quality & Evaluation, you will play a pivotal role in ensuring the robustness, accuracy, and fairness of our AI models and systems. Your work will directly contribute to accelerating customer onboarding and enhancing the overall user experience.

• Drive High-Impact Projects: Own data science and AI evaluation projects end-to-end—from defining the problem and developing the methodology to executing the analysis, communicating results, and driving implementation.
• Drive Continuous Improvement of LLM-as-a-Judge: Build the data and evaluation infrastructure for a closed-loop AI quality system, including golden datasets, ground-truth benchmarks, judge calibration, error analysis, and production feedback.
• Develop Evaluation Frameworks: Design and implement robust evaluation frameworks, methodologies, and metrics for LLM- and speech-based AI systems.
• Build and Manage Evaluation Datasets: Design, curate, and maintain high-quality datasets for training, testing, benchmarking, and evaluating conversational AI systems and their components.
• Create and Manage Data Pipelines and Dashboards: Build and maintain scalable data pipelines and intuitive dashboards that enable reliable data analysis, ongoing AI quality monitoring, and data-driven decision-making.
• Collaborate and Innovate: Work closely with AI/ML Engineers and Product Managers to integrate your findings into product enhancements.
• Mentor and Lead: Mentor junior data scientists, establish best practices, and raise the technical bar across the team.

Qualifications

• 5+ years of industry experience in data science, machine learning, or a related quantitative field.
• Bachelor's or Master's degree in Computer Science, Data Science, Statistics, Mathematics, or a related quantitative field.
• Strong proficiency in programming languages such as Python (with libraries like Pydantic, Pandas, NumPy, Scikit-learn).
• Strong foundation in statistical analysis, experimental design, hypothesis testing, and A/B testing methodologies.
• Experience working with large-scale datasets, data pipelines, and analytical tools to generate actionable insights.
• Familiarity with evaluation metrics (precision, recall, agreement metrics like Cohen's Kappa) to calibrate judge models against human ground truth.
• Demonstrated ability to independently own and drive projects from problem definition through execution and implementation.
• Excellent communication and interpersonal skills, with the ability to collaborate effectively with AI/ML Engineers, Product Managers, and other cross-functional stakeholders in a fast-paced environment.

Requirements

• Experience with Natural Language Processing (NLP) techniques and understanding of LLM architectures and their evaluation challenges.
• Experience with Speech Technologies and their evaluation is a significant advantage.
• Experience in a regulated industry (e.g., healthcare, finance). Familiarity with data governance principles, particularly concerning PII/PHI.
• Experience with MLOps practices and deploying models to production.
• Familiarity with Langfuse, cloud platforms (GCP, AWS, Azure) and data platforms such as Databricks.

Benefits

• Competitive salary and benefits, including 401k matching up to a certain percentage of your salary.
• Health, vision, and dental insurance.
• Very flexible paid time off.
• The typical salary range for this role is $180k-230k USD.

Background Checks

As a health technology company, we reserve the right to run a background check on any applicant to which we extend an offer and to re-perform any such check at any time during the course of employment.

Assistance

If you have a disability or otherwise require any assistance whatsoever in the application or recruitment process, please feel free to submit a request to [email protected].