[Hiring] Senior Software Engineer - Model Evaluation & AI Systems @Deepgram

Role Description

Deepgram is looking for a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers. This team owns the evaluation and quality assurance surfaces that ensure Deepgram's models — across speech-to-text, text-to-speech, and increasingly LLM- and multimodal-powered systems — meet their performance targets in both batch and streaming environments.

• Define evaluation methodology and build the infrastructure that measures model quality at scale.
• Create evaluation pipelines and define pass/fail criteria grounded in Research benchmarks.
• Build monitoring systems that keep models honest in production.
• Provide trusted signals that inform release and optimization decisions, protecting the customer experience.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems.
• Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.
• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results.
• Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

Requirements

• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems.
• Design, build, and maintain automated evaluation pipelines across batch and streaming.
• Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines.
• Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates.
• Build and operate canaries and continuous-monitoring systems that detect quality regressions in production.
• Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure.
• Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals.
• Integrate evaluation and quality gates into CI/CD so quality is verified continuously.
• Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.

Nice to Have / Ways to Stand Out

• Hands-on experience evaluating modern AI systems such as LLMs, RAG pipelines, agents, or multimodal models.
• Experience with React Native or other cross-platform mobile frameworks.
• Experience building or improving evaluation frameworks, benchmarks, or ML infrastructure.
• A strong appreciation for evaluation quality — correctness, reproducibility, and consistency across environments.
• Experience with voice, audio, speech recognition, or real-time systems.
• Prior involvement in open-source projects, through contributions, reviews, maintenance, or community engagement.
• Experience acting as a technical bridge across teams or platforms.
• Familiarity with cloud infrastructure, containerized/ephemeral environments, and monitoring tooling.