[Hiring] Senior Principal AI Engineer @Cerence

Role Description

A Moving Experience.

What You Will Work On:

• Design and operate distributed training systems for large neural networks (autoregressive, diffusion, State Space Models, etc.) across GPU clusters.
• Optimise multi-node, multi-GPU execution to maximize throughput and utilization.
• Diagnose & resolve bottlenecks across compute, memory, and network.
• Improve training stability and fault tolerance at scale.
• Partner with research and applied ML teams to productionize large-model training pipelines.

Core Responsibilities

• Distributed Training Infrastructure:
  • Build and optimize GPU cluster orchestration using: Slurm, Kubernetes, Ray, RunAI.
  • Ensure efficient scheduling, isolation, and fairness across training workloads.
• Communication & Networking:
  • Optimize and debug distributed communication using: NCCL, RDMA, InfiniBand, NVLink.
  • Minimize networking bottlenecks that dominate end-to-end training time.
• Training Frameworks:
  • Scale large-model training using: PyTorch, Distributed Megatron-LM, DeepSpeed.
  • Own multi-node launch configurations, failure recovery, and performance tuning.
• Memory & Performance Optimization:
  • Apply advanced memory optimization techniques: Activation checkpointing, ZeRO (Stage 1–3) and offload strategies.
  • Balance compute, memory, and communication to push model size and batch scale.

What Success Looks Like

• GPU utilization consistently stays high (>80–90%).
• Training scales cleanly from single node to dozens or hundreds of GPUs.
• Communication overhead is minimized and predictable.
• Large training jobs run stably for days or weeks without failure.
• New models can be trained faster, larger, and more reliably than before.

Qualifications

• Deep hands-on experience with distributed systems or ML systems.
• Experience running large-scale workloads on GPU clusters.
• Production experience with PyTorch distributed training.
• Strong understanding of parallelism strategies (data, tensor, pipeline parallelism).
• Low-level understanding of GPU communication and networking.

Requirements

• Critical Technical Skills:
  • GPU orchestration: Slurm, Kubernetes, Ray, RunAI.
  • Communication libraries: NCCL, RDMA, InfiniBand, NVLink.
  • Training frameworks: PyTorch Distributed, Megatron-LM, DeepSpeed.
  • Memory optimization: activation checkpointing, ZeRO offload techniques.

Common Problems You’ll Be Solving

• Many teams fail at scale because:
• GPU utilization is low despite large clusters.
• Networking and communication dominate training time.
• Training jobs crash or become unstable at large scale.
• You will be explicitly focused on eliminating these failure modes.

Ideal Background

• This role is a strong fit for individuals who have worked as:
• ML Systems Engineer.
• Distributed Systems Engineer.
• AI Infrastructure Engineer.
• HPC Engineer transitioning into ML.
• Experience working with large language models or foundation models is a strong plus, but deep systems expertise is valued over pure model architecture experience.

Why This Role Matters

• Without robust distributed training infrastructure, progress on large models stalls. This role directly enables:
• Larger models.
• Faster iteration cycles.
• More reliable research-to-production pipelines.
• You will be building the foundation that makes large-scale AI possible.