[Hiring] Senior ML Engineer @Cast AI

Role Description

Throughput. Latency. Cold starts. Cost per token.

• Move those four numbers in the right direction and two things happen: customers get faster, cheaper inference, and our margins improve.
• Your mandate is the inference stack's optimization, whatever it takes, and the technical direction of how we optimize it is yours to argue with data.
• The problem: the checkpoints run 0.3-1.5 TB, and the "right" serving configuration depends on various shifting variables.
• We took Kimi-K3 (1.5 TB) from over an hour cold-to-serving to under twelve minutes with a shared filesystem, O_DIRECT weight loaders, and node-local JIT caches.
• Prefill/decode disaggregation and tiered KV caches are next.

Stack:

• Python
• SGLang
• vLLM
• TensorRT-LLM
• PyTorch
• fastsafetensors
• CUDA-adjacent tooling
• Kubernetes
• virtiofs shared filesystems
• ClickHouse
• OpenTelemetry
• Prometheus
• Grafana
• GitLab CI

Qualifications

• You've built real ML systems for 5+ years, with depth in inference or training infrastructure.
• Your Python is production-grade: services, not scripts.
• You've run vLLM, SGLang, or TensorRT-LLM for real, with a working mental model of engine performance on GPUs.
• You've measured quantization regressions, not just compression ratios.
• Multi-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.
• You know where the bytes live and how fast they move.
• You instrument before you optimize, distinguishing real wins from benchmark artifacts.
• A wide mandate excites you more than it unsettles you.

Requirements

• You've worked on LLM routing specifically: RouterBench-class evals, cost/quality/latency Pareto fronts.
• Calibration, abstention, or conformal methods in production decision systems.
• You've run online A/B loops with drift monitoring on a model that touches real money.

Responsibilities

• Push throughput: continuous batching, chunked prefill, speculative decoding, CUDA-graph capture, and scheduler tuning across the engine fleet.
• Cut latency: profile to find and fix the actual bottleneck (compute, memory bandwidth, scheduling, storage, or network).
• Get more out of the KV cache: hierarchical and paged attention, prefix caching, eviction policies, quantized KV.
• Kill cold starts: optimize weight loaders, JIT and compile caches, shared-filesystem read paths.
• Quantize without regressing quality: INT8/INT4, FP8/FP4 across weights, activations, and KV.
• Scale across nodes: distributed inference topologies, network-aware placement, prefill/decode disaggregation.
• Set the technical direction: decide what to benchmark, adopt, and build, with strong writeups and reproducible experiments.

Benefits

• Competitive salary (depending on the level of experience).
• Flexible, remote-first global environment.
• Collaborate with a global team of cloud experts and innovators.
• Equity options.
• Quick feedback with a fast-paced workflow.
• Spend 10% of your work time on personal projects or self-improvement.
• Learning budget for professional and personal development.
• Annual hackathon to spark new ideas and strengthen team bonds.
• Team-building budget and company events.
• Equipment budget to ensure you have everything you need.
• Extra days off to help maintain a healthy work-life balance.

Hiring Process

• Screening call with Recruiter
• Hiring Manager interview
• Technical interview (system design)
• Live coding
• Culture Check interview with an executive

As part of our standard hiring process, a background check may be conducted at the final stage of recruitment through our third-party provider, Checkr.

Please note that Cast AI does not provide any form of visa sponsorship/work permit.