[Hiring] Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering @Together AI

Role Description

We're looking for a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet. You'll design a manifest-driven API where the inference team declares what they need, whether that's a cluster, a model deployment, or a capacity change, and our controllers handle the reconciliation, provider/runtime selection, and lifecycle management underneath.

• Build the provisioning state machine:
  • Design and implement the software that models the full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA.
• Build the self-service API:
  • Design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call.
• Automate self-healing:
  • Detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
• Own reliability of the pipeline:
  • Idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
• Partner with the inference/ML platform team:
  • Understand the cluster shapes they need — topology, interconnect, scheduling constraints — and encode them as first-class abstractions in the platform.
• Engineer it like software:
  • Strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code.

Qualifications

• Strong software engineering background in Go, Python, Rust, or similar.
• Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent.
• Experience building software control planes or orchestration systems that model state and reconcile it over time.
• Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub.
• A product mindset — you've built internal platforms or APIs consumed by other engineering teams.

Requirements

• Nice to have:
  • Exposure to bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC) and/or networking fundamentals (VLANs, BGP, fabric design), or GPU/accelerator infrastructure.
  • Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).
  • Prior work at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.
  • Systems programming in Rust or Go.

Company Description

Together AI, the AI Native Cloud, is purpose-built for AI engineers. AI application developers get high-performance inference that scales reliably, fine-tuning and reinforcement learning for creating frontier-level specialized models, and pre-training at massive scale for fully custom intelligence.

• Trusted by Cursor, Decagon, ElevenLabs, Salesforce, and Zoom.
• Together serves 400+ trillion tokens a month.

Equal Opportunity

Together AI is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.

Please see our privacy policy at https://www.together.ai/privacy .