[Hiring] Customer Reliability Engineer @Andromeda Cluster

Role Description

Our customers run large AI training and inference workloads on GPU clusters we source from providers worldwide. When a node goes dark or a job dies eight hours into a run, the Customer Reliability Engineer is who they hear from, and who gets it sorted.

The job has three parts:

• Triage incoming issues and debug them at the Linux and Kubernetes layer.
• Work provider-side to figure out whose fault something actually is and push external providers to fix it.
• Build the monitoring and scripts that catch problems before a customer has to tell us.

You need to be comfortable in a Linux shell and know how Kubernetes works. You don't need GPU or HPC experience. Most people pick that up here.

What You’ll Do

• Triage and fix customer issues.
• Own issues start to finish: reproduce, diagnose, fix or escalate, close the loop.
• Debug at the Linux layer: processes, networking, storage, kernel logs, resource contention, systemd, journald.
• Dig into Kubernetes problems like pods stuck pending or crash-looping, node conditions, scheduling failures, resource limits.
• Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, nodes that need cordoning or draining, jobs failing across multiple nodes.
• Escalate when you're past your depth, with the evidence already gathered.
• Handle incidents.
• Take part in a 24/7 on-call rotation.
• First response on alerts and customer-reported outages: assess impact, set severity, pull in the right people.
• Keep customers updated during incidents. Clear status, honest unknowns, no silence.
• Write up what happened, then turn it into a runbook, an alert, or a fix so it costs less next time.
• Push providers to resolution.
• Work out whether a fault is provider-side, ours, or the customer's before it gets handed anywhere.
• Open tickets with compute providers and chase them down rather than waiting.
• Track recurring provider failures and flag the patterns to the people making sourcing decisions.
• Build the tooling.
• Write Python or Bash to automate the checks you'd otherwise run by hand.
• Build and improve monitoring: cluster and node health checks, GPU telemetry, dashboards, alerts that fire on real problems.
• Keep runbooks and customer docs current as you go.

Qualifications

• Real Linux troubleshooting ability from the command line.
• Working knowledge of Kubernetes: pods, nodes, deployments, services, scheduling, and how to investigate when one of those breaks.
• Can write a script in Python or Bash to automate something repetitive.
• Strong writing skills to explain technical problems clearly to customers.
• Good judgment under pressure.
• Willingness to join a 24/7 on-call rotation.

Requirements

• Hands-on time with NVIDIA GPUs in production: drivers, CUDA, DCGM, the Kubernetes device plugin.
• Experience with high-performance networking (InfiniBand, RoCE) or NCCL.
• Experience with HPC or batch schedulers like Slurm.
• A previous customer-facing technical role: support engineering, TAM, solutions, professional services.
• Knowledge of Prometheus, Grafana, Datadog, or similar.
• IAC: Terraform, Ansible, or Helm.
• Genuine interest in AI infrastructure and how big training jobs behave.

Benefits

• High-growth environment: Get in early at a company at the center of the AI infrastructure boom.
• Competitive compensation: + meaningful equity.
• Comprehensive benefits: for you and your dependents, including healthcare, dental, and vision coverage, 401(k), and unlimited PTO.