[Hiring] HPC Engineer @Cudo Ventures

Role Description

CUDO is delivering a large-scale GPU infrastructure project, taking it from hardware specification through to production clusters running real AI workloads. Getting there means more than installing GPUs. The infrastructure needs to be stable, tuned, monitored and built to operate reliably from day one.

As an AI infrastructure owner-operator, CUDO doesn't design something and hand it over. We own it and we run it. So the GPU clusters we build need to be specified properly, deployed cleanly, automated from day one and supported by processes our service team can operate.

This role exists to address a very specific engineering challenge. Delivering GPU infrastructure at this scale takes hands-on engineering across every layer of the cluster:

• Compute
• Interconnect
• Storage
• Scheduling
• Monitoring

You will work alongside our Senior HPC Engineers to get these clusters built, tuned and running reliably. Just as importantly, you will help make sure the way we deliver them is repeatable, documented and ready for whatever CUDO builds next.

Requirements

• Shaping the build before the hardware arrives. Work with OEMs and vendors on bills of materials, data centre layouts and power and cooling specifications.
• Deploying and tuning the clusters. Install and tune Linux, design and test InfiniBand and RoCE networking, configure SLURM partitions and priorities, and set up parallel file systems such as Ceph, Lustre, WEKA and VAST.
• Automating the delivery. Use Ansible, Terraform and scripting to standardise deployment, configuration and maintenance, so every build is faster and more consistent than the last.
• Making it operable. Build proactive monitoring with our SREs, so issues are caught and fixed before users feel them. Keep clusters secure, patched, resilient and highly available.
• Handing over well. Write clear architecture, configuration and procedure documentation, train the service team, move routine work to them and act as their escalation point for AI and HPC incidents.
• Advising customers. Support the sales team in pre-sales conversations, helping to architect solutions that CUDO can deliver and run.
• Building for what comes next. Track developments from NVIDIA, AMD and Intel and bring that knowledge into how we plan and build.