[Hiring] Senior or Staff AI Infrastructure Engineer @TRM Labs

Role Description

As a Senior or Staff AI Infrastructure Engineer , you’ll be at the core of building and scaling the technical infrastructure for AI/ML systems. You will:

• Build reusable CI/CD workflows for model training, evaluation, and deployment — integrating Langfuse, GitHub Actions, and experiment tracking, etc.
• Automate model versioning, approval workflows, and compliance checks across environments.
• Build out a modular and scalable AI infrastructure stack — including vector databases, feature stores, model registries, and observability tooling.
• Partner with engineering and data science to embed AI models and agents into real-time applications and workflows.
• Continuously evaluate and integrate state-of-the-art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML, etc.).
• Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime.
• Build and enhance AI/ML Model Performance.
• Ensure data accuracy, consistency, and reliability, leading to better model training and inferencing.
• Deploy infrastructure to support offline and online evaluation of LLMs and agents — including regression testing, cost monitoring, and human-in-the-loop workflows.
• Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments.

Qualifications

• Write high-quality, maintainable software — primarily in Python, but we value engineering ability over language familiarity.
• Strong background in scalable infrastructure, including:
  • Containerization and orchestration (e.g. Docker, Kubernetes)
  • Infrastructure-as-code and deployment (e.g. Terraform, CI/CD pipelines)
  • Monitoring and logging frameworks (e.g. Datadog, Prometheus, OpenTelemetry)
• Understand and implement ML Ops best practices, including:
  • Model versioning and rollback strategies
  • Automated evaluation and drift detection
  • Scalable model and agent serving infrastructure (e.g. vLLM, Triton, BentoML)
• Deploy and maintain LLM and agentic workflows in production, including:
  • Monitoring cost, latency, and performance
  • Capturing traces for analysis and debugging
  • Optimizing prompt/response flows with real-time data access
• Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact.

Requirements

• Rapid Issue Resolution: TRM Engineers identify and resolve critical onsite issues in minutes to hours, not weeks.
• Navigating Bureaucracy: Anticipate and address procedural hurdles, build trust with key stakeholders, and find alternative pathways to approvals.
• Efficient Knowledge Transfer: Document and share updates in real time, ensuring full visibility into plans, blockers, and resolutions.

Benefits

• Estimated base salary range for this role is $200,000 - $275,000.
• This role may be eligible to participate in TRM’s equity plan.

Company Description

TRM Labs provides blockchain analytics and AI solutions to help law enforcement and national security agencies, financial institutions, and cryptocurrency businesses detect, investigate, and disrupt crypto-related fraud and financial crime.

• TRM’s blockchain intelligence and AI platforms include solutions to trace the source and destination of funds, identify illicit activity, build cases, and construct an operating picture of threats.
• TRM is trusted by leading agencies and businesses worldwide who rely on TRM to enable a safer, more secure world for all.