[Hiring] Senior Software Engineer, Machine Learning Infrastructure & Automation @fal

Role Description

Help fal's ML team move faster by building the automation, infrastructure, and developer tooling that makes developing, testing, and deploying generative AI models seamless.

• Own and improve the CI/CD systems supporting our rapidly growing collection of ML models and inference pipelines.
• Focus on eliminating manual work, accelerating development cycles, and building reliable systems that allow ML engineers to ship new models and optimizations with confidence.
• Work closely with our Applied ML and ML Performance teams.
• Build everything from automated model validation and performance benchmarking to AI-powered development workflows that help engineers iterate faster.

What you’ll do:

• Own ML CI/CD infrastructure.
• Design, build, and maintain automated testing, validation, and deployment pipelines for our ML models and inference services.
• Accelerate development cycles by dramatically reducing CI execution times through intelligent parallelization, caching, test selection, and efficient use of compute resources.
• Build automated model validation systems that test model outputs, detect quality regressions, and validate changes across different models, GPU architectures, and configurations.
• Automate performance benchmarking to detect regressions in inference latency, throughput, GPU utilization, and cost.
• Build automated pricing and deployment checks to ensure model pricing, billing configurations, API schemas, and deployments are validated automatically before reaching production.
• Extend our agentic engineering workflows by developing AI-powered automation and agentic coding systems that automatically diagnose CI failures, identify regressions, propose fixes, and streamline engineering workflows.
• Improve deployment reliability by building automated safeguards, deployment verification, rollback mechanisms, and monitoring to ensure new model releases are reliable.
• Eliminate engineering toil by identifying repetitive tasks across the ML team and building tools and systems that automate them, allowing engineers to focus on developing new models and improving performance.

Qualifications

• 5+ years of software engineering background with proficiency in Python and experience building production infrastructure and developer tooling.
• Experience designing and operating CI/CD systems using GitHub Actions or comparable technologies.
• Deep understanding of automated testing, build systems, dependency management, caching, and parallel execution.
• Experience working with containerized workloads, Docker, and cloud infrastructure.
• Ability to design reliable distributed systems and debug complex infrastructure failures.
• Strong understanding of observability, including logs, metrics, tracing, and automated alerting.
• A passion for developer productivity and a demonstrated ability to eliminate manual processes through automation.
• Comfortable working independently, identifying high-impact problems, and building end-to-end solutions.
• Experience with ML infrastructure, PyTorch, GPU workloads, or model-serving systems.
• Familiarity with NVIDIA GPU architectures and multi-GPU environments.
• Experience building automated inference benchmarks or ML quality evaluation frameworks.
• Experience with agentic coding tools such as Codex or Claude Code, or building custom AI engineering agents.
• Experience optimizing CI/CD pipelines at scale, including distributed test execution and ephemeral compute environments.
• Experience developing internal developer platforms or infrastructure-as-code tooling.

Tech Stack

• Python, PyTorch, Docker, GitHub Actions
• NVIDIA GPUs and distributed GPU infrastructure
• Model serving and inference pipelines
• Observability and performance monitoring tools
• AI coding agents and automated engineering workflows
• Access to fal's massive GPU cluster for testing and development

Benefits

• Interesting and challenging work
• A lot of learning and growth opportunities
• Health, dental, and vision insurance (US)
• Regular team events and offsites

U.S. EQUAL EMPLOYMENT OPPORTUNITY INFORMATION

fal provides equal employment opportunities to applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability, or any other classification protected by applicable law.