[Hiring] Senior Storage Production Engineer @NVIDIA

Role Description

Production engineering is a field that involves crafting, building, and maintaining large-scale production systems with high efficiency and availability. It encompasses various areas, including software and systems engineering practices, storage, data management, and services. Professionals in the role of Production Engineers hold specialized knowledge and expertise across various domains, including:

• Storage architecture
• High-performance distributed storage
• Data management
• Systems
• Networking
• Coding
• Database management
• Prioritization
• Continuous delivery and deployment
• Open-source cloud-enabling technologies (e.g., Kubernetes, containers, virtualization)

Their responsibilities include:

• Ensuring storage architectures are reliable, scalable, and efficient
• Optimizing data placement and access patterns
• Managing large-scale distributed storage systems
• Ensuring low-latency data access for HPC and AI/ML workloads

Storage Production Engineers at NVIDIA ensure that our internal and external-facing GPU cloud services meet reliability and uptime goals as promised to the users while enabling developers to make changes to the existing system through careful preparation and planning while keeping an eye on capacity, latency, and performance. This role also requires a mindset focused on:

• Automating storage operations
• Improving data access efficiency
• Optimizing storage performance

Much of our software development focuses on:

• Optimizing operations through automation
• Enhancing system responsiveness
• Improving the efficiency of storage and production systems

Since Production Engineers are responsible for the big picture of how our systems interface with each other, we use a breadth of tools and approaches to tackle a broad spectrum of challenges. Practices such as:

• Proactive storage performance monitoring
• Automated fault detection and remediation
• Scalable data redundancy methods
• Integration of intelligent caching mechanisms

factor into iterative improvements that are key to system reliability and efficiency.

What You Will Be Doing:

• Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity
• Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues
• Work with AI/ML workloads to improve storage architectures for low-latency access, efficient caching, and high-throughput performance
• Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization
• Support storage services before they become available through activities such as system build consulting, developing automation frameworks, capacity management, and launch reviews
• Maintain production storage infrastructure by supervising availability, latency, and system health, leveraging predictive analytics and AI-driven automation
• Optimize storage efficiency through compression, deduplication, tiering strategies, and intelligent workload placement
• Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques
• Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems
• Practice sustainable incident response and blameless root cause analysis
• Be part of an on-call rotation to support storage and production systems

Qualifications

• BS degree or equivalent experience in Computer Science, Storage Systems, or a related technical field with 8+ years of practical experience
• Experience with distributed and high-performance storage solutions, including clustered and parallel file systems, distributed object storage, and enterprise-grade storage systems
• Solid understanding of block, file, and object storage technologies, including their scalability, reliability, and performance characteristics and standard processes
• Experience with storage networking protocols such as NFS, SMB, iSCSI, S3, Fibre Channel, RDMA, and NVMe over Fabrics
• Expertise in algorithms, data structures, complexity analysis, software design, and automating maintenance of large-scale Linux-based storage systems
• Experience in one or more of the following: C/C++, Java, Python, Go, NodeJS, and Bash for storage automation, monitoring, and performance tuning
• Hands-on experience with infrastructure configuration management tools like Ansible, Chef, Puppet, and Terraform for automating storage deployments
• Experience with observability and tracing tools like InfluxDB, Prometheus, Grafana, and the Elastic stack for monitoring storage system health
• Excellent written and oral communication skills
• Excellent work ethics and a deep sense of teamwork
• Commitment to producing quality work and finishing tasks every single day

Ways to stand out from the crowd:

• Deep understanding of extensive distributed storage systems, replication strategies, and erasure coding techniques
• Experience in capacity planning, performance tuning, and troubleshooting high-throughput storage systems
• Experience with Git, code review, pipelines, and CI/CD for handling infrastructure as code
• Experience in analyzing and improving distributed storage system performance at scale
• Strong debugging skills with a systematic problem-solving approach to identify sophisticated storage issues
• Proven understanding of network protocols, architectures, and troubleshooting techniques, especially as it relates to storage performance, stability, and availability
• Experience using or operating private and public cloud storage solutions based on Kubernetes, OpenStack, or hybrid cloud architectures
• Ability to design and implement automated storage migration, backup, and disaster recovery strategies
• Thrive in collaborative environments and enjoy working with various teams to optimize storage performance
• Flexible in adapting to different working styles and emerging storage technologies