Role Description
We are seeking a Principal Site Reliability Engineer (SRE) to lead the design and implementation of resilient, observable, and high-performing systems across our organization. This role is ideal for a strategic thinker and hands-on technologist who thrives in complex environments and is passionate about reliability, automation, and innovation—especially at the intersection of SRE and AI.
You’ll enjoy the flexibility to work remotely from anywhere within the U.S. as you take on some tough challenges. For all hires in the Minneapolis or Washington, D.C. area, you will be required to work in the office a minimum of four days per week.
Primary Responsibilities:
-
Observability & Monitoring:
-
Lead the implementation and standardization of OpenTelemetry across services to enhance observability and traceability.
-
Define and enforce SLIs, SLOs, and error budgets in collaboration with engineering teams.
-
Resiliency Engineering:
-
Design and execute resiliency tests, disaster recovery (DR) exercises, and chaos engineering game days to proactively identify and mitigate system weaknesses.
-
Develop automated failure injection and recovery validation tools.
-
CI/CD & Performance Engineering:
-
Enhance CI/CD pipelines with automated performance and load testing to ensure reliability and scalability before production deployment.
-
Collaborate with DevOps and QA to integrate performance benchmarks into release gates.
-
Cloud Architecture & Reliability:
-
Drive cloud adoption strategies with a focus on resiliency patterns, multi-region failover, and cost-effective scaling.
-
Partner with cloud architects to design fault-tolerant infrastructure and services.
-
AI & Innovation in SRE:
-
Explore and implement AI-driven solutions for anomaly detection, incident prediction, and intelligent alerting.
-
Innovate with AI agents to automate routine SRE tasks and improve incident response efficiency.
-
Leadership & Mentorship:
-
Serve as a thought leader and mentor for SRE best practices across the organization.
-
Lead cross-functional initiatives to improve system reliability, developer productivity, and customer experience.
Qualifications
-
10+ years of experience in software engineering, DevOps, or SRE roles, with at least 3+ years in a principal or lead capacity.
-
5+ years of experience with CI/CD tooling (e.g., Jenkins, GitHub Actions, ArgoCD).
-
5+ years of experience with container orchestration in cloud platforms (Azure or AWS preferred).
-
3+ years of deep experience in observability and monitoring tools (e.g., OpenTelemetry, Prometheus, Grafana, Datadog).
-
3+ years of experience with chaos engineering, DR planning, and performance testing.
Preferred Qualifications
-
Bachelor's degree in Computer Science, Information Technology or related field.
-
Hands-on experience with infrastructure as code (Terraform, Pulumi) and automation tools such as Ansible, Helm.
-
Experience with service mesh technologies (e.g., Istio, Linkerd).
-
Familiarity with AI/ML concepts and experience applying them in operational contexts.
-
Proven excellent communication and leadership skills.
Benefits
-
Comprehensive benefits package.
-
Incentive and recognition programs.
-
Equity stock purchase and 401k contribution (all benefits are subject to eligibility requirements).