[Hiring] Senior Principal / Architect Engineer, Software - Backend, AI Observability/Evaluations @Palo Alto Networks

Role Description

We are looking for experienced engineers with a strong product sense who asks “why” as often as “how”. You’ll partner with product managers, engineers, and customers to translate emerging AI observability and evaluation needs into scalable backend systems and cohesive developer workflows. You’ll help shape the technical roadmap and architecture, taking features from early exploration through production delivery.

• Build backend capabilities that help developers observe and evaluate AI models, agents, and RAG applications before and during production.
• Design systems for systematic testing, measurement, and benchmarking of AI agents and applications accuracy, relevance, performance, and safety.
• Build scalable APIs and data processing systems that contextualize AI telemetry and evaluation results, helping customers understand application behavior and identify opportunities for improvement.
• Help shape the technical roadmap and architecture for AI Observability and Evaluation, building on Chronosphere’s rapidly scaling platform.
• Collaborate with product managers, deeply technical teammates, and customers to scope, validate, and deliver integrated features and developer workflows.
• Improve our own system observability (we use our own product heavily).
• Improve the performance, reliability, and cost efficiency of the distributed systems powering AI Observability and Evaluation.
• Mentor engineers and help maintain our engineering standards.

Qualifications

• 12-15+ years of experience in software engineering.
• Strong experience in one or more backend languages (e.g., Go, Java, Python, Rust).
• Some experience developing LLM powered product features is desirable.
• Experience using AI coding assistants (e.g., Cursor, Codex, Claude) as part of day-to-day software development.
• Experience building and operating highly available systems in production.
• Experience working with cloud infrastructure (AWS, GCP, Azure).
• Experience designing APIs (REST, gRPC, GraphQL) for scale.
• Experience working effectively with cross functional partners.
• Experience owning and executing projects from inception to delivery.
• Experience working with a high degree of autonomy with good judgement on when to push forward and when to ask for help.
• An effective collaborative working style and willingness to build and maintain a positive and supportive environment.
• An ability to operate effectively in a remote first environment.

Company Description

Chronosphere, a Palo Alto Networks company, is the observability platform built for control in the modern, containerized world. Chronosphere empowers customers to focus on the data and insights that matter by reducing complexity, optimizing costs, and remediating issues faster. Chronosphere reduces data volumes and associated costs by 84% on average while saving developers thousands of hours. Recognized as a leader by major analyst firms, Chronosphere is trusted by the world's most innovative brands, including DoorDash, Affirm, and Zillow.