[Hiring] Principal Engineer - Agentic Security Operations @First American

Role Description

As a member of our Threat Detection Engineering team operating within our SOC, you will build governed agent capabilities to support data onboarding, detection engineering, threat hunting, investigation, response, automation, and platform operations. The Principal Engineer provides hands-on technical leadership to turn this strategy into secure, reliable, and supportable production capabilities across the SOC.

• Lead the architecture and hands-on implementation of Agentic Security Operations capabilities across the SOC.
• Define and evolve the reference architecture for governed data services, agent capabilities, workload routing, execution environments, and deterministic orchestration.
• Build reusable capabilities that translate operational intent into tested, reviewable, and auditable security outcomes.
• Establish engineering standards for creating, testing, registering, approving, releasing, monitoring, and retiring agents and their capabilities.
• Advance detection-as-code practices, including versioning, simulation, deployment, tuning, migration, and rollback.
• Build governed knowledge and context services that provide authoritative documentation, policies, operational history, citations, freshness information, and permission-aware retrieval.
• Design workload identity, least-privilege access, short-lived authorization, spend controls, approval gates, audit trails, rollback, and emergency shutdown mechanisms.
• Define autonomy levels that determine which activities may execute automatically, require human approval, or remain analyst led.
• Establish agent evaluation and observability through historical replay, synthetic scenarios, expected-versus-actual comparisons, policy tests, traces, outcome verification, and regression testing.
• Measure quality, security, reliability, latency, cost, analyst acceptance, and operational impact.
• Engineer reliable integrations across search, case management, workflow, communications, and response systems, including retries, idempotency, partial failures, and reconciliation.
• Ensure consequential response actions remain deterministic, attributable, reversible, and subject to appropriate human oversight.
• Design production reliability, continuity, recovery, and support models for agent capabilities and security data services.
• Lead proofs of concept and architecture evaluations using measurable criteria for security, fidelity, scale, reliability, portability, operating effort, and economic value.
• Collaborate with enterprise architecture, infrastructure, data, risk, privacy, legal, and compliance teams.
• Mentor engineers, establish technical standards, review designs, and influence Threat Detection Engineering strategy.
• Present complex technical decisions, risks, tradeoffs, and investment recommendations clearly to engineering and executive stakeholders.
• Partner across Threat Detection and Response groups, including engineering, threat hunting, security analysis, investigation, and incident response stakeholders on initiatives and to translate operational feedback into improvements within our agentic harness, evaluations, safeguards, workflows, and other platforms.

Qualifications

• Extensive experience designing and operating enterprise security, software, data, or distributed systems at scale.
• Demonstrated success as a principal-level technical leader while remaining actively involved in architecture, prototyping, coding, testing, and production delivery.
• Hands-on experience delivering production AI-assisted or agentic systems that use tools, memory, retrieval, planning, and structured workflows.
• Strong understanding of security operations, including telemetry management, detection engineering, threat hunting, triage, investigation, response, and automation.
• Experience with high-volume streaming and batch data pipelines, object-based storage, elastic analytics, distributed query, schema management, and data-quality controls.
• Experience designing modular agents, reusable capabilities, workload routing, local and managed execution, and human-in-the-loop workflows.
• Deep understanding of grounding, retrieval, source attribution, context freshness, permission inheritance, and protection against untrusted or manipulated context.
• Experience evaluating nondeterministic systems using versioned test sets, historical replay, synthetic scenarios, regression testing, and measurable outcome criteria.
• Strong software-engineering practices, including source control, automated testing, continuous delivery, infrastructure automation, peer review, and rollback.
• Experience designing secure APIs, event-driven integrations, workload identities, authorization boundaries, secrets management, and auditable action execution.
• Knowledge of AI security risks such as prompt injection, data poisoning, excessive agency, sensitive-data exposure, insecure tool use, and model or workflow drift.
• Strong production-engineering experience covering observability, service objectives, capacity, failure isolation, continuity, recovery, and operational support.
• Ability to evaluate emerging capabilities critically, distinguish demonstrated functionality from product claims, and validate architectural assumptions through controlled testing.
• Excellent written and verbal communication skills, including the ability to document architectures, operating models, engineering standards, and executive recommendations.
• Ability to lead through influence and work effectively across organizational and technical boundaries.
• Hands-on experience with cloud-scale data and analytics platforms supporting security operations, including distributed query, high-volume data processing, object storage, schema management, data pipelines, and the operational patterns required to run security analytics at scale (e.g., Databricks, Snowflake, etc.).
• Experience operating and migrating security analytics and detection workloads across modern platforms, along with strong working knowledge of major public cloud environments (AWS, Azure, GCP) and the identity, networking, logging, data protection, and workload-security controls required to secure them.
• A minimum of 10 years’ experience in cybersecurity or relevant fields.
• Bachelor’s degree in computer science, Information Technology, Cybersecurity, a related field, or equivalent work experience.
• Preferred certifications such as, GASAE, GAIPS, GDSA, GCDA, CEH, OSCP, CISSP, etc.

Benefits

• Comprehensive benefits package including medical, dental, vision, 401k, PTO/paid sick leave.
• Employee stock purchase plan.

What Success Looks Like

• A leadership-approved Agentic Security Operations reference architecture and governance model.
• Governed agent prototypes progressing into reliable production capabilities.
• Reusable capabilities spanning data, detection, automation, and other SOC functions.
• Measurable reductions in engineering effort and operational cycle time without sacrificing security or human accountability.
• Continuous evidence that agent capabilities are accurate, secure, reliable, explainable, cost-effective, and safe to operate.
• A production platform with clear ownership, support procedures, recovery controls, and an intentional path away from vendor or architecture lock-in.