[Hiring] Gen AI Site Reliability Engineer @PwC

Monitoring, Alerting & Service Health:

• Build and maintain dashboards and alerts for availability, latency, error rates, and overall service health for in-scope GenAI services.

• Tune thresholds and alert routing to reduce noise and improve actionable detection, improving MTTA and MTTR.

Incident Triage, Investigation & Restoration:

• Triage incidents, gather evidence, and perform structured troubleshooting using logs/metrics/traces and documented runbooks.

• Execute restoration steps and coordinate escalations to platform owners, engineering teams, or vendors for complex issues.

• Provide clear technical updates during live events and document resolution details for future reference and trend analysis.

Problem Prevention & Reliability Improvements:

• Contribute to root-cause investigations and implement corrective actions (monitoring improvements, configuration changes, resilience enhancements).

• Identify recurring failure modes and propose fixes that reduce repeat incidents and improve overall service stability.

• Support verification of corrective actions by monitoring outcomes and validating that improvements reduce incident recurrence.

Performance Troubleshooting Support:

• Assist with latency and error investigations by gathering diagnostics, isolating contributing factors, and proposing mitigations.

• Partner with engineering teams to validate fixes and monitor post-deployment impact on service health and performance.

Automation & Scripting:

• Automate diagnostics and routine operational tasks to reduce manual effort and improve consistency (scripts, repeatable checks, standardized steps).

• Maintain and document operational scripts and ensure they are usable and supportable by the broader team.

Documentation & Knowledge Management:

• Maintain runbooks, troubleshooting guides, and knowledge articles for frequent scenarios and standard operating procedures.

• Document known issues, standard resolutions, and escalation paths to improve first-time fix rate and onboarding efficiency.

Change Readiness & Post-Change Validation:

• Support operational readiness for changes by validating monitoring readiness, runbook updates, and post-change verification steps.

• Execute post-change checks and report regressions or unexpected behavior promptly to ensure rapid remediation.

Continuous Improvement & Service Reporting Inputs:

• Identify operational pain points and recommend improvements to monitoring, alerting, runbooks, and support workflows.

• Provide inputs to service reporting on incident trends, recurring issues, and improvement opportunities related to GenAI reliability.

Quality, Controls & Operational Discipline:

• Follow defined operational processes (incident, request, change) and maintain high-quality ticket hygiene and documentation discipline.

• Comply with security and access controls for supported tools and environments; proactively raise operational risks or control gaps for mitigation.

Collaboration & Team Support:

• Collaborate with peers and leads to coordinate workload, share knowledge, and support consistent execution standards across the pod.

• Support onboarding and knowledge transfer by maintaining clear documentation and participating in team enablement activities.

Required Skills:

• Hands-on experience supporting production services in a cloud environment, including monitoring, troubleshooting, and incident response.

• Experience building dashboards and alerts and working with logs/metrics/traces to diagnose issues and reduce time to restore.

• Strong analytical problem-solving skills and ability to implement reliability improvements and corrective actions in a controlled manner.

• Experience working within ITIL-aligned processes (incident, request, change) and maintaining runbooks/knowledge articles.

• Preferred: experience with ITSM and observability tooling (e.g., client ITSM and monitoring tools; ServiceNow, CloudWatch, Datadog, Splunk, New Relic). Familiarity with GenAI services (AWS Bedrock, OpenAI/ChatGPT Enterprise) is desirable. AWS certifications are highly preferred.