[Hiring] Applied AI Engineer @Upstream Works Software Careers

Role Description

Upstream Works Software is hiring a contract Applied AI Engineer to contribute to the setup of our agentic AI evaluation and demo environment on Amazon Bedrock AgentCore. You will own the hands-on build:

• Setting up AgentCore
• Creating a golden dataset and evaluation process
• Delivering an end-to-end demo shaped around a real business use case

You will work closely with our Product Lead, who will define the business requirements the demo needs to meet.

Key Responsibilities

• Setup AWS AgentCore
  • Lead the setup of Amazon Bedrock AgentCore in our AWS environment, including Runtime, Observability, and Evaluations.
  • Configure secure access, permissions, and logging following AWS best practices.
  • Document the setup so our team can maintain and extend it.
• Build the golden dataset and evaluation process
  • Work with our Product Lead and subject matter experts to define realistic scenarios, expected responses, and the steps the agent should take.
  • Build and version a golden dataset that reflects real contact centre interactions, using anonymized or synthetic data.
  • Design the evaluation approach, including success criteria, built-in and custom evaluators, and a repeatable process for regression testing as the agent changes.
  • Report on agent quality in a way business stakeholders can understand.
• Build an end-to-end demo in AgentCore
  • Build a working agent and connect it to the tools and data the use case needs.
  • Shape the demo around the business requirements defined with our Product Lead, and iterate based on feedback.
  • Prepare and present the demo to internal stakeholders.

Throughout the engagement, you will share regular progress updates, flag risks early, and leave behind clear documentation and reusable components.

Qualifications

• 3 or more years of hands-on software or AI engineering experience, including building at least one LLM based application or agent that reached production or real users
• Hands-on experience with Amazon Bedrock and Amazon Bedrock AgentCore
• Strong Python skills, including APIs, SDKs, and data handling
• Experience building AI agents with tool or function calling, retrieval, and at least one agent framework (for example, Strands, LangGraph, CrewAI or similar)
• Experience designing evaluations for non-deterministic AI systems, such as golden datasets, success criteria, LLM as a judge, and regression testing
• Working knowledge of core AWS services, including IAM, CloudWatch, Lambda, and S3
• Ability to turn business requirements into a working technical solution with limited direction
• Excellent written and verbal English, and the ability to explain technical results to non-technical stakeholders
• Availability for regular meetings during Eastern Time business hours

Requirements

• Experience with conversational AI or voice agents (for example, Amazon Connect, Cisco or Genesys)
• AWS certifications, such as AWS Certified AI Practitioner, Machine Learning Engineer Associate or Solutions Architect Associate
• Experience with Model Context Protocol (MCP) or AgentCore Gateway
• Experience setting up evaluation in CI/CD pipelines
• Experience handling sensitive customer data, including anonymization and privacy requirements