[Hiring] Software Development Engineer in Test @Newfold Digital

Role Description

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

What you'll do & how you'll make your mark

• AI/LLM Quality Engineering
  • Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
  • Develop AI evaluation (Evals) frameworks for:
    • Hallucination detection
    • Groundedness
    • Toxicity & bias validation
    • Response quality
    • Context relevance
    • Prompt regression testing
    • Determinism & consistency
    • Safety and guardrail validation
  • Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
  • Create synthetic datasets and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and agent memory/context retention.
  • Validate MCP (Model Context Protocol) integrations and tool interoperability.
  • Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
  • Evaluate latency, token usage, model cost optimization, throughput, and scalability.
• Automation Engineering
  • Develop and maintain scalable UI/API automation frameworks using:
    • Python (mandatory)
    • Robot Framework (preferred)
    • Playwright/Selenium/Cypress.
  • Automate testing for:
    • REST APIs
    • GraphQL APIs
    • Event-driven systems
    • Streaming systems
    • Webhooks
    • Microservices
    • MFE (Micro Frontend) architectures.
  • Build reusable automation libraries, utilities, and AI-assisted testing workflows.
  • Integrate automation into CI/CD pipelines and quality gates.
• AI Engineering & Platform Validation
  • Validate RAG architectures including:
    • Chunking strategies
    • Embedding quality
    • Retrieval accuracy
    • Re-ranking
    • Vector search relevance
  • Work with vector databases such as:
    • Pinecone
    • Weaviate
    • ChromaDB
    • Qdrant
    • FAISS
  • Build or validate AI agents leveraging:
    • LangChain
    • LangGraph
    • CrewAI
    • AutoGen
    • Semantic Kernel
    • OpenAI SDKs
  • Validate AI workflows involving:
    • Tool calling
    • Function calling
    • Agent memory
    • Agent planning/execution
    • Multi-modal AI systems
    • Structured outputs
• DevOps, Observability & Reliability
  • Build and maintain quality workflows integrated with:
    • GitHub Actions
    • Jenkins
    • GitLab CI/CD
    • Azure DevOps
  • Work with Docker and Kubernetes environments.
  • Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
  • Validate distributed systems reliability and resiliency.
  • Monitor AI system behaviors using observability platforms/logging/tracing tools.
  • Contribute to shift-left and quality engineering best practices.

Qualifications

• Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred.
• Equivalent hands-on experience and demonstrated projects are highly valued.
• Strong programming experience in Python.
• Strong understanding of Software Testing & Quality Engineering principles.
• Hands-on experience with:
  • AI/LLM testing
  • Prompt Engineering
  • Context Engineering
  • AI Evals
  • RAG systems
  • AI Agents
• Experience with:
  • CI/CD pipelines
  • Docker & Kubernetes
  • Cloud platforms (AWS/Azure/OCI/GCP)
  • API testing & automation
  • Microservices testing
  • UI automation
• Experience with SQL and data validation.
• Strong debugging, analytical, and troubleshooting skills.
• Experience working in Agile/Scrum environments.

Requirements

• Good to have skills:
  • Experience with Robot Framework.
  • Experience with Playwright.
  • Knowledge of:
    • LangChain
    • LangGraph
    • LlamaIndex
    • CrewAI
    • Semantic Kernel
    • MCP (Model Context Protocol)
  • Hands-on experience with:
    • Cursor
    • GitHub Copilot
    • Claude
    • Gemini
    • OpenAI APIs
  • Experience with vector databases and embeddings.
  • Exposure to MLOps workflows and ML lifecycle testing.
  • Knowledge of performance testing and security testing.
  • Experience validating AI governance/compliance requirements.
  • Experience with contract testing, chaos testing, or resilience engineering.
  • Familiarity with accessibility testing and cross-browser/device testing.
  • Exposure to synthetic monitoring and production observability.

Benefits

• Work-life balance.
• Competitive HMO benefits – 175k MBL with one free dependent upon one year of service.
• Punctuality Bonus.
• Generous Vacation policy.
• Programs that foster a culture of belonging and provide learning and development opportunities for all employees.
• 20+ affinity groups for networking and connection globally.