[Hiring] Principal Software Quality Engineer @Weekday AI

Role Description

We are looking for an experienced Principal Software Quality Engineer to define and drive the quality strategy for a distributed database platform. This is a senior individual-contributor role requiring deep expertise in distributed database testing, systems engineering, debugging, and test automation.

The ideal candidate will have hands-on experience working with distributed database or storage systems, preferably NoSQL platforms. They should be comfortable reading server-side code, tracing complex defects to their root cause, designing fault-injection scenarios, and collaborating with QE, client, and server engineering teams to ensure product reliability.

The role will also lead the integration of AI agents into testing frameworks, enabling automated test generation, failure triage, and evaluation of agent reliability.

Qualifications

• 10–18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, with a strong focus on quality systems and demonstrated Staff or Principal-level technical impact.
• Deep hands-on experience testing distributed databases or storage systems, preferably NoSQL platforms such as Cassandra, ScyllaDB, MongoDB, Couchbase, Redis, DynamoDB, or HBase.
• Strong experience testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.
• Solid understanding of distributed-systems fundamentals, including replication protocols, consensus, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock skew, and failure/recovery semantics.
• Ability to read and debug server-side code written in C, C++, Go, Rust, or Java and trace defects to specific functions and root causes.
• Understanding of protocol and serialization edge cases, including wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.
• Strong software engineering and automation skills in Python, with proficiency in at least one systems programming language.
• Practical experience with fault injection, GDB, chaos-style testing, process termination, network partitions, latency and packet-loss injection, disk faults, corrupted inputs, version skew, and boundary-value testing.
• Experience across multiple testing layers, including unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.
• Proven ability to lead debugging and root-cause investigations across QE, client, and server teams, providing actionable technical analysis and following defects through to resolution.
• Strong CI/CD experience with GitHub Actions, Jenkins, or equivalent tools, along with experience managing shared test infrastructure.
• Practical experience using AI agents for testing, including agent workflows, tool integration, prompt and evaluation design, and integration with real-world testing frameworks.
• Excellent written and verbal communication skills, with the ability to produce clear technical tickets, investigation reports, Slack discussions, and RFCs.

Requirements

• Experience testing client SDKs across multiple programming languages, including Python, Java, Go, C, Node.js, and .NET.
• Knowledge of compatibility matrices, upgrade/downgrade scenarios, and behavioral consistency across client SDKs.
• Experience testing backup and restore, checkpointing, snapshotting, and WAL/journal correctness, including performance at terabyte scale.
• Expertise in query-engine and secondary-index testing, including query planning, index selection, pagination, boundary values, and encoding/collation correctness.
• Kubernetes and database operator testing experience using tools such as KUTTL, Chainsaw, or envtest.
• Cloud infrastructure experience across AWS, GCP, or Azure, including TLS/mTLS, IAM, secrets management, and air-gapped installations.
• Security-focused testing experience, including fuzzing, CVE reproduction, and identifying memory-safety issues such as out-of-bounds reads and use-after-free defects in native code.

Success Metrics

• Critical database workflows, including replication, rebalancing, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication, have comprehensive and reliable test coverage.
• Critical regressions are identified before release through high-signal, low-flakiness automated tests.
• Defect reports consistently include minimal reproducible cases, root-cause analysis, and suspected source-code locations.
• Development teams experience shorter defect-resolution cycles across the areas owned by the role.
• AI agents become an integral part of the testing framework, supporting test generation and failure triage with measurable reliability through evaluation suites.
• Test-farm flakiness decreases, upgrade/downgrade coverage improves, and new features have test plans before implementation is complete.
• Engineering teams across QE, client, and server functions adopt the testing frameworks, practices, and quality standards introduced by the role.

Role Expectations

This is a senior technical individual-contributor position, not a manual QA or people-management role. The candidate is expected to write framework code, debug server processes, investigate low-level system behavior, evaluate AI-agent performance, and build reliable testing infrastructure that other engineers can operate and extend.

Must-Have Skills

• Distributed Database
• Software Testing

Good-to-Have Skills

• Database Testing