Note: The job is a remote job and is open to candidates in USA. Ruya AI is a defense-tech company building AI-native intelligent software for sovereign institutions and mission-critical environments. The Senior QA and AI Evals Engineer will build automated test suites, AI evaluation pipelines, and release gates to ensure the reliability of deterministic software behavior and non-deterministic AI outputs. The role also involves validating security and failure scenarios, integrating testing into CI, and partnering with engineers on testable interfaces and observability.
Responsibilities
- Build E2E coverage for investigations, maps, graphs, authentication, and chat
- Develop API, contract, integration, and component tests across TypeScript and Python services
- Create Gherkin/BDD scenarios and build Playwright test cases for important UI flows
- Build AI evaluation datasets, metrics, experiments, and regression gates using Langfuse
- Test streaming disconnects, replay, cancellation, retries, race conditions, and long-running jobs
- Validate authorization boundaries and sensitive-data handling
- Integrate test suites into GitHub Actions and manage flaky-test diagnosis
- Partner with engineers early to create testable interfaces and useful observability
Skills
- Strong Playwright and API/integration testing experience
- Experience with pytest and TypeScript test frameworks such as Vitest
- Understanding of asynchronous and distributed-system failure modes
- Experience evaluating LLM or other non-deterministic systems
- Ability to design high-signal test suites rather than maximizing test counts
- Experience with Langfuse, playwright-bdd, fault injection, contract testing, security testing, durable workflows, or model/prompt deployment gates
Benefits
- Total compensation may also include equity through Ruya AI's employee stock ownership plan (ESOP), a sign-on bonus, and other incentives over time.
- Employees (and their eligible dependents) can enroll in medical, dental, and vision insurance
- Commuter benefits
- Relocation assistance
- Paid time off built into the end of each year (subject to team and business needs) and based on location
- A supportive leave-of-absence program, including time off for military service and medical events
- Most teams offer one or two remote days a week
Company Overview