About Bidgely
Bidgely is a Silicon Valley AI company transforming how utilities and their customers understand and manage energy. The UtilityAI platform turns smart-meter data into appliance-level insights, and Agentic CX brings GenAI assistants to utility chat, voice IVR and customer service. Leading utilities across North America, Europe and Asia rely on Bidgely, and the Bengaluru engineering center is a certified Great Place to Work.
The Role
We are hiring a senior SDET to own quality across customer-facing products: web dashboards, APIs, notifications, and GenAI products such as chatbots, voice IVR and AI agents. You will define how AI agents are tested, build automated evals, and set the quality criteria for release. You will also own test strategy for your area, and build and improve UI, API and end-to-end automation and regression coverage. This is a hands-on engineering role working closely with GenAI engineering, CX engineering and product teams.
Key Responsibilities
Own test strategy and release quality criteria for your product area.
Build and maintain UI automation (Selenium or Playwright), API automation and end-to-end regression suites.
Design evaluation strategy for AI agents using LLM-as-judge, rubrics and other eval techniques.
Build eval datasets and automated evaluation suites in Python, integrated with CI/CD.
Test AI agents for accuracy, consistency, tool usage, safety and data handling.
Debug agent failures using LangSmith and similar AI observability tools.
Analyse release-over-release quality and identify regressions.
Review product requirements and identify gaps in requirements and test coverage.
Review test code, set testing standards and mentor SDETs.
Communicate findings and recommendations to engineering, product and leadership.
Requirements
6–9 years in software testing, including hands-on SDET experience.
Good working knowledge of both Python and Java.
Experience owning test strategy and improving automation and release quality.
Experience with UI automation using Selenium or Playwright.
Experience building API automation and end-to-end frameworks with CI/CD.
Ability to analyse product requirements and identify testing gaps.
Prior experience testing chatbots, IVRs and GenAI agents.
Experience designing evals, rubrics and LLM-as-judge evaluations.
Working knowledge of LangChain, LangGraph and LangSmith or similar tools.
Experience building eval harnesses or tooling for AI agents.
Strong analytical and communication skills.
Tech You Will Work With
Python, Java, Selenium, Playwright, REST API testing, Postman, TestNG, PyTest, LangChain, LangGraph, LangSmith, LLM evaluation (evals), LLM-as-judge, RAG testing, AI agent testing, end-to-end testing, regression testing, CI/CD, GitHub Actions, Jenkins, AWS.
6-9 years software testing experience; Python and Java; UI automation with Selenium or Playwright; API and end-to-end frameworks with CI/CD; experience testing chatbots, IVRs, GenAI agents; knowledge of LangChain, LangGraph, LangSmith.
Key Responsibilities
- owning test strategy
- building automation
- evaluating AI agents
Skills & Tools
Python, Java, Selenium, Playwright, REST API, Postman, TestNG, PyTest, LangChain, LangGraph, LangSmith, LLM evaluation, LLM-as-judge, RAG testing, AI agent testing, end-to-end testing, regression testing, CI/CD, GitHub Actions, Jenkins, AWS
Job Details
- Category: Software Development
- Seniority: Senior Level
- Commitment: Full Time
- Workplace: Remote — India
- Languages: English
About Bidgely
A Silicon Valley AI company transforming how utilities and customers understand and manage energy. — Industry: Information Technology