Note: The job is a remote job and is open to candidates in USA. Data Nexus AI builds AI-powered solutions, including conversational AI and enterprise automation platforms. The AI Quality & Automation Engineer role focuses on testing and optimizing Generative AI and conversational systems, ensuring production-ready systems through quality, benchmarking, safety, and automation.
Responsibilities
- Test Generative AI and conversational systems
- Perform LLM evaluation and benchmark testing
- Validate RAG (retrieval quality, relevance, grounding)
- Build automation frameworks (Playwright/Selenium)
- Conduct API, regression, and E2E testing
- Perform AI safety, red teaming, and bias validation
- Support observability, monitoring, and HITL workflows
- Collaborate with engineering, QA, and DevOps teams
- Support CI/CD quality pipelines
Skills
- QA & automation testing
- Playwright/Selenium, API testing (Postman, REST)
- GenAI & RAG validation, LLM evaluation
- Prompt engineering, conversational AI testing
- HITL testing, AI safety & observability
- SQL, Git, Python, CI/CD concepts
- OpenAI, Azure OpenAI, ChatGPT, Claude
- LangChain, CrewAI, LangGraph, MCP
- Vector DBs (Pinecone, ChromaDB, Weaviate)
- RAGAS, DeepEval
- Python/JS, Docker, Kubernetes, GitHub Actions
Company Overview