Note: The job is a remote job and is open to candidates in USA. Invovia is seeking professionals to support the evaluation and measurement of AI-powered software development systems. The roles focus on developing evaluation metrics and analytical methods, building reproducible benchmarking and automation tooling, validating results, and generating insights into AI system performance, quality, productivity, and efficiency.
Responsibilities
- Support development and refinement of evaluation metrics, scoring approaches, and benchmark methodologies
- Validate datasets, data quality criteria, and evaluation inputs to improve reliability of results
- Analyze evaluation data, performance trends, and measurement signals to generate actionable insights
- Apply statistical analysis and experimentation techniques to assess AI system effectiveness
- Develop analytical approaches to identify trends, patterns, and opportunities for improvement
- Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks
- Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time
- Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable
- Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency
- Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and find ways to make the workflows more reliable and more automated
- Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences
Skills
- Strong background in data science, analytics, statistics, or related fields
- Experience designing metrics, analytical frameworks, or measurement approaches
- Proficiency in Python, SQL, and data analysis techniques
- Experience working with large datasets and extracting meaningful insights
- Strong communication skills with the ability to explain analytical results clearly
- Strong software engineering background, with real experience building automation, developer tooling, or test and validation systems
- Proficient in Python, and comfortable in at least one of Java, JavaScript, or a similar language
- Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker
- Experience with APIs, development environments, CI/CD pipelines, and typical engineering workflows
- Some familiarity with how AI, LLM, or agent evaluation works and where it goes wrong, such as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens
- Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully
- Experience evaluating AI/ML systems and agentic AI applications
- Experience with experimentation, benchmarking, or performance analysis
- Experience working with software engineering, productivity, or quality metrics
Company Overview
Company H1B Sponsorship