Note: The job is a remote job and is open to candidates in USA. An unnamed client is seeking engineering experts to design and validate complex simulation problems for training and evaluating advanced AI models and coding agents. The role involves engineering simulation, benchmark development, and analysis of AI execution logs and failure patterns.
Responsibilities
- Develop original, self-contained engineering design problems with realistic technical requirements, competing constraints, and clearly defined optimization objectives
- Create challenging tasks that require AI models to interpret specifications, evaluate design trade-offs, configure simulation environments, and iteratively improve engineering solutions
- Establish validated reference solutions and objective automated grading mechanisms to assess correctness, performance, and compliance with engineering requirements
- Build, configure, execute, and validate simulation-based problem environments using relevant open-source engineering tools
- Develop Python scripts, custom test benches, and supporting utilities to automate simulations and verify expected results
- Ensure that simulation outputs satisfy physical plausibility, dimensional consistency, boundary conditions, convergence requirements, and other domain-specific constraints
- Troubleshoot simulation failures, numerical instability, dependency issues, and computational errors
- Evaluate the performance of large language models (LLMs) and coding agents across repeated engineering problem-solving trials
- Analyze execution trajectories, tool interactions, simulator feedback, and intermediate outputs to identify systematic reasoning failures
- Investigate issues such as incorrect interpretation of engineering specifications, premature convergence, invalid geometries, inconsistent units, and physically impossible solutions
- Apply evaluation concepts such as pass@k, failure-mode classification, and analysis of nondeterministic model behavior
- Calibrate task difficulty using empirical model performance and evaluation results
- Refine engineering problems to improve their ability to distinguish between successful and unsuccessful AI solutions
- Ensure that tasks remain unambiguous, technically rigorous, reproducible, and objectively gradable
- Collaborate with AI researchers, technical leads, and subject-matter experts to integrate validated benchmarks into AI evaluation workflows
Skills
- 10+ years of hands-on engineering design experience
- Master's degree or PhD in Electrical, Mechanical, Aerospace Engineering, or a closely related discipline
- Master's degree or PhD in Electrical Engineering, Mechanical Engineering, Aerospace Engineering, or a closely related technical discipline
- At least 10 years of practical engineering design experience
- Strong Python programming and scripting skills
- Hands-on proficiency with at least one relevant open-source engineering simulation or design tool, such as:
- + **Electrical simulation:** ngspice, PySpice
- + **Mechanical and computational engineering:** OpenFOAM, FEniCSx, CalculiX
- + **Control systems:** python-control, OpenModelica
- + **CAD and design automation:** CadQuery, build123d
- + **Thermodynamics and meshing:** Cantera, Gmsh
- Experience developing, executing, debugging, and validating simulation-based engineering workflows
- Familiarity with modern LLMs, AI coding agents, and AI evaluation methodologies
- Ability to analyze model execution logs and identify underlying reasoning, planning, and tool-use failures
- Strong understanding of physical constraints, units, boundary conditions, numerical convergence, and engineering validation
- Excellent analytical, problem-solving, and technical documentation skills
- Ability to work independently and deliver technically accurate results within defined timelines
- Candidates should have strong expertise in one or more of the following areas:
- * Electrical Engineering
- * Mechanical Engineering
- * Aerospace Engineering
- Experience in AI model evaluation, benchmark development, automated grading, or coding-agent assessment
- Familiarity with repeated-trial evaluation, pass@k metrics, failure taxonomies, and nondeterministic model behavior
- Experience building automated testing frameworks and reproducible computational environments
- Knowledge of optimization methods, numerical analysis, simulation automation, and engineering design validation
- Experience collaborating with multidisciplinary engineering and AI research teams
Benefits
- Remote work arrangement
- Part-time engagement may be acceptable where agreed
Company Overview