Note: The job is a remote job and is open to candidates in USA. Crossing Hurdles is seeking a contract Software Engineer to evaluate AI-powered developer tools and technical outputs. The role involves creating reproducible development and CI environments, troubleshooting software integrations, documenting product behavior, and participating in evaluation calibration sessions.
Responsibilities
- Execute structured evaluation assignments simulating developer activities using AI-powered tools
- Assess and grade technical correctness of outputs within development environments
- Design and maintain reproducible test environments with real commit histories and CI workflows
- Configure and troubleshoot integrations across multiple software services for seamless connectivity
- Investigate and document undocumented product behaviors through systematic testing
- Participate in calibration sessions to ensure consistent evaluation methods
Skills
- Have strong relevant experience in software engineering with Git/GitHub workflows
- Proficient with CI/CD systems, ideally GitHub Actions, and authoring custom workflows
- Advanced scripting abilities in Python or Bash, with understanding of REST APIs and OAuth
- Fluency with major enterprise SaaS platforms; administrative experience in cloud workspaces is advantageous
- Background in technical QA, evaluation, or data annotation with consistency in applying rubrics
- Experience leveraging AI assistants or agent tools in developer tasks is a plus
Benefits
- Remote work
Company Overview