Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC connects experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. The AI Software Engineering Domain Expert will review and improve AI-generated technical content, develop and refine prompts, evaluate outputs against rubrics, and produce technical documentation and architecture analysis. The role also involves research, fact checking, data annotation, and professional technical writing.
Responsibilities
- Review AI-generated software-engineering content for accuracy, clarity, and completeness
- Evaluate technical recommendations against established engineering best practices
- Identify incorrect assumptions, architectural weaknesses, or incomplete reasoning
- Assess whether technical outputs are practical and internally consistent
- Provide structured feedback supporting measurable improvements in quality
- Create prompts and instructions for complex software-engineering tasks
- Refine existing prompts to improve clarity, scope, and technical depth
- Design tasks that test analytical reasoning and engineering judgement
- Evaluate how prompt structure influences model behaviour and output quality
- Contribute to development of robust software-engineering evaluation scenarios
- Assess AI-generated outputs using detailed evaluation rubrics
- Evaluate correctness, completeness, relevance, and technical quality
- Apply consistent judgement across repeated evaluation tasks
- Identify edge cases or ambiguities requiring additional interpretation
- Provide clear written justification for evaluation decisions
- Draft and review technical documentation
- Develop and evaluate architecture plans
- Review RFCs, design specifications, and engineering proposals
- Improve technical materials for structure, clarity, and precision
- Ensure documentation reflects sound software-engineering reasoning and implementation considerations
- Apply professional engineering judgement to technical scenarios
- Evaluate architectural and implementation trade-offs
- Analyse system-design and software-development decisions
- Identify maintainability, scalability, reliability, and performance considerations
- Translate complex engineering concepts into clear analytical conclusions
- Conduct independent research to validate technical assertions
- Fact-check engineering concepts, technologies, and implementation claims
- Compare outputs against reliable technical documentation where appropriate
- Identify outdated, unsupported, or misleading technical information
- Maintain high standards of technical accuracy throughout project work
- Interpret and annotate technical data and engineering content
- Support nuanced labelling and evaluation workflows
- Review content against project-specific standards and rubrics
- Identify inconsistencies across datasets or evaluation examples
- Contribute to quality-assurance and calibration processes
- Produce clear professional and technical written materials
- Edit complex engineering content for accuracy and readability
- Improve technical reports, documentation, and analytical summaries
- Adapt explanations for technical and non-technical audiences
- Maintain consistent editorial and technical standards
Skills
- 3+ years of professional software-engineering experience
- Background as a Software Engineer, Senior Software Engineer, Staff Engineer, Technical Lead, Engineering Manager, Solutions Architect, or comparable role
- Strong critical-thinking and analytical-reasoning skills
- Demonstrated ability to solve complex technical problems logically
- Strong attention to detail and commitment to quality assurance
- Experience authoring or reviewing technical documentation
- Familiarity with architecture documents, design specifications, RFCs, engineering proposals, postmortems, technical blogs, or code reviews
- Excellent written communication and technical-editing skills
- Ability to communicate sophisticated engineering concepts clearly
- No prior AI-training experience is required
- Fully remote
- Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party
- Experience with prompt authoring, content evaluation, data annotation, or rubric-based assessment is advantageous
- Familiarity with AI coding assistants or automated documentation tools is beneficial but not required
- Experience with AI-model evaluation, RLHF, or prompt engineering is advantageous
- Advanced academic qualifications such as a Master's, JD, MBA, or PhD are valued but not required
Benefits
- Fully remote
- Collaboration will primarily occur asynchronously with project leads and other subject-matter experts
Company Overview