Note: The job is a remote job and is open to candidates in USA. ADVATIX - Advanced Supply Chain and Logistics is seeking a Forward Deployed Machine Learning Engineer to establish the technical foundation for AI model benchmarks and evaluations. The role combines machine learning engineering, backend infrastructure, data pipelines, evaluation systems, and customer-facing technical delivery, including owning projects from feasibility through production deployment.
Responsibilities
- Partner directly with the General Manager, researchers, and early customers to define, design, and build AI benchmarks and evaluation systems
- Develop benchmarks and evaluation frameworks across multiple AI domains and modalities
- Build and own backend infrastructure supporting AI model evaluation
- Design and maintain data pipelines, execution environments, storage systems, and orchestration infrastructure
- Build sandboxed environments for agentic evaluations involving tools, code execution, and multi-step tasks
- Develop and deploy end-to-end evaluation systems for foundation models
- Own the engineering component of customer engagements from initial requirements through technical delivery
- Work directly with enterprise customers to understand technical requirements and develop practical evaluation solutions
- Identify repeatable evaluation patterns that can be transformed into scalable infrastructure and product capabilities
- Identify infrastructure gaps and opportunities that can inform future product development
- Scope ambiguous technical problems, assess feasibility, and independently drive solutions through implementation and delivery
- Collaborate with AI researchers and other technical stakeholders to translate research concepts into reliable production systems
- Develop scalable systems capable of supporting large-scale machine learning evaluation and benchmark workloads
- Communicate technical concepts, project status, tradeoffs, and solutions clearly to customers and internal stakeholders
Skills
- Minimum 4+ years of professional engineering experience with hands-on machine learning model evaluation experience
- Minimum 3–8 years of relevant experience across machine learning engineering, evaluation systems, benchmark development, and end-to-end technical ownership
- Demonstrated experience deploying end-to-end ML evaluation or benchmark systems to production for foundation models
- Strong hands-on experience with ML evaluation frameworks and benchmark design, including approaches such as LLM-as-a-judge
- Prior ownership of backend and infrastructure systems, including:
- Data pipelines
- Execution environments
- Storage
- Orchestration
- Experience building data pipelines capable of supporting large-scale workloads
- Experience working with or deploying machine learning systems in production
- Customer-facing engineering experience, including direct interaction with enterprise customers
- Demonstrated ability to independently scope and execute ambiguous technical problems from feasibility through delivery
- Strong written communication skills for customer-facing and cross-functional technical work
- Strong bias toward action and ability to operate effectively in fast-moving, high-ambiguity environments
- Bachelor's degree or higher in Computer Science, Physics, or a related technical field
- Early-stage B2B startups with demonstrated traction
- Forward-deployed engineering organizations or companies with a strong FDE model
- High-ownership generalist roles within larger organizations, such as an Office of the CTO or internal machine learning evaluation team
- Fast-moving engineering environments where individuals have owned projects from initial concept through production deployment
- Working directly with enterprise customers
- Translating customer requirements into technical specifications
- Managing technical components of customer engagements
- Explaining complex ML evaluation concepts clearly to technical and non-technical stakeholders
- Balancing customer-specific requirements with the development of reusable infrastructure
- Working under tight customer deadlines while maintaining engineering quality
- Taking ownership of technical delivery from initial discovery through deployment
- Experience building benchmarks, evaluations, or human-data pipelines for large language models (LLMs) is strongly preferred
- Experience working directly with AI researchers or foundation model labs
- Experience developing evaluation systems for foundation models or LLM-based applications
- Published research, papers, or open-source contributions related to ML evaluations or benchmarks
- GitHub contributions involving ML evaluation, benchmark systems, LLM evaluation, or related infrastructure
- Experience with agentic AI evaluation environments involving tools, code execution, or multi-step workflows
- Experience developing human-data pipelines for machine learning evaluation
- Experience working across multiple AI modalities or evaluation domains
Benefits
- Remote work setup
Company Overview