Note: The job is a remote job and is open to candidates in USA. Upscale AI is building high-performance infrastructure for demanding artificial intelligence workloads. The Forward-Deployed Engineer owns complex AI fabric and system issues end-to-end, working directly with customers, reproducing issues in the lab, developing workarounds, and contributing fixes and diagnostic capabilities to the product and AI monitoring agent.
Responsibilities
- Triage and resolve complex AI fabric issues: including silent packet drops, queue anomalies, NCCL stalls, gray failures, and performance degradation in production environments. You own the problem from intake to resolution
- Reproduce issues in the lab: building test scenarios that isolate L2 and L3 problems (packet loss, latency, retransmits, ECMP behavior, PFC/ECN interactions) and deliver reproducible cases to the development team when code fixes are needed
- Act as the escalation buffer for engineering: resolving issues without engaging development when possible, and packaging clean, reproducible problem statements when development engagement is required
- Train and improve the on-box and off-box AI agents: contributing field-validated detection signatures, classification logic, and resolution recommendations based on real cases. Your field experience directly shapes what the agent can identify and handle autonomously
- Validate AI agent accuracy: reviewing the agent's data collection, anomaly detection, and triage classifications against real-world outcomes. You determine when the agent is ready to advance from data collection to active triage to mitigation
- Work with customers directly: understanding their fabric topology, workload patterns, and operational constraints. Communicate findings clearly to both technical and non-technical stakeholders
- Contribute to the product: identifying supportability gaps, proposing diagnostic improvements, and feeding field insights into the product management process. You are an active voice in what the product needs to become
- Maintain and evolve the AI agent's detection capabilities: update detection signatures, retrain classification models, and tune thresholds as customer fabrics scale, new hardware is deployed, and new failure modes are discovered in the field. The AI agent is an evolving system, not a shipped product
- Read and work with source code: engaging with developers at the code level when needed to understand behavior, identify root causes, or validate fixes. You do not need to be a full-time developer, but you must be comfortable in the codebase
Skills
- 5+ years of hands-on experience with high speed data center switching platforms at scale. Experience with AI fabric deployments is a strong plus but not required if switching and troubleshooting background depth is there
- People who take ownership. The FDE role requires someone who treats every problem as theirs until it's resolved, regardless of where the root cause sits. If the issue crosses into ASIC behavior, NOS code, optics firmware, or customer configuration, you follow it there. Pointing to another team is not a resolution
- Deep Ethernet troubleshooting experience, including L2/L3 forwarding, ECMP, and packet-level analysis
- BGP operations experience, including route reflectors, convergence behavior, and fabric-scale deployments
- QoS configuration and troubleshooting: PFC, ECN, DSCP, queue management
- Hardware support background: ASICs, FPGAs, chassis-based platforms, pluggable optic modules
- Fiber and optical link troubleshooting, including DOM telemetry interpretation
- Software support history: working with NOS, firmware, and driver-level issues
- Linux proficiency (command line, system administration, log analysis)
- Telemetry and monitoring: experience with streaming telemetry, counters, and event-driven diagnostics
- Experience contributing to or training ML/AI systems (classification models, labeled data, feedback loops)
- Lab skills: ability to design, build, and execute complex test scenarios that isolate specific failure modes
- Clear written and verbal communication, including customer-facing interaction
- Where you fall within that range depends on your experience, skills, and impact— we benchmark against internal levels to keep things fair and consistent
- Experience with SONiC or other open network operating systems
- Familiarity with AI data center designs: low-latency fabrics, GPU cluster networking, NCCL, RDMA/RoCEv2
- Experience with NVIDIA Spectrum switch family, BlueField SuperNICs, or ConnectX adapters
- Understanding of Ultra Ethernet Consortium specifications and goals
- PCIe architecture knowledge (relevant to NIC and accelerator integration)
- Experience with Ixia/Keysight or similar network test equipment and test suites
- Exposure to high-frequency telemetry (HFT) or WJH (What Just Happened) event data
Company Overview
Company H1B Sponsorship