Forward-Deployed Engineer, AI Fabric

Upscale is building the engineering team that drives product improvements though hands-on field engineering. The Forward-Deployed Engineer (FDE) owns complex fabric and system issues end-to-end: from first report through root-cause analysis, workaround delivery, and verified fix. You resolve them with the rest of the team, or you drive them to resolution across whatever boundary stands in the way.

This role is new to UpscaleAI and sits at the intersection of support, engineering, and product management. You will work directly with customers running production AI fabrics, reproduce issues in the lab, develop workarounds under pressure, and contribute fixes and diagnostic content back into the product and our AI-driven monitoring agent.

The environment is often unstructured. Problem definitions are incomplete. Documentation may not exist yet. The right candidate sees that as an opportunity, not an obstacle.

Large AI infrastructure operators increasingly expect their vendor's engineering team to function as an extension of their own infrastructure organization. This role is built to meet that expectation.

This is a small team. There will be an on-call component, but we are building a global team to reduce out-of-hours calls. We work with customers directly, so some travel is involved. Remote meeting tools handle much of the collaboration but building strong trust relationships will require time on site.

Key Responsibilities

• Triage and resolve complex AI fabric issues: including silent packet drops, queue anomalies, NCCL stalls, gray failures, and performance degradation in production environments. You own the problem from intake to resolution.

• Reproduce issues in the lab: building test scenarios that isolate L2 and L3 problems (packet loss, latency, retransmits, ECMP behavior, PFC/ECN interactions) and deliver reproducible cases to the development team when code fixes are needed.

• Act as the escalation buffer for engineering: resolving issues without engaging development when possible, and packaging clean, reproducible problem statements when development engagement is required.

• Train and improve the on-box and off-box AI agents: contributing field-validated detection signatures, classification logic, and resolution recommendations based on real cases. Your field experience directly shapes what the agent can identifyand handle autonomously.

• ValidateAIagent accuracy: reviewing the agent's data collection, anomaly detection, and triage classifications against real-world outcomes. You determinewhen the agent is ready to advance from data collection to active triageto mitigation

• Work with customers directly: understanding their fabric topology, workload patterns, and operational constraints. Communicate findings clearly to both technical and non-technical stakeholders.

• Contribute to the product: identifyingsupportability gaps, proposing diagnostic improvements, and feeding field insights into the product management process. You are an active voice in what the product needs to become.

• Maintain and evolve the AI agent's detection capabilities:update detection signatures, retrain classification models, and tune thresholds as customer fabrics scale,new hardware is deployed, and new failure modes are discovered in the field. The AIagent is anevolvingsystem, not a shipped product.

• Read and work with source code: engaging with developers at the code level when needed to understand behavior, identifyroot causes, or validatefixes. You do not need to be a full-time developer, but you must be comfortable in the codebase.

Requirements

5+ years of hands-on experience with high speed data center switching platforms at scale. Experience with AI fabric deployments is a strong plus but not required if switching and troubleshooting background depth is there.

People who take ownership. The FDE role requires someone who treats every problem as theirs until it's resolved, regardless of where the root cause sits. If the issue crosses into ASIC behavior, NOS code, optics firmware, or customer configuration, you follow it there. Pointing to another team is not a resolution.

• Deep Ethernet troubleshooting experience, including L2/L3 forwarding, ECMP, and packet-level analysis

• BGP operations experience, including route reflectors, convergence behavior, and fabric-scale deployments

• QoS configuration and troubleshooting: PFC, ECN, DSCP, queue management

• Hardware support background: ASICs, FPGAs, chassis-based platforms, pluggable optic modules

• Fiber and optical link troubleshooting, including DOM telemetry interpretation

• Software support history: working with NOS, firmware, and driver-level issues

• Linux proficiency(command line, system administration, log analysis)

• Telemetry and monitoring: experience with streaming telemetry, counters, and event-driven diagnostics

• Experience contributing to or training ML/AI systems (classification models, labeled data, feedback loops)

• Lab skills: ability to design, build, and execute complex test scenarios that isolate specific failure modes

• Clear written and verbal communication, including customer-facing interaction

Nice to have

• Experience with SONiCor other open network operating systems

• Familiarity with AI data center designs: low-latency fabrics, GPU cluster networking, NCCL, RDMA/RoCEv2

• Experience with NVIDIA Spectrum switch family, BlueFieldSuperNICs, or ConnectXadapters

• Understanding of Ultra Ethernet Consortium specifications and goals

• PCIe architecture knowledge (relevant to NIC and accelerator integration)

• Experience with Ixia/Keysight or similar network test equipment and test suites

• Exposure to high-frequency telemetry (HFT) or WJH (What Just Happened) event data

What This Role Is Not

This is not a traditional TAC or support queue role. You will not be working from runbooks or following scripts. Many of the problems you encounter will not have documented solutions. You will be expected to figure it out, document it, and make sure the next engineer (or the agent) can handle it, next time, without you.

This is also not a development role. You will read code and work closely with developers, but your primary output is resolved customer issues, validated agent training data, and product improvement recommendations, not shipped features.