The Model Is the Easy Part: Building Effective and Useful Agents That You Can Trust
How we built our own agents to keep network operations moving as fast as our datacenter buildouts.
I work on the network automation team at Fluidstack. We build the datacenters behind some of the largest AI training clusters in the world, and under every one of them sits a core network, the routed backbone everything else depends on. Traditionally when something looks wrong, the job is investigation: remote into a device, run show commands, compare what you see against what the fabric is supposed to look like, hop to the next device, keep a mental model of the whole path, and land on a conclusion you'd stake a change window on.
That work doesn't scale the way our buildouts do. We deliver gigawatts of compute in months, not years. The fleet grows faster than any team of network engineers can, and the investigation toil grows with it.
So we built agent-deploy, a terminal TUI where an operator can ask a network question in plain English — "overall fleet status?", "why is this rack's ToR down?" — and an agent fans out across the fleet, gathers device state through typed tools, and comes back with a grounded conclusion and a recommendation. It runs the entire troubleshooting path end to end, never stopping to ask what to check next, until it lands on an answer the evidence supports. And it never, structurally, changes the network without operator approval.
What it does
It's in daily use across the team. An investigation ends in an artifact you can act on: a report, a table export for the field team, or a Jira ticket the agent files itself, with findings and evidence attached, ready for datacenter operations to pick up. Repeatable procedures live as runbooks you invoke by name, and the catalog grows as engineers contribute their own. And when a device needs human hands, the app opens a live terminal session to it, full TTY emulation in the same interface, and closes the loop between the investigation and the remedy: the agent finds it, you fix it, without ever leaving the app.
An investigation that ends in a ticket: a dedupe check first, then a Jira issue with every evidence line traced to the device it came from.
Incidents are where this pays for itself fastest. When something pages, we can spin up an ephemeral agent scoped to that investigation: it sweeps the affected devices, assembles a picture of what changed, and hands the on-call engineer a grounded starting point instead of a blank terminal. It's operator-assisted by design: the agent informs, the engineer decides. The value is time. Minutes to a supported hypothesis and a head start on root-cause analysis, instead of an hour of manual state-gathering before the real thinking starts.
The hand-off: the agent concludes a fault needs hands-on work and flags an interactive SSH session. The agent flags, a human picks up.
One keypress later: a live TTY to the flagged device, inside the same interface. Investigation and remedy in one place.
There's a quieter payoff we didn't fully anticipate: the tool meets operators at their experience level. A new engineer can ask the network the questions they might hesitate to ask a person. What does this alarm mean? Why is it designed this way? What is healthy supposed to look like? It answers from the live fleet. People ramp at the speed of their own curiosity instead of the availability of a senior engineer, and new team members are using it to learn the network itself. When you're delivering the largest datacenters in the world on compressed timelines, how fast an engineer becomes productive is as much a scaling constraint as the automation.
A plain-English fleet question: parallel tool fan-out, then a per-pod summary anyone on the team can read.
Why we wrote the loop ourselves
The obvious question: why not point Claude Code, Codex, or any general coding agent for that matter at the network and call it a day? Public disclosure of popular agent loops puts roughly 1.6% of the code in AI decision logic and ~98.4% in deterministic infrastructure: permission gates, context management, tool routing, recovery.
The model is the easy part. The product is the scaffolding around the loop. And the scaffolding a network agent needs is different from what a coding agent needs, in three ways:
- Determinism. Network operations are repeatable procedures. The same question about the same fleet state should take the same path to the same answer.
- Eval-ability. Every run is a recorded artifact we can replay and grade. We gate changes to the agent on evals the same way we gate code on tests.
- Impact foresight. Before this thing ever earns write access, it needs to predict blast radius deterministically, not vibes about whether a BGP change is safe.
A general coding agent's safety is a polite request in the system prompt. Ours is enforced in code and covered by a test.
Determinism in practice: a runbook backed by a deterministic audit tool, where every NOT_READY verdict comes back with a failing criterion and a fix owner.
Context matters
The quality of an agent's conclusion is bounded by the quality of what's in its context window. And the window is finite. So the agent loop treats the context window like a budget: it decides what gets in and what gets summarized away.
Every device query runs through typed MCP tools that return structured results, each tagged with metadata naming the device it came from, so the model never has to guess which switch said what, and neither do we when we audit a run afterward. That attribution is the backbone of the provenance we've built into the tooling: an exported table or a saved config has to trace back to tool output captured from the specific device it claims to describe, checked in code before anything is written. The model can't sneak a guess into an artifact.
The same discipline governs what never enters the window and what doesn't stay. Secrets stay out entirely: credentials live server-side behind the tool layer, tool results are treated as untrusted data rather than instructions, and run logs are redacted before anything is recorded.
Bulk data mostly bypasses the window entirely: a fleet-wide export comes back as a file on disk with a one-line note to the model, not thousands of rows paged through as tokens. And when a long investigation does fill the window, the loop compacts it: stale tool-result bodies are elided while the reasoning trail stays intact, so the agent keeps its train of thought without dragging every raw show-command output behind it.
Config drift against intended state, rendered as a colored diff, with a recommendation.
Evals decide, not vibes
Every run is recorded at two tiers: a full transcript for replay and debugging, and a redacted, context-free log of who asked what, which tools ran, what it cost, and how it ended. That corpus feeds three eval layers: a deterministic offline regression suite (scripted model, mock devices, graded trajectories) that gates CI; a live grounding eval measuring fabrication rate; and a live behavior baseline we run across models. That corpus is also how changes earn their way in: a new tool, prompt tweak, or model upgrade that regresses a graded trajectory doesn't ship.
The future of agents at Fluidstack
Traditional automation isn't going anywhere. Templates, pipelines, that kind of thing: still essential, and the best of what our agents discover gets promoted right back into it. But traditional automation scales linearly with the engineers writing it: every failure mode has to be anticipated, encoded, and maintained ahead of time, and at the pace our fleet grows we will never write playbooks fast enough. Agentic automation changes the ratio, and it has become a key driver of how we operate. Enough that we've built a team around it and are hiring quickly into it.
The work changes what a domain expert's job looks like. Instead of encoding every branch in advance, you build the tools, guardrails, and evals that let an agent cover the long tail, then promote what it discovers into deterministic code. That takes a particular kind of engineer: a genuine domain expert with a deep interest in developing these tools and a desire to share that craft with a team doing the same. That's who we're looking for.
We build civilization-scale infrastructure for AI, and we build it fast: gigawatts of compute in months, including the buildout behind Anthropic's $50 billion compute commitment. Automation like this is how a small team operates a network that grows that fast without burning out. If building the tools that run that network (including agents you can actually trust with it) sounds like your kind of problem: fluidstack.com/jobs.