Most agent demos are toys. Ai2's Shippy is not. It's a maritime domain awareness agent that answers questions like "show me fishing activity in Panama's EEZ last month" by querying live satellite data, resolving geospatial boundaries, and linking back to verifiable map coordinates—because when you're directing patrol vessels in the real world, getting it wrong wastes resources and potentially puts people in harm's way.
The Ai2 team's tech blog post about building Shippy is one of the most honest accounts I've seen of what it takes to ship an agent into a high-stakes operational environment. This isn't about prompt engineering your way to AGI. It's about reliability, isolation, deterministic tooling, and evaluations that actually measure whether the thing works.
The soul, skills, and config decomposition
Shippy's architecture splits an agent into three components: soul (the system prompt that defines persona and behavioral boundaries), skills (task-specific capabilities), and config (runtime settings like which LLM to use and which agent harness to run).
The soul and skills get baked into a versioned Docker image. Config stays external—so swapping from Claude Opus 4.6 to another model is a configuration change, not a rebuild. Secrets like API keys are injected at runtime.
This matters because it makes the what of the agent (its identity and capabilities) separate from the how (which model, which runtime). You can version, test, and deploy the agent artifact independently of the infrastructure running it.
Skills follow the same agent-skills spec used by tools like Claude Code—plain markdown files with structured frontmatter. Shippy's skills include querying the Skylight API for vessel events, looking up Exclusive Economic Zone and Marine Protected Area boundaries, interpreting vessel track data, and generating interactive map links.
A single user question can chain multiple skills. "Are there vessels operating near the Cordillera de Coiba MPA?" triggers the Skylight data query skill, ProtectedSeas' MPA boundary database, and the vessel track interpretation skill—all in one dialogue turn.
Deterministic tools wrapping nondeterministic models
Here's the key insight: you can't control what an LLM decides to do, but you can make the tools it reaches for predictable.
Early prototypes let Shippy construct raw API calls to Skylight's backend. The API has dozens of input types, nested filter objects, pagination cursors, and complex geometry inputs. The result? A steady stream of subtle bugs—malformed pagination that silently dropped results, geometry encoding errors, correct-looking queries that returned wrong data.
The solution was a purpose-built CLI that wraps the API. Shippy issues a single command—skylight events search with typed filter flags—and the CLI handles authentication, pagination, and structured output. The CLI is self-documenting with extensive --help text and detailed error messages, giving both the agent and human developers enough context to recover from mistakes.
All output goes to a local JSON file instead of piped through the shell. Early on, large result sets would hit pipe buffer limits or break downstream tools like jq. Writing to disk sidesteps both problems and lets the agent programmatically access query results across subsequent steps.
Layered abstraction narrows the failure surface
Underneath the CLI is a standardized API where multiple resource types (events, vessels, regions, satellite imagery, tracks) are accessible through a common pair of operations: search and aggregate. Inputs and outputs are defined as typed schemas with field-level descriptions.
This layering—typed API, deterministic CLI, agent skills that reference CLI commands—means each component can be tested independently. The API has its own test suite. The CLI can be exercised by a human or an agent. The agent skills reference CLI commands that handle the plumbing.
Each layer narrows what the next layer can get wrong. This is the opposite of "let the LLM figure it out"—it's aggressively reducing the search space of possible mistakes.
Sandboxed hosting and per-user isolation
Skylight serves hundreds of government agencies and NGOs across over 70 countries. A fisheries officer in the Philippines has their own areas of interest, vessel watchlists, and alert configurations. When they ask Shippy a question, the agent's API calls must return their data, and their conversation history must never leak to anyone else.
Every user gets an ephemeral, isolated session. Ai2 built Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for each session. When a user opens a conversation, the system spins up pods packaging the agent runtime, its skills, and the Skylight CLI.
The user's Skylight JWT is injected at provision time so the agent's API calls are scoped to that user's data. Files the agent writes during multi-step analysis exist only within that session and are never shared across users.
Inside the sandbox, the agent can write and run code, install dependencies, pull in datasets, and work through multi-step analyses. At the network level, the sandbox is restricted to only the services it needs.
This is not optional hygiene. Multi-tenancy and data isolation are table stakes for deploying agents in regulated or sensitive domains. The fact that Ai2 invested serious engineering effort into per-user sandboxing tells you they're serious about production.
Evaluating the whole agent, not just the model
Most benchmarks rank general-purpose AI on static questions. They don't capture how an agent behaves once it's wired into a real workflow—how it selects tools, queries live data, acts on results, and knows where to stop.
So Ai2 built their own eval system that scores the whole agent—model, skills, and sandbox together—against live data.
Subject-matter experts write scenarios and rubrics, choosing which criteria apply to each task and setting the weights. A fishing-events query weights data accuracy most heavily, with boundary resolution and timeframe next, and source attribution and response style carrying less weight. Experts also annotate individual responses as correct or incorrect, giving the LLM judge ground truth to score against.
The eval pipeline
The pipeline is straightforward: a natural-language prompt runs through the sandbox, an LLM judge grades each criterion from 0 to 1 with written reasoning, and the weighted aggregate is checked against a fixed pass threshold.
Tasks are executed through Harbor, an open evaluation framework. Ai2 wrote a Harbor plugin that spins up a real Shippy session on the exact version being tested, against the same real data a user would encounter. The suite runs in parallel against a specific versioned Shippy build, producing a timestamped results file and a report of score changes against the previous run.
They rerun the suite whenever the skills, model, or underlying data change. A version of Shippy that regresses on their eval criteria doesn't reach end users.
Shippy scores consistently across data retrieval and guardrail tasks, correctly refusing military intelligence requests, maintaining user data isolation, and attributing sources accurately. The clearest failure modes in the latest run: patrol-planning tasks where Shippy overstepped into tactical recommendations rather than decision support, geometry-sensitive queries where boundary simplification caused missed events, and one case where the agent invented a CLI command that didn't exist.
Each failure points directly to the next round of skill improvements. This is what iterative agent development looks like when you're measuring the right things.
Behavioral boundaries are explicit, not implicit
Shippy won't make legal determinations about whether a vessel is breaking the law—that's a determination for people, not an agent. It also won't speculate beyond what the data supports.
These boundaries are explicit in the system prompt, not implicit in fine-tuning. That makes them auditable and easy to revise. This is an underrated design choice.
When you fine-tune boundaries into a model, you lose visibility into what the model will refuse and why. When you put them in the system prompt, you can read them, version them, and adjust them as policy evolves. You're treating the agent's behavioral constraints as configuration, not as opaque model behavior.
What's next
Ai2 is opening Shippy to early adopters on a rolling basis. Here's what they're building:
- Agent-driven UI control: Shippy returns map links today; next it will drive the Skylight map itself, moving to regions, applying filters, and adjusting time ranges.
- Model routing: Not every question needs a frontier model. Simple lookups can go to smaller, faster models; complex investigations get the full-weight model.
- Cross-thread memory: Conversation history persists within a thread, but context doesn't carry across threads. They're building memory so Shippy carries persistent facts (an analyst's jurisdiction, preferred sources) and applies them automatically.
The work on Shippy is already shaping how Ai2 thinks about agents elsewhere. That's the real takeaway here.
The broader lesson
Shippy is interesting because it's not about the model. Claude Opus 4.6 is doing the reasoning, but the value is in everything around it: the deterministic CLI that makes API calls predictable, the per-user sandboxing that makes multi-tenancy safe, the eval framework that tests the whole system against live data, the explicit behavioral boundaries that make the agent's limits auditable.
This is what building production-grade agents looks like. It's not prompt engineering. It's systems engineering with an LLM in the loop. The model is nondeterministic, so you make everything else as deterministic as possible. You isolate, you version, you test, you measure.
If you're serious about shipping agents into high-stakes domains, this is the architecture you study.