Building Reliable AI Agents Lessons from a High-Stakes Maritime Domain

Why Reliable Agents Are an Architecture Problem, Not a Model Problem

When you build an AI agent for a domain where a wrong answer sends a patrol vessel miles in the wrong direction, the model is the least interesting part of the system. The real engineering challenge is building something you can trust to be correct, to stay within its limits, and to hold up across wildly different tasks—against live data, not a static snapshot.

This is the story of how the team behind an open maritime domain awareness platform built their agent. Their lessons apply to anyone shipping agents into production, whether you're building for healthcare, finance, or customer support.

The core insight: agents are nondeterministic, but the tools they reach for don't have to be. Every reliability gain came from narrowing what each layer of the stack could get wrong.

If you've been thinking about how agents learn from expert knowledge without retraining, this architectural pattern is a natural companion to the organizational second brain approach.

The Agent Anatomy: Soul, Skills, and Config

The team decomposes an agent into three parts:

• Soul — the system prompt that frames persona and sets behavioral boundaries. Explicitly states what the agent won't do (e.g., "never make legal determinations," "never speculate beyond the data").
• Skills — markdown files with structured frontmatter that encode how to handle specific request types. Same spec used by Claude Code and Codex.
• Config — harness choice, LLM choice, runtime settings. Swapping the model is a config change, not a rebuild.

The soul + skills are baked into a Docker image—a versioned, deployable artifact. Secrets are injected at runtime.

Why This Separation Matters

When boundaries live in the system prompt (not fine-tuning), they're auditable and easy to revise. When skills are plain markdown, each one is comprehensible, versioned, and diffable in a PR.

Deterministic Tools for a Nondeterministic Agent

The single biggest reliability win: wrap your API in a purpose-built CLI instead of letting the agent construct raw calls.

# BEFORE: agent constructs API calls from scratch
# Result: malformed pagination, geometry encoding errors,
# silent filter-type bugs that return *wrong* data

# AFTER: agent issues one typed command, CLI handles plumbing
# skylight events search --region "Panama EEZ" \
#   --type fishing --since 2024-01-01 --output results.json

Why the CLI approach works:

1. Typed filter flags collapse complex nested query objects into a predictable interface.
2. Self-documenting — extensive --help text and structured error messages let the agent recover without guessing.
3. Write to disk, not stdout — piping large JSON through a shell hits buffer limits and breaks downstream tools like jq. Writing to a local file sidesteps both and lets the agent read results across multi-step workflows.

Underneath the CLI sits a typed API with a common search/aggregate operation pair across all resource types. Each layer narrows what the next can get wrong: typed API → deterministic CLI → agent skills referencing CLI commands.

Sandboxing, Isolation, and Evaluating the Whole Agent

Per-User Sandboxed Sessions

When your agent serves hundreds of government agencies and NGOs across 70+ countries, data isolation is non-negotiable. The pattern:

• Each user session gets a dedicated Kubernetes deployment with its own pods.
• The user's JWT is injected at provision time, scoping all API calls to their data.
• Files written during analysis exist only within that session.
• Network access is restricted to only the services the agent needs.

This is the kind of infrastructure decision that looks boring on a slide but is where most production agent projects quietly fail.

⚠️ The Limits You Need to Know About

Even a well-architected agent has failure modes. From their own eval data:

• Overstepping into recommendations — the agent drifted from decision support into tactical advice. Guardrails must be tested, not assumed.
• Geometry simplification bugs — spatially simplified boundaries caused the agent to miss real events near edges.
• Invented tool calls — the agent hallucinated a CLI command that didn't exist. This is exactly why your CLI must return a clear "unknown command" error with a list of valid ones.

Evaluating the Agent, Not the Model

Static benchmarks won't tell you if your agent works in production. The team built their own eval pipeline:

1. Subject-matter experts write scenarios and rubrics, and weight each criterion per task.
2. A natural-language prompt runs through the real sandbox against live data.
3. An LLM judge grades each criterion 0–1 with written reasoning.
4. The weighted aggregate is checked against a fixed pass threshold.
5. Results are timestamped and diffed against the previous run.

A version that regresses on eval criteria never reaches end users. This is the discipline that separates a demo from a product.

For another angle on making infrastructure behavior transparent and debuggable, see this writeup on making DNSSEC trust anchor failures visible.

What to Steal for Your Own Agent

If you're building a production agent, here's the checklist that emerged from this project:

1. Separate soul, skills, and config. Bake soul+skills into a versioned image. Make model swaps a config change.
2. Never let the agent construct raw API calls. Wrap everything in a typed, self-documenting CLI with structured errors.
3. Write large outputs to disk, not stdout. Shell pipes will betray you at scale.
4. Sandbox every session. Per-user isolation is not optional when real data is involved.
5. Evaluate the whole agent, not the model. Rubrics, LLM judges, weighted scores, version diffs.
6. Version-gate on eval regressions. No green, no ship.

What's Next in This Space

Three patterns worth watching: agent-driven UI control (the agent doesn't just return links, it drives the map), model routing (cheap models for lookups, frontier models for investigations), and cross-thread memory (persistent facts like a user's jurisdiction carried automatically across conversations).

Where to Go From Here

• Start by instrumenting your current agent with an eval harness before adding features.
• If you don't have a CLI wrapper for your API yet, that's step one.
• Read the original engineering writeup for the full context: evidence and source material.

Reliability isn't a feature you add at the end. It's the architecture you choose at the beginning.