Skip to content
RivetOS

Mesh Networking

RivetOS supports a multi-node mesh that lets agents delegate tasks to each other across instances. One node can ask another node’s agent to handle a task, even if that agent runs on different hardware or uses a different LLM provider.

How it works

Section titled “How it works”

Every mesh-enabled node runs an agent channel, an HTTPS server that receives delegated tasks and routes them to the local DelegationEngine. When one node needs an agent it doesn’t host locally, it looks up the target node in the shared mesh.json registry and sends the task over mTLS.

The mesh listener is AgentChannelServer in @rivetos/core, started by boot when mesh.enabled and mesh.tls are set. It binds port 3000 by default (mesh.agent_channel_port, or RIVETOS_AGENT_PORT). That is a different server from the standalone @rivetos/channel-agent plugin, whose bind default is 3100 (channels.agent.port). rivetos mesh join defaults to 3000 to match the mesh listener.

┌─────────────────────────────────┐     HTTPS/mTLS      ┌─────────────────────────────────┐
│  <node_name> — opus             │  ──────────────────▶ │  <peer_name> — grok             │
│                                 │  POST /api/message   │                                 │
│  MeshDelegationEngine           │                      │  AgentChannelServer (port 3000) │
│  mesh.json (shared r/w)         │◀──────────────────── │  mesh.json (shared r/w)         │
└─────────────────────────────────┘    delegation result └─────────────────────────────────┘

Shared registry

Section titled “Shared registry”

All nodes read and write a single mesh.json file in the shared mesh data directory ($RIVETOS_SHARED_DIR/mesh.json, unset → product default). The datahub host typically mounts that directory for the mesh. This is the source of truth; no extra coordination service needed.

Discovery modes

Section titled “Discovery modes”
Mode How it works
static Peer list hard-coded in config. Good for stable infra.
seed New node contacts a seed’s /api/mesh endpoint to bootstrap its view.
mdns mDNS discovery (local network).

mTLS authentication: Phase 0.5

Section titled “mTLS authentication: Phase 0.5”

Starting from Phase 0.5, all mesh agent-channel traffic is mutual TLS. There is no plaintext fallback and no bearer-token authentication on the agent channel. CA-signed certificate = trusted. Everything else = rejected at the TLS handshake level.

How it works

Section titled “How it works”
  1. Each node has a certificate issued by the mesh CA (under the mesh cert directory you configured, typically $RIVETOS_SHARED_DIR/rivet-ca/).
  2. The agent channel server requires a client cert and verifies it against the CA chain.
  3. The delegation client builds an mTLS connection using the same cert pair.
  4. Connections to remote nodes use <node_name>.mesh DNS names so the cert SANs match.

Certificate layout

Section titled “Certificate layout”
$RIVETOS_SHARED_DIR/rivet-ca/
  intermediate/
    ca-chain.pem          ← CA chain (validates all node certs)
  issued/
    <node_name>.crt       ← node cert (CN=<node_name>, SAN=<node_name>.mesh + mesh IP)
    <node_name>.key       ← node private key
    <peer_name>.crt / .key
    <agent>@<node>.crt    ← agent certs (reserved, unused on the wire in Phase 0.5)

Make the tree readable by the service user on every node that mounts the share.

Configuration

Section titled “Configuration”

Minimal mesh config (tls: true → default paths)

Section titled “Minimal mesh config (tls: true → default paths)”
mesh:
  enabled: true
  node_name: <node_name>    # must match the cert CN
  tls: true                 # default mesh.tls paths under $RIVETOS_SHARED_DIR/rivet-ca
  agent_channel_port: 3000
  # storage_dir omitted → $RIVETOS_SHARED_DIR (unset → product default shared root)
  heartbeat_interval_ms: 30000
  stale_threshold_ms: 90000
  discovery:
    mode: seed
    seed_host: <node_name>.mesh   # use .mesh hostname — matches cert SAN
    seed_port: 3000

Custom cert paths

Section titled “Custom cert paths”

tls: true derives these from $RIVETOS_SHARED_DIR (unset → product default). Override mesh.tls.* only when your mesh cert directory lives elsewhere.

mesh:
  enabled: true
  node_name: <node_name>
  tls:
    ca_path: $RIVETOS_SHARED_DIR/rivet-ca/intermediate/ca-chain.pem
    cert_path: $RIVETOS_SHARED_DIR/rivet-ca/issued/<node_name>.crt
    key_path: $RIVETOS_SHARED_DIR/rivet-ca/issued/<node_name>.key

Config reference

Section titled “Config reference”
Key Type Default Description
mesh.enabled bool false Enable mesh networking.
mesh.node_name string hostname Node identifier — must match cert CN.
mesh.tls bool | object — mTLS config. Required — mesh refuses to start without it.
mesh.tls.ca_path string $RIVETOS_SHARED_DIR/rivet-ca/intermediate/ca-chain.pem CA chain PEM path. Unset RIVETOS_SHARED_DIR → product default.
mesh.tls.cert_path string $RIVETOS_SHARED_DIR/rivet-ca/issued/<node_name>.crt Node cert PEM path.
mesh.tls.key_path string $RIVETOS_SHARED_DIR/rivet-ca/issued/<node_name>.key Node private key PEM path.
mesh.agent_channel_port number 3000 HTTPS port for the agent channel.
mesh.storage_dir string $RIVETOS_SHARED_DIR (unset → product default) Directory containing mesh.json.
mesh.heartbeat_interval_ms number 30000 How often to write a heartbeat.
mesh.stale_threshold_ms number 90000 Age before a node is considered stale.
mesh.discovery.mode string — seed | static | mdns.
mesh.discovery.seed_host string — Seed node hostname. Use <nodeName>.mesh.
mesh.discovery.seed_port number 3100 Seed node port. Client-side fallback when seed_port omitted; set it to match the seed’s listener.
mesh.secret string — Deprecated — no longer used for agent-channel auth. Retained for update --mesh orchestration.

.mesh DNS names

Section titled “.mesh DNS names”

dnsmasq on every node resolves <node_name>.mesh to the node’s mesh IP. Always use .mesh names for seed hosts and anywhere you reference a peer by URL. This ensures the cert SAN matches the connection hostname and TLS succeeds without rejectUnauthorized: false.

Endpoints

Section titled “Endpoints”

All endpoints are served over HTTPS. The TLS handshake requires a valid client certificate; connections without one are rejected before any HTTP code runs.

Method Path Description
GET /api/mesh/ping Liveness probe. Returns { status, node, tls, cn }.
POST /api/message Receive a delegated task. Body: MessageRequest.
GET /api/mesh Return mesh registry for seed sync.
GET /api/agents List local agents.

Audit / logging

Section titled “Audit / logging”

Every accepted request logs peer.cn=<nodeName>. You can grep for it in journalctl -u rivetos or wherever your log sink is:

INFO [AgentChannel] Received mesh delegation peer.cn=<node_name> from opus → grok: Summarise...

TLS handshake failures log at WARN:

WARN [AgentChannel] TLS handshake failed from 192.0.2.112: peer did not return a certificate

Cutover

Section titled “Cutover”

The Phase 0.5 cutover (all nodes upgraded together for shared-CA mTLS) is complete on the supported releases. The historical procedure was documented in MIGRATION.md, which has since been removed; see CHANGELOG.md (Phase 0.5 entry) for the original steps and rationale.

TermsPrivacyGitHub