Agents and MCP

The runtime serves MCP, the protocol AI assistants use to call tools. An agent connected to it can work with running streams the way you do with the CLI: look at a stream’s graph, sample what its outputs carry, see its frames and read its logs. It can also load and stop streams, and add, connect and remove nodes in a running one.

Any agent that can run a local MCP server works, including these:

• Claude Code
• Claude Desktop
• Cline
• Codex
• Cursor
• Devin Desktop
• Gemini CLI
• GitHub Copilot
• Grok Build
• Zed

Computer-use agents such as Grok Bot and Muse also open a stream by its URL.

Connect

Claude Code

claude mcp add tatolab -- tatolab mcp

Run it in your project’s folder: by default, Claude Code keeps an MCP server for the folder you added it in. Add --scope user to have it in every folder. Then start a new Claude Code session.

Codex, Gemini CLI and Grok Build

Each adds the server with one command:

codex mcp add tatolab -- tatolab mcp
gemini mcp add tatolab tatolab mcp
grok mcp add tatolab -- tatolab mcp

Codex keeps it in ~/.codex/config.toml, which its CLI and IDE extension share. Gemini CLI adds it to the project’s .gemini/settings.json; add -s user to have it in every project. Grok Build keeps it in ~/.grok/config.toml; add --scope project to keep it with the project instead.

Clients configured with JSON

Claude Desktop, Cursor, GitHub Copilot in VS Code, Cline and Devin Desktop take:

{
  "mcpServers": {
    "tatolab": { "command": "tatolab", "args": ["mcp"] }
  }
}

• Claude Desktop: Settings → Developer → Edit Config opens claude_desktop_config.json. Restart the app after you save it.
• Cursor: .cursor/mcp.json in the project, or ~/.cursor/mcp.json for every project.
• GitHub Copilot in VS Code: .mcp.json at the top of the workspace, or ~/.copilot/mcp-config.json for every workspace.
• Cline: the MCP Servers icon in Cline’s panel opens cline_mcp_settings.json.
• Devin Desktop, formerly Windsurf: ~/.config/devin/mcp_config.json.

Zed

Add it in Settings → AI → MCP Servers → Add Server → Add Local Server, or in Zed’s settings.json:

{
  "context_servers": {
    "tatolab": { "command": "tatolab", "args": ["mcp"], "env": {} }
  }
}

How it connects

tatolab mcp connects to the runtime through its local socket and passes messages between it and the client. The MCP server belongs to the runtime: it’s there whenever the runtime is running.

If a client can’t find tatolab, give its full path as the command: /usr/local/bin/tatolab on a Mac, where the app links it, or /home/you/.local/bin/tatolab on Linux, with your user name in place of you.

Another machine

Run tatolab mcp on the other machine over ssh:

claude mcp add lab-camera -- ssh lab-camera .local/bin/tatolab mcp

A command run over ssh gets a non-interactive shell, and on Debian and Ubuntu that shell doesn’t have ~/.local/bin on its PATH, so give the path to tatolab, relative to your home folder on that machine. (After a source build, it’s wherever you put the binaries.) On a tailnet, use the machine’s Tailscale name as the host.

Computer-use agents

Computer-use agents, such as Grok Bot from xAI and Muse from Meta, open an exposed output by its URL, on its viewer page, the way a person would. What they reach follows the output’s exposure: a private output’s URL works on its own machine, and a public one from your other machines and through a relay the machine joins. See Across machines.

The tools

Observe

Tool	Does
graph	Returns the live graph, of one stream or of all of them
tap	Samples, for half a second, the bags an output sends. The output needs a reader: a link in its stream, or another stream reading it.
logs	Reads a stream’s log records, from a given point onward
exchange	Turns a frame’s surface ID, from a tapped bag, into an image the agent can see, scaled down to a size the model can take

Change a running stream

Tool	Does
add_node	Adds a node by its class path, such as tatolab.stream:DisplayWindow or nodes.detector:Detector, with optional settings
connect	Links an output to an input, within a stream, or from another stream’s exposed output
disconnect	Removes a link
remove_node	Removes a node

Manage streams

Tool	Does
list_streams	Lists the streams the runtime holds, and their states
run_stream	Loads a project’s stream, attached or kept
stop_stream, start_stream, remove_stream	Like tatolab stop, start and rm
expose_port	Sets an output’s exposure level

Agents also get two resources: the node catalog (streamlib://node-catalog), which lists every node type the runtime can add, with its description, settings and ports; and the live graph (streamlib://graph).

Recipes

The runtime offers four prompts, step-by-step recipes made of the tools above, which clients show as ready-made tasks:

Prompt	Does
look_at_what_a_channel_carries	Samples one bag from an output, decodes it, and shows the frame it names, if any
insert_node_between_linked_nodes	Puts a new node into an existing link, such as an effect between the camera and the window
fan_output_to_another_consumer	Adds a node as one more reader of an output, leaving the existing readers alone
show_channel_on_virtual_camera	Presents an output’s video as a camera other apps can select (Linux)

Things to ask

• “What’s running on this machine?”
• “Is the camera stream healthy? Has anything dropped frames?”
• “Show me what the detector is outputting right now.”
• “Put a grayscale effect between the camera and the window.”
• “Why is the window black?”
• “Write a node that blurs faces, add it after the camera, and show me a frame.”

That last one needs an agent that can also edit files in your project, like Claude Code running on the same machine. It writes a node class into a module in the project, or installs a package with uv add, then adds the node to the running stream by its class path. An agent connected over ssh with only tatolab mcp has the runtime’s tools, and can’t write files there. Like every Python node, it runs in its own process, from your project’s .venv. Each add_node imports the module fresh, so an agent can edit its node’s code and add it again to try the change. A node’s ports are read the first time it’s added, so changing them needs a new class name.

Changes, and what lasts

• Live changes apply to the running stream only. A kept stream reloads the graph recorded at its last tatolab run -d, so when an agent’s change should stay, put it in stream.py (or ask the agent to) and run tatolab run -d again.
• A stream an agent loads attached lives as long as the agent’s connection. When the client exits or restarts its MCP server, the stream stops. A stream loaded as kept stays.

Access and safety

• What the agent sees leaves the machine. Frames it exchanges, bags it taps, the logs it reads and the node settings in the graph all go into the agent’s context, and so to its model provider. Decide for each camera and stream whether that’s acceptable before you connect an agent.
• Control stays on the machine. The runtime opens no network port for MCP or the CLI. Only processes running as your user account can open its socket. Reaching another machine is ssh’s job, with ssh’s keys and permissions.
• An agent can change and stop streams. If you’d rather approve each change, use your client’s permission settings: Claude Code, for example, asks before running a tool unless you’ve allowed it.
• Settings are visible. The graph includes each node’s settings. Don’t pass secrets to a node as settings if agents you don’t trust can connect.

Write streams agents understand

Agents read what you write. A few habits make them much more useful:

• Node docstrings. A node’s docstring is its description in the catalog. Say what it does in a sentence. (A stream function’s docstring stays in your code: agents don’t see it.)
• Describe settings. Annotated[float, "Ignore detections scored below this."] on a config field puts that description in the catalog, next to the type and default.
• Describe ports. @node.output(description="One bag per frame: label, score and box for each object") tells an agent what to expect before it taps.
• Name nodes. stream_builder.add(CameraSource, name="front_door") gives addresses that explain themselves.
• Log what happened. log.info("motion started", changed=0.14) gives an agent something to read with logs long after the moment has passed; a tap only sees the last half-second.