How Tatolab works

This page explains the ideas the rest of the docs build on. If you’ve run the quickstart, you’ve already met most of them.

The short version

• A stream is a function that adds nodes and connects their ports with links.
• Data moves along links as bags: maps of named values, like a dict.
• The runtime, tatolabd, runs every stream on the machine. You talk to it with the tatolab CLI, and agents talk to it over MCP.
• Every output has an address, like motion_watcher/detector/motion. Expose an output, and other streams on the machine can read it too.
• Built-in nodes (camera, window, codecs, recorder) run inside the runtime. Every Python node runs in its own process, started from your project’s virtual environment.

Streams

A stream is a function decorated with @stream. It receives a StreamBuilder and describes a graph: which nodes to create, and how to connect them.

stream.py

from tatolab.stream import DisplayWindow, StreamBuilder, TestPatternSource, stream


@stream
def bars(stream_builder: StreamBuilder) -> None:
    """Color bars in a window."""
    pattern = stream_builder.add(TestPatternSource)
    window = stream_builder.add(DisplayWindow)
    stream_builder.connect(pattern.output("video"), window.input("video"))

• The function runs once, when the stream loads. It doesn’t run for each frame or each bag. The runtime records the graph it built and starts it.
• The function’s name is the stream’s name (bars here). Load it under another name with tatolab run --name. Its docstring documents the stream in your code; agents read node descriptions, not stream docstrings.
• A project is an ordinary Python project: a stream.py, a pyproject.toml and a .venv. There’s no manifest and no build step. tatolab run and tatolab dev load the one @stream in stream.py; for a file with several, name one, as in tatolab run stream.py:bars.

Nodes

A node is one step of a stream. There are two kinds:

• Built-in nodes ship inside the runtime: CameraSource, TestPatternSource, DisplayWindow, MicrophoneSource, SpeakerSink, the H.264, H.265 and Opus encoders and decoders, Mp4Sink, and VirtualCameraSink on Linux. They’re native code, configured from Python with a config dict. See Built-in nodes.
• Python nodes are classes you write, decorated with @node, or classes from a package you install.

A Python node can handle any kind of data. This one watches temperature readings and sends an alarm when one runs too hot:

nodes/temperature_alarm.py

from dataclasses import dataclass

from tatolab.stream import RuntimeContextLimitedAccess, node


@dataclass
class TemperatureAlarmConfig:
    threshold_celsius: float = 60.0


@node
class TemperatureAlarm:
    """Sends an alarm for each reading above the threshold."""

    @node.input(delivery_profile="ordered")
    def readings(self) -> dict: ...

    @node.output()
    def alarms(self) -> dict: ...

    def __init__(self, config: TemperatureAlarmConfig) -> None:
        self.threshold_celsius = config.threshold_celsius

    def process(self, ctx: RuntimeContextLimitedAccess) -> None:
        reading = ctx.inputs.read("readings")
        if reading is not None and reading["celsius"] > self.threshold_celsius:
            ctx.outputs.write("alarms", {"sensor": reading["sensor"], "celsius": reading["celsius"]})

Each Python node runs in its own operating-system process, with its own Python interpreter started from your project’s .venv. If a node hangs, crashes or leaks memory, the problem stays in that process, the log names the node, and the rest of the stream keeps running. It’s also why a node class lives in a module of its own (a file under nodes/, say) rather than in stream.py: the node’s process imports the class by its module path.

Write a node covers everything a node can declare.

Ports, links and bags

A node declares input ports and output ports, and stream_builder.connect(output, input) makes a link between them.

• One output can feed many inputs (a fan-out). The output isn’t split between them: each reader reads it at its own pace.
• Many links can enter one input (a fan-in). Mp4Sink, for example, records one track per link into its tracks input.
• Ports have no types. The annotation on a port method (-> dict) is for you and your type checker; the runtime never compares types between nodes.

What travels along a link is a bag: a map of named values. There’s no schema to register; a node writes a dict and the next node reads it. A bag can hold numbers, strings, booleans, lists, nested dicts and bytes. Built-in nodes use a few well-known shapes. A video bag names a frame held in GPU memory, which the runtime calls a surface: the bag carries the surface’s ID (surface_id) with width, height and timestamp_ns, never the pixels. An audio bag carries its samples inline.

When you read, you choose how strict to be. ctx.inputs.read("readings") gives you the dict. ctx.inputs.read("readings", into=Reading) builds a dataclass or pydantic model from it, and raises an error if the bag doesn’t fit.

Delivery profiles

Every input port chooses how it reads:

Profile	Reads	Use it for
newest	The most recent bag. Older ones are skipped.	Anything that only needs the current state: a window, a detector, a dashboard.
ordered	Every bag, in the order it was sent, unless the reader falls far behind.	Anything that must not skip: logs, encoders, recorders, audio.

There’s no default; each input must choose. No link ever slows down the node that sends on it. If a reader falls far behind, bags are dropped at that reader’s input and counted, and tatolab graph shows the count in the reading node’s metrics. One output can feed a newest reader and an ordered reader at the same time.

Exposing outputs

Every output port has an exposure level, which decides whether other streams can read it:

Level	Who can read the port	How to set it
internal	Only nodes in the same stream. This is the default.	Nothing to do
private	Other streams on this machine	stream_builder.expose(output), or tatolab expose while it runs
public	Readers on other machines too, by URL.	stream_builder.expose(output, Exposure.PUBLIC), or tatolab expose --public

An exposed port has an address: <stream>/<node>/<port>, such as motion_watcher/detector/motion. Node names default to the class name in lowercase; pass name= to stream_builder.add for an address that reads well. Every part is made URL-safe for you: lowercased, accents dropped, and anything other than letters, digits, -, ., _ and ~ turned into -. The full form starts with the machine’s name, <machine>/<stream>/<node>/<port>; the machine’s name is its Tailscale machine name, or its hostname when it isn’t on a tailnet.

Another stream on the machine reads an exposed port by naming its address:

stream_builder.connect(
    stream_builder.remote_output("motion_watcher/detector/motion"),
    event_log.input("motion"),
)

The reading stream owns the link: streams pull from each other, and no stream can push into another’s input. The two streams can live in different projects, with different dependencies, and you can stop, change and restart either one without touching the other. Share between streams covers the details.

Looking is different from linking. tatolab tap, tatolab exchange and an agent’s sampling tools take the same addresses, and can sample any output that a link reads, exposed or not, because they’re for inspecting your own streams. An output that nothing reads has nothing flowing to sample.

URLs, over MoQ

Streams on one machine read each other through shared memory. Everything else reads an exposed output by its URL. The runtime runs a MoQ (Media over QUIC) relay on the machine, and the relay serves each exposed output at a URL whose path is the output’s address.

The runtime's MoQ relay gives an exposed output a URL, and a browser, a laptop and an agent all read it there.

• A private output’s URL works on its machine. A browser, a script or any other tool there can read it.
• A public output’s URL works from your other machines too, over your tailnet, where Tailscale provides the reach, names, identity and encryption. Once the machine joins a MoQ relay on the internet, it works from there as well, and that relay’s access rules decide who can read.
• Nothing is sent for an output nobody reads. Readers pull, just as streams on one machine pull from each other.

Across machines has the details, including reaching another machine’s runtime over ssh.

The runtime

tatolabd is the runtime: one program per machine that runs every stream on it.

Each Python node runs in its own process. When the detector crashes, the camera and the recorder keep going.

• It’s always on. The Mac app and the Linux installer start it for you and restart it if it crashes. On a machine where nothing installed it as a service, you run tatolabd in a terminal. No tatolab command ever starts a runtime.
• There’s one per machine. Every stream on the machine runs in it. A second tatolabd is refused with a message naming the one already running, and so is another user’s: one user owns the runtime on a machine at a time.
• It never runs your Python in its own process. To learn a node’s ports, it asks your project’s Python. To run the node, it starts a process from your .venv.
• Control stays on the machine. The CLI and agents reach the runtime through a local socket that only your user account can open. No network port is opened for control. To work on another machine, you ssh to it.

Attached and kept streams

A stream can be loaded in two ways:

• Attached: tatolab run or tatolab dev. The stream lives as long as the command. Press Ctrl-C, close the terminal, or kill the command, and the stream stops.
• Kept: tatolab run -d. The runtime records the stream and keeps it running after the command returns. It loads it again every time the runtime starts, including after a crash or a reboot, until you tatolab stop or tatolab rm it.

Manage streams covers both.

Seeing inside

Everything a running stream does can be looked at from outside, from a terminal or by an agent:

• tatolab graph prints the live graph as JSON: every node and link, their states, and counters such as dropped bags.
• tatolab tap samples the bags an output sends, without slowing it down.
• tatolab exchange saves the frame a video bag names as a PNG, with no window needed.
• Each stream writes a JSON Lines log, one record per line, in its project’s .streamlib/logs/ folder. tatolab logs reads and follows it, and any log tool can ship it, including into OpenTelemetry. See Logs.

Inspect a running stream shows each in use.

Video, audio and the GPU

Streams that carry video or audio get help from the runtime. You don’t have to use any of it:

• Frames stay in GPU memory. A video bag only names its frame. A model gets it as a GPU tensor through DLPack, with no CPU copy (torch.from_dlpack(frame)), a pixel effect is one GLSL function, and CPU code asks for pixels explicitly with frame.cpu(). See GPU effects and Use your model.
• Hardware codecs and recording. H.264 and H.265 use the GPU’s encoder and decoder, Opus handles audio, and Mp4Sink writes a fragmented MP4 that stays playable if the stream stops abruptly. See Record and encode.
• Audio arrives as you ask for it. An input can ask for a sample rate, a format and a window size, and the runtime converts and cuts the audio to match. See Audio.

One clock

Every timestamp in a stream (timestamp_ns on frames, first_sample_timestamp_ns on audio, ctx.time in a node) comes from the machine’s monotonic clock, in nanoseconds. Timestamps from different nodes and streams on one machine compare directly, so lining up a camera frame with an audio block is a subtraction.

What Tatolab leaves to you

You keep	Tatolab’s part
Your model runtime: PyTorch, ONNX Runtime, TensorRT, MLX	It hands the model the frame in GPU memory. No inference stack ships with Tatolab.
Your Python packages	A node imports whatever your project’s .venv holds; uv add is how a dependency arrives.
Your network: Tailscale, ssh	Control is reachable only on the runtime’s own machine. Reaching another machine is ssh’s job, and the tailnet’s.
Your message queues and databases	Tatolab doesn’t replace them. A node you write sends to them with the client library you already use.