Logs

Each stream writes its own log, as JSON Lines: one JSON object per line, in a file in the stream’s project. The format is a stable, versioned contract, so you can read it with tatolab logs, query it with jq, or ship it to the log tools you already run.

What gets logged

A stream’s log holds every record about that stream, from every process involved:

• The runtime and the built-in nodes: a node starting or failing, a camera opening, a recording’s progress.
• Your nodes: each call to log.trace, log.debug, log.info, log.warn or log.error, with the fields you pass.
• Anything a node prints: print(), a library writing to stdout or stderr, even native code. It’s captured, logged at warn, and marked "intercepted": true, so you can find it and replace it with a proper log call.

from tatolab.stream import log

log.info("motion started", changed=0.1391, zone="door")

The message becomes the record’s message, and the keyword fields go into its attrs.

Read it with tatolab logs

tatolab logs --stream motion_watcher -f

--stream reads a loaded stream’s records from the runtime, which keeps each stream’s most recent 4096 in memory, and -f keeps following. The output is one readable line per record. Narrow it down with:

Option	Shows
--level warn	warn and error only. Levels are trace, debug, info, warn and error.
--source python	Your nodes’ records only; --source rust for the runtime’s and the built-ins’
--processor ID	One node’s records, by its ID
--intercepted-only	Only captured output: print() calls and anything written to stdout or stderr

To read a stream that isn’t loaded any more, read its file. In the project folder:

tatolab logs --list                                        # the logs in this project's .streamlib/logs/
tatolab logs Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher -f   # read one, and follow it

Where the files are

Each stream’s log is in its own project, under .streamlib/logs/:

motion-watcher/.streamlib/logs/
├── Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher-1791727378117.jsonl     the one being written
└── Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher-1791727378117.1.jsonl   an older segment

The name is the runtime’s ID, the stream’s name, and when the stream loaded, in milliseconds since 1970. Loading the stream again starts a new log. When a file reaches 100 MiB, it’s renamed with a number, the higher the newer, and a new one is started; each log keeps its 10 most recent files. To read a log in order, read the numbered files from lowest to highest, then the unnumbered one.

The runtime’s own records, the ones that belong to no stream, go to tatolabd-<millis>.jsonl in its state folder: ~/.local/state/tatolab/logs/ on Linux, ~/Library/Application Support/Tatolab/logs/ on macOS. It also prints every record in readable form on its standard error, which on Linux the service sends to the journal (journalctl --user -u tatolabd).

What a record holds

{"schema_version":1,"host_ts":1791727402203914311,"runtime_id":"Rk3x9q2m7v1c8b4n6p0w5z2ty","stream":"motion_watcher","source":"python","level":"info","message":"motion started","target":"streamlib::polyglot::python","pipeline_id":null,"processor_id":"Pq8vbx4t1m0c7n2k5w9r3j6dz","rhi_op":null,"intercepted":false,"attrs":{"changed":0.1391,"zone":"door"}}

Field
schema_version	The format’s version, 1. It changes only when a field is renamed or removed.
host_ts	When the record was made: wall-clock time, in nanoseconds since 1970
runtime_id	The runtime the stream ran in
stream	The stream’s name. Absent on the runtime’s own records.
source	python for your nodes, rust for the runtime and the built-in nodes
level	trace, debug, info, warn or error
message	The message
attrs	The fields passed with it, as a JSON object
processor_id	The ID of the node the record came from, as tatolab graph shows it, or null
target	Where in the code it came from: a module path for the runtime’s records, streamlib::polyglot::python for your nodes’
intercepted, channel	true, with fd1 for stdout or fd2 for stderr, when the record is captured output
pipeline_id, rhi_op, source_ts, source_seq	Detail for debugging the runtime itself, often null

Fields may be added over time; a reader should ignore ones it doesn’t know. Within one file, records are in the order they were written. host_ts can be out of order by a little, because many threads stamp records before they’re written, so sort by it only to merge logs from several streams.

Query it with jq

cd motion-watcher

# Warnings and errors, newest last
jq -c 'select(.level == "warn" or .level == "error") | {level, message, attrs}' .streamlib/logs/*.jsonl

# Everything your nodes printed instead of logging
jq -c 'select(.intercepted) | {processor_id, message}' .streamlib/logs/*.jsonl

# Records per node
jq -r '.processor_id // "runtime"' .streamlib/logs/*.jsonl | sort | uniq -c

Ship it to OpenTelemetry

Tatolab doesn’t send OpenTelemetry itself. The OpenTelemetry Collector reads the files instead, with the filelog receiver from the Collector’s contrib distribution, and turns each record into an OpenTelemetry log record, which it can send to any backend that accepts OTLP:

collector.yaml

receivers:
  filelog:
    include:
      - /home/you/motion-watcher/.streamlib/logs/*.jsonl
      - /home/you/.local/state/tatolab/logs/*.jsonl
    start_at: beginning
    operators:
      - type: json_parser
        parse_ints: true
        timestamp:
          parse_from: attributes.host_ts
          layout_type: epoch
          layout: ns
        severity:
          parse_from: attributes.level
      - type: move
        from: attributes.message
        to: body

exporters:
  otlphttp:
    endpoint: https://otel.example.com:4318

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlphttp]

Each record arrives with host_ts as its timestamp, to the nanosecond, level as its severity (info is INFO, warn is WARN), message as its body, and every other field, including stream, processor_id and your attrs, as attributes. parse_ints: true keeps host_ts exact; without it, the nanoseconds are rounded. To check what the Collector makes of your logs before sending them anywhere, use the debug exporter with verbosity: detailed. We checked this configuration with otelcol-contrib 0.162.0.

Any other log shipper that tails JSON Lines files works the same way: the timestamp is host_ts in nanoseconds, the severity is level, and the message is message.

Durability

• Records are written in batches, every 100 milliseconds or every 64 KiB, whichever comes first.
• When a stream unloads normally, every record it made is written and synced to disk before the unload finishes.
• If the runtime is killed or the machine loses power, the last batch, up to about 100 milliseconds of records, can be lost. A file never ends with half a record.
• If records arrive faster than they can be written, the oldest are dropped, and the log says how many with a dropped= record.

Tune it

The runtime reads these environment variables when it starts:

Variable	Default
STREAMLIB_LOG_ROTATE_BYTES	104857600 (100 MiB)	Size at which a file is set aside and a new one started. 0 never rotates.
STREAMLIB_LOG_RETAIN_SEGMENTS	10	Files kept per log, the one being written included. 0 keeps them all.
STREAMLIB_LOG_BATCH_MS	100	How often records are written
STREAMLIB_LOG_FSYNC_ON_EVERY_BATCH	0	1 syncs every batch to disk: nothing is lost in a power cut, at a large cost in speed
STREAMLIB_QUIET	0	1 stops the readable copy on the runtime’s standard error. The files are still written.

For the Linux service, set them with systemctl --user edit tatolabd, adding for example:

[Service]
Environment=STREAMLIB_LOG_RETAIN_SEGMENTS=30

then systemctl --user restart tatolabd. For a tatolabd you start yourself, set them in its environment.