---
title: "Logs"
description: "Every stream writes a JSON Lines log in its project. Read it with tatolab logs, query it with jq, or ship it to your own tools, including OpenTelemetry."
type: "docs-page"
url: "https://tatolab.com/docs/run/logs"
site: "Tatolab"
section: "Run"
---

# Logs

> Every stream writes a JSON Lines log in its project. Read it with tatolab logs, query it with jq, or ship it to your own tools, including OpenTelemetry.

Each stream writes its own log, as JSON Lines: one JSON object per line, in a file in the stream's project. The format is a stable, versioned contract, so you can read it with `tatolab logs`, query it with `jq`, or ship it to the log tools you already run.

## What gets logged

A stream's log holds every record about that stream, from every process involved:

- **The runtime and the built-in nodes**: a node starting or failing, a camera opening, a recording's progress.
- **Your nodes**: each call to `log.trace`, `log.debug`, `log.info`, `log.warn` or `log.error`, with the fields you pass.
- **Anything a node prints**: `print()`, a library writing to stdout or stderr, even native code. It's captured, logged at `warn`, and marked `"intercepted": true`, so you can find it and replace it with a proper `log` call.

```python
from tatolab.stream import log

log.info("motion started", changed=0.1391, zone="door")
```

The message becomes the record's `message`, and the keyword fields go into its `attrs`.

## Read it with `tatolab logs`

```sh
tatolab logs --stream motion_watcher -f
```

`--stream` reads a loaded stream's records from the runtime, which keeps each stream's most recent 4096 in memory, and `-f` keeps following. The output is one readable line per record. Narrow it down with:

| Option | Shows |
|---|---|
| `--level warn` | `warn` and `error` only. Levels are `trace`, `debug`, `info`, `warn` and `error`. |
| `--source python` | Your nodes' records only; `--source rust` for the runtime's and the built-ins' |
| `--processor ID` | One node's records, by its ID |
| `--intercepted-only` | Only captured output: `print()` calls and anything written to stdout or stderr |

To read a stream that isn't loaded any more, read its file. In the project folder:

```sh
tatolab logs --list                                        # the logs in this project's .streamlib/logs/
tatolab logs Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher -f   # read one, and follow it
```

## Where the files are

Each stream's log is in its own project, under `.streamlib/logs/`:

```txt
motion-watcher/.streamlib/logs/
├── Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher-1791727378117.jsonl     the one being written
└── Rk3x9q2m7v1c8b4n6p0w5z2ty-motion_watcher-1791727378117.1.jsonl   an older segment
```

The name is the runtime's ID, the stream's name, and when the stream loaded, in milliseconds since 1970. Loading the stream again starts a new log. When a file reaches 100 MiB, it's renamed with a number, the higher the newer, and a new one is started; each log keeps its 10 most recent files. To read a log in order, read the numbered files from lowest to highest, then the unnumbered one.

The runtime's own records, the ones that belong to no stream, go to `tatolabd-<millis>.jsonl` in its state folder: `~/.local/state/tatolab/logs/` on Linux, `~/Library/Application Support/Tatolab/logs/` on macOS. It also prints every record in readable form on its standard error, which on Linux the service sends to the journal (`journalctl --user -u tatolabd`).

## What a record holds

```json
{"schema_version":1,"host_ts":1791727402203914311,"runtime_id":"Rk3x9q2m7v1c8b4n6p0w5z2ty","stream":"motion_watcher","source":"python","level":"info","message":"motion started","target":"streamlib::polyglot::python","pipeline_id":null,"processor_id":"Pq8vbx4t1m0c7n2k5w9r3j6dz","rhi_op":null,"intercepted":false,"attrs":{"changed":0.1391,"zone":"door"}}
```

| Field | |
|---|---|
| `schema_version` | The format's version, `1`. It changes only when a field is renamed or removed. |
| `host_ts` | When the record was made: wall-clock time, in nanoseconds since 1970 |
| `runtime_id` | The runtime the stream ran in |
| `stream` | The stream's name. Absent on the runtime's own records. |
| `source` | `python` for your nodes, `rust` for the runtime and the built-in nodes |
| `level` | `trace`, `debug`, `info`, `warn` or `error` |
| `message` | The message |
| `attrs` | The fields passed with it, as a JSON object |
| `processor_id` | The ID of the node the record came from, as `tatolab graph` shows it, or `null` |
| `target` | Where in the code it came from: a module path for the runtime's records, `streamlib::polyglot::python` for your nodes' |
| `intercepted`, `channel` | `true`, with `fd1` for stdout or `fd2` for stderr, when the record is captured output |
| `pipeline_id`, `rhi_op`, `source_ts`, `source_seq` | Detail for debugging the runtime itself, often `null` |

Fields may be added over time; a reader should ignore ones it doesn't know. Within one file, records are in the order they were written. `host_ts` can be out of order by a little, because many threads stamp records before they're written, so sort by it only to merge logs from several streams.

## Query it with `jq`

```sh
cd motion-watcher

# Warnings and errors, newest last
jq -c 'select(.level == "warn" or .level == "error") | {level, message, attrs}' .streamlib/logs/*.jsonl

# Everything your nodes printed instead of logging
jq -c 'select(.intercepted) | {processor_id, message}' .streamlib/logs/*.jsonl

# Records per node
jq -r '.processor_id // "runtime"' .streamlib/logs/*.jsonl | sort | uniq -c
```

## Ship it to OpenTelemetry

Tatolab doesn't send OpenTelemetry itself. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) reads the files instead, with the `filelog` receiver from the Collector's contrib distribution, and turns each record into an OpenTelemetry log record, which it can send to any backend that accepts OTLP:

```yaml title="collector.yaml"
receivers:
  filelog:
    include:
      - /home/you/motion-watcher/.streamlib/logs/*.jsonl
      - /home/you/.local/state/tatolab/logs/*.jsonl
    start_at: beginning
    operators:
      - type: json_parser
        parse_ints: true
        timestamp:
          parse_from: attributes.host_ts
          layout_type: epoch
          layout: ns
        severity:
          parse_from: attributes.level
      - type: move
        from: attributes.message
        to: body

exporters:
  otlphttp:
    endpoint: https://otel.example.com:4318

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlphttp]
```

Each record arrives with `host_ts` as its timestamp, to the nanosecond, `level` as its severity (`info` is INFO, `warn` is WARN), `message` as its body, and every other field, including `stream`, `processor_id` and your `attrs`, as attributes. `parse_ints: true` keeps `host_ts` exact; without it, the nanoseconds are rounded. To check what the Collector makes of your logs before sending them anywhere, use the `debug` exporter with `verbosity: detailed`. We checked this configuration with `otelcol-contrib` 0.162.0.

Any other log shipper that tails JSON Lines files works the same way: the timestamp is `host_ts` in nanoseconds, the severity is `level`, and the message is `message`.

## Durability

- Records are written in batches, every 100 milliseconds or every 64 KiB, whichever comes first.
- When a stream unloads normally, every record it made is written and synced to disk before the unload finishes.
- If the runtime is killed or the machine loses power, the last batch, up to about 100 milliseconds of records, can be lost. A file never ends with half a record.
- If records arrive faster than they can be written, the oldest are dropped, and the log says how many with a `dropped=` record.

## Tune it

The runtime reads these environment variables when it starts:

| Variable | Default | |
|---|---|---|
| `STREAMLIB_LOG_ROTATE_BYTES` | `104857600` (100 MiB) | Size at which a file is set aside and a new one started. `0` never rotates. |
| `STREAMLIB_LOG_RETAIN_SEGMENTS` | `10` | Files kept per log, the one being written included. `0` keeps them all. |
| `STREAMLIB_LOG_BATCH_MS` | `100` | How often records are written |
| `STREAMLIB_LOG_FSYNC_ON_EVERY_BATCH` | `0` | `1` syncs every batch to disk: nothing is lost in a power cut, at a large cost in speed |
| `STREAMLIB_QUIET` | `0` | `1` stops the readable copy on the runtime's standard error. The files are still written. |

For the Linux service, set them with `systemctl --user edit tatolabd`, adding for example:

```ini
[Service]
Environment=STREAMLIB_LOG_RETAIN_SEGMENTS=30
```

then `systemctl --user restart tatolabd`. For a `tatolabd` you start yourself, set them in its environment.