---
title: "GPU effects"
description: "Write pixel effects as one GLSL function, steer them from Python with dials, and know when to reach for a full compute kernel."
type: "docs-page"
url: "https://tatolab.com/docs/build/gpu-effects"
site: "Tatolab"
section: "Build"
---

# GPU effects

> Write pixel effects as one GLSL function, steer them from Python with dials, and know when to reach for a full compute kernel.

Anything that touches every pixel of every frame belongs on the GPU. Tatolab makes the common case short: a **pixel effect** is one GLSL function, and `GlslPixelEffect` supplies everything else a GPU program needs.

You don't need GLSL experience to follow this page. GLSL looks like C, and the examples here are short enough to read line by line.

## A complete effect node

This node darkens the edges of the picture, with a setting for how much:

```python title="nodes/vignette.py"
"""Darkens the edges of the picture."""

from dataclasses import dataclass
from typing import Annotated

from tatolab.stream import (
    GlslPixelEffect,
    RuntimeContextFullAccess,
    RuntimeContextLimitedAccess,
    VideoFrame,
    node,
)

VIGNETTE_GLSL = """
vec4 effect(vec4 source, ivec2 at) {
    // Where this pixel is, from (0, 0) at one corner to (1, 1) at the other.
    vec2 uv = (vec2(at) + 0.5) / vec2(streamlib_extent);
    // 0.0 at the center, about 1.0 in the corners.
    float distance_from_center = length(uv - 0.5) * 1.4142;
    float shade = 1.0 - dials.amount * smoothstep(0.4, 1.0, distance_from_center);
    return vec4(source.rgb * shade, source.a);
}
"""


@dataclass
class VignetteConfig:
    amount: Annotated[float, "How dark the corners get: 0.0 is none, 1.0 is black."] = 0.6


@node
class Vignette:
    """Darkens the edges of the picture."""

    @node.input(delivery_profile="newest")
    def video(self) -> VideoFrame: ...

    @node.output()
    def vignetted_video(self) -> VideoFrame: ...

    def __init__(self, config: VignetteConfig) -> None:
        self.amount = config.amount

    def setup(self, ctx: RuntimeContextFullAccess) -> None:
        self.effect = GlslPixelEffect.compile(
            ctx.gpu_full_access, effect_glsl=VIGNETTE_GLSL, dials={"amount": "float"}
        )

    def process(self, ctx: RuntimeContextLimitedAccess) -> None:
        frame = ctx.inputs.read("video", into=VideoFrame)
        if frame is None:
            return
        ctx.outputs.write(
            "vignetted_video",
            self.effect.apply_to_frame(ctx.gpu_limited_access, frame, dials={"amount": self.amount}),
        )
```

Wire it between a source and a window:

```python title="stream.py"
from tatolab.stream import CameraSource, DisplayWindow, StreamBuilder, stream

from nodes.vignette import Vignette


@stream
def vignette(stream_builder: StreamBuilder) -> None:
    """The camera, with darkened edges."""
    camera = stream_builder.add(CameraSource)
    effect = stream_builder.add(Vignette, config={"amount": 0.8})
    window = stream_builder.add(DisplayWindow)
    stream_builder.connect(camera.output("video"), effect.input("video"))
    stream_builder.connect(effect.output("vignetted_video"), window.input("video"))
```

Every effect node follows this pattern: compile in `setup`, apply in `process`, write the result. Only the GLSL changes.

## The `effect` function

```glsl
vec4 effect(vec4 source, ivec2 at)
```

The function runs once for every pixel of the output, in parallel on the GPU.

- `source` is the input pixel at the same position, as red, green, blue and alpha, each from 0.0 to 1.0.
- `at` is the pixel's position in pixels, from `(0, 0)` at the top-left corner.
- It returns the output pixel, also as RGBA from 0.0 to 1.0. Keep `source.a` as the alpha unless you mean to change it; an alpha of 0.0 makes the picture disappear.

Inside the function you can also use:

| Name | What it is |
|---|---|
| `streamlib_extent` | The frame's width and height in pixels, an `ivec2` |
| `streamlib_elapsed_seconds` | Seconds since the effect first ran, a `float`; use it for animation |
| `streamlib_source_at(ivec2 p)` | The input pixel at another position, clamped to the frame's edge |
| `streamlib_source_uv(vec2 uv)` | The input sampled at normalized coordinates (0.0 to 1.0), smoothly interpolated |
| `dials.<name>` | A value you pass from Python on every frame |

You can define helper functions above `effect`, as the edge-detection recipe below does.

The output is a new 8-bit RGBA frame of the same size, carrying the input frame's timestamp. The input frame isn't changed, so other nodes reading the same frame aren't affected.

## Dials

Dials pass values from Python into the shader on every frame. Declare their names and GLSL types when you compile:

```python
self.effect = GlslPixelEffect.compile(
    ctx.gpu_full_access,
    effect_glsl=TINT_GLSL,
    dials={"color": "vec4", "amount": "float"},
)
```

and supply every one of them each time you apply the effect:

```python
self.effect.apply_to_frame(
    ctx.gpu_limited_access, frame, dials={"color": [1.0, 0.5, 0.0, 1.0], "amount": 0.3}
)
```

- The types are `float`, `int`, `vec2` and `vec4`. Pass a number for `float`, a Python `int` for `int`, and a list of 2 or 4 numbers for the vectors.
- Dials don't persist between frames, so pass all of them every time. An undeclared, missing or wrongly typed dial raises an error that names it.
- All the dials together must fit in 128 bytes, the most every GPU guarantees, and the effect keeps 4 of those for `streamlib_elapsed_seconds`. That's room for 31 `float`s or 7 `vec4`s. Going over raises an error that says by how much.

Dials are how a node's settings, its other inputs or its own logic steer the effect. The [tutorial's motion tint](https://tatolab.com/docs/tutorial/gpu-tint) sets a dial from a second input.

## Recipes

Each of these is a drop-in replacement for the GLSL in the node above. Where a recipe uses dials, declare them with the types shown.

### Grayscale

```glsl
vec4 effect(vec4 source, ivec2 at) {
    float gray = dot(source.rgb, vec3(0.299, 0.587, 0.114));
    return vec4(vec3(gray), source.a);
}
```

### Sepia

```glsl
vec4 effect(vec4 source, ivec2 at) {
    vec3 c = source.rgb;
    vec3 sepia = vec3(
        dot(c, vec3(0.393, 0.769, 0.189)),
        dot(c, vec3(0.349, 0.686, 0.168)),
        dot(c, vec3(0.272, 0.534, 0.131)));
    return vec4(min(sepia, vec3(1.0)), source.a);
}
```

### Color tint

Dials: `color` (`vec4`), `amount` (`float`).

```glsl
vec4 effect(vec4 source, ivec2 at) {
    return vec4(mix(source.rgb, dials.color.rgb, dials.amount), source.a);
}
```

### Pixelate

Dial: `block_size` (`float`), the size of each block in pixels.

```glsl
vec4 effect(vec4 source, ivec2 at) {
    vec2 block = vec2(max(dials.block_size, 1.0));
    vec2 center = (floor(vec2(at) / block) + 0.5) * block;
    return streamlib_source_uv(center / vec2(streamlib_extent));
}
```

### Mirror

```glsl
vec4 effect(vec4 source, ivec2 at) {
    return streamlib_source_at(ivec2(streamlib_extent.x - 1 - at.x, at.y));
}
```

### Chromatic aberration

Dial: `offset` (`float`), how far apart the red and blue channels drift, in pixels.

```glsl
vec4 effect(vec4 source, ivec2 at) {
    ivec2 shift = ivec2(int(dials.offset), 0);
    float red = streamlib_source_at(at + shift).r;
    float blue = streamlib_source_at(at - shift).b;
    return vec4(red, source.g, blue, source.a);
}
```

### Edge detection

A Sobel filter: bright where the picture changes sharply, dark where it's flat.

```glsl
float luma(ivec2 p) {
    return dot(streamlib_source_at(p).rgb, vec3(0.299, 0.587, 0.114));
}

vec4 effect(vec4 source, ivec2 at) {
    float gx = luma(at + ivec2(1, -1)) + 2.0 * luma(at + ivec2(1, 0)) + luma(at + ivec2(1, 1))
             - luma(at + ivec2(-1, -1)) - 2.0 * luma(at + ivec2(-1, 0)) - luma(at + ivec2(-1, 1));
    float gy = luma(at + ivec2(-1, 1)) + 2.0 * luma(at + ivec2(0, 1)) + luma(at + ivec2(1, 1))
             - luma(at + ivec2(-1, -1)) - 2.0 * luma(at + ivec2(0, -1)) - luma(at + ivec2(1, -1));
    float edge = clamp(length(vec2(gx, gy)), 0.0, 1.0);
    return vec4(vec3(edge), source.a);
}
```

### Moving scan lines

```glsl
vec4 effect(vec4 source, ivec2 at) {
    float line = 0.85 + 0.15 * sin(float(at.y) * 0.5 - streamlib_elapsed_seconds * 8.0);
    return vec4(source.rgb * line, source.a);
}
```

## Develop effects quickly

- Run the stream with `tatolab dev` and keep the effect's file open. Every save reloads the stream, so you see each change in a second or two.
- A compile error names the line of *your* GLSL, counting from the first line of the string.
- For a still frame to compare against, use `tatolab exchange` to save the effect's output as a PNG (see [Inspect a running stream](https://tatolab.com/docs/run/inspect)).
- If you don't have a camera handy, use `TestPatternSource`: its color bars make color mistakes obvious.

## Chain effects

Give each effect its own node and connect them in a row. Each node is small and testable, you can reorder them in `stream.py`, and each can be reused in other streams:

```python
stream_builder.connect(camera.output("video"), grain.input("video"))
stream_builder.connect(grain.output("grainy_video"), vignette.input("video"))
stream_builder.connect(vignette.output("vignetted_video"), window.input("video"))
```

Frames stay on the GPU the whole way.

## When one function isn't enough

`GlslPixelEffect` covers effects where each output pixel depends on the input around it. When you need more, such as several passes, your own textures, writing to arbitrary positions, or reading a buffer, the GPU context offers full kernels:

- **Compute kernels:** `ctx.gpu_full_access.create_compute_kernel(source=..., push_constant_size=..., bindings=...)` in `setup`, then `kernel.dispatch(...)` in `process`. The [camera-compute-kernel](https://github.com/tatolab/streamlib/tree/main/examples/camera-compute-kernel) and [camera-halftone](https://github.com/tatolab/streamlib/tree/main/examples/camera-halftone) examples show the whole pattern, including managing output textures.
- **Graphics kernels:** vertex and fragment shaders, for drawing geometry. See [camera-python-effects](https://github.com/tatolab/streamlib/tree/main/examples/camera-python-effects).
- **Ray tracing kernels:** on Linux with a GPU that supports hardware ray tracing; see [raytracing-showcase](https://github.com/tatolab/streamlib/tree/main/examples/raytracing-showcase). They aren't available on macOS.

Tatolab compiles your GLSL itself, so none of these need a shader toolchain installed.

To hand a frame to your own GPU code instead, such as PyTorch or a CUDA library, see [Use your model](https://tatolab.com/docs/build/models).