GPU effects

Anything that touches every pixel of every frame belongs on the GPU. Tatolab makes the common case short: a pixel effect is one GLSL function, and GlslPixelEffect supplies everything else a GPU program needs.

You don’t need GLSL experience to follow this page. GLSL looks like C, and the examples here are short enough to read line by line.

A complete effect node

This node darkens the edges of the picture, with a setting for how much:

nodes/vignette.py

"""Darkens the edges of the picture."""

from dataclasses import dataclass
from typing import Annotated

from tatolab.stream import (
    GlslPixelEffect,
    RuntimeContextFullAccess,
    RuntimeContextLimitedAccess,
    VideoFrame,
    node,
)

VIGNETTE_GLSL = """
vec4 effect(vec4 source, ivec2 at) {
    // Where this pixel is, from (0, 0) at one corner to (1, 1) at the other.
    vec2 uv = (vec2(at) + 0.5) / vec2(streamlib_extent);
    // 0.0 at the center, about 1.0 in the corners.
    float distance_from_center = length(uv - 0.5) * 1.4142;
    float shade = 1.0 - dials.amount * smoothstep(0.4, 1.0, distance_from_center);
    return vec4(source.rgb * shade, source.a);
}
"""


@dataclass
class VignetteConfig:
    amount: Annotated[float, "How dark the corners get: 0.0 is none, 1.0 is black."] = 0.6


@node
class Vignette:
    """Darkens the edges of the picture."""

    @node.input(delivery_profile="newest")
    def video(self) -> VideoFrame: ...

    @node.output()
    def vignetted_video(self) -> VideoFrame: ...

    def __init__(self, config: VignetteConfig) -> None:
        self.amount = config.amount

    def setup(self, ctx: RuntimeContextFullAccess) -> None:
        self.effect = GlslPixelEffect.compile(
            ctx.gpu_full_access, effect_glsl=VIGNETTE_GLSL, dials={"amount": "float"}
        )

    def process(self, ctx: RuntimeContextLimitedAccess) -> None:
        frame = ctx.inputs.read("video", into=VideoFrame)
        if frame is None:
            return
        ctx.outputs.write(
            "vignetted_video",
            self.effect.apply_to_frame(ctx.gpu_limited_access, frame, dials={"amount": self.amount}),
        )

Wire it between a source and a window:

stream.py

from tatolab.stream import CameraSource, DisplayWindow, StreamBuilder, stream

from nodes.vignette import Vignette


@stream
def vignette(stream_builder: StreamBuilder) -> None:
    """The camera, with darkened edges."""
    camera = stream_builder.add(CameraSource)
    effect = stream_builder.add(Vignette, config={"amount": 0.8})
    window = stream_builder.add(DisplayWindow)
    stream_builder.connect(camera.output("video"), effect.input("video"))
    stream_builder.connect(effect.output("vignetted_video"), window.input("video"))

Every effect node follows this pattern: compile in setup, apply in process, write the result. Only the GLSL changes.

The effect function

vec4 effect(vec4 source, ivec2 at)

The function runs once for every pixel of the output, in parallel on the GPU.

• source is the input pixel at the same position, as red, green, blue and alpha, each from 0.0 to 1.0.
• at is the pixel’s position in pixels, from (0, 0) at the top-left corner.
• It returns the output pixel, also as RGBA from 0.0 to 1.0. Keep source.a as the alpha unless you mean to change it; an alpha of 0.0 makes the picture disappear.

Inside the function you can also use:

Name	What it is
streamlib_extent	The frame’s width and height in pixels, an ivec2
streamlib_elapsed_seconds	Seconds since the effect first ran, a float; use it for animation
streamlib_source_at(ivec2 p)	The input pixel at another position, clamped to the frame’s edge
streamlib_source_uv(vec2 uv)	The input sampled at normalized coordinates (0.0 to 1.0), smoothly interpolated
dials.<name>	A value you pass from Python on every frame

You can define helper functions above effect, as the edge-detection recipe below does.

The output is a new 8-bit RGBA frame of the same size, carrying the input frame’s timestamp. The input frame isn’t changed, so other nodes reading the same frame aren’t affected.

Dials

Dials pass values from Python into the shader on every frame. Declare their names and GLSL types when you compile:

self.effect = GlslPixelEffect.compile(
    ctx.gpu_full_access,
    effect_glsl=TINT_GLSL,
    dials={"color": "vec4", "amount": "float"},
)

and supply every one of them each time you apply the effect:

self.effect.apply_to_frame(
    ctx.gpu_limited_access, frame, dials={"color": [1.0, 0.5, 0.0, 1.0], "amount": 0.3}
)

• The types are float, int, vec2 and vec4. Pass a number for float, a Python int for int, and a list of 2 or 4 numbers for the vectors.
• Dials don’t persist between frames, so pass all of them every time. An undeclared, missing or wrongly typed dial raises an error that names it.
• All the dials together must fit in 128 bytes, the most every GPU guarantees, and the effect keeps 4 of those for streamlib_elapsed_seconds. That’s room for 31 floats or 7 vec4s. Going over raises an error that says by how much.

Dials are how a node’s settings, its other inputs or its own logic steer the effect. The tutorial’s motion tint sets a dial from a second input.

Recipes

Each of these is a drop-in replacement for the GLSL in the node above. Where a recipe uses dials, declare them with the types shown.

Grayscale

vec4 effect(vec4 source, ivec2 at) {
    float gray = dot(source.rgb, vec3(0.299, 0.587, 0.114));
    return vec4(vec3(gray), source.a);
}

Sepia

vec4 effect(vec4 source, ivec2 at) {
    vec3 c = source.rgb;
    vec3 sepia = vec3(
        dot(c, vec3(0.393, 0.769, 0.189)),
        dot(c, vec3(0.349, 0.686, 0.168)),
        dot(c, vec3(0.272, 0.534, 0.131)));
    return vec4(min(sepia, vec3(1.0)), source.a);
}

Color tint

Dials: color (vec4), amount (float).

vec4 effect(vec4 source, ivec2 at) {
    return vec4(mix(source.rgb, dials.color.rgb, dials.amount), source.a);
}

Pixelate

Dial: block_size (float), the size of each block in pixels.

vec4 effect(vec4 source, ivec2 at) {
    vec2 block = vec2(max(dials.block_size, 1.0));
    vec2 center = (floor(vec2(at) / block) + 0.5) * block;
    return streamlib_source_uv(center / vec2(streamlib_extent));
}

Mirror

vec4 effect(vec4 source, ivec2 at) {
    return streamlib_source_at(ivec2(streamlib_extent.x - 1 - at.x, at.y));
}

Chromatic aberration

Dial: offset (float), how far apart the red and blue channels drift, in pixels.

vec4 effect(vec4 source, ivec2 at) {
    ivec2 shift = ivec2(int(dials.offset), 0);
    float red = streamlib_source_at(at + shift).r;
    float blue = streamlib_source_at(at - shift).b;
    return vec4(red, source.g, blue, source.a);
}

Edge detection

A Sobel filter: bright where the picture changes sharply, dark where it’s flat.

float luma(ivec2 p) {
    return dot(streamlib_source_at(p).rgb, vec3(0.299, 0.587, 0.114));
}

vec4 effect(vec4 source, ivec2 at) {
    float gx = luma(at + ivec2(1, -1)) + 2.0 * luma(at + ivec2(1, 0)) + luma(at + ivec2(1, 1))
             - luma(at + ivec2(-1, -1)) - 2.0 * luma(at + ivec2(-1, 0)) - luma(at + ivec2(-1, 1));
    float gy = luma(at + ivec2(-1, 1)) + 2.0 * luma(at + ivec2(0, 1)) + luma(at + ivec2(1, 1))
             - luma(at + ivec2(-1, -1)) - 2.0 * luma(at + ivec2(0, -1)) - luma(at + ivec2(1, -1));
    float edge = clamp(length(vec2(gx, gy)), 0.0, 1.0);
    return vec4(vec3(edge), source.a);
}

Moving scan lines

vec4 effect(vec4 source, ivec2 at) {
    float line = 0.85 + 0.15 * sin(float(at.y) * 0.5 - streamlib_elapsed_seconds * 8.0);
    return vec4(source.rgb * line, source.a);
}

Develop effects quickly

• Run the stream with tatolab dev and keep the effect’s file open. Every save reloads the stream, so you see each change in a second or two.
• A compile error names the line of your GLSL, counting from the first line of the string.
• For a still frame to compare against, use tatolab exchange to save the effect’s output as a PNG (see Inspect a running stream).
• If you don’t have a camera handy, use TestPatternSource: its color bars make color mistakes obvious.

Chain effects

Give each effect its own node and connect them in a row. Each node is small and testable, you can reorder them in stream.py, and each can be reused in other streams:

stream_builder.connect(camera.output("video"), grain.input("video"))
stream_builder.connect(grain.output("grainy_video"), vignette.input("video"))
stream_builder.connect(vignette.output("vignetted_video"), window.input("video"))

Frames stay on the GPU the whole way.

When one function isn’t enough

GlslPixelEffect covers effects where each output pixel depends on the input around it. When you need more, such as several passes, your own textures, writing to arbitrary positions, or reading a buffer, the GPU context offers full kernels:

• Compute kernels: ctx.gpu_full_access.create_compute_kernel(source=..., push_constant_size=..., bindings=...) in setup, then kernel.dispatch(...) in process. The camera-compute-kernel and camera-halftone examples show the whole pattern, including managing output textures.
• Graphics kernels: vertex and fragment shaders, for drawing geometry. See camera-python-effects.
• Ray tracing kernels: on Linux with a GPU that supports hardware ray tracing; see raytracing-showcase. They aren’t available on macOS.

Tatolab compiles your GLSL itself, so none of these need a shader toolchain installed.

To hand a frame to your own GPU code instead, such as PyTorch or a CUDA library, see Use your model.