Writing a shader for filmed footage¶
This guide is for writing a SarabOrtex shader from scratch, or adapting a Shadertoy shader, so that it lives inside a shot resolved by SarabResolver: standing on the real floor, hidden behind real objects, lighting the real walls, moving with the real camera, and stacking with other SarabFX nodes. Every rule here comes from the 160+ samples that ship in samples/sarabresolver/; read one of those next to this page.
If you only want to use the samples, start with Shaders in real footage. This page is the author's side of it.
The node setup you write for¶
Read ──> SarabResolver ──> SarabOrtex ──> Viewer
Output: Source Shader File = yours
with depth in Tracking Source = None
alpha Blend Mode = Replace
SarabResolver writes three things into the clip your shader receives as its source: the footage (RGB), the depth of every pixel (alpha), and the camera of every frame (a few pixels in the bottom-left corner that SarabOrtex reads and removes before your shader sees the frame). Your shader gets all of it through the scene functions below; you never read the alpha or those pixels yourself.
The world¶
Everything is in SarabResolver world units: the camera of the first frame sits at the origin looking down +Z, Y is up, +X is screen-right. One unit is roughly the size of a room, so a coffee cup is ~0.08 and a person ~0.35. The numbers you write into a shader are in those units.
The shader also has an object space, which is where you actually write your maths. SarabOrtex's Place in Footage group (Object Position, Rotation, Scale, Anchor, and the Plane Mapper) maps object space into the world: world = Position + Scale · Rotation · (object − Anchor). Every scene function returns values already in object space, so you write your shader as if it were the world and the user moves it with the gizmos. With the defaults the two coincide.
The one point you should care about is the anchor, sarabObjA. It is where Place Plane puts the object when the user clicks a surface, and where the Object Position gizmo drags. Anchor your geometry to it:
const vec3 HALL_ANCHOR = vec3(-0.06, -0.13, 1.05); // the demo hall's floor point (default anchor)
const vec3 OFFSET = vec3(0.30, 0.25, 0.20); // where the object sits relative to it
vec3 center = sarabObjA + OFFSET; // moves with Place Plane / the gizmos
An object built around sarabObjA itself (offset zero) lands exactly where the user clicks.
The scene functions¶
SarabOrtex injects these into any shader that mentions them. Call them; declare nothing. Never declare uniforms named cameraPosition, cameraRotation or cameraFOV: if SarabOrtex sees those names it assumes you supplied them and skips its own.
| Function | Gives you |
|---|---|
vec3 sarabCameraPos() |
The real camera position |
vec3 sarabGetRay(vec2 fragCoord) |
The normalised view ray through the pixel |
vec3 sarabCameraForward() |
The camera's viewing direction |
mat3 sarabCameraMatrix() |
Camera-to-object rotation (for camera-facing cards) |
vec4 sarabSource(vec2 fragCoord) |
The footage under the pixel (use .rgb) |
float sarabDepth(vec2 fragCoord) |
Real depth at the pixel, along the camera axis |
vec3 sarabWorldPos(vec2 fragCoord) |
The real surface point at the pixel |
vec3 sarabSceneNormal(vec2 fragCoord) |
The real surface normal at the pixel |
vec2 sarabProject(vec3 p) |
Where a 3D point lands on screen (a fragCoord) |
float sarabViewDepth(vec3 p) |
Depth of a 3D point along the camera axis |
float sarabVisibility(vec2 fragCoord, float depth, float soft) |
1 if nothing real is in front of depth, 0 if something is; soft over soft units |
bool sarabPlaneActive(), vec2 sarabPlaneHalfSize() |
The Plane Mapper rectangle: on/off, and its half extents in X and Z |
vec3 sarabObjA |
The anchor (a uniform, not a function) |
vec4 sarabOutput(vec3 col, float coverage, float depth, vec2 fragCoord) |
Your final colour, plus the object's depth for the next node (see Stacking) |
"Depth" throughout means distance along the camera axis, not along the ray. For a hit at ray parameter t: depth = t * dot(rd, sarabCameraForward()).
The skeleton every footage shader shares¶
void mainImage(out vec4 fragColor, in vec2 fragCoord) {
vec3 scene = sarabSource(fragCoord).rgb; // 1. the footage is the background
vec3 ro = sarabCameraPos(); // 2. the real camera
vec3 rd = sarabGetRay(fragCoord);
float cosF = dot(rd, sarabCameraForward());
float sceneD = sarabDepth(fragCoord); // 3. how far the real world is, on this pixel
// 4. bound your work: a sphere/box around the object (ray/box span), march only inside it
// 5. for every sample or hit at ray distance t:
// float vis = smoothstep(-EDGE_SOFT, EDGE_SOFT, sceneD - t * cosF); // hidden behind real things?
// accumulate colour and coverage × vis; stop marching once t * cosF > sceneD
// 6. light the real surfaces near the object (sarabWorldPos / sarabSceneNormal)
// 7. composite over `scene`
fragColor = sarabOutput(col, coverage, hitDepth, fragCoord); // 8. colour + depth for the next node
}
Steps 3–5 are what make an object part of the shot instead of a sticker on it: a pillar between the camera and the object hides exactly the part of it behind the pillar, sample by sample, and the object can walk behind things.
Occlusion, in detail¶
The comparison is always between the real depth on this pixel and the depth of your sample:
const float EDGE_SOFT = 0.008; // softness where you meet real surfaces (units)
float vis = smoothstep(-EDGE_SOFT, EDGE_SOFT, sceneD - t * cosF);
- Solid surface hit: multiply the hit's coverage by
vis. - Volume (fire, smoke, fur): multiply each sample's density by
vis, and stop the march at the real surface (tEnd = min(tExit, sceneD / cosF)), so nothing is ever drawn inside a wall. - Particles and sparks: test each particle at its own depth. A cloud of embers then loses only the ones behind the pillar.
- Camera-facing cards (2D effects): test the card's hit point. A flat flame on a card is hidden by anything in front of the card.
- Things that sit ON a real surface (water on the floor, a picture on a wall): the surface itself must not hide them. Add a tolerance: treat real depth within
PLANE_TOL(~3 cm) of the plane as the plane, or bias the test byDEPTH_BIAS(~0.03). Objects standing on that surface, like table legs, still cut it.
The depth is 8-bit in the clip, so it steps by about a centimetre at room scale. EDGE_SOFT of 0.5–1 cm hides that.
Standing on the floor, sitting on the table¶
The floor is not at a fixed height in real footage; SarabResolver's floor drifts by a few centimetres between frames. Two approaches:
- Let the user place it: anchor your base at
sarabObjAand tell them to Place Plane on the floor. That fits the plane to the real depth at the click and puts the anchor on it. - Snap every frame: scan a short vertical line through your object's axis with
sarabProject+sarabDepth, find where the real surface starts, and stand the object there. The candle sample (Fire/candle_flame) does this to stay on the table top.
Add a contact shadow where the object touches: darken the real surface within a small radius of the base, more the closer it is (exp(-r²/R²) × (1 − smoothstep(0, 0.06, |P.y − base.y|))).
Lighting the room, and being lit by it¶
This is what sells a fire or a neon sign. For every pixel that shows a real surface near your object:
vec3 P = sarabWorldPos(fragCoord);
vec3 N = sceneNormalWide(fragCoord); // see the note below
vec3 L = lightPos - P;
float dist = length(L);
float lam = max(dot(N, L / dist), 0.0);
float x = dist / FALLOFF;
scene += SPILL_COLOR * SPILL * flicker * lam / (1.0 + x * x) * (0.3 + scene);
- Use a wide-footprint normal.
sarabSceneNormal()differences neighbouring pixels of an 8-bit depth, which bands on flat walls. Difference ±6 pixels instead:
vec3 sceneNormalWide(vec2 fc) {
vec3 dx = sarabWorldPos(fc + vec2(6.0, 0.0)) - sarabWorldPos(fc - vec2(6.0, 0.0));
vec3 dy = sarabWorldPos(fc + vec2(0.0, 6.0)) - sarabWorldPos(fc - vec2(0.0, 6.0));
vec3 n = normalize(cross(dx, dy));
return dot(n, sarabCameraPos() - sarabWorldPos(fc)) < 0.0 ? -n : n;
}
- Wrap the lambert (
(dot(N,L) + 0.5) / 1.5) for grazing walls, or the spill vanishes on them. - Flicker for fire: a product of two or three sines at unrelated rates.
- Flashes (lightning, fireworks): a brief room-wide lift of the plate plus the local spill.
- Shadows on the floor: from the real surface point, march towards the key light through your object's distance field (
bodyShadow) and darken by the result. For a sphere an analytic term is enough. - Being lit: match the plate. Sample the footage at five points, average it, and tint your object's ambient with that colour (
PLATE_TINT); scale its exposure to the plate's level. An object lit for a black Shadertoy background looks pasted on a sunlit courtyard. - Reflecting the room: walk the reflected ray and compare
sarabViewDepth(q)withsarabDepth(sarabProject(q))at each step. Once it passes behind the real surface, bisect and takesarabSourcethere. Fall back to the plate average when the ray leaves the frame.
Compositing¶
- Solid, opaque:
col = mix(scene, objectColour, coverage). - Volumes: march front to back, accumulate premultiplied colour and
alpha = 1 − transmittance; thencol = scene × (1 − alpha) + accumulated. - Emissive (fire, lasers, neon, holograms, anything drawn on black): screen it, so bright plates don't clip:
col = scene + glow − clamp(scene, 0, 1) × glow. Black in the original becomes see-through. - Matte prints on a surface (tiles, patterns):
picture × plate local brightness, so the real light and shadows stay on it.
Whatever the original's "sky" or "background" was, remove it. The footage is the background.
Cards: 2D effects in 3D¶
Most Shadertoy fire, flare and sign shaders are 2D. Put them on a card:
- Camera-facing: build the card's axes from
sarabCameraMatrix()and intersect the ray with its plane. Good for fireballs, flares, sparks. - Upright, turning about Y to face the camera: for things standing on the floor (a flame, a sign). It reads right from every side of the walk.
- World-fixed: a plane through the anchor with a fixed normal. Portals, screens.
Map the card's local coordinates to the original's fragCoord on a virtual resolution (e.g. 960×540), evaluate the original's mainImage as a function vec4 picture(vec2 fc) with iResolution replaced by that constant, feather the card edge, and depth-test the hit. Keep +Z of the card as "up" so it stands upright when the user tilts the plane onto a wall.
Surfaces: the Plane Mapper rectangle¶
For effects that are a surface (water, a projected screen, a rug), the plane is the placement:
vec2 hs = sarabPlaneActive() ? sarabPlaneHalfSize() : PLANE_HALF; // the rectangle, half extents in X and Z
The rectangle is the object-space plane through the anchor with normal +Y, spanning X and Z. The user puts it on the floor or a wall with Place Plane and drags its corners; your shader sees it as its local X/Z plane either way.
- Water: evaluate the height field on the rectangle (scale the waves to 10–30 cm), march the ray through the slab above it and bisect to the surface, then reflect the room, refract the real floor through
WATER_DEPTHof absorbing water, blend by Fresnel, add the key-light glint. Keep the water level a few millimetres above the plane so no trough dips under the real floor. - Screen or print: ray/plane hit → local (x, z) → the picture's fragCoord; screen emissive pictures with a glow spill, multiply matte ones by the plate.
Motion and animation¶
- The camera moves for free. You never animate the camera; SarabResolver's camera is applied for you, so a static object already tracks the shot. Remove the original's orbit, mouse and fly-through.
- Loop cleanly with
mod(iTime, CYCLE)for walks, flights and cycles, and keep anything random keyed to the cycle time so the loop is seamless. - Move through the room, not the screen: a meteor flies from one world point to another; a character walks along X. It then passes behind pillars on its own.
- Static jitter, never temporal: dither marches with
hash(fragCoord), nothash(fragCoord + iTime), or the object shimmers between frames. - Secondary motion sells characters: evaluate the pose twice (
tandt − DELAY) and let tips, hair or cloth trail by the difference. - Timed events (a strike every 3 s, a click at 6.0 s): expose the times as consts so they can be matched to the footage.
Stacking with other nodes¶
End with sarabOutput(col, coverage, depth, fragCoord). It writes your colour and, into the alpha, the nearer of your object's depth and the real depth. With Pass Scene Downstream on, the next SarabOrtex (or SarabHyper, SarabInator, Sarabography, SarabBrush3D) then hides its object behind yours as well as behind the real scene, in either node order. It costs one comparison per pixel. coverage is how much of the pixel your object covers (a hit = 1; a volume = its alpha); depth is the first hit or first dense sample.
Performance¶
- Bound every march to the object: ray/box or ray/sphere first, march only the span.
- 40–140 steps per pixel is the usual budget; a 1080p frame should stay under ~0.4 s on an M3 Max so Pre-Render fills the cache quickly.
- Refresh expensive things (normals, lag vectors) every few centimetres along the ray, not every sample.
- Depth-test before shading a sample; a hidden sample costs nothing.
- Replace texture noise with in-shader value noise. It removes the preset, and the converter handles it better.
Rules the converter imposes¶
SarabOrtex converts your GLSL to SkSL and Metal. These break the conversion:
- Never write
fragCoord - 0.5*iResolution(any spacing) anywhere, comments included: it switches the automatic Y-flip off and the image renders upside down. Write(2.0 * fragCoord - iResolution.xy) / (2.0 * iResolution.y). - Comments are parsed. Don't quote golf loops (
for(O*=i; i++<9e1;…)),max(0,,tanh(or the wordfragCoordinside a helper's comment; the converter acts on them. Writemax(0.0, …). - No function-like
#definemacros. Use functions. - Variable types are tracked by name, globally. Don't reuse a name with two types in different functions.
- Metal reserves
and,or,not,this,char,half,saturate: don't use them as identifiers. - Constant loop bounds (a
breakis fine); guardpowof negatives,sqrt,normalize(0), and divisions by values that can reach zero. - Buffer passes have no camera or footage. Everything that depends on either belongs in the Image pass.
Licensing and credit¶
Shadertoy's default licence is CC BY-NC-SA 3.0 (non-commercial). Keep the author, the URL and the licence in the header, list what you changed, and replace any snippet whose author forbids altered versions (Inigo Quilez's noise, SDF and palette helpers are the ones you meet most; write your own). Shaders you write yourself, and MIT / Apache / CC BY ones, can ship.
Header, in the house style¶
// <Effect name> in filmed space — one line on what it does in the shot.
//
// Original: "<title>" by <author> (<Shadertoy URL>), <licence as the file states, or the default>.
// What changed from the original:
// - the camera is the real one (sarabCameraPos / sarabGetRay); orbit and mouse removed;
// - the black background is gone: the footage is the background, the object is screened over it;
// - every march sample is depth-tested, so pillars hide the part behind them;
// - warm flickering light on the real floor and walls; contact darkening under the base.
// Kept: the field, the colour ramp, the tone map.
//
// Load: SarabOrtex → Shader File = samples/sarabresolver/<Category>/<name>/<name>.glsl
// Chain: Read → SarabResolver (Output = Source with depth in alpha) → SarabOrtex (Tracking Source = None, Blend Replace).
const vec3 OFFSET = vec3(0.0, 0.0, 0.0); // position relative to the anchor
const float SCALE = 0.3; // object units → world units
const float EDGE_SOFT = 0.008; // occlusion softness
Put every number a user might tune in that block with a one-line comment.
Testing without a host¶
build_shader_tests/shader_test_engine renders a shader on a real frame with its camera and depth, exactly as the plugin would:
shader_test_engine validate my.glsl --gpu
shader_test_engine render my.glsl -o out.png --size 960x540 --time 3 \
--source frame.png --source-depth 0,2 \
--camera "px,py,pz,rx,ry,rz,fov" \
--object "px,py,pz,rx,ry,rz,scale,ax,ay,az" --plane hw,hd --gpu
--source is a frame exported from SarabResolver (Source with depth in alpha), --camera its recovered camera for that frame, --object and --plane the Place in Footage values. Render a frame with an object in front and one from the side, and look at both: if the pillar cuts your object cleanly and nothing floats where the wall is, it will work in the host.
See also: Shaders in real footage · SarabOrtex · SarabResolver