Skip to content

Shaders in real footage

SarabOrtex can place a shader inside your shot, not just on top of it. The object follows the real camera, sits at a real distance, is hidden by real objects that pass in front of it, and lights, shadows and reflects the real room.

The 3D information comes from SarabResolver. This page is for anyone writing or porting a shader for that workflow, whether you start from a Shadertoy shader or write one from scratch.

The node setup

Read ──> SarabResolver ──> SarabOrtex ──> Viewer / Output
Node Settings
SarabResolver Output = Source with depth in alpha. Camera = Tracked. Keep Lock Lens and Restore Plate Shake on and Camera Smoothing at 4. Click Resolve Clip Now once from the first frame.
SarabOrtex Shader File = your shader, or Load Preset for multi-pass shaders and shaders with textures. Tracking Source = None: a shader that uses the SarabFX camera follows SarabResolver's camera automatically. Blend Mode = Replace: the shader outputs the finished composite.

Nothing else is needed. You don't need a Merge, Grade or holdout matte, because the shader composites itself.

The world your shader lives in

  • Units are SarabResolver's. Tracked mode keeps one consistent scale for the whole clip. The first frame's camera sits at the origin and looks down +Z, with Y up.
  • Finding a position: set SarabResolver's Depth Mapping to Raw values and read a pixel with the viewer's colour picker on the first frame. A pixel at the image centre with depth d is at world point (0, 0, d). For example, the floor of the demo hall is at y ≈ -0.13, and its back wall is at z ≈ 1.6.
  • Depth means distance along the camera's viewing axis, not along the ray.

Placing the shader in the shot

A shader keeps its own coordinates. SarabOrtex's Place in Footage group decides where they land in the real world. The camera, the depth and the room it hands the shader are all transformed to match, so the object moves, turns and scales, and the footage does not move.

Control What it does
Object Position / Rotation / Scale Where the shader's Anchor sits, how it is turned and how big it is. Drag them in the viewer with the Gizmo Mode entries Object Position (Shader Only), Object Rotation (Shader Only) and Object Scale (Shader Only).
Shader Anchor The point of the shader's own space that is placed at Object Position. The SarabFX samples use the floor point under the object, so they stand on whatever they are placed on.
Reset Placement Puts the shader back where it was written.
Place Plane Click it, then click a surface in the viewer, such as a table top, a wall or the floor. SarabOrtex fits a plane to the real depth there, and the shader stands on it.
Make Level Makes the plane exactly horizontal and keeps its position and heading.
Track Point Click it, then click a textured detail of something that moves in the shot (a badge on a car, the corner of a box, a knuckle). From then on the shader rides that detail: see Motion beyond keyframes.
Follow Tracked Point Turned on by Track Point. Off keeps the track but stops moving the shader.
Follow Mode Position: one tracked point moves the shader. Position + Rotation (planar): a small planar patch (the click and four satellites) is tracked; its templates turn and grow with the detail, and its measured 3D rotation turns the shader with it — a label on a turning box, a hand that rolls. Planar Surface (drawn area): the surface you drew with Draw Track Area is tracked with up to 32 feature points and a homography — it may turn, tilt, come closer and be partly covered.
Draw Track Area Click it, then drag a rectangle over a flat, textured surface in the viewer (a poster, a phone screen, a table top, the side of a van). Feature points appear inside it: cyan while they agree with the surface, red when they don't, grey while re-projected behind an occluder. Drag the corner handles to refine the area on the frame you drew it on. The surface's 3D motion moves and turns the shader.
Follow Scale Planar mode: also scale the shader by the patch's measured size change. Off by default — a real thing keeps its size in a solved shot.
Follow Smoothing Frames of smoothing on the tracked motion, each way. 0 follows every jitter; 2–4 steadies a hand-held track.
Plane Size / Scale Shader with Plane The rectangle's size, and whether resizing it scales the shader.

Viewer gizmos

  • Object Position mode:
  • Drag the centre to move the object across the view at its current depth.
  • Drag the Z square to bring it nearer or push it farther.
  • Object Rotation mode:
  • Drag the centre to turn it left, right, up or down.
  • Drag the ring to roll it.
  • Plane Mapper mode: the plane is drawn as a glowing rectangle in perspective.
  • Drag a corner to resize it.
  • Drag the centre to slide it along its own surface.
  • Drag N to push it along its normal.
  • Drag the ring to turn it within its surface.
  • Drag the trackball (bottom right) to tilt it.
  • The coloured axes at its centre show the applied rotation.

All of these are ordinary animatable parameters, so placements can be keyframed.

Shaders can read the mapped plane: sarabPlaneActive() and sarabPlaneHalfSize(), the half extents along X and Z in shader units. The plane is the shader-space plane through the Anchor with normal +Y.

Several objects in one shot. Chain SarabOrtex nodes:

Read ──> SarabResolver ──> SarabOrtex (fire, Pass Scene Downstream ON) ──> SarabOrtex (character, OFF) ──> Viewer

Turn on Pass Scene Downstream on every SarabOrtex except the last. That node's output keeps the footage depth in alpha and carries the camera on. The next SarabOrtex then sees the shot, with the first object already in it, as if it came from SarabResolver. Give each node its own placement.

When a shader reports its object's depth with sarabOutput (the SarabFX samples do), the depth passed on is the nearer of the object and the real scene, so the objects hide each other correctly whatever the node order. A shader that doesn't report it passes the real scene's depth only: its object hides behind real objects, but a later node draws over it where they overlap on screen.

Pre-Render (Frame Cache group) renders the whole clip into the frame cache with the host's progress bar, so heavy shaders play smoothly from the first loop.

Motion beyond keyframes

Object Position, Rotation and Scale take host keyframes like any parameter, and the viewer gizmos set a key at the current frame when the parameter already has keys. For an object that has to ride something moving in the shot, keyframing by hand is slow and never quite locks, so Place in Footage has a tracked motion path:

  1. Park on the frame where the object should be placed, and place it (gizmos or Place Plane).
  2. Click Track Point, then click a textured detail of the moving thing in the viewer. The pen shows a cyan crosshair while it waits for the click.
  3. That frame is the reference: the object stays exactly where you put it there. On every other frame it is moved by the detail's real 3D displacement — the detail is followed through the footage (normalised cross-correlation of a small template around a constant-velocity prediction) and lifted into the world with SarabResolver's depth, so a box coming towards the camera brings the object with it, not just across the screen.
  4. A cyan diamond marks the tracked detail; it turns red on frames where the detail could not be found (the position is held there) and a faint line joins it to the object. If the marker wanders, park on a good frame and click Track Point again on a better detail.

For something that turns — a box on a conveyor, a hand that rolls over — set Follow Mode to Position + Rotation (planar) before clicking Track Point, and click on a flat textured area at least a tenth of the frame across. Four satellite markers appear around the click; the patch's rotation is measured from their depths and turns the object with it. Satellites that lose the detail turn red and are re-projected from the others.

For a whole surface — a screen to replace, a poster on a wall the camera walks past, the side of a van — use Draw Track Area instead of a point: drag a rectangle over the surface on a frame where it is fully visible. A cyan outline with dots inside it follows the surface through turns, tilts, perspective changes and partial occlusion; the outline turns red on frames where it is held. This is the same idea as planar tracking in Mocha, with the difference that SarabResolver's depth turns the 2D track into the surface's real 3D motion, so the object placed on it is truly on it, not projected onto the screen.

Keyframes still apply on top of the track, so the object can ride a car roof and be animated relative to it. The track is walked frame by frame the first time a frame is rendered and shared between the host's render threads; scrubbing far ahead in a long clip takes a moment the first time. Follow Smoothing averages the tracked point over a few frames each way when the footage is hand-held.

Depth at the tracked pixel is what turns the 2D track into 3D: the source must carry SarabResolver's Source with depth in alpha, and a detail on the moving thing itself gives a steadier depth than a detail at its silhouette.

Scene functions

SarabOrtex injects these functions into any shader that calls them. Call them by name; there is nothing to declare. They all work in the shader's own space, after the placement above.

Function Returns
vec3 sarabCameraPos() Camera position (world)
vec3 sarabGetRay(vec2 fragCoord) Normalised view ray through the pixel (world)
mat3 sarabCameraMatrix() Camera-to-world rotation
vec2 sarabPlaneHalfSize(), bool sarabPlaneActive() The Plane Mapper's rectangle, in shader units (see above)
vec3 sarabCameraForward() Camera viewing direction
vec4 sarabSource(vec2 fragCoord) The footage under the pixel (RGB; A holds the depth)
float sarabDepth(vec2 fragCoord) Real depth at the pixel, in world units, whatever Depth Mapping is set
vec3 sarabWorldPos(vec2 fragCoord) The 3D point of the real surface at the pixel
vec3 sarabSceneNormal(vec2 fragCoord) Normal of the real surface, rebuilt from the depth
vec2 sarabProject(vec3 worldPos) Where a world point lands on screen (fragCoord)
float sarabViewDepth(vec3 worldPos) Depth of a world point along the camera axis
float sarabVisibility(vec2 fragCoord, float depth, float softness) 0 when a real object is in front of depth, 1 when nothing is; soft over softness world units
vec4 sarabOver(vec4 premul, vec2 fragCoord) A premultiplied colour composited over the footage (alpha 1)

A minimal object

const vec3  CENTER = vec3(0.0, 0.0, 1.05);   // where the object floats (world units)
const float RADIUS = 0.09;

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
    vec3 ro = sarabCameraPos();
    vec3 rd = sarabGetRay(fragCoord);
    vec3 scene = sarabSource(fragCoord).rgb;              // the footage is the background

    // ray / sphere
    vec3 oc = ro - CENTER;
    float b = dot(oc, rd), h = b * b - dot(oc, oc) + RADIUS * RADIUS;
    if (h < 0.0) { fragColor = vec4(scene, 1.0); return; }
    float t = -b - sqrt(h);
    vec3 n = normalize(ro + rd * t - CENTER);
    vec3 col = vec3(0.9) * (0.3 + 0.7 * max(dot(n, normalize(vec3(-0.3, 1.0, -0.2))), 0.0));

    // hide behind real objects
    float vis = sarabVisibility(fragCoord, t * dot(rd, sarabCameraForward()), 0.015);
    fragColor = vec4(mix(scene, col, vis), 1.0);
}

Integration recipes

Solid objects

  • Shadow: for every real-surface pixel, take P = sarabWorldPos(fc) and N = sarabSceneNormal(fc), then trace from P towards the light through your object's shape (a soft-shadow march or an analytic sphere shadow) and darken the footage.
  • Contact darkening: darken real surfaces close to the object, for example with an analytic sphere-occlusion term.
  • Reflections of the room: march the reflected ray and, at each step, compare sarabViewDepth(q) with sarabDepth(sarabProject(q)). Once the ray passes behind the real surface, bisect a few times and take sarabSource at that point. Rays that leave the frame need a fallback, such as the plate's average colour or an environment cubemap.

Glowing objects

  • Composite: add the light on top, or use a screen blend (plate + glow * (1 - 0.5 * plate)) so bright areas don't clip.
  • Light spill on the room: add colour * max(dot(N, normalize(C - P)), 0) / (1 + (|C - P| / falloff)^2) to the real surfaces.

Volumes (smoke, fire, plasma)

  • Bounding: limit the march to the volume's bounding sphere or box.
  • Occlusion per sample: compare each sample's depth with sarabDepth(fragCoord) so a pillar cuts the volume exactly where it passes through it.

2D effects (portals, rings, flares)

  • Placement: evaluate the pattern on a world-anchored plane using a ray–plane intersection. The effect then gets perspective and parallax as the camera moves.
  • Facing the camera: for effects that should always face the viewer, build the plane from sarabCameraMatrix().

Anti-aliased edges

  • One pixel in world units: length(sarabGetRay(fc + vec2(1, 0)) - rd) * t. Use it to soften silhouettes.

Other SarabFX plugins in the shot

The same Place in Footage group, buttons and viewer gizmos are in Sarabography, SarabInator, SarabHyper, SarabBrush and SarabBrush3D. Connect SarabResolver (Output: Source with depth in alpha) to the plugin's Source input, and the plugin's content appears in the shot:

Plugin What stands in the shot Examples
Sarabography The text, as a flat card. Screen and Add blend modes are added as light A title on a wall that a pillar passes in front of
SarabInator The Lottie animation, as a flat card An animation inside a picture frame on a wall
SarabHyper The HTML page, as a flat card. Pages rendered on black (Alpha Output off) are added as light, so black is see-through A HUD or hologram screen floating in front of an actor
SarabBrush The strokes, as a flat card. Add and Screen strokes glow into the shot; Over and Multiply cover it Strokes drawn on a table top or on the floor
SarabBrush3D The 3D strokes themselves, in depth A path curling around a pillar like a snake
  • Where it starts: a card stands upright facing the first frame's camera. Place Plane puts it on a wall, a table or the floor. The card is the Plane Mapper rectangle: its corners set the card's size.
  • Scene Gizmo (in the group) picks what the viewer gizmo edits: Object Position, Rotation, Scale or Plane Mapper. Set it to Off to get the plugin's own gizmos back.
  • Place in Footage (checkbox) turns the placement off. Plain footage, without SarabResolver's camera, always gets the old flat overlay.
  • Occlusion: real objects nearer than the card, or than a SarabBrush3D stroke, hide it.
  • Motion: Track Point / Follow Tracked Point work the same in every one of these plugins and in SarabMesh (see Motion beyond keyframes).
  • Chains: these plugins also have Pass Scene Downstream. SarabInator, SarabHyper and SarabBrush pass their card's depth on. SarabBrush3D passes the real scene's depth and camera on unchanged.
  • SarabBrush3D: the drawing lies on the placement's X/Z plane, with Z up the drawing, like a card. The brush's Depth Spread pushes strokes away from the viewer, behind the plane. Object Scale starts at 0.15, because a drawing spans about 2 brush units. The brush camera, camera motion, environment and ground plane are ignored while the strokes are in the shot. The real camera replaces them.
  • DaVinci Resolve: the viewer gizmos appear when the viewer's overlay menu (bottom left) is set to OpenFX Overlay. They appear once the node has rendered a frame, because they are drawn with the camera that frame was rendered with.
  • SarabBrush, SarabBrush3D: draw and edit strokes with Scene Gizmo Off. The drawing overlay is in the layer's own coordinates, not the card's.

Rules and pitfalls

  • Never write the uniform names cameraPosition, cameraRotation or cameraFOV in your shader. Use the functions above: if SarabOrtex sees the names, it assumes you declared them yourself, and the shader fails to compile.
  • Don't centre coordinates with fragCoord - 0.5*iResolution. That pattern turns off the automatic Y-flip and turns the camera upside down. Use sarabGetRay.
  • Buffer passes (A–D and Cubemap A) have no camera and no footage. Anything that depends on either belongs in the Image pass. A buffer can hold work that doesn't depend on the camera, such as textures, simulations, or a feedback effect kept in the object's own coordinates. A screen-space feedback or blur will smear when the real camera pans.
  • Textures come from shared asset folders. Reference them in the preset with a path relative to the preset file.
  • Metal is strict: avoid and, or, not, this and char as names. Keep loop bounds constant (break is fine). Guard pow() on negative numbers and normalize() of zero vectors.
  • Keep it fast: bound every march to the object. A frame should render in well under 0.15 s at 1080p for smooth playback once the cache is warm.

Output

Write the whole composite with alpha 1: footage plus object plus its light and shadow. Leave SarabOrtex's Blend Mode on Replace. The camera-data pixels SarabResolver writes in the bottom-left corner are removed before your shader sees the footage.


Writing your own: Writing a shader for footage · See also: SarabOrtex · SarabResolver · All plugins