Vapoursynth Expressions

Vapoursynth expressions (expr) are a powerful feature that allows you to perform per-pixel operations using a simple expression syntax. Common use cases for expressions include:

• Arithmetic operations on pixel values (addition, subtraction, multiplication, etc.)
• Combining multiple clips through mathematical operations
• Creating masks based on pixel values or clip differences
• Simple color adjustments and corrections
• Implementing basic pixel-based filters

Expressions are much more powerful than this, and can be used for even more complex operations, such as:

• Complex color space conversions and transformations
• Creating adaptive masks based on multiple conditions
• Noise detection and filtering
• Edge detection and enhancement
• Complex mathematical operations for specialized filters
• Frame property storing and comparison using single pixel clips

There are two different implementations of expressions in Vapoursynth:

1. The standard std.Expr function from the core library
2. The optimized akarin.Expr function from the akarin plugin

Both implementations provide the same core functionality, but akarin.Expr is an optimized extension that offers enhanced performance and additional features through a slightly modified syntax.

This guide will use the akarin.Expr implementation as it is more powerful and covers all the use-cases the standard library does and more.

Postfix and Infix Notation

Basic Syntax

If you've studied any algebra, you've probably seen expressions written in infix notation, where the operator is placed between the operands: $$ x * 1.5 - 10 $$ Infix notation is easy to understand for humans, but it can be more difficult for machines to parse. That's why expressions in Vapoursynth use postfix notation, where the operator is placed after the operands: $$ x\ 1.5\ *\ 10\ - $$ This can be a bit confusing at first, but it's actually quite simple once you get the hang of it.

Stack Evaluation

Postfix notation works by using a stack, a data structure where values are pushed on top and popped off from the top. We can see how this works with a simple example: $$ 5\ 10\ 3\ -\ * $$ The expression is evaluated from left to right:

1. When we see a number, we push it onto the stack:

   1. Push \(5\): Stack is \([5]\)
   2. Push \(10\): Stack is \([5, 10]\)
   3. Push \(3\): Stack is \([5, 10, 3]\)

2. When we see an operator, we pop the required number of values off the stack, apply the operator, and push the result back.

   • See -:

     1. Pop \(3\), then \(10\) from the stack
     2. Calculate \(10 - 3\)
     3. Push \(7\): Stack is \([5, 7]\)

   • See *:

     1. Pop \(7\), then \(5\) from the stack
     2. Calculate \(5 * 7\)
     3. Push \(35\): Stack is \([35]\)

The final value on the stack (\(35\)) is our result.

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#b257e6',
      'primaryTextColor': '#ffffff',
      'primaryBorderColor': '#9333ea',
      'lineColor': '#a0aec0',
      'secondaryColor': '#2d3748',
      'tertiaryColor': '#4a2d5d'
    },
    'flowchart': {
      'curve': 'basis',
      'padding': 10,
      'rankSpacing': 30,
    }
  }
}%%

graph TD
    classDef default fill:#2d3748,stroke:#4a5568,stroke-width:1px;
    classDef operation fill:#b257e6,stroke:#9333ea,color:#ffffff,stroke-width:2px;
    classDef stack fill:#4a2d5d,stroke:#b257e6,color:#ffffff,stroke-width:2px;

    subgraph "Stack Operations"
        A["Initial Stack"]:::operation
        B["Push (5)"]:::operation
        C["Push (10)"]:::operation
        D["Push (3)"]:::operation
        E["Subtract (10 - 3)"]:::operation
        F["Multiply (5 * 7)"]:::operation
    end

    subgraph "Expression: 5 10 3 - *"
        G["Stack: []"]:::stack
        H["Stack: [5]"]:::stack
        I["Stack: [5, 10]"]:::stack
        J["Stack: [5, 10, 3]"]:::stack
        K["Stack: [5, 7]"]:::stack
        L["Stack: [35]"]:::stack
    end

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F

    G --> H
    H --> I
    I --> J
    J --> K
    K --> L

    B -.-> H
    C -.-> I
    D -.-> J
    E -.-> K
    F -.-> L

Reading Postfix Expressions

If you have trouble reading a postfix expression, JET has a simple website where you can enter an expression and hover over operators to show which values on the stack are being operated on. Using our previous expression: $$ 5\ 10\ 3\ -\ * $$ Hovering over the * and - operators shows affected values:

Plugin Syntax

To use expressions in a script, you can use the akarin.Expr plugin. The std.Expr function can also be used, but it has fewer features.

Clips

The plugins accept a list of VideoNodes, and will apply the expression to each pixel of the first given clip. Note that the plugins always return a single VideoNode that is the result of applying the expression to the first input clip, regardless of how many input clips are provided.

In standard Vapoursynth expressions, each VideoNode is represented by a single character. This starts from \(x\) and goes through the alphabet up to \(u\).

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#b257e6',
      'primaryTextColor': '#ffffff',
      'primaryBorderColor': '#9333ea',
      'lineColor': '#a0aec0',
      'secondaryColor': '#2d3748',
      'tertiaryColor': '#4a2d5d'
    },
    'flowchart': {
      'curve': 'basis',
      'padding': 8,
      'nodeSpacing': 4,
      'rankSpacing': 20
    }
  }
}%%

graph TD
    classDef node fill:#b257e6,stroke:#9333ea,color:#ffffff,stroke-width:1px;
    classDef char fill:#4a2d5d,stroke:#b257e6,color:#ffffff,stroke-width:1px;
    classDef skip fill:none,stroke:none;

    N1["N1"]:::node
    N2["N2"]:::node
    N3["N3"]:::node
    N4["N4"]:::node
    N5["N5"]:::node
    N6["N6"]:::node
    skip1["⟶"]:::skip
    N24["N24"]:::node
    N25["N25"]:::node
    N26["N26"]:::node

    C1["x"]:::char
    C2["y"]:::char
    C3["z"]:::char
    C4["a"]:::char
    C5["b"]:::char
    C6["c"]:::char
    C24["w"]:::char
    C25["v"]:::char
    C26["u"]:::char

    N1 --> C1
    N2 --> C2
    N3 --> C3
    N4 --> C4
    N5 --> C5
    N6 --> C6
    N24 --> C24
    N25 --> C25
    N26 --> C26

akarin.Expr extends the standard Vapoursynth expression syntax by adding an additional operator that makes these values more human-readable, while also allowing you to pass a theoretically arbitrary number of clips. This operator is srcN, where N is the index value.

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#b257e6',
      'primaryTextColor': '#ffffff',
      'primaryBorderColor': '#9333ea',
      'lineColor': '#a0aec0',
      'secondaryColor': '#2d3748',
      'tertiaryColor': '#4a2d5d'
    },
    'flowchart': {
      'curve': 'basis',
      'padding': 8,
      'nodeSpacing': 4,
      'rankSpacing': 20
    }
  }
}%%

graph TD
    classDef node fill:#b257e6,stroke:#9333ea,color:#ffffff,stroke-width:1px;
    classDef char fill:#4a2d5d,stroke:#b257e6,color:#ffffff,stroke-width:1px;
    classDef skip fill:none,stroke:none;

    N1["N1"]:::node
    N2["N2"]:::node
    N3["N3"]:::node
    N4["N4"]:::node
    N5["N5"]:::node
    skip1["⟶"]:::skip
    N253["N253"]:::node
    N254["N254"]:::node
    N255["N255"]:::node

    C1["src0"]:::char
    C2["src1"]:::char
    C3["src2"]:::char
    C4["src3"]:::char
    C5["src4"]:::char
    C253["src252"]:::char
    C254["src253"]:::char
    C255["src254"]:::char

    N1 --> C1
    N2 --> C2
    N3 --> C3
    N4 --> C4
    N5 --> C5
    N253 --> C253
    N254 --> C254
    N255 --> C255

Expressions

Expressions are the core of these plugins. It makes use of operators to manipulate pixel values. Below is a list of operators, small descriptions on what they do, which plugins support which, and how many values they take from the stack.

Both expression plugins accept either a single string or a list of strings as the expression argument. When passing a list, the number of expressions can match the number of planes in the first clip. For example, a YUV clip would accept up to 3 expressions: one for the Y plane, one for the U plane, and one for the V plane.

If fewer expressions are provided than planes, the last expression in the list will be used for any remaining planes. When passing just a single expression string, that expression will be applied to all planes.

1

clip = akarin.Expr([clip1, clip2], "x 1 +")

This will apply the expression x 1 + to every plane of the first input clip. This is equivalent to the following:

1

clip = akarin.Expr([clip1, clip2], ["x 1 +", "x 1 +", "x 1 +"])

If we want to only apply an expression to a single plane, we can pass empty strings. For example, if we only want to apply an expression to the U plane, we can pass the following:

1

clip = akarin.Expr([clip1, clip2], ["", "x 1 +", ""])

When passing values in an expression, you should be aware of the input and output formats. While integer formats automatically clamp values to their valid ranges, floating point formats do not. This means that for float clips, you must manually clamp values to avoid invalid results.

For float clips, there are two ways to clamp values in an expression:

1. Using min and max operators
2. Using the clamp (or clip) operator (akarin.Expr only)

In akarin.Expr, the following are equivalent:

• 'x 0.0 max 1.0 min'
• 'x 0.0 1.0 clamp'

If necessary, you can also define custom clamp ranges for each plane.

• ['x 0.0 max 1.0 min', 'x -0.5 max 0.5 min']
• ['x 0.0 1.0 clamp', 'x -0.5 0.5 clamp']

Similarly, you must be aware of the formats of the input clips when writing expressions. If there's an unaccounted format mismatch, the expression may produce garbage data.

Simple Expressions

There are a couple of simple expressions that are commonly used.

Making diffs

Expressions can be used to make a diff between clips, just like the std.MakeDiff/MergeDiff functions.

1

core.akarin.Expr([clip1, clip2], "x y = 0 255 ?")

The current expression catches a lot of noise (due to dithering applied during filtering and encoding). If we want to add a threshold, we can make the following changes:

1. First, we define a threshold value (thr) that determines how much difference we allow between pixels
2. Instead of checking for exact equality between pixels, we:
   • Subtract one pixel value from the other (x y -)
   • Take the absolute value of that difference (abs) to handle both positive and negative differences
3. We compare the absolute difference against our threshold (<= thr)
   • If the difference is less than or equal to the threshold, the pixels are considered "similar enough"
   • If it's greater than the threshold, the pixels are considered different

1
2

thr = 24
core.akarin.Expr([clip1, clip2], f"x y - abs {thr} <= 0 255 ?")

Complex Expressions

Expressions give you a lot of power and freedom to manipulate pixels however you want.

Range Compression and Decompression

Expressions can be used to handle range compression issues in video clips, serving as an alternative to functions like std.Levels. Range compression occurs when a clip's pixel values are incorrectly scaled between limited range and full range.

For example, when a limited range clip is incorrectly interpreted as full range, its values get compressed into a smaller range, resulting in loss of contrast. While std.Levels is commonly used to fix this, we can achieve the same results using expressions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from vstools import vs, get_peak_values, get_lowest_values, ColorRange


# Get full range peak and lowest values for Y and UV planes
peak_full_y, peak_full_uv, _ = get_peak_values(clip, range_in=ColorRange.FULL)
lowest_full_y, lowest_full_uv, _ = get_lowest_values(clip, range_in=ColorRange.FULL)

# Get limited range peak and lowest values for Y and UV planes
peak_limited_y, peak_limited_uv, _ = get_peak_values(clip, range_in=ColorRange.LIMITED)
lowest_limited_y, lowest_limited_uv, _ = get_lowest_values(clip, range_in=ColorRange.LIMITED)

# Apply range scaling formula to convert from limited to full range
# Formula: (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min
clip.akarin.Expr(
    [
        f"x {lowest_limited_y} - {peak_full_y - lowest_full_y} * {peak_limited_y - lowest_limited_y} / {lowest_full_y} +",
        f"x {lowest_limited_uv} - {peak_full_uv - lowest_full_uv} * {peak_limited_uv - lowest_limited_uv} / {lowest_full_uv} +",
    ]
)

The inverse of this can be done by swapping the peak and lowest values, as well as swapping full and limited in the formula.

1
2
3
4
5
6

clip.akarin.Expr(
    [
        f"x {lowest_full_y} - {peak_limited_y - lowest_limited_y} * {peak_full_y - lowest_full_y} / {lowest_limited_y} +",
        f"x {lowest_full_uv} - {peak_limited_uv - lowest_limited_uv} * {peak_full_uv - lowest_full_uv} / {lowest_limited_uv} +",
    ]
)