Motion Tracking Workshop April 2022 Report

A few weeks ago (8-10 April), to spontaneously celebrate the 10th anniversary of the Tears of Steel project, a mini-workshop dedicated to VFX-related topics was organised at the Blender HQ. During this workshop me and Sebastian König went over the accumulated backlog of tasks and ideas, and came up with an updated roadmap for the VFX module. This is a summary of the results of this workshop.

Sergey Sharybin

Mask editor readability patches

We kick-started the workshop by going over open patches which got stalled due to lack of clarity in the design. There is a set of patches done by Simon Lenz:

• D13317 Mask editor: add mask spline width slider
• D13314 Mask Editor: Add toggle for mask spline drawing
• D13284 Add mask blending factor to mask editor
• D12776 ClipMasks: add color spline outline mode

The main challenge was to define what should be configurable on per-file basis, and what should be part of a theme. With the feedback of Sebastian, and by checking on test footage, it turned out that per-file configuration is the better solution. This is due to the different types of footage and ways of color-coding splines (green screen footage, blue screen footage, regular footage, a single character in the shot, multiple characters in the shot, etc). So it is not really different to how track colors or viewport color for objects is used.

The agreed design is that the system will automatically suggest automatic not-very-similar color, which then can be overwritten by the artist.

Proportional editing

This is an old open patch (D2771). Proportional editing is designed to smooth out jumps when track slides off from its feature and an artist needs manually re-position it. Currently, adjusting such discontinuities can cause further jumps in the camera solution.

Unlike other cases where proportional editing happens in 2D or 3D space, in motion tracking we need to do proportional editing in the time domain. This poses the interesting challenge of finding a good way to visualise the area of effect.

The solution we agreed on is to use a color overlay for the track path: a solid color at the center of the path, which fades out to default red/blue splines as an effect of proportional editing. Using cache line for visualizing affected frames was also discussed. It does sound like an interesting idea to explore deeper, although with low priority. The issue with the cache line is that it is almost out of sight when artists work on an individual tracks.

Last but not least: there are many ways to transform tracks. This can be done with a preview widget, or with a click-slide operator. Both ways of interaction use different code paths from the general transform system. On a user level they would behave same as regular transform, but under the hood it we need to disentangle some code and share as much of it as possible.

Migrate Movie Clip Editor to new tool system

During 2.8 days, a migration of the Clip Editor to the tool system was started (D10198). The challenge here is that motion tracking tools works different from generic 3D modeling tools.

I did some improvements and cleanup in the tracking_tools branch before the workshop so now the existing tools works nicely (unfortunately, operation with left mouse select configuration is broken, temporarily!). During the workshop we went over the current state of the project, and realised the follwing:

• Pie menus are very powerful in the clip editor!
• We’re much closer to be able to switch to the tool system than it seemed

We agreed that as a first step only a handful of tools will exist: add/slide marker in tracking mode, and spline operation tools in masking mode. The rest of the tacking tools will be accessible via the header, menu, and a pie-menu (depending on how frequently those tools are used). This will de-clutter the interface and make space for further tools.

An example of such new tool is the Orientation tool. But more about it later, as it might not be coming as an initial “toolification” of the Clip Editor.

For the Masking Mode a better add primitive tool is possible:

• For adding a circle, using click-drag will define the circle center and its radius.
• Similarly for rectangle: define 2 corners of the rectangle.

General improvements and features

When it comes to improvements, there are a few small things which will improve quality of life. For example, implement double-click to close masks, or introduce a fix for the single point handle which currently is 90 degrees out of phase when tweaking it with a mouse. But there are also bigger new features to be implemented.

Tracking: guided tracks. The idea is to use the “flow” of the already-tracked features to improve robustness of motion tracker. In shaky (hand-held) shots Kalman-filter based prediction might fail, and using information from adjacent tracks can give more reliable information on where to look for a feature on the next frame.

This will allow the following workflow: On a shaky shot, creating a first good track will still be challenging for artists, requiring a lot of manual correction. Adding more tracks will become easier as the tracker algorithm will have more information to work with.

Tracking: Scene orientation tool. The idea is to offer a more interactive experience which will involve an interactive visualization of floor/walls and allow easier tweaking of scale and possible offset from reconstructed track position (i.e. there might not be a good track where the scene origin is desired to be). The scale, orientation, and such will be able to be tweaked via widgets.

In order to get a good experience of scene orientation, extra smartness will be added to the solver: the goal is to preserve scene orientation as good as possible when scene is re-solved after manual orientation is performed.

Tracking: dopesheet. Default to sort by average error, with highest error on top.

Tracking: “bake” plane track. Allow the creation of an image datablock from what the plane track “sees”. This will enable a much easier workflow of cleaning up billboards (or walls, or screens, or anything what somewhat represents flat rectangular shape in the real world :)

Masking: mask direction should not matter. Not sure there is much more to be said here. It is annoying for artists to worry about winding of masking points. We need to do something about it :)

Masking: feather inside. Current masking design is that feather can only be added to the outside. For some shots it works fine, for others it adds a lot of planning to be done before artist starts to define mask. Being able to define outer spline of an object and feather inside will bring joy to artists! In technical terms it is something like inverting meaning of mask spline and feather spline when passing to the rasterizer.

Masking: feather offset. Currently feather fade-out always begin at the mask spline. But it doesn’t have to! The idea is to allow a “dual-band” feather control: by default fade-out will be defined by the mask spline and feather, but it will be possible to “offset” feather relative to the mask.

Masking: transform cage. A tool to allow transform masking points using a cage, similar to how it is done for edit mode for objects.

Masking: inset tool. Shrink or expand mask by offsetting points from their original position in the direction of normal.

Masking: layer sets. Currently the active mask layer only defines order of drawing. Editing outside masks from non-active layer is still possible by default and it is only possible by per-layer Lock control. This leads to cumbersome workflow when multiple layers are used for an object. While the final design still needs polishing, the current idea behind it is to allow multiple selection in the mask layer list and disallow editing anything that does not belong to any of the selected layers.

Masking: An improvement is needed for parenting a mask to a rotating object. Currently a plane track is the only way to do so in an automated manner, but in cases like face masking it is hard to find 4 points to solve a plane.

Compositor: active scene clip. The idea is to pre-fill all motion-tracking related nodes with an active movie clip from scene. Small feature, but saves some clicks.

VFX workspace: add Image editor. The workspace does need image editor, especially after addition of plane track backing.

Green button

We’ve discussed how to move forward with the “green button” project: implementing a single button which takes care of tracking and solving the entire shot. This is ideal when dealing with long shots with many features appearing and disappearing. This project is more about math, rather than UI/UX. The latter one is simple: just a single button in the interface, right? :)

Digital sets

This is an exciting topic! It is a very interesting collaboration of various projects which are already happening or are planned to happen within Blender.

Depending on who you ask, a digital set might imply different things, but the common ground is that it is a technique to simplify/streamline the VFX process. In practice it could either be a special setup which takes care of keying and compositing live action footage with CG content. More exiting aspect of digital set will imply extending the live action set with a CG content via a projection of some sort (projector or a large screen).

In any case, it is essential to acquire or solve actual camera motion, and do compositing tricks. We’ve looked into different ways of doing so, and the most promising from both ease of setup and accuracy of solver is to use VR trackers (or even controllers) attached to the real camera. The alternative could be to use lidar data and video stream from a phone, and solve everything manually, but this has feedback loop issues with digital set projector, and is also prone to solver stability issues. Leveraging VR technology has the advantage of using an existing rock-solid technology, and, drumroll, allows to put DP into the VR world!

This is already kind of possible, but the setup is tricky and there is room for streamlining and making it easier to setup. The project consists of the following steps:

• Streamline controllers and VR trackers setup in Blender XR
• Implement XR data “capture” so that it is possible to re-create it exactly as a post-process (i.e. improve preview quality with the most realistic rendering which can not happen at realtime)
• Similar thing would need to happen for capturing video stream from camera
• A real-time compositing (keying node) is needed. This seems to be aligned with the realtime compositing project
• Some sort of viewport split is needed: output compositing setup for a director’s monitor, but also output separate viewport which will be used for digital set projection

This is just a quick overview. A more clear design presentation deserve its own dedicated blog post.

Closing words

It was very exciting to have such a workshop! We will surely be sharing more details as we get into each of the projects.

Here is a preview of the upcoming tracking UI, tested during the workshop. Enjoy!