Asset management and Pipeline

Here is a quick report/summary of work and discussions about Data-block handling, Assets management and Pipeline, that happened at the Blender Institute during my four days stay in Amsterdam, in September 2016.

Reconfirmed Principles

Maybe the most important point is the clear separation between low-level tools and APIs to allow data-blocks and assets management, and the high-level and complex setup that is a production pipeline. Blender itself should not enforce any type of pipeline or asset management, it must only provide tools to make them possible. The “Asset Engine” idea springs from this principle, quite similar to the “render engine” system used integrate external renderers into Blender.

The second point is that we want to keep the linking system already existing in Blender as basis for the new features, this ensures us both a degree of backward and forward compatibility, and excellent performances. However, this implies we keep the .blend file format at the core of the system.

Current Status

Claude Cloud Asset Engine in action.

Claude Cloud Asset Engine in action.

Past year has been mostly spent on fixing and enhancing how Blender handles its data-blocks. Most of this work as been merged in master now, and is part of 2.78 release.

The asset engine system/API is also mostly finalized, and has two testing/demo engines now:

  • Amber, designed as a simple local asset repository (asset meta-data being stored in a JSON file along with library .blend files).
  • Claude, mimicking the Cloud Add-on, to allow browsing and importing textures from the Blender Cloud on-line repository.

Roadmap and TODO’s

Main goal in coming months is to finalize Asset Engine API, and as a side track smaller goal, enhance our ‘pose library’ feature.

Asset Engine API is now for the most part well defined, and (limited) tests have proved it to be working so far. What remains to do here is mostly implement last bits of it, and a lot of UI/UX design and code work (currently it does not expose much more than basic linking to users, access to things like variants and revisions are not yet possible e.g.).

The Sub-Data Issue

One big topic that is still somewhat open is how to handle non-data-blocks data (good example: texture image files). So far, asset-engine branch handles them by:

  • Defining a ‘virtual library’ data-block (one per asset engine repository).
  • When loading image file, it creates a ‘faked’ linked Image data-block, using relevant virtual library.

This is how things should work in a more generic way, but it poses a problem: linked data-blocks cannot be edited at all, which, in case of image files e.g., means you cannot edit any setting of the Image data-block, and have to go with the default values.

Current asset-engine branch works around this with a hack – it detects those faked-linked data-blocks and allows their edition. Further more, it also fully saves them in .blend file (instead of only saving a ‘reference’ as usually done for linked data).

However, this is not an acceptable solution for final code, here we’ll need the ‘override’ feature, which would allow us to define a purely linked Image data-block (still using the virtual library), and then add a local overriding Image data-block that would allow edition of everything but the image file path.

Pose Library Enhancement

As a smaller, more close-goal project, we want to enhance Pose Library system in Blender, with both some general UI/UX improvement, and by adding a preview system for each pose.

This can also be used as some occasion to investigate and experiment things that should be used more widely in final Asset Management feature (that is, easy and friendly access to data, with icons, drag’n’drop, etc.).

More detailed plan can be found about this project on Sybren’s page.

  1. Schema5ic overviewv would be really great for this kinda thing. I wonder if stealing from OpenToonz might help make this possible.

    We already have the nodes feature written. Setting scenes up through data block nodes could be super handy.

    Maybe animation nodes could help pipe this one out?

  2. Will the new asset engine show imported videos? If so, than the VSE will also really benefit from it as the engine could be used as a media pool (like other editors). This is especially true if the asset engine supports bins.

  3. I come to Blender from Daz Studio and iClone where character “presets” (partial instance actors sharing rig), UV/material presets, and pose presets and animation clip previews are not mere afterthought. Asset management is big deal, for Blender to progress from a DCC tool to a more efficient character animation suite. Glad to see Blender devs are working on Assets preview and Pose Library…

    Please see how Daz Studio plugin Animate2 (2010 gen app) allows real time preview of animation right on thumbnails – prior to loading into scene/ onto selected character. If rendering high poly characters in action is too performance intensive, then I’ll settle for just 3K standard cartoon proxy actions.

    Also see Daz’s Puppeteer panel used to quickly load, audition, or store multiple poses and RESIZABLE thumbnails.

    Thanks for making life easier for riggers and animators! Keep up the already awesome works!

  4. This Unreal 4 workflow would be AWESOME to make part of it :

    I got amazed to see how easy it was to distribute the assets..

  5. i love what john said. thanks john, and thanks for post and working on something, which we all will be using in future.

  6. Please simplify linking rigged meshes and the need to make proxy rig. I should be able to link one character multiple times. Also when the rig changes I don’t want to re-create the proxy and assign action to it, loosing all constraints animators added to the old proxy (child_ofs,..). This is the biggest pain in the pipeline and the whole system could be done so much more better.

  7. It would be great to allow local overrides for all linked datablocks. It would allow to easily create variations of a base data while keeping the advantages of linked data (getting updates, lower file size, etc…).
    Is there a way to test amber?

    • Overrides sure aim at that – but this is slightly different area (also involves depsgraph, etc.).

      As for testing Amber… you may build asset-engine branch, you can also find at a small demo archive of amber repo (you need to uncompress it first). But you’ll have to create the JSON repository index all by hand for now, so would not call this really usable yet ;)

In order to prevent spam, comments are closed 15 days after the post is published.
Feel free to continue the conversation on the forums.