Asset management and Pipeline

Here is a quick report/summary of work and discussions about Data-block handling, Assets management and Pipeline, that happened at the Blender Institute during my four days stay in Amsterdam, in September 2016.

Reconfirmed Principles

Maybe the most important point is the clear separation between low-level tools and APIs to allow data-blocks and assets management, and the high-level and complex setup that is a production pipeline. Blender itself should not enforce any type of pipeline or asset management, it must only provide tools to make them possible. The “Asset Engine” idea springs from this principle, quite similar to the “render engine” system used integrate external renderers into Blender.

The second point is that we want to keep the linking system already existing in Blender as basis for the new features, this ensures us both a degree of backward and forward compatibility, and excellent performances. However, this implies we keep the .blend file format at the core of the system.

Current Status

Claude Cloud Asset Engine in action.

Claude Cloud Asset Engine in action.

Past year has been mostly spent on fixing and enhancing how Blender handles its data-blocks. Most of this work as been merged in master now, and is part of 2.78 release.

The asset engine system/API is also mostly finalized, and has two testing/demo engines now:

  • Amber, designed as a simple local asset repository (asset meta-data being stored in a JSON file along with library .blend files).
  • Claude, mimicking the Cloud Add-on, to allow browsing and importing textures from the Blender Cloud on-line repository.

Roadmap and TODO’s

Main goal in coming months is to finalize Asset Engine API, and as a side track smaller goal, enhance our ‘pose library’ feature.

Asset Engine API is now for the most part well defined, and (limited) tests have proved it to be working so far. What remains to do here is mostly implement last bits of it, and a lot of UI/UX design and code work (currently it does not expose much more than basic linking to users, access to things like variants and revisions are not yet possible e.g.).

The Sub-Data Issue

One big topic that is still somewhat open is how to handle non-data-blocks data (good example: texture image files). So far, asset-engine branch handles them by:

  • Defining a ‘virtual library’ data-block (one per asset engine repository).
  • When loading image file, it creates a ‘faked’ linked Image data-block, using relevant virtual library.

This is how things should work in a more generic way, but it poses a problem: linked data-blocks cannot be edited at all, which, in case of image files e.g., means you cannot edit any setting of the Image data-block, and have to go with the default values.

Current asset-engine branch works around this with a hack – it detects those faked-linked data-blocks and allows their edition. Further more, it also fully saves them in .blend file (instead of only saving a ‘reference’ as usually done for linked data).

However, this is not an acceptable solution for final code, here we’ll need the ‘override’ feature, which would allow us to define a purely linked Image data-block (still using the virtual library), and then add a local overriding Image data-block that would allow edition of everything but the image file path.

Pose Library Enhancement

As a smaller, more close-goal project, we want to enhance Pose Library system in Blender, with both some general UI/UX improvement, and by adding a preview system for each pose.

This can also be used as some occasion to investigate and experiment things that should be used more widely in final Asset Management feature (that is, easy and friendly access to data, with icons, drag’n’drop, etc.).

More detailed plan can be found about this project on Sybren’s page.

4 comments 11,727 Views
  1. It would be great to allow local overrides for all linked datablocks. It would allow to easily create variations of a base data while keeping the advantages of linked data (getting updates, lower file size, etc…).
    Is there a way to test amber?

    • Overrides sure aim at that – but this is slightly different area (also involves depsgraph, etc.).

      As for testing Amber… you may build asset-engine branch, you can also find at http://download.blender.org/ftp/mont29/amber/ a small demo archive of amber repo (you need to uncompress it first). But you’ll have to create the JSON repository index all by hand for now, so would not call this really usable yet 😉

  2. Please simplify linking rigged meshes and the need to make proxy rig. I should be able to link one character multiple times. Also when the rig changes I don’t want to re-create the proxy and assign action to it, loosing all constraints animators added to the old proxy (child_ofs,..). This is the biggest pain in the pipeline and the whole system could be done so much more better.

  3. i love what john said. thanks john, and thanks for post and working on something, which we all will be using in future.

  1. Leave a Reply

    Your email address will not be published. Required fields are marked *

     

share this story on