As part of the four 2022 Blender strategic targets, a workshop to help refine the Overrides topic was held during the week of the 17th of January at the Blender HQ in Amsterdam. The general idea behind overrides is to enable artists to edit and animate their assets, while still keeping them in sync with the library files content.
The current Library Overrides (as in Blender 3.0/3.1) were reviewed, and several possible developments, both in Library Overrides and in new areas, were investigated. Some indirectly related topics were also discussed.
This blog post is a summary of the topics investigated during the workshop week. Some are very well defined and can be implemented immediately, others will require further design work, and only a subset of those tasks and ideas will be addressed this year.
Library Overrides: Introduction
In the original library file the data-blocks are local and can be edited directly.
The different datablocks are related by dependencies. In this example the collection depends on an object that it contains, which depends on a mesh.
In the set file the data-blocks are linked from the library file.
Data-blocks cannot just be edited and are bound by their dependencies to each other.
To change the data of the object a user override is created (e.g. to move the object around). To be able to create this override, however, the collection that depends on this object needs to be overridden as well.
As this is only necessary to access the object inside, this is applied as a system override. The mesh data-block is not overridden in this case, as it does not depend on any overridden data.
The overridden data from the set file is then linked into the shot file. In this file none of the datablocks are overridden, as they are directly linked. However, the object that hase been changed with an override in the set file does keep the changes it received in the set file.
In the shot file the data is then overridden again, now the mesh data is changed (e.g. to activate a shapekey). To be able to access the mesh data, system overrides are created for the object and the collection and a user override is created for the mesh.
The changes that were made in the set file are still intact and the changes that are made locally are added on top.
Current State
Overall the feedback – mainly from the Blender Studio artists team – over the current library overrides was very positive.
However, several critical pain points and potential improvements were identified, such as performance during resync.
Resync Performance
Performance of Library Overrides is generally good, except when the linked library data is modified. This often requires a resync of the overrides using the modified data, and this process can take a lot of time in the current Blender (3.0), depending on the complexity of the file.
This is a matter of optimizations and implementing known pending TODO’s in the current codebase. It is already being worked on, Blender 3.1 will ship with some improvements in that regard, and much more speed-up is expected in the coming weeks.
Reliability
Library overrides are not only about single, isolated data-blocks. In almost all use cases, a group of data-blocks need to be overridden together.
For example, in a character, there will be overrides of its root collection and sub-collections, the rig object(s), and most of the geometry objects. Those groups are called “override hierarchies”, as there is typically a single “root” (the main collection of a character for example), and a tree of dependencies.
Keeping this hierarchy information in a clean state is crucial, especially for the resync process required when the linked data changes. The current Blender (3.0) is not doing a great job at that. Furthermore, the current implementation does not properly support relationships between different hierarchies (like parenting a library override of a character to another one). There are several ways to improve this situation.
Reduce Possibilities for Wrong Manipulations
There are many minor tweaks to be done in existing operators to reduce the likeliness of wrong operations that will break override hierarchies.
For example, in the Outliner, it should not be possible to easily delete the root collection of a library override, while deleting its whole hierarchy should be allowed.
Stronger Definition of Override Hierarchies
A stronger definition of library override hierarchies at the code/data level will also help on several aspects:
- Simplify managing code
- Help detecting conflicts
- Help handling those broken situations in a better way
- Add support for relationships between different hierarchies (for example parenting an overridden character to another).
NOTE: Technically, in each library override data-block, a pointer to the root of the hierarchy it belongs to will be added.
Reduce Visibility of System Overrides
“System” library overrides are data-blocks (and their relationships) that are created purely for technical reasons under the hood, and are not supposed to be editable by the user. A good example of this is the tree of collections in the override of a character: they are required for the system to work, but they are not needed from the user’s perspective.
By preventing modifications of those system overrides (by default), and not showing them so prominently in the user interface (for example with a filter toggled off by default), both robustness and usability of the system will be increased.
Visualization & Management
The Outliner view of Library Overrides is currently very limited. It needs improvements on both the visualization and the management tools aspect.
Besides reducing the visibility of “system overrides” as discussed in the previous section, the Override view of the Outliner should be divided in two modes, one dedicated to the visualization and handling of override hierarchies, and the other as a flat view listing all overrides with their overridden properties. These properties should be editable when possible, to become the one place to see and edit all overridden properties at once.
USD Compatibility
USD (Universal Scene Description) was not discussed during the workshop, but it is definitely an important topic to keep in mind.
Last year, in February, Jeroen Bakker did a short experiment on mapping USD and Blender overrides, which was fairly successful. This can be further investigated in collaboration with the team implementing USD support in Blender.
Restrictive Library Overrides
An alternative way of using Library Overrides is by letting the user specify what can be overridden in the source library.
The main goal of the restrictive workflow is to encourage collaboration between the artists producing an asset, and the artists using this asset.
How to achieve this goal is still under discussion. One of the main challenges of restrictive overrides is to provide an easy and comprehensive user interface and experience.
Extending Library Overrides Support
Materials & Shading
While being able to edit materials, shading and lighting of linked data is an important feature, it appears that most of the use cases could also be covered using Dynamic Overrides. The main case that requires Library Overrides is the re-assignment of materials.
Furthermore, supporting this in Library Overrides requires a way to select and expose the key parameters of a material, since allowing the editing of all nodes’ parameters in an override of a node tree is not a realistic scenario due to its complexity.
Geometry
Being able to tweak the shape of a character is an important feature for animators.
The natural solution would be to support adding new Shape Keys in Library Overrides. However, Shape Keys have several issues currently in Blender, on both usability and architectural levels which would need to be addressed first, before adding yet another layer of complexity.
Alternative solutions also need to be investigated, like using Geometry Nodes or sculpting tools.
Geometry Data Layers Addition
Adding new data layers to overrides of meshes would open the doors for flexible and powerful pipelines. For example, an asset creation pipeline could be split into different steps, each using overrides from the previous step while adding new extra data layers.
This example could go like so:
- A modeler creates the geometry.
- A rigger adds new vertex groups.
- A texture artist adds new UV maps and Vertex Color layers.
Supporting this feature will require quite some technical and design work.
Dynamic Overrides
While generally based on the same idea, Dynamic Overrides are a fairly different approach than Library Overrides. Here is a list of the key differences:
Library Overrides
- Work only on linked data
- Allows modifying ID relationships
- Allows adding complex entities like Modifiers or Constraints
- Are handled on file save/load
- Are stored in the ID they affect
- Directly linkable
Dynamic Overrides
- Work on any data, linked or local
- No ID relationships changes
- Can only modify simple ‘value’ properties
- Are handled as part of evaluation
- Are stored in a different ID
- Linkable only if the ID owning them is linked
Conceptually, Dynamic Overrides are also quite similar to animation data, the main difference being that they are stored in a different data-block than the one they affect.
Being evaluated as part of the depsgraph, after the handling of animation, also implies that Dynamic Overrides:
- Can be animated, but…
- Will supersede any animations or drivers defined on the properties they affect.
Dynamic Overrides will be stored in Scene data-blocks, on two levels:
- A global level, applying the overrides to all view layers in the scene.
- A per-view layer level, replacing the global one when defined.
A Dynamic Override contains the data-block it affects, an RNA path leading to the affected property in that data-block, and the value of the override.
Further Required Design Topics
Finding out how critical it is to be able to link or export/import those dynamic overrides, and how this could be done, needs to be further investigated.
Another idea to explore is to find out how much Library Overrides and Dynamic Overrides could be presented as a single override interface to the user, at least on a high level of usability.
Side Topics
While investigating overrides, other somewhat-related topics were also discussed.
Data-Blocks Relationships Visualization
While the Outliner provides a way to visualize and manage data in .blend files, its tree nature has some limitations.
Several years ago, in the 2.3x/2.4x series, Blender used to have a graph representation of its data-blocks and their relationships, called the Oops (Object Oriented Programming Schema).
The need for this type of visualization is still there, although in very complex files a simple Oops (or DOT graph) quickly becomes confusing and hardly useful.
An idea could be to implement a dedicated view/tree type in the Node Editor, where each node would be a data-block. The sockets and edges would then be their relationships.
To keep a better control over visualization, node groups could be used to represent some level of hierarchy (e.g. top node group for a scene, sub-node groups for collections, etc.). This could help both keep the graph cleaner, and help identify unexpected connections between data-blocks from unrelated groups.
Such a visualization/management tool would help to manage Library Overrides.
File Loading Performance
While file loading speed became a big issue with Library Overrides, even without the resync process, loading a full production file can take several tens of seconds.
Making this process non-blocking by showing any available data as fast as possible, would improve the feeling of responsiveness and reduce user frustration.
This is not a trivial task, it has several technical challenges touching many areas of the codebase.
What’s Next
Design and development tasks are coordinated in the Overrides project on developer.blender.org.
Performance and reliability topics have already been investigated at the end of last year, and are being worked on. Great improvements on that front are expected in Blender 3.1 (to be released early March), and even more in Blender 3.2 (June).
In February, further planning will be done with the development team as well as some design work on the remaining targets. The usability topic could be the target of a first development sprint, including tools and UI improvements. Later this year a second sprint could cover the initial dynamic overrides implementation.
This Library Override system very helps in my daily tasks. In the simple scenes I use the Material Override approach with one master material. But when the complexity increase I use linked collections where each object has an unique Object Index and an unique Attribute (Geometry Nodes helps me apply this integer factor to the Face Cornel domain). This way helps to keep the principles of encapsulation when only a master file has materials. When I run the master file at the first time the python script goes trough all objects in linked collections and apply material using index comparing (for example: OB Index 1 = Material 1). Thanks to Library Override system my project splits on the directory level and thanks to Geometry Nodes I use attributes as mix factors in the shader editor bypassing the standard options like material indexes or slots. Thank Blender Developers for the work done!
Does this help to improve performance when dealing with large rigged assets?
I do visualization of mechanical assemblies, and many my rigged assets consist of several thousand objects. And since Blender 2.93, if I remember correctly, re-opening a file with multiple rigs linked in can take over an hour to open. Previously the Proxy system was capable of handling this fine when linking in the entire asset, and just making a proxy out of the rig. But now that it seems like Blender is overriding all objects instead of just the rig, it just about kills my machine.
While that kind of use case (thousands of Objects) is not fully supported in Blender (in general, not only for library overrides), the current master (3.2) builds should heavily improve the situation already, yes.
That’s great! Looking forwards to an overrides system. I strongly believe that these are the steps towards building a core which will help with fully implementing a functioning USD importer/ exporter
Will we be able to override materials and other data blocks per view layer?
I was about to ask the same question. For me it’s super important. Also, the ability to override render settings. An occlusion render layer may not necessarily need as many samples as a beauty shot.
Right, I never really understood how that wasn’t a thing before -overriding materials seems to me one of the main use cases of render layers. I’m very happy it is part of the list
It’s on the roadmap as stated in the blog post.
For the time being, the add-on Amaranth (that ships with Blender) has this functionality. In the Sampling panel (Cycles) you can specify the samples per ViewLayer. I made this nearly 10 years ago but I just tested it now in master and still works.
Why no properly implement USD the way Houdini, Maya and Katana are doing right now?
This would create an ecosystem of tools that we could actually use.
None of the commercial software is perfect or free, so Blender would be a great addition to them in a USD pipeline (effectively every sane and manageable pipeline).
Right now, Blender is unusable because its USD export features are horrible and wrong, so what’s the point of creating data in Blender that can’t be used downstream? Also there is nothing else in Blender regarding USD.
The developer guys should just state the obvious: They’re making Blender for their own pipelines and don’t respect anything else used by the industry, which makes the financial backing by the industry rather stupid.
Blender should change its priorities if it wants to offer a usable tool to the industry.
Agree.
You should implement true USD management and do a PROPER SOLARIS-LIKE SOLUTION.
Blender (2022) still can’t share shaders and assign them procedurally in an easy way, still manages render settings worse than Maya back in 2008, still talk about everything nodes but forget the nodal rendering managment.
Blender is great and super modern in some aspects and super-duper obsolete in some other aspects that are actually more important. The sense of priority is a bit off with you guys.
But it’s good to see that part of the issue is being tackled at least.
Overrides are great, when they work, but they are exacerbating a more basic interoperability problem. As Blender grows ever larger and more popular it is beginning to become a parallel universe. The continuing use of a constantly evolving .blend file format and clunky support for import/export from competing packages is starting to build walls around the ecosystem. It is not realistic for one organisation to develop the world’s best tools for all the things Blender does, but the closed pipeline, rapid evolution and “free” status tends to kill cooperation.
There is an opportunity to change that, but it involves rebuilding the Blender core to become a launcher/linker/importer/exporter and recognising that the tools such as the 3D editor, renderer, and compositor could be completely separate applications. This would allow the organisation to become the guardians of industry interface standards whilst providing space for other players.
How about an “open” movie to demonstrate a mixed pipeline integrating paid-for apps? That might open a few eyes.
Having mesh data editable/overridable right there in the animation scene would be invaluable. This is the biggest takeaway for me. A new “OOPS” view would be nice too, for troubleshooting rig behaviour for instance.
I begged for the return of some sort of OOPS schematic ever since it was removed a long time ago.
Coming from Softimage and its schematic view it helped a lot to visualize and debug the relationships of datablocks etc.
Outstanding blog post, I am captivated by the direction the Library override project is going.
On one note, I think this topic relates to a “Freeze Geometry” node concept.
– When you freeze the geo node graph and “override” some attributes (the changes would be stored in the Freeze Node data).
I don’t have a specific implementation\proposal though IMAHO having that in mind would benefit the design of both.
This is likewise linked to improved Shapekeys and attributes editing.
Excited to see all the work being done on overrides, especially with USD compatibility in mind.
Really looking forward to some work being done on shape keys as well. Having live shape keys preferably as a modifier would be really helpful for shotsculpting purposes as well which seem to be planned in the sculpt module.
Please get material overrides working with the current system before leaving it to some future potential design-level refactor; they’re the last barrier to the prospect of using multiple instances of the same collection (character rig) in the same scene. Just being able to change the material assigned to a slot on a linked mesh, without regard to adding or removing material slots themselves, would be sufficient.
USD will solve all the problems above
use open source industry standards
Why when there is a industry standard?
Its called USD and its open source and all the above Problems are solved :)
No, iam not talking about the simple importer/exporter , use the full power and make a full integration.
https://graphics.pixar.com/usd/release/index.html
You’re right though, USD was built exactly for this. However I understand this implies remaking all of Blender’s scene structure.
No, it doesn’t require remaking Blender’s scene structure.
The way Maya is implementing USD right now is that you can load a hugely complex USD scene with hundreds and thousands of instances, references, materials, textures etc. and all displayed similar to an efficient cache. And if you want to edit for example a rigged character, you can pull just the Maya scene file that contains the rigged character and tweak the animation and publish back to USD in your department layer.
https://area.autodesk.com/blogs/the-maya-blog/maya-usd/
What Blender would need to do is to primarily create a Hydra viewport that displays normal Blender and USD data together and a series of push and pull tools.
That is correct Hadrien. For USD to be properly integrated into Blender, Blender firsts needs a core overrides system. Once that is there, USD data model can be glued to Blender’s data model.
Thank you developers for taking steps to improve Blender’s core. I am looking forward to the developments in 2022 and beyond.
I also think USD is the way to go. It is made for overrides natively.
The dynamic overrides approach seems close to this, and I think it could be used for nodal render scene description ( a la solaris or katana)
In order to prevent spam, comments are closed 15 days after the post is published. Feel free to continue the conversation on the forums.