Assets – FileBrowser Preliminary Work – Experimental Build II

So, as some of you may know already, since December 2014 and my three weeks spent in Amsterdam at BI, I’ve started working on the asset topic.

An ‘Append’ file browser with two blender files showing all their materials, objects and textures. Note the renamed bookmark on the left too.

So far, I did not do anything really directly related to assets (except early designing) – rather, I’ve been improving the editor I intend to use later for assets handling, i.e. the FileBrowser. I have already ported some cleanup/refactor and some minor features (like search field in header, operator to enforce mat/tex/etc. previews generation, …) in master, but most work is still in the dedicated ‘assets-experiments’ git branch, so I’ve made some experimental builds to (try to ;) ) get a bit of testing. In those builds you’ll find:

  • Bookmarks and co using UIlists, with possibility to rename and reorganize bookmarks, plus the default features of UILists (filtering by name, and some sorting).
  • Possibility to list the whole content of a blend file (in append/link mode) at once (set ‘Recursion Level’ setting to 1), and in any mode, to list several levels of the directory tree in a “flat” mode (including blend files content if relevant, set ‘Recursion Level’ to 2 or more).
  • Consequently, possibility to append/link at once several items, either from a same .blend lib, or even from different ones.
  • Also, filtering by datablock types was added (so that you can see only e.g. materials and textures from all .blend libs from a same directory…).
  • Previews were added to object and group datablocks. Generation of those is handled by a python script (note: only handles BI renderer in this build, Cycles is yet to be added).

Note about previews of datablocks like materials, textures, etc., you have to manually generate them (from File -> Data Previews main menu), and then save the .blend file. On the other side, preview generation of objects and groups work with separated automated tasks ran on selected .blend files (which should rather not be opened at that time). This is quite inconsistent and shall be fixed for sure! On a more technical aspect (though it can also have effects on user PoV):

  • Directory listing is now also a background job (like thumbnail generation of images and .blend files), which means listing huge directories, or remote ones, does not lock the UI anymore.
  • Previews of datablocks (IDs) are now exposed in RNA, so third party scripts will also be able to generate their own previews if needed. Not all ID types have previews though (only object, group, material, lamp, world, texture and image ones currently), this is likely to change though.

So, as usual, any feedback is more than welcome! Note too that Windows behavior was not tested at all yet (don’t like starting my win WM :/ ), do not expect (too much) issues on this platform, but you never know with Windows. Cheers, and hope that new year will be full of good things for Blender and all of you!

Updates from May 2015

Warning: this will be mostly technical, since nearly nothing changed feature-wise. Testing is really needed though, internal changes are quite heavy!

You may find the updated builds at the usual place: https://download.blender.org/ftp/mont29/

From the “outside”, user point of view, this build does not differ much from previous one from January. Only changes (outside of master work, like preview size) are:

  • Some data was removed from the detailed view (modes and owner – UNIX-only, and Blender is not an OS file browser!).
  • Columns of compact (default) and detailed views are now fixed-size (among four choices, like for previews view). This is due to internal changes detailed below.

Internal changes were mostly done for future asset engines, but they also improve regular file browser experience, in a nutshell:

  • Smaller memory footprint when browsing directories with huge number of items (several thousands and more) – especially in case of preview display mode.
  • Much quicker display of those previews.

Plan is to merge this branch (or as much as possible of it) into master after 2.75 release.

Now for the technical details!

The main changes are under the hood – a full rewrite of our file listing code, to reduce memory usage and global computing effort. The idea is to keep as few as possible data from actual directory(-ies) listing, and to generate the full stuff needed to draw in filebrowser window only when needed, following the “sliding window” principle.

So, let’s say your file browser is currently centered on item 4000 in a directory containing 10000 files. We do store minimal data for those 10000 files (what’s needed for sorting and filtering), but only generate complete info (detailed types, strings for size, previews, etc.) for items through 3489 to 4512. Then, if user scrolls slightly to center item 4500, we only trash items up to 3988, and only have to generate those from 4513 to 5012… As a bonus, this means e.g. previews are generated for visible areas first, instead of ‘flat’ top-to-bottom process which would sometimes takes several tens of seconds to reach the bottom of the listing.

This allowed us to reduce our `direntry` struct size (it now basically wraps path and stat info). But the main improvement is with previews – since a few days previews are now 256*256 pictures in master. For a directory with 10000 images in preview mode, that means 2.6GiB of ram – just for previews! While with a sliding window of about one thousand items, we can limit this to a maximum of 266MiB – should we be browsing a directory with tens of thousands of items.

This is a rather extreme example of course, but not uncommon (think e.g. to render directories, or one with one or two hundreds of library .blend files read in new ‘flat’ mode…). And there is another reason why this change is needed: future asset engines. One can easily imagine on-line asset repositories with tens of thousands of items, you just cannot generate in Blender items for all of them at once! With that new code, asset engine only gives Blender the total number of entries, and then Blender requests for a limited range of those as needed (note that ordering and filtering will obviously also be deferred to asset engines).

.blend file items preview has also been moved into IMB_thumb area. This was mandatory, as we do not store previews for all items, we could not read them anymore while doing the initial listing. The main drawback is that it means on first run, .blend files will be read twice. However, there are several great advantages:

  • Consistency: .blend library items previews are now handled as any other (image, font, .blend file itself…) – this also means other areas of the code can easily get previews for them.
  • Performances: since they are handled by IMB_thumb, they also use the thumbnail caching system – in other words, .blend file is read once, regenerating the thumbnail later will only be a matter of reading the cached .png file!

Also, one side effect of those heavy changes is that previews should be generated much more quickly, since they now use a much lighter `BLI_task`-based threading, instead of the complex and heavy `Job`-based one.

Duplicated from https://mont29.wordpress.com/2015/01/14/assets-filebrowser-preliminary-work-experimental-build-i/ and https://mont29.wordpress.com/2015/05/12/assets-filebrowser-preliminary-work-experimental-build-ii/

23 comments
  1. And also remember sorting settings from last time you opened the window! It’s useful to have last changed items on top in large directories.

  2. Being able to increase the size of the thumbnails would be useful. It’s often difficult for me to judge textures based on their small preview thumnails.

  3. Have you thought of select folder for image sequence? so I don’t even have to look at all the files inside.

  4. one and only one question regarding asset acces – network paths, please?

  5. I’m not sure If I missed it, but will this add a similar functionality like Maya’s/ C4D’s hypershade shader/material viewer to Blender where you can make new shaders/materials and also see what materials are used in the scene?

  6. Excellent, will be very useful ! Maybe adding a video preview in the file browser (with mouse drag controlled frame scolling) would be very powerful when used with the VSE too ..

  7. Sorry but I just want to point out the Cube.001, Cube.002, Cube.003, Lamp.001, Lamp.002 & Lamp.003. :v

  8. How about implementing turnable preview (turntable) like shotgun software?
    it would be good for review purpose.

    • turnable preview -> turnable preview generation

      • This is a (very) long term TODO. It’s nice eye-candy feature, but far from trivial to implement well, and not really that crucial, at least in a first time. ;)

  9. Looks great!

    On a slightly related note: It would great if we could see full file folder names while browsing through files and folders. If you have few items in a related project that begin the same (like me:0) it can be difficult to look through several folders listed as MyProjectFolde…

    Or is this already possible and I am just unaware of the setting?

    • No, currently not possible. Think will rather add it as a tooltip, having full paths directly in UI would really be ugly imho.

  10. Salut Bastien. Je me suis empressée de tester la génération de preview pour les objets et les groupes. Or celà semble ne pas fonctionner sous windows 7. Le script se lance, crée un premier rendu dans le répertoire temp de windows et plus rien, si ce n’est un message d’erreur dans la console et Blender qui ne répond plus.

  11. It would be very useful if Blender would be capable of save files for materials, with specific format. It would save us a lot of time.

  12. Its great that the File Browser does a wonderful job!! Feeling content for the availability of Directory Listing back!!

  13. Will the asset manager be usefull for BGE game asset management? That would be great.

    • I see absolutely no reason why it would not be able to do so. ;)
      BGE data are the same as ‘regular’ Blender ones, so…

  14. is it also planned to update the keybindings so its streamlined with all other software functionality? like shift+lmb to select multiple files and such?

    • Think shortcuts rework is a separate project, already ongoing (mainly by Jonathan Williamson). What could be done on filebrowser side itslef though is exposing more customizable shortcuts (we generally still have way too much hardcoded stuff in our UI :/ ).

  15. ” listing huge directories, or remote ones, does not lock the UI anymore” – I love responsive interfaces! : )

In order to prevent spam, comments are closed 15 days after the post is published.
Feel free to continue the conversation on the forums.