Assets – FileBrowser Preliminary Work – Experimental Build II

So, as some of you may know already, since December 2014 and my three weeks spent in Amsterdam at BI, I’ve started working on the asset topic.

So far, I did not do anything really directly related to assets (except early designing) – rather, I’ve been improving the editor I intend to use later for assets handling, i.e. the FileBrowser. I have already ported some cleanup/refactor and some minor features (like search field in header, operator to enforce mat/tex/etc. previews generation, …) in master, but most work is still in the dedicated ‘assets-experiments’ git branch, so I’ve made some experimental builds to (try to ;) ) get a bit of testing. In those builds you’ll find:

• Bookmarks and co using UIlists, with possibility to rename and reorganize bookmarks, plus the default features of UILists (filtering by name, and some sorting).
• Possibility to list the whole content of a blend file (in append/link mode) at once (set ‘Recursion Level’ setting to 1), and in any mode, to list several levels of the directory tree in a “flat” mode (including blend files content if relevant, set ‘Recursion Level’ to 2 or more).
• Consequently, possibility to append/link at once several items, either from a same .blend lib, or even from different ones.
• Also, filtering by datablock types was added (so that you can see only e.g. materials and textures from all .blend libs from a same directory…).
• Previews were added to object and group datablocks. Generation of those is handled by a python script (note: only handles BI renderer in this build, Cycles is yet to be added).

Note about previews of datablocks like materials, textures, etc., you have to manually generate them (from File -> Data Previews main menu), and then save the .blend file. On the other side, preview generation of objects and groups work with separated automated tasks ran on selected .blend files (which should rather not be opened at that time). This is quite inconsistent and shall be fixed for sure! On a more technical aspect (though it can also have effects on user PoV):

• Directory listing is now also a background job (like thumbnail generation of images and .blend files), which means listing huge directories, or remote ones, does not lock the UI anymore.
• Previews of datablocks (IDs) are now exposed in RNA, so third party scripts will also be able to generate their own previews if needed. Not all ID types have previews though (only object, group, material, lamp, world, texture and image ones currently), this is likely to change though.

So, as usual, any feedback is more than welcome! Note too that Windows behavior was not tested at all yet (don’t like starting my win WM :/ ), do not expect (too much) issues on this platform, but you never know with Windows. Cheers, and hope that new year will be full of good things for Blender and all of you!

Updates from May 2015

Warning: this will be mostly technical, since nearly nothing changed feature-wise. Testing is really needed though, internal changes are quite heavy!

You may find the updated builds at the usual place: https://download.blender.org/ftp/mont29/

From the “outside”, user point of view, this build does not differ much from previous one from January. Only changes (outside of master work, like preview size) are:

• Some data was removed from the detailed view (modes and owner – UNIX-only, and Blender is not an OS file browser!).
• Columns of compact (default) and detailed views are now fixed-size (among four choices, like for previews view). This is due to internal changes detailed below.

Internal changes were mostly done for future asset engines, but they also improve regular file browser experience, in a nutshell:

• Smaller memory footprint when browsing directories with huge number of items (several thousands and more) – especially in case of preview display mode.
• Much quicker display of those previews.

Plan is to merge this branch (or as much as possible of it) into master after 2.75 release.

Now for the technical details!

The main changes are under the hood – a full rewrite of our file listing code, to reduce memory usage and global computing effort. The idea is to keep as few as possible data from actual directory(-ies) listing, and to generate the full stuff needed to draw in filebrowser window only when needed, following the “sliding window” principle.

So, let’s say your file browser is currently centered on item 4000 in a directory containing 10000 files. We do store minimal data for those 10000 files (what’s needed for sorting and filtering), but only generate complete info (detailed types, strings for size, previews, etc.) for items through 3489 to 4512. Then, if user scrolls slightly to center item 4500, we only trash items up to 3988, and only have to generate those from 4513 to 5012… As a bonus, this means e.g. previews are generated for visible areas first, instead of ‘flat’ top-to-bottom process which would sometimes takes several tens of seconds to reach the bottom of the listing.

This allowed us to reduce our `direntry` struct size (it now basically wraps path and stat info). But the main improvement is with previews – since a few days previews are now 256*256 pictures in master. For a directory with 10000 images in preview mode, that means 2.6GiB of ram – just for previews! While with a sliding window of about one thousand items, we can limit this to a maximum of 266MiB – should we be browsing a directory with tens of thousands of items.

This is a rather extreme example of course, but not uncommon (think e.g. to render directories, or one with one or two hundreds of library .blend files read in new ‘flat’ mode…). And there is another reason why this change is needed: future asset engines. One can easily imagine on-line asset repositories with tens of thousands of items, you just cannot generate in Blender items for all of them at once! With that new code, asset engine only gives Blender the total number of entries, and then Blender requests for a limited range of those as needed (note that ordering and filtering will obviously also be deferred to asset engines).

.blend file items preview has also been moved into IMB_thumb area. This was mandatory, as we do not store previews for all items, we could not read them anymore while doing the initial listing. The main drawback is that it means on first run, .blend files will be read twice. However, there are several great advantages:

• Consistency: .blend library items previews are now handled as any other (image, font, .blend file itself…) – this also means other areas of the code can easily get previews for them.
• Performances: since they are handled by IMB_thumb, they also use the thumbnail caching system – in other words, .blend file is read once, regenerating the thumbnail later will only be a matter of reading the cached .png file!

Also, one side effect of those heavy changes is that previews should be generated much more quickly, since they now use a much lighter `BLI_task`-based threading, instead of the complex and heavy `Job`-based one.

Duplicated from https://mont29.wordpress.com/2015/01/14/assets-filebrowser-preliminary-work-experimental-build-i/ and https://mont29.wordpress.com/2015/05/12/assets-filebrowser-preliminary-work-experimental-build-ii/