Viewport project – targets, current state of the code

Encompassing a broad issue with decentralized code such as real time drawing under the umbrella of the “Viewport” project, might be slightly misleading. The viewport project, essentially encapsulates a few technical and artistic targets such as:

• Performance improvement in viewport drawing, allowing greater vertex counts
• Shader driven drawing – custom/user driven or automatic for both internal materials and postprocessing in viewport (includes eye candy targets such as HDR viewport, lens flares, PBR shaders, depth of field)
• Portability of drawing code – this should allow us to switch with as little pain as possible to future APIs and devices such as OpenGLES compatible devices

These targets include code that has already been written as part of blender, as part of the viewport GSOC projects by Jason Wilkins, and will also require more code and a few decisions on our part to make them work. One of those decisions is about the version of OpenGL that will be required for blender from now on. First, we should note that OpenGL ES 2.0 for mobile devices is a good target to develop for, when we support mobile devices in the future, given those stats. OpenGL ES 2.0 means, roughly, that we need programmable shading everywhere – fixed function pipeline does not exist in that API. Also, using programmable shading only will allow us to easily upgrade to a pure OpenGL 3.0+ core profile if/when we need to, since modern OpenGL also has no fixed pipeline anymore. For non-technical readers, OpenGL 3.0+ has two profiles, “compatibility” and “core”. While compatibility is backwards compatible with previous versions of OpenGL, core profile throws out a lot of deprecated API functionality and vendors can enable more optimizations in those profiles, since they do not need to take care of breaking compatibility with older features. Upgrading is not really required, since we can already use an OpenGL 3.0+ compatibility profile in most OS’s (with the exception of OSX), and OpenGL extensions allow us to use most features of modern OpenGL. Upgrading to core 3.0 would only enforce us to use certain coding paradigms in OpenGL that are guaranteed to be “good practice”, since deprecated functionality does not exist there. Note though, that those paradigms can be enforced now (for instance, by using preprocessor directives to prohibit use of the deprecated functions, as done in the viewport GSOC branch), using OpenGL 2.1. So let’s explore a few of those targets, explaining ways to achieve them:

• Performance:

This is the most deceptive target. Performance is not just a matter of upgrading to a better version of OpenGL (or to another API such as Direct X, as has been suggested in the past). Rather, it is a combination of using best practices when drawing, which are not being followed everywhere currently, and using the right API functions. In blender code we can benefit from:

1. Avoid CPU overhead. This is the most important issue in blender. Various drawing paths check every face/edge state that is sent to the GPU before sending them. Such checks should be cached and invalidated properly. This alone should make drawing of GLSL and textured meshes much faster. This requires rethinking our model of derivedmesh drawing. Current model uses polymorphic functions in our derived meshes to control drawing. Instead, drawing functions should be attached to the material types available for drawing instead and derived meshes should have a way to provide materials with the requested data buffers for drawing. A change that will drastically improve the situation for textured drawing is redesigning the way we handle texture images per face. The difficulty here is that every face can potentially have a different image assigned, so we cannot make optimizing assumptions easily. To support this, our current code loops over all mesh faces every frame -regardless of whether the display data have changed or not- and checks every face for images. This is also relevant to minimizing state changes – see below.
2. Minimize state changes between materials and images. If we move to a shader driven pipeline this will be important, since changing between shaders incurs more overhead than simply changing numerical values of default phong materials.
3. Only re-upload data that need re-uploading. Currently, blender uploads all vertex data to the GPU when a change occurs. It should be possible to update only a portion of that data. E.g, editing UVs only updates UV data, if modifiers on a mesh are deform type only, update only vertices etc. This is hard to do currently because derivedmeshes are completely freed on mesh update, and GPU data reside on the derivedmesh.
4. Use modern features to accelerate drawing. This surely includes instancing APIs in OpenGL (attribute, or uniform based) – which can only be done if we use shaders. Direct state access APIs and memory mapping, can help eliminate driver overhead. Uniform buffer objects are a great way to pass data across shaders without rebinding uniforms and attributes per shader, however they require shading language written explicitly for OpenGL 3.0+. Transform feedback can help avoiding vertex streaming overhead in edit mode drawing, where we redraw the same mesh multiple times. Note that most of those are pretty straightforward and trivial to plug in, once the core that handles shader-based, batch-driven drawing has been implemented.

• Shader Driven Drawing

The main challenge here is the combinatorial explosion of shaders (ie shader uses lighting or not, uses texturing or not, is dynamically generated from nodes etc,etc). Ideally we want to avoid switching shaders as much as possible. This can be trivially accomplished by drawing per material as explained above. We could probably implement a hashing scheme where materials that share the same hash also share the same shader, however this would incur its own overhead. Combinations are not only generated by different material options, but also from various options that are used in painting, editors, objects, even user preferences. The aspect system in the works in the GSOC viewport branch attempts to tackle the issue by using predefined materials for most of blender’s drawing, where of course we use parameters to tweak the shaders. Shader driven materials open the door to other intersting things, such as GPU instancing, and even deferred rendering. For the latter we do some experiments already in the viewport_experiments branch. For some compositing effects, we can reconstruct the world space position and normals even now using a depth buffer, but this is expensive. Using a multi-render target approach here will help with performance but again, this needs shader support. For starters though we can support a minimum set of ready-made effects for viewport compositing. Allowing full blown user compositing or shading requires having the aforementioned material system where materials or effects can request mesh data appropriately. Shader driven drawing is of course important for real time node-driven GLSL materials and PBR shaders too. These systems need a good tool design still, maybe even a blender internal material system redesign, which would be much more long term if we do it. Some users have proposed a separate visualization system than the renderers themselves. How it all fits together and what expectations it creates is still an open issue – will users expect to get the viewport result during rendering, or do we allow certain shader-only real time eye candy, with a separate real time workflow?

• Portability

Being able to support multiple platforms – in other words multiple OpenGL versions or even graphics APIs – means that we need a layer that handles all GPU operations and allows no explicit OpenGL in the rest of the code, allowing us to basically replace the GPU implementation under blender transparently. This has already been handled in the GSOC viewport 2013 branch (the 2014 branch is just the bare API at the moment, not hooked into the rest of blender), with code that takes care of disallowing OpenGL functions outside the gpu module. That will mean GLES and mobile device support support, which is something Alexandr Kuznetsov has worked on and demostrated a few years back.

• Conclusion

As can be seen some of those targets can be accomplished by adjusting the current system, while other targets are more ambitious and long term. For gooseberry, our needs are more urgent than the long term deliverables of the viewport project, so we will probably focus on a few pathological cases of drawing and a basic framework for compositing (which cannot really be complete until we have a full shader-driven pipeline). However in collaboration with Jason and Alexandr we hope to finish and merge the code that will make those improvements possible on a bigger scale.