Hi everybody, last few weeks I’ve been working under problem with sculpting on deformed mesh and here is first patch going to SVN. This patch mostly changing internal implementation of  brush strokes applying, so only few cases of modifiers are covered now

Unfortunately, not every modifier could be handled correct in sculpt mode and trying to support them could work acceptable for one case but will lead to incorrect and unpredictable result in other cases. All feedback I’ve got was about restricting some cases of modifiers but make things predictable for all other cases. So with new approach of supporting modifiers in sculpt mode you will always see exact mesh shape you’re sculpting on. Technically it means that brush will always know exact position at original mesh and will also know translation which should be applied on displacement made against deformed mesh to be applied on original mesh. The only way to reach this is to disable some “unsupported” modifiers in sculpt mode. Currently, only Armature modifier is supported, but hopefully all deformation modifiers would be supported. Why only armature you’ll ask. It’s simple – this modifier already has got that transformation information to re-calculate displacement to be applied on original mesh, and for most artists I’ve been talking with it’s the most important modifier to be supported (i.e. level small artifacts of rigged characters). Also, due to multires implementation you’ll be also unable to sculpt on deformed multires (I’ll write a bit more about this issue in implementation part of this post and I hope it’ll be also supported in the future).

What about constructive modifiers? I can’t see any suitable way of handling them, so most likely they’ll be permanently disabled in sculpt mode. There is no way to know which part of original mesh should be deformed with stroke. Note, that all modifiers, which can’t be predictable handled in sculpt mode have warning message. I don’t think it’ll be a big problem with unsupported modifiers if there’ll be a hotkey to switch between modes. In this case you’ll be sculpting in predictable way and be able to see final result relatively fast.

So, summarized changes:

  • Sculpting on actual shape
  • Deal with armature modifier (which could be applied on shape key)
  • Quite nice warning message about unsupported modifiers

Implementation

Let’s talk a bit about implementation. Things are quite easy: calculate displacement made against deformed mesh and translate it to original mesh space.  This happens in function named sculpt_flush_deformation() which is called at each brush step. It’s logic very simple – displacement is calculated between current PBVH’s coordinates and coordinates of deformed mesh from previous step. Then this displacement gets multiplied by transformation matrix (crazyspace) and applied on original mesh. At the last, if you’re sculpting on locked key, all the rest relative key block would be updated.

Crazyspace construction isn’t very difficult at this moment – it’s based on modifier’s deformMatricies callback. It’s new callback, but it’s made practically the same things as existing deformMatriciesEM with the only difference – it works with object-mode data. I’m planning to add crazyspace correction like it’s done for edit mode to deal with any deformation modifiers.

And last thing i wanted to describe here is multires deformation. Multires stores coords in grids, and crazyspace stores per-vertex matrices. Sounds like vertexes order in grid and crazyspace could be mapped, but not sure how vertexes are mapped in PBVHVertexIter, so I decided to keep this issue for further work.

P.S. Thanks to Daniel Salazar (ZanQdo) for rigged dragon 🙂

-Sergey-