This is a story about Collada, its current status and what we possibly can do in the future.
Where we are
The current implementation of the Blender Collada module is based on the OpenCollada C++ Library. OpenCollada’s goal is to fully implement the Collada standard. This library has been developed slowly over the years with a significant drop of activities in the years 2011-2014. Since then it slowly gained speed again, see the OpenCollada Contributor Chart.
The initial reason for why the OpenCollada library was chosen for Blender is somewhat shady, but the 3 main reasons seem to be:
- It was available when Blender 2.5 was born
- It provided an Import part and an Export part “out of the box”
- There was someone who got it to work with Blender in a GSOC project
How it is implemented
Blender normally uses Python based Addons for implementing Data Importers and Exporters. But the OpenCollada Library is a C++ Library and it does not come with a Python API. Because of that the Collada Module is also written in C++ and it is fully integrated into the Blender main Program itself.
So what we call the Collada Importer/Exporter is a set of C++ Classes sitting in between the Blender Program and the OpenCollada Library, mediating data between the 2.
What’s wrong with it?
There are a couple of downsides with the Collada Module as we have it today. First thing is that on the Blender side many very basic functions are used which make it hard to follow what is actually happening and why things have been done as they are done. There is about zero documentation about why parts of the code are implemented as they are (in short form: Odd stuff )
On the other side the OpenCollada library seems to be used in some weird way. I am still not sure why the Blender interface is doing so much on its own, why the imported files are always read twice, why so many things are done in the module where i suspect the OpenCollada library has it already solved for us.
So here is the list:
- The Module must be maintained by C++ People
- The Maintenance is awkward due to self made Code Complexity (too much c++ inside)
- The Library adds about 35 MByte to the Blender Distribution
- The Importer and the Exporter have never been fully developed
What we can do about it
I see 3 basic options how we could possibly improve:
Full Rewrite
After fiddling with this module for about 3 years i finally feel confident enough to try a full rewrite. The idea is to reorganize the module such that it becomes easier to maintain and easier to improve. And at the same time make it easier to replace the current OpenCollada Library by something new (also based on C and C++) for example for a …
Hybrid Approach
We make use of the rather complete feature set of the OpenCollada Library (or a more recent replacement if such exists) on one side, but also add a Python API to it so that we get the advantages of having a Python Addon which can possibly be maintained by more people.
Full Replace
We can remove the Module and create a full replacement in Python. In fact for the Exporter side there are already a couple of alternatives available. However as far as i can tell none of the python based alternatives seems capable to fully replace what we have by now.
For the Importer side all i can tell so far is: This is where most of the trouble seems to be located. Collada is complex and very flexible. Trying to create a full featured importer based on Python might be a huge task.
What we will do about it
In the very first place we will attempt to simplify what we have and reorganize the library with the Hybrid approach in mind but without actually implementing any Python API. I think the best we can do here is to add 2 organizational layers between the importer/exporter and their counterparts (Blender and the Library).
- IO – Shield is the only passway that we will allow to get stuff out of Blender or into Blender. This can be a very thin API (if needed at all) based on the Blender RNA module.
- Compatibility Cloack is the only passway that we will allow to get stuff from Collada or to export stuff to Collada.
The cool thing about this approach is that we can put all Blender dependencies into the IO-Shield while we can put library dependent stuff into the compatibility Cloak. When we later decide to step out of the C world then we can convert the IO Shield into a Python API and then go crazy with our Python Addon.
Actually the IO-Shield might even serve as generic API for Importers and Exporters as the general questions about how to feed data to Blender or pull data from Blender should remain more or less the same for all formats… no?
What we could need from you
- In first place we ask for your opinion about this approach. Is it feasible, can it be improved, are there alternative approaches worthwhile to look at?
- But we also want to know in which ways Collada is actually used. All i know so far is that those who use it make heavy use of it.
- I also know there are other groups out there in the wilderness of the internet who can not live with our Current implementation for one or the other reason. It would be helpful to get detailed information in which way the current implementation does not work (including demos how it fails).
Why i am doing this?
I think that Collada is an open standard and Blender should support this standard in the best way it can be done. I also believe that we can get most out of this by embracing existing stuff like OpenCollada or possibly other existing libraries (c or python) and help to improve what exists and not to get into “This stuff is odd, i can do better then that, take mine its awesome” attitudes.
And besides all of the above i just want to learn how its done. If anybody feels like they can actually take the lead here please step in. I would be more than happy with that. Until this happens i will do my best to keep this going and make it a successful learning experience :)
I will add my activities to this article as time goes by. But now its time for opening the editor and start doing.
I am glad you realise this now.
python is better for such a thing.
If afterwards you realise as well that not using Qt for the interface was a mistake, cut the game engine and invest in a decent partnership with ogre3d or other engine, develop a better and more generic node editor… still there is hope.
I followed the blender project for a long time and I use it regularly. Some amazing people here, but is very discouraging you take so many bad decisions and don’t listen.
Please, add less “awesome” stuff and focus a bit on make work what you have. Focus on the community.
My two cents.
We regularly export our Blender models to Collada to import them into ArcGIS 3D, which supports Collada.
This works very well, and carries texture information too.
Hey, Gaia, I’ve emailed you before sometime mid last year. I did not get the impression you were super interested in Blender’s COLLADA situation.
So you can imagine my surprise to find this blog post in my search results just now.
I am very interested in getting COLLADA working on Blender because of its status as a premier inexpensive modeling suite. I just can’t seem to use Blender for the life of me, and I don’t have time to get bogged down in its largely C core code. (I’ve taught myself Blender on multiple occasions but I don’t use it constantly and it’s the opposite of learning to ride a bicycle.)
I intend to approach the Blender devs after I get further the work I am doing to provide alternatives to OpenCOLLADA that I think will make it possible to move beyond import/export. COLLADA just isn’t an import/export paradigm. It’s more like the WWW. You’ve never heard of importing a webpage, because it’s just not done. It doesn’t make sense. And it’s not useful.
I did not know of any useful COLLADA software until last week or so I discovered Nvidia’s FX Composer software. It is designed as any software that boasts meaningful COLLADA support must be. COLLADA was never really intended for storing anything. But that’s what people expect of it, and I think for good reason, which is why I am trying to work with Khronos to revitalize it.
Last week I noticed that COLLADA didn’t seem able to support morph-animations, like Blender’s ShapeKeys. I hope to use this to publish a new official document on its homepage, in order to retroactively enable morph animations without changing the schema or this questionable extension:
https://www.khronos.org/collada/wiki/Morph_weights_KHR_extension
Ideally that will be a show of life yet in COLLADA. I encounter many developers eager to write COLLADA handling software. I think it shows that there’s a real need and a keen interest in it or something like it. But I don’t think it’s the most useful instinct to approach the problem as they are.
I do not know the extent of high-end (expensive) commercial software’s adoption of COLLADA or if people who use that software use COLLADA. It can be very hard to tell from the outside looking in.
I think C++ is a prerequisite for coding against COLLADA. It’s too complicated to waste time with “string” level code. If Blender cannot handle COLLADA losslessly, it’s pointless. “Roundtripping” doesn’t really apply to COLLADA. It’s extensible and multi-representational, so Blender cannot include all of its features by definition. It must implement a side-by-side COLLADA database, or better an API for writing side-by-side databases, which COLLADA can use.
https://www.khronos.org/collada/wiki/ColladaDOM_3 is my primary contribution to COLLADA. It lets you write high level algorithms against it very naturally. I developed it so that I can work with COLLADA without wasting decades of my life. There’s source code up. But it’s unstable/debug-only/preview status for now and a large part of this year.
It fully works, but needs some more features, and I think the code-generator needs to have a synthetic schema feature before I will present it to Blender’s community. The new style eschews class methods so that they do not name-clash with schemas. It requires a procedural API that wraps around the low-level API. There doesn’t have to be just one. Users can create their own. But there should be a basic one provided.
There is a portable protocol I invented called DAEP. Procedural APIs are not part of the protocol. It uses C++’s operator overloading, since it cannot use “identifiers” for many things. There are some times when it’s possible to like .size() to get the number of children with the same name, or the number of items in a list-value. Often it’s not possible. In that case, it’s necessary to use a non-portable cast using a dae() wrapper to access the low-level implementation layer, or xs::any currently acts like a low-level daeElement. Those are the tools used write a procedural API in terms of one or more implementations of DAEP. Then that API is itself a portable API.
Armed with this it’s just a matter of adding a significantly large new aspect of Blender. It couldn’t be a small project. It’ll take a commitment to the field itself. It’s bigger than just supporting a novel file-format.
Hi, Mick;
Thanks for this very detailed comment. Here is just a very brief answer for now.
My primary intention was to shield the Collada module from Blender in such a way that its heart (the OpenCollada library) eventually can be replaced by something different (maybe smaller, more capable, better suited…)
I have only the usage scenario in mind where it comes to transfer data between applications. But your view on this seems quiet a bit different and i admit that i have not yet fully understood what that actually means for the Collada module specifically and for Blender in general.
However, whenever we switch to another basic Collada library, or another implementation, this will certainly only be accepted when whatever is done creates a better situation with more quality. “Instable”, “Still in development” and “Not maintained” are certainly anti features of the future implementation :)
My favorite approach is still to have one part that is an Addon that handles the user interface, file handling, sending/receiving data from Blender and so on. And another part that is the basement (the Collada engine) which does all the data interpretation and conversion from Collada to an internal representation that can then be used to feed Blender.
What this means in detail and where it will eventually end i do not yet know.
Hi again Gaia,
In my brief interaction with the Blender community I took away from it that there were not really programming people in the COLLADA export/import feature’s corner so to speak.
At that time I went away thinking that it would not be too long before I can have something better suited for the job than OpenCOLLADA, but I didn’t realize how long the journey ahead of me would be then.
Today the work is much further along, but there’s still a ways to go. But there is code that programmers can familiarize themselves with, to get an idea if it can be a better avenue than OpenCOLLADA.
That said, the COLLADA-DOM code is not a layer on top of COLLADA; it’s just a way to greatly bolster programmers ability to tackle COLLADA. In Blender’s case, I think the COLLADA code needs to be much closer to Blender. In fact, it should be a core component of Blender. And the Blender Foundation should take the view that this is the future direction of noncommercial 3-D, and it needs to be invested in it, even if the current standards right now are very embryonic.
I think in reality COLLADA adoption has been very poor for many reasons. I don’t think Blender should be ashamed that it has not further embraced it. It’s actually a shrewd and realistic view of the situation at hand.
As for what I mean. Take a look at Nvidia’s FX Composer software. Blender needs to integrate with COLLADA like that. COLLADA wasn’t designed for import/export any more than it was designed to be a storage format. It was designed as a transient communication protocol. This dim view of it severely hampered early implementation of COLLADA because this notion of “transience” seeds in the minds of programmers that the quality of their code, need be even more shoddy than it normally be, because the data is just going to be vaporized at the end of its route anyway.
To say import/export connects applications is not even the real view of it. That is a secondary market. It’s very difficult to use COLLADA that way, because it has to be implemented very thoroughly, and its open-ended and XML nature makes that a very, very difficult challenge. The XML looks inviting if you think of it has hand-manipulating text; but in the real world it has to be manipulated in software, and XML sets a deceptively high standard that makes lossless transformation very challenging.
I take an opposite and very common view that COLLADA should’ve been a storage/archival format from the beginning. If the authors had taken this view it would’ve put the onus on developers to be much more thorough. This gradual shift in perspective is a new era of COLLADA that is rising out of noncommercial artists’ frustrations with this age old problem, as old as 3-D itself.
It’s hard to make this shift. It will require ecosystems of software that implements COLLADA correctly; and ultimately better versions of COLLADA; but the former must precede the latter. Blender is the first and most obvious place to establish a watershed for a future ecosystem. I don’t know how it will be done, but it’s a logical ground zero for COLLADA.
The basic problem with import/export is it sheds data, which prevents artists from truly using COLLADA, just as you’d not invest too much creative energy or anything into something that will just shed your efforts as soon as you try to do something with it. I need to do real work with COLLADA. Which is why import/export doesn’t interest me at all. We need at a minimum a successor to Nvidia’s FX Composer. Ultimately I want that to be an alternative to Blender that will be called Daedalus. But like I say, we need an ecosystem, and everything that can be done should be done, the sooner the better.
P.S. I’m not sure what you intended to communicate by: ‘“Instable”, “Still in development” and “Not maintained” are certainly anti features of the future implementation.’
But upon rereading what I’d written, I noticed this (I’d noticed before also) and today for whatever reason I feel like just making sure you know that in software speak “unstable” is a technical description, that means A) the code if not fully codified, so clients may need to adjust their code as the library comes along. B) the library has not seen enough use to gauge if it’s safe to say it’s ready to be recommended for set it and forget it systems. Often times past version are stable, but new versions are unstable. It doesn’t mean the code base is of poor quality or crashes or anything like that.
I described COLLADA-DOM 2.5 as unstable simply because it’s not been tested enough. And because I want to get more features into it. That said, it’s very likely better than Blender’s present situation, and it probably has all of the features Blender would require for your what you want to do with it.
My concerns for it are more forward looking. What I would recommend for a serious push is very different than what I’d recommend for experimentation, review, and to get a leg up on a hard problem.
COLLADA-DOM 2.4 is stable by contrast. But I’d not recommend it personally, for a whole host of reasons.
Often time stable and unmaintained go hand in hand. Since once something is stable, it no longer requires maintenance. The other side of that is it is no longer improved, unless there is a new version in chamber. In which case, if you intend to benefit from the new version, then it’s functionally the same as being unstable if what you’re looking for is not having to maintain your own client/user code.
Used to great care was given to binary compatibility across versions. But that practice is mostly being abandoned by modern software projects, because in truth, it is not so difficult for clients to either keep their old version, or make the necessary changes to keep up with the new version. Even if that is a chore, it keeps future versions free of legacy barnacles and warts. I try to get the design perfect on the first attempt. But I’m an odd duck.
Ultimately someone has to be willing to investigate the off the shelf solutions if not willing to do the work from scratch–which I don’t recommend in COLLADA’s case. In which case I highly recommend the my new effort with COLLADA-DOM. It is the original COLLADA library. And it’s a DOM like how JavaScript animates webpages. It’s ideal for what Blender needs to do. There’s no way to have a one-size-fits-all COLLADA solution. And if Blender intends to use OpenCOLLADA then it needs to get its own arm of the OpenCOLLADA project, like 3DSMax and Maya and other software have. Otherwise it should use something else I think.
Could you make some description of the format that is directly parsable by code? I mean, for portability,
I mean, for example, here to try to implement a custom C# dae importer, what i do is trying to find examples of collada with a lot of things in them, in order to generate some classes from the xml, if it could be easier to know like:
mesh
VERTEX => float[,,]
NORMAL => float[,,]
UV => float[,]
like preventing a developer to read documentation to know what can contain each sub elements, by making a complete and concise universally parsable description, and maybe, if needed remove the “source” elements, in order to have simpler things.
Could be really cool, so even if a new programming language come, we have all we need to quickly say “hop”, and a blender importer/exporter is made in this language.
On your question in which way Collada is actually used: we are planning kitchens with BricsCAD, a CAD program similar to AutoCAD. When we want to make “beautiful visualizations” for our customers, we take the CAD version over to Blender, where we can render it more realistically, adding lights and colours, backgrounds and maybe even DOF etc.
The problem is, the only built-in export (at least the only working well for Blender) in BricsCAD is Collada, while for others you have to pay extra to get e.g. 3DS export – however, the resulting mesh objects in 3DS are much worse after importing them in Blender than the Collada meshes.
I don’t know if these problems when you’re using AutoCAD or other CAD software as well, but we definitely need the Collada support in Blender. Well, the importer that is, not necessarily the exporter.
What about pycollada? It provides creating, editing and loading capabilities for COLLADA.
Here is a Blender collada importer built on pycollada, but it seems to be RIP for 5 years. https://github.com/skrat/bpycollada
you mean 35mb out of the ~100mb download size?
with 7.000.000 downloads per year that’s like 250TB
Hello Gaia,
If the Better Collada is worthy enough you can use it anyway. The code is distributed under GPL 2 so it is free to use, distribute and modify it (unless the license is not changed). And i think the devs should be happy if you embrace the module without conditions.
Hi, Tessalonicas
This is not about being or not being allowed to use the code, its about who is going to actively maintain it and how they work together with the Blender developers. Please check here for further information:
https://developer.blender.org/D1787
This should make it more clear why this is not a “just take the code and be happy” case :)
Hello Gaia,
the Godot devs are actively discussing about getting BCE into Blender:
https://github.com/godotengine/collada-exporter/issues/31
why don’t you or a Blender dev with a Github account go on there and try to reach an agreement with them? It seems that both you and them want the same thing, and although there have been some disagreements in the past, their intentions are good, otherwise they would have not implemented all the changes the Blender devs requested on https://developer.blender.org/D1787
Better Collada support would be a great feature for the future of Blender. Let’s get the ball rolling as soon as possible! :)
Hi, Pax;
Thanks for the pointer to that other discussion. I took a quick look and i saw Ideasman has already answered in september about how to proceed.
Anyways i want to make it clear again that if an Addon shall be integrated into the Blender distribution then there also must be continuous support for keeping it working, see also https://wiki.blender.org/index.php/Dev:Doc/Process/Addons
I can understand when the Godot people are not interested in maintaing such an Addon for Blender because this adds a lot of additional workload, especially because this is not only about Exporting to Godot, but its about exporting to anywhere and importing from anywhere…
And just for the records: Noone here is criticizing the “better collada exporter” except for its name maybe :)
Hallelujah, Gaia: THANK you for caring!
One of the more common reasons to want Collada as complete and functional as possible, I suspect, is to bridge Blender with the world of Poser “content” such as sold and given on Renderosity, Hivewire, Cornucopia3d, or daz3d via DAZ Studio, and yes: I’m all for it!
PS. My daughter’s name is Gaia, too. ^_^
To preface this, I’m not familiar with the Blender code base. But when I read your post I was reminded of the old saying “you can solve every problem with another layer of software abstraction”. If the other importers are Python-based, I can only encourage you to write a custom importer instead of trying to add more layers of abstraction in front of a 3rd party library from which you probably only need a fraction of the functionality. I tried to use Assimp with my own stuff and have worked on products which build on top of OpenCOLLADA but both seemed to produce more issues then building a custom importer.
In my experience, there are three things to consider with COLLADA:
1. The spec is pretty bloated. It has grown over the years and there is a lot of functionality in there which I have never seen in the wild. So OpenCOLLADA’s benefit of potentially supporting all of COLLADA becomes irrelevant.
2. COLLADA at first seems to be flexible to a point where it produces serious headaches but it is actually not so bad once you wrap your head around the concepts. Took me three attempts at though to get to that point ;)
3. There are various “accents”: some older Sketch Up versions export the opacity incorrectly, Blender had it’s bugs in export at some point in time, etc. This in addition to programs writing their own meta data to have a more expressive material system for example.
But by focusing on the core functionality, implementing COLLADA isn’t soooo complex (as compared to JT, STEP or some other formats) and you have to have some “custom repair code” anyway to deal with point 3 as well as the translation to blender-specific data structures. So you’re writing half of your own importer anyway…
For reference, the importer I wrote for Shapeflow3D ( http://metashapes.com/products/shapeflow3d/ ) is about 1000 lines of C++ in total (200 of which are for performance optimized low-level parsing of float values, etc) and handles all of the meshes I have encountered so far (no support for animations/rigs yet but n-gons, materials, transformations, multiple UV sets, cameras, model units work fine).
It might take a bit of coding (or rather a lot of testing) to get back to feature-parity with what is supported in Blender right now but – at least from the outside – it seems worth it…
Hope this helps!
Hi;
Thanks for taking care!
In fact i believe that writing an exporter or an importer for mesh objects is a task that is not overly complicated. Well, there are some quirks mainly to get things working smoothly where we have to work around features which are not available in Blender (like for example faces with holes) or where the Collada specification itself could be interpreted so and so. I recall for example we had(have…) some issues with the interpretation of the unit attribute in the Asset section.
Regarding the layering: it is meanwhile clear that the IO Shield that i introduced is more or less what blender provides with its RNA Interface. So we have that already in Blender and just can use it. On the other side the Compatibility layer is probably also very thin and i only want it to avoid getting dependencies that later make it hard to kick out OpenCollada by whatever may come.
The people behind the Godot engine have already written a pretty good Collada exporter in Python:
https://github.com/godotengine/collada-exporter
I know a lot of work has been done to improve it since it was last submitted to the attention of Blender devs and the people working on it want it to be merged into Blender, so I think you could consider it before writing one from scratch.
Hi, Pax;
Thanks for the info. We know about the Godot Exporter and we have been in touch with the developer. But it looks like there was no agreement possible how to step forward. The main 3 reasons why Godot was not (yet) chosen as alternative:
1.) It is for now only an exporter, there was no Importer part available when we last talked to the developer
2.) It is unclear who is going to maintain the addon
3.) The developer wanted us to first drop the current module, then work on making Godot fully functional for the use cases supported by the current Blender module. While we wanted it to go the other way (first make the new one working, then drop the old one)
We still have it on the list though.
Gaia, one of the Godot devs answered your questions:
https://github.com/godotengine/collada-exporter/issues/31#issuecomment-256402373
I think they are willing to work with you to get BCE into Blender. Maybe one of the Blender devs could go on their issue tracker on Github and try to discuss the finer points with them? :)
What industry/app adoption does Collada have. From what I can see working in TV 3D animation OBJ, FBX and Alembic are the defacto standards for Mesh, Animation and caching.
Surely development would be better spent on interchange formats that are strong and growing.
* OBJ has no support for animation data.
* Alembic (as far as i know) can be used to transport “baked data” along a production pipeline and it looses development information.
* FBX is not open and i have only heard rumors about it might possibly become an open format sometime in the future.
Surely this project is not about competing with other formats. it is just about making something better that we already have.
Hi Gaia, Your absolutely right, I wasn’t clear enough about the formats. I was trying to articulate that there are better places to focus development on (for the widest possible payback).
I’m probably thinking of this from the wrong perspective, If your doing this out of personal interest in improving collada then I hope it goes well and pays off :), If you’re trying to improve general I/O for blender id go FBX.
Only one (but important) note: we already have an “IO-shield” in Blender – it’s called RNA, used for both Python access (nearly all IO add-ons), and C++ access (Cycles uses it to get data from Blender).
In order to prevent spam, comments are closed 7 days after the post is published. Feel free to continue the conversation on the forums.