Gitea Diaries: Part 1

First installment of a series of articles about moving Blender’s main development website from Phabricator software to Gitea.

Blender’s website developer.blender.org is the well-known go-to place for reporting bugs, submitting patches and checking for updates on fixes for problems people have reported.

It’s clear that a project the scale of Blender involves countless interactions between our users, artists, tinkerers, testers and developers, bugfixers and other contributors.

developer.blender.org has been doing this for our community for years now, and successfully so. Sadly Phabricator, the software running behind the scenes, has come to be discontinued; the announcement about this arrived a bit over a year ago as of writing this post. The writing on the wall was clear: we’d have to look for a replacement.

Early 2022, I have, among other things, been put on the task of looking into this, checking what’s available, what our choices are and help us get to a choice on what to do. This brings us to the title of this post.

From all the horses we’ve looked at, Gitea is the software we’ve decided to put our faith into, being able to provide us with a choice that’ll serve as our next-generation development-forge.

Over the first half of 2022 , work’s been put into understanding what options there are out there that we could live with, that would function well and that would be future-proof enough to not require such an involving task of moving as we face currently. There are the obvious answers out there, of course Github, Gitlab, Atlassian all have something that works well as a forge, and they assuredly have happy customers.

For the Blender project, however, just being available and usable are not the only requirements; it became clear that if we wanted to adhere to principles regarding the betterment of open-source software, there was an obvious option available to us that we should be taking seriously. Hence, Gitea has been chosen, also because it is mature, feature-packed and built around modern development idioms which give the promise of it being built to last us a good long time.

Next to this, it promises to deliver on a number of things we’ve been wanting to have Phabricator do for us; including more straight-forward project-management, better CI/CD integration, and a more well-understood/well-known process for people to use for submitting code.

This article aims to be the first in a series of posts about the journey that we have set out on, now with Gitea as the focus.

A journey which we aim to have us have a working replacement for Phabricator during the course of October 2022 (in time for Blender Conference, in fact!).

---

First Steps

Now that the tool-of-choice has been chosen, we get to look at how to get it to do all that we want of it. Does it perform? Does it handle a load? Can it handle our data? Will we be able to keep our development history after we’ve moved? How hard will that be? What compromises might have to be made?

For that, we need data. In particular, our current data inside of Gitea. It’s time for a first attempt at importing data into Gitea from Phabricator.

Sadly, no finished, off-the-shelf, importer exists for Phabricator here. Not many other platforms do, in fact. There may be several reasons for this, but one of them definately is that the way Phabricator “does things” is just very different from how most other platforms choose to do things. In other words, some of the core concepts of a forge like repositories, projects, teams, tasks/issues, labels, statuses have different meanings, different relationships between each other in Phabricator and Gitea.

The biggest hurdle that needs to be tackled is just how Gitea may or may not support importing pull-requests and review-data (called Differentials in Phabricator) in an ‘elegant way’™. Let me explain…

Gitea has support for the Fork → Branch → Pull-request → Merge workflow as is common on other platforms. This is the workflow we intend to use for future development; though there are some challenges to be worked out; particularly regarding the amount of disk-space required to support this workflow as in the default configuration of Gitea, it allocates a full copy of the data for each fork. With a current repository-size of around 800MB, it is easy to understand that this presents scaling challenges.

However, given that Phabricator has patches that dont have a ‘source’ reference, these would be somewhat awkward to import.

The main avenues of investigation regarding this issue are currently:

• See if we can import them in a way that’d look the same as what you’d get if you’d accept a pull request from a branch in a fork that’d get deleted afterwards. That’s essentially ‘the same kind of thing’ (nothing to point to anymore).
• See if we can make a ‘synthetic’ fork-repo of the main blender repository that’d reflect all the changes of each ‘Differential’ that ever got created in Phabricator. This’d be ‘technically a cool feat’, but not expected to be possible without a serious amount of hackery/wizardry.
• Gitea might support the concept of an ‘imported’ Pull-request, migrated from a different platform, without needing to have the context originally available on the platform migrated away from; this would be an almost direct fit.
• Last but not least: request doing some customization on the Gitea code to support having PR’s without context to refer to.

These are just the approaches we’ve currently identified and it’s clear there’s going to be some way to handle this in a way we’d like. It’s less a showstopper or even a hurdle than that it is just where we’re at at the moment.

There’s a good list of things still to be worked out; here’s a short-list of some of the things to tackle:

• Consolidating (most!) users between Phabricator and Blender ID so that when you login to Gitea, you get to do everything as you used to, just via Blender ID.
• Getting workboards into Gitea that are based on a query and not just ‘project’ and be able to save them for re-use on dashboards, etc. Allow queries to include ‘labels’.
• Importing attachments in a way that their use as ‘inline media’ can be supported as was the case in Phabricator.
• Badge-support (in conjunction with Blender ID badges).
• As noted above already: Find a way to mitigate disk-space usage implications when forking a repository.
• Have issues be able to belong to multiple projects.
• Possibility of supporting ‘priorities’ in another way than using ‘labels’ for that.

---

Meanwhile

Wait; does all of this mean that all that’s been done is just hypothetical/theoretical work and no actual things have been made yet?

Well, no, not quite.

So far, this is what we already have:

• There’s a Blender ID authentication module that allows you to login natively.
• There’s a Python-based exporter/converter tool that takes data out of Phabricator and is able to mush things into Gitea’s ‘restore-repo’ format with (partial) incremental-fetch functionality (think ‘rsync’). This is being prepared for being hosted as open source project for re-use by others with the same challenge.
• There’s a mapping of Phabricator projects towards Gitea labels
• There’s running instance of all Blender-project issues (around 72K) imported into https://gitea1.dev.blender.org/blender-foundation/blender (login with Blender ID)
• And, quite significantly, we’re in the finishing stages of having a support agreement with the Gitea project to have them support us in our migration by funding work on missing features, code and bugfixes that will be available 100% to Gitea users under Gitea’s MIT license.

---

Next Steps

For the coming week(s), most of my time will be going towards making sure that we can have the right feature descriptions and understand just how important (or not) they are for us so that we can understand what to work on first, what to get ready and make sure we have functioning before anything else.

There are going to be things we can leave for later; implement after a possible migration date; but a number of things are obvious ‘must-haves’ before even considering being able to switch. Getting that right is important.

With that, I also expect that somewhere at the end of next week, there might be enough new things to report that it deserves another code-blog post – Gitea Diaries: part 2 !

Until then, happy coding!
Arnd