Michal Čihař - Blog Archives for Weblate

Today, I wondered how much translations were done using Weblate since it was born. Of course I can give only numbers for services I maintain, the biggest one being translation server for phpMyAdmin, Gammu and Weblate.

Looking at the logs, the server l10n.cihar.com helped to contribute over 20000 (actually little bit more as tracked are only changes after releasing Weblate 0.8), what is quite impressive number. Of course biggest amount of contributions came in time of preparing phpMyAdmin 3.5.0 release, where I gave it huge amount of publicity (AKA spamming my blog), but number of people accessing the website still grows since then (still number of submitted translations is slightly lower).

Now as 1.0 release proved to be quite stable, it's time to focus on 1.1 development, which will bring offloading of fulltext indexing (it was proven to be unreliable to update index online for bigger sites). Almost all issues targeted for 1.1 are already fixed and all what needs to be done is testing in real life situations.

As Weblate 1.0 is out, it's time to compare it to existing solutions. As first I've chosen Pootle as this is tool which we have been using for quite some time and I used it as an inspiration for Weblate.

First of all, both Weblate and Pootle use same backend for manipulating with translation files - Translate Toolkit. On the way I've learned that I've chosen different way of manipulating with backend files, what seem to lead to much smaller changes (no more reformatting of whole po file as Pootle does). Also Weblate does not suffer from some Pootle issues like not properly handling changes to fuzzy flag, which has bitten us several times at phpMyAdmin.

Another shared thing is web framework Django. For me it was obvious choice as I like writing application with it. The biggest difference here is that Weblate relies on Django admin interface for maintenance, while Pootle has everything in it's interface. Pootle has great advantage here for having consistent interface for everything, what was sacrificed in Weblate for less code to maintain (while having powerful Django admin interface). Also sharing same framework allows quite easy migration of user accounts.

Now let's functional side of things - this is where I've tried to take all good things I've seen in Pootle, while adding features I've missed there. What makes biggest difference here are project/subproject classification in Weblate, which was originally started as way to handle more project branches, but proved to be useful for translating related projects as well. This allows automatic propagation of changes across project, what is really great for translating more branches of same program or different interfaces to same functionality (eg. GUI application and command line tool).

The other big difference (and I would say key feature of Weblate) is Git integration - all changes are commited to Git with correct authorship, it can help in merging po files or automatically pull upstream changes (eg. from GitHub). This all makes integration into (Git based) development process really easy.

Other features like consistency checks or dictionary (terminology) are quite similar, though different. There is more consistency checks in Pootle, but the main reason was that I found some of them too annoying. User dictionaries are still pretty basic in Weblate and that's definitely something what should improve in future versions.

On configuration side, Pootle is much more flexible. The main reason here is that I've never used most of the options there. So there are no per project settings of checks or privileges. There is also no way to add language from Weblate - usually adding translation is not just matter of adding appropriate po file, but also other steps, which anyway had to be done manually. So Weblate leaves this up to user. On the other side as soon as new language pops up in Git repository, it is shown in the interface.

Both projects have quite good documentation - Pootle has most things covered on wiki while Weblate relies on separate documentation. However there are definitely some bits missing on both sides.

PS: If you want to want to have more "facts" in comparison, you can check Ohloh :-).

After few weeks of heavy testing, Weblate 1.0 has been released today.

Compared to 0.9 there are just minor changes and bug fixes. The most important thing is that Weblate should be now really ready to use :-).

Full list of changes for 1.0:

• Improved validation while adding/saving subproject.
• Experimental support for Android resource files (needs patched ttkit).
• Updates from hooks are run in background.
• Improved installation instructions.
• Improved navigation in dictionary.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances can be found in SUSE Studio Gallery.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited).

I think Weblate is really ready to be released as 1.0 release. It runs pretty stable on https://l10n.cihar.com, I got some bug reports from others, but these were quickly fixed and now nobody complains.

The issue tracker lists few improvements which I plan after 1.0 which will help in user experience and performance, but I see nothing what would be needed right now. So I'm ready to release (even before deadline I have set to this milestone).

Anyway you're welcome to give it some testing and complain if anything is broken. One area where I desperately need feedback are checks, which probably still miss variations for some languages.

I did sort of commit to at least trying to add support for Android string resource to Weblate on Stack Overflow, now while waiting for bugs to fix in 1.0 release it was right time to look at it.

It turned out to be few lines of code but it took me some time to figure them out as translate-toolkit internals are not really good documented. Anyway I've hacked something what works for Weblate (and one random project I've tried) so it might be actually useful. You can find the code for it in pull request, let's see if it will get upstream.

To use that, you need patched translate-toolkit (see above) and latest Git checkout from Weblate. As the changes on Weblate side were quite minor and not intrusive, I've decided to put this into 1.0 release, even if I originally did not plan to do so.

Thanks to great SUSE Studio I've made available appliance with ready to run Weblate. It's based on openSUSE 12.1, with few packages coming from Python devel repository (where I had to push some package updates) and Weblate package, which is currently available in my home.

After booting the appliance, you will get Weblate running as web service (it takes some time on first boot as database setup is done in that time). In case you will use it for more than playing, please remember to change default passwords as described in our documentation.

Anyway let's stop talking, you can get the appliance at SUSE Studio Gallery.

Pretty much on schedule, Weblate 0.9 has been released today.

Besides dozen of bug fixes, the project has been changed to recommended file layout for Django 1.4. In case you are upgrading you will need to adjust few bits, please check documentation.

Full list of changes for 0.9:

• Fixed import of unknown languages.
• Improved listing of nearby messages.
• Improved several checks.
• Documentation updates.
• Added definition for several more languages.
• Various code cleanups.
• Documentation improvements.
• Changed file layout.
• Update helper scripts to Django 1.4.
• Improved navigation while translating.
• Better handling of po file renames.
• Better validation while creating subproject.
• Integrated full setup into syncdb.
• Added list of recent changes to all translation pages.
• Check for not translated strings ignores format string only messages.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

I still get lot of questions why I wrote Weblate instead of staying with some existing solution. Most of the reasons were already addressed in previous posts, where I looked for Pootle alternative or announced Weblate, but still it might be worth to summarize reasons once more.

Git integration

None of existing tools integrate with distribute version control, while I find this integration somewhat obvious and necessary. Having translation system as one of remote branches makes things really easy to handle for developers. Merging translations is nothing more than git merge.

Attribution to translator

I think it's fair to request that translator gets appropriate attribution for his contribution. Once we rely on Git, it is quite obvious that he will be author for Git commit doing changes. This way you can find author purely with Git and don't need to look for other tools.

Handling of several branches

In phpMyAdmin we heavily use branches to maintain stable releases. On the other side we want to be able to push translations to both stable and testing branches without translators having to do the work twice. Weblate does automatically push changes to all branches. Well this is not limited to branches but to subprojects as they don't have to be only branches, they can be also just related projects like with Gammu and Wammu.

Displaying context

Most of tools I tried fail to provide way to show related source code even when Gettext po files contain such references. With Weblate this is just matter of configuring URL of viewer, what nicely integrates with existing services like Github.

Sane merging of uploaded translations

With Pootle, it is possible to upload incomplete po file and it will luckily take it and replace existing one. This is quite dangerous as you would not notice it is incomplete unless you do some translation validation before release. Weblate does merge uploaded translations unit by unit, what makes it slower, but generates minimal diff and works also across branches.

Consistency checks

There are always things which can go easily wrong in the translation like format strings and translation tool should catch these as early as possible. Weblate semi-enforces checks - if check has failed, you will be redirected back to previous translation to possibly correct it (or mark check for ignoring). At current state checks are not as good as I would like to, but they are completely modular and can be easily extended and improved.

Bonus points

While working on Weblate I realized that some things could be done even better than I originally anticipated. For example with Github web hooks, it is possible to receive notifications on upstream repository updates and merge changes as they appear, leaving to really live translation process. Git allows to write custom merge drivers and this can be utilized for less merge conflicts on po files (they are usually caused by timestamps in headers), so that's another handy you get with Weblate.

Of course there is much more built in, just check the documentation.

After quite big delay, Weblate 0.8 is finally ready. There are many fixes and improvements like grouping of Git commits, improved checks or activity logging. All in all this should be feature complete for 1.0 release.

Now it's time to stabilize all the features and release 0.9 (AKA 1.0rc) and then we'll be ready to fire 1.0.

Full list of changes for 0.8:

• Replaced own full text search with Whoosh.
• Various fixes and improvements to checks.
• New command updatechecks.
• Lot of translation updates.
• Added dictionary for storing most frequently used terms.
• Added /admin/report/ for overview of repositories status.
• Machine translation services no longer block page loading.
• Management interface now contains also useful actions to update data.
• Records log of changes made by users.
• Ability to postpone commit to Git to generate less commits from single user.
• Possibility to browse failing checks.
• Automatic translation using already translated strings.
• New about page showing used versions.
• Django 1.4 compatibility.
• Ability to push changes to remote repo from web interface.
• Added review of translations done by others.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

PS: If you're about to use Weblate, rather use Git version than tagged 0.8, you will get bunch of bug fixes for free :-).

One of features I wanted to introduce with Weblate is ability to do some quality checks while translating. We want contributors to have easy access to translating, but still we need to ensure some quality of translations. That's why Weblate comes with set of quality checks which are performed on every translation unit. The current set of checks (see documentation) is pretty basic and more might be easily added in the future pretty easily.

There are two ways how checks are used in Weblate. During translating, if any of checks fires, translator stays on current translation with warning that something might be wrong. This is not a hard error as he still can be correct, but should at least prevent accidental mistakes like corrupted format strings or missing trailing colon.

The other way is to do review of a translation. Failing checks are shown on each translation page or you can browse them by project to give you overview what can be wrong.

For example with phpMyAdmin this helped me discover that there were few languages where almost half of translated strings are actually not translated but simply copied from English. There were also other problems like inconsistent translations across branches. All in all using these features helped me to improve quality of translations in phpMyAdmin quite a lot and I hope we will keep it for the future.