Michal Čihař - Blog Archives for Weblate

Hackweek is over

10th hackweek is over and I think it has been again great chance to hack on something. This year we even had better food supplies so interruptions from hacking were even less frequent.

As you've might have already noticed, I was working on Weblate whole week and I think it worked pretty well and I've implemented all what I wanted.

First of all, Weblate now supports login using lot of third party services (like GitHub, Facebook, Google, ...). This was achieved by using python-social-auth for that. It is quite new module for this, so hopefully it's API will stay stable enough to be usable in the long term. It was surprisingly easy to implement, though I've spent quite a lot of tweaking of the login and registration process to make it work according to my expectations.

After doing this quite big change, I thought it's about time to restructure the documentation and document new features in it. I think it now covers all important things, but if you can't find something or some parts are hard to understand, just let me know, I'll fix it.

Another quite big feature (though it won't be much visible in upcoming 1.8 release) is source string tracking. This is prerequisite for many features people have requested in Weblate's issue tracker, but these will have to wait for next releases. If you want to see some feature earlier, you can support it by money on Bountysource :-).

Weblate can now also search in all strings, which might become handy if grepping over dozen of Git repositories is not your favorite game.

And last but not least, I've implemented simple Captcha protection for new registrations as the demo server is full of bots who register there and do nothing afterward.

Basically I think this makes Weblate 1.8 feature complete and I'd like to stabilize it in upcoming weeks to release. Right now it is deployed on the demo server, where you can play with it and discover bugs :-). Also it's now time to work on Weblate translations!

First day of Hackweek

First day of hackweek did not really go as planned. I had to spend too much time on tasks which I did not now I will have to do in advance.

I've started with releasing Weblate 1.7. As a part of that I update SUSE Studio images with Weblate. Unfortunately this turned out to be more challenging than usual as with new version the build script always failed but without any obvious error. After some time, I've realized that it just does not show whole output, so the last (and most important) lines are missing.

Then it was easy to spot that the problem is in Whoosh version and it's just matter of providing newer version.

After fixing the image, I could finally focus on Weblate's issue tracker and resolved few issues and questions. I've also improved some of the in application documentation and links to documentation or other resources.

Meanwhile I could also setup translation hosting for monkeysign, happy translating!

Weblate 1.7

Weblate 1.7 has been released today. It comes with lot of improvements, especially in translation speed and many new features.

Full list of changes for 1.7:

  • Please check manual for upgrade instructions.
  • Support for checking Python brace format string.
  • Per subproject customization of quality checks.
  • Detailed per translation stats.
  • Changed way of linking suggestions, checks and comments to units.
  • Users can now add text to commit message.
  • Support for subscribing on new language requests.
  • Support for adding new translations.
  • Widgets and charts are now rendered using Pillow instead of Pango + Cairo.
  • Add status badge widget.
  • Dropped invalid text direction check.
  • Changes in dictionary are now logged in history.
  • Performance improvements for translating view.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far!

Call for Weblate translations

Weblate, a free web-based translation management system, of course also needs to be translated. Now it's right time to complete translation into your language before Weblate 1.7 will be released.

The release is currently planned on Monday, to have clean table before hacking on Weblate during Hackweek. There is not much time left so jump in right now and start translating :-).

Translation status

Weblate free hosting

The amount of projects using Hosted Weblate grows every month so it seems like there is quite interest in that. This growth will however bring some challenges in the future.

Currently everything is managed by myself and that really does not scale well. There should be some automation in the process of adding new projects and there should be more control given to project admins, so that they can change some things on them own. This is quite some coding, but there is another Hackweek on the way, so my plan is to implement at least something in this area as well.

Other side is unsurprisingly money - even though I just got new hardware to run this service (which will be hopefully deployed in a month or so), in the long term it might need other investments as well. That's why I've rewritten the donation page for Weblate and added some more options.

Most importantly the goal is now to get some regular income which would allow sustainable development of both Weblate and free hosting service. I think Gittip is great service for such goal, so let's see how it works.

Roadmap for Weblate 1.7

Weblate 1.6 is already two weeks out without any regressions, so now it's time to think about features for 1.7.

I definitely want to improve support for monolingual translation formats as they currently suffer by various metadata issues (eg. checks not working correctly). This requires quite big change in underlying database, but I think it's quite manageable.

When talking about checks, I want to make them configurable per project as global configuration is quite limiting for shared sites like hosted Weblate.

I will definitely need to touch registration and authentication, either by supporting latest django-registration API or to move some other solution with support for federated login. Which of these approaches will be used depends on my time, what is always the most limiting factor.

This might sound as too few features, but I'd like to make releases more often, so that should fit into this plan. However if you want to push for some feature, you still have two days to fund something on Catincan!

Weblate 1.6 in SUSE Studio

Finally, Weblate 1.6 is available as ready to use appliance in SUSE Studio.

This took quite long as openSUSE 12.3 does not work as expected inside SUSE Studio and I was too lazy to find all necessary tweaks earlier. The same problem actually prevented from Weblate 1.5 being there at all.

The biggest obstacle was MySQL setup - after enabling "Set up MySQL" in studio, you end up with MySQL database with empty mysql.users table, which obviously can not authenticate anybody. As I've found out, the only way around is manual setup.

Once this was figured out, the appliance only needed minor tweaking and is now ready to use in SUSE Gallery in all formats used for virtualization.

Weblate 1.6

Weblate 1.6 has been released today. It comes with lot of improvements, especially adds voting for suggestions, improves import performance and brings more helpful admin interface. This is also first release with crowdfunded feature, but there are still some features to fund!

Full list of changes for 1.6:

  • Nicer error handling on registration.
  • Browsing of changes.
  • Fixed sorting of machine translation suggestions.
  • Improved support for MyMemory machine translation.
  • Added support for Amagama machine translation.
  • Various optimizations on frequently used pages.
  • Highlights searched phrase in search results.
  • Support for automatic fixups while saving the message.
  • Tracking of translation history and option to revert it.
  • Added support for Google Translate API.
  • Added support for managing SSH host keys.
  • Various form validation improvements.
  • Various quality checks improvements.
  • Performance improvements for import.
  • Added support for voting on suggestions.
  • Cleanup of admin interface.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you.

Speed improvements in Weblate

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features.

I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds.

Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds...

With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster.

Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import.

Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds.

In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Weblate has suggestion voting

The first crowdfunded feature for Weblate is here - Voting for suggestions.

Once you enable this for your translation, you can vote for translation suggestions and once they reach defined number of votes, it will get accepted. You can however combine this with other Weblate features to achieve different setups, see our documentation for some other use cases.

This is also most likely last feature for Weblate 1.6, so you can expect release in next week or two in case no major problems appear.

PS: There are still some features to fund.