Michal Čihař - Blog Archives for Weblate

Speed improvements in Weblate

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features.

I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds.

Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds...

With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster.

Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import.

Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds.

In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Weblate has suggestion voting

The first crowdfunded feature for Weblate is here - Voting for suggestions.

Once you enable this for your translation, you can vote for translation suggestions and once they reach defined number of votes, it will get accepted. You can however combine this with other Weblate features to achieve different setups, see our documentation for some other use cases.

This is also most likely last feature for Weblate 1.6, so you can expect release in next week or two in case no major problems appear.

PS: There are still some features to fund.

Fund Weblate features

Weblate is now year and half old, has quite many users and almost all features I wanted it to have. Now it's time to push it further to fit also other projects.

Therefore I've decided to try crowdfunding on Catincan to see which of requested features are actually wanted most. Yes, you can vote with your money :-).

The first set of features to fund is:

This is my first experiment with crowdfunding, so I'm looking forward to it's results.

Weblate 1.5

Weblate 1.5 has been released today. It comes with lot of improvements, especially in performance, reporting and support for machine translations.

Full list of changes for 1.5:

  • Please check manual for upgrade instructions.
  • Added public user pages.
  • Better naming of plural forms.
  • Added support for TBX export of glossary.
  • Added support for Bitbucket notifications.
  • Activity charts are now available for each translation, language or user.
  • Extended options of import_project admin command.
  • Compatible with Django 1.5.
  • Avatars are now shown using libravatar.
  • Added possibility to pretty print JSON export.
  • Various performance improvements.
  • Indicate failing checks or fuzzy strings in progress bars for projects or languages as well.
  • Added support for custom pre-commit hooks and commiting additional files.
  • Rewritten search for better performance and user experience.
  • New interface for machine translations.
  • Added support for monolingual po files.
  • Extend amount of cached metadata to improve speed of various searches.
  • Now shows word counts as well.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery.

Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others.

If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you.

Hackweek 9 is over

Hackweek 9 is over and it's time to share what I've done on Weblate during that.

I think everything went quite well and Weblate is now ready for 1.5 release. I'm slowly deploying it on my installations (unfortunately this release migration will need some noticeable downtime for bigger installations) and everything seems to work fine so far. I believe this is possible thanks to massive test coverage - all important code is covered by testcases.

So what you can expect in 1.5 release? The most visible change is probably new machine translation support, providing support for way more backends and allow you to plug in own services as well. The other changes include word counting (what might give you more idea how much work is remaining) or fancy progress bars in all places (they used to be available for translations only).

From the other side, Weblate can now run custom scripts to pre-process translations before commit, what can be used for various things from generating byte compiled files to sorting or cleaning up the translation files.

Also Weblate should be now much faster - there were dozen of optimizations done, leading to much lower press on database server.

If you want to see more detailed work progress, check Hackweek project page or Weblate changelog.

PS: In case no problems appear, Weblate 1.5 should be released on Sunday.

Weblate and Hackweek 9

You might have already noticed that there is Hackweek 9 coming next week. At SUSE we will get pizzas, icecream and other nice stuff, but most importantly we can spend the week on hacking anything we want.

Same as last year, I want to spend most of my Hackweek on Weblate, nice crowdsourcing tool for translations. The major goal is to finish 1.5 release, what should not be that hard. The most challenging bits for new machine translation interface are already implemented, and the rest is pretty much only tweaking of existing code.

Another thing we want to explore is possibility of using Weblate for openSUSE translations. Currently they are mostly kept in SVN, what is blocker for using Weblate, but we will see what can be done there.

Server troubles

This night you might have noticed that some services like hosted Weblate or phpMyAdmin wiki being quite flaky.

It was caused by DDOS attack which pretty much busted firewall and thus made unable to open new connections. In the morning when I've realized this, I introduced another breakage by too strict filtering on firewall :-). However now everything should be back to normal.

Libravatar support in Weblate

For some time, Weblate was showing avatars for users. Just as I've discovered Libravatar - free and federated alternative to Gravatar, I thought it would be better replacement.

Quickly looking at their website, it seems that they transparently provide all avatars from Gravatar as well, so the migration seems to be pretty much painless. Basic replacement to use their server is just matter of changing base URLs, however to support federated behavior, you have to install pyLibravatar. Weblate in Git now supports both these ways.

While implementing the client side, I did also setup my own instance of Surrogator to provide avatars for some of my domains. Surprisingly this worked fine without problems, but let's see how much used this will actually become.

Weblate with Mercurial or Bazaar

Recently, I've learned about git remote helpers feature in Git, which allows to transparently use Git with other version control systems. As Weblate currently supports only Git, it was quite obvious to give this approach a try.

After some testing, it actually worked just fine - everything works as expected and you can use Weblate with Mercurial or Bazaar with these helpers. Of course there might be some rough edges, but all standard things I've tried worked just fine.

This is now also covered in Weblate's FAQ, which includes basic instructions on how to setup this.

Weblate charts of activity

Weblate recently got another new feature - charting of translation project activity.

It counts and charts daily contributions in last month and weekly contributions in last year and draws nicely looking charts out of that:

Weblate activity chart

Originally I wanted to use some existing charting library and first implementation was actually done using PyCha, but in the end I ended up rendering charts using completely custom code, what allowed me to draw better graphs.