Michal Čihař - Blog Archives for Debian

Hosted Weblate provides also free hosting for free software projects. I'm quite slow in processing the hosting requests, but when I do that, I process them in a batch and add several projects at once.

This time, the newly hosted projects include:

• JaMuz - Music management tool
• World-Weather - An Android app that uses the Open Weather Map API
• GNUSim8085 - A graphical simulator, assembler and debugger for the Intel 8085 microprocesso
• Commons Mobile App - Upload pictures from Android to Wikimedia Commons
• Debian Edu Documentation
• Duplicati - Store securely encrypted backups on cloud storage services

Yesterday, the motranslator 2.0 has been released. As the version change suggests there are some important changes under the hood.

Full list of changes:

• Consistently use camelCase in API
• No more relies on using eval()
• Depends on symfony/expression-language for calculations

As you can see, yesterday announced SimpleMath is not used in the end and I've moved to use existing library. Somehow I misunderstood library description and I thought that it works as PHP, what would be problem for us (or would bring need to add parenthesis around ternary operator as we did with eval()). But this is not the case and ternary operator behaves sane in ExpressionLanguage, so we're good too use it.

Anyway if you were using MoTranslator, it might be good idea to upgrade and check if API changes affect you.

For quite some time we've been relying on using eval() function in phpMyAdmin in two places. One of them is gettext library, where we have to evaluate plural forms and second of them is MySQL configuration advisor, which does it's suggestions based on text file (the original idea was to make this file shared with other tools, but it never really worked out).

Using eval() in PHP is something what is better to avoid, but we were using it on data we ship, so it was considered safe. On the other side, there are hostings which deny using eval() altogether (as many of exploits are using this function), so it's better to avoid that. I've been looking for options for replacing eval() in motranslator (library we use for handling Gettext MO files) for quite some time, but never found library which would support all operators needed in Gettext plural formulas.

Yesterday I finally came to conclusion that writing own library to do this is best approach. This way it can in future extended to work with Advisor as well. Also we can make it pretty lightweight without additional dependencies (what was problem in some existing libraries I've found).

To make the story short, this is how SimpleMath was born. As of now, it has grown to version 0.2 (you can use Packagist to install it). For now it's really simple and it can be probably confused by various strange inputs, but it seems for work pretty well for our case. Currently supported features:

• Supports basic arithmetic operations +, -, *, /, %
• Supports parenthesis
• Supports right associative ternary operator
• Supports comparison operators ==, !=, >, <, >=, <=
• Supports basic logical operations &&, ||
• Supports variables (either PHP style $a or simple n)

Maybe it will be usable for somebody else as well, but even if not, it's the way for us to get rid of using eval() in our codebase.

Update

It seems that Symfony ExpressionLanguage Component is doing pretty much same, but more flexible and faster, so SimpleMath will be probably dead soon and we will switch to using Symphony component.

Stardicter 0.10, the set of scripts to convert some freely available dictionaries to StarDict format, has been released today. There are mostly minor changes and it's time to push them out in official release.

There is one change worth mentioning though - the original site for English - Czech dictionary (http://slovnik.zcu.cz/) has stopped to work and has been moved to https://www.svobodneslovniky.cz/. Hopefully this new location will live at least as long as the original one and will bring back new contributors (honestly the original dictionary gained mostly spam entries in last months). The dictionary data are now hosted in Git repository on GitHub.

wlc 0.6, a command line utility for Weblate, has been just released. There have been some minor fixes, but the most important news is that Windows and OS X are now supported platforms as well.

Full list of changes:

• Fixed error when invoked without command.
• Tested on Windows and OS X (in addition to Linux).

wlc is built on API introduced in Weblate 2.6 and still being in development. Several commands from wlc will not work properly if executed against Weblate 2.6, first fully supported version is 2.7 (it is now running on both demo and hosting servers). You can usage examples in the wlc documentation.

Quite on schedule (just one day later), Weblate 2.7 is out today. This release brings Subversion support or improved zen mode.

Full list of changes:

• Documentation improvements.
• Translations.
• Updated bundled javascript libraries.
• Added list_translators management command.
• Django 1.8 is no longer supported.
• Fixed compatibility with Django 1.10.
• Added Subversion support.
• Separated XML validity check from XML mismatched tags.
• Fixed API to honor HIDE_REPO_CREDENTIALS settings.
• Show source change in zen mode.
• Alt+PageUp/PageDown/Home/End now works in zen mode as well.
• Add tooltip showing exact time of changes.
• Add option to select filters and search from translation page.
• Added UI for translation removal.
• Improved behavior when inserting placeables.
• Fixed auto locking issues in zen mode.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Four months after 1.0 release, motranslator 1.1 is out. If you happen to use it for untrusted data, this might be as well called security release, though this is still not good idea until we remove usage of eval() used to evaluate plural formula.

Full list of changes:

• Improved handling of corrupted mo files
• Minor performance improvements
• Stricter validation of plural expression

The motranslator is a translation library used in current phpMyAdmin master (upcoming 4.7.0) with focus on speed and memory usage. It uses Gettext MO files to load the translations. It also comes with testsuite (100% coverage) and basic documentation.

Recommended way to install it is using composer from Packagist repository:

composer require phpmyadmin/motranslator

The Debian package will be available probably at point phpMyAdmin 4.7.0 will be out, but if you see need to have it earlier, just let me know.

Since I've created the phpMyAdmin container for Docker I've always felt strange about using PHP's built in web server there. It really made it poor choice for any production setup and probably was causing lot of problems users saw with this container. During the weekend, I've changed it to use more complex setup with Supervisor, nginx and PHP FPM.

As building this container is one of my first experiences with Docker (together with Weblate container), it was not as straightforward as I'd hope for, but in the end is seems to be working just fine. While touching the code, I've also improved testing of the Docker container to tests all supported setups and to better report in case of test fails.

The nice side effect of this is that the PHP code is no longer being executed under root in the container, so that should make it more sane for production use as well (honestly I never liked this approach that almost everything is executed as root in Docker containers).

Once I got CI working on multiple platforms the obvious next step was to be able to aggregate coverage reports across them. This should not be that hard, right? Well I've spent couple of hours on that during last few days.

On Linux and OSX it was pretty much straightforward. Both GCC and Clang do support coverage, so it's just matter of configuring them properly and collect the coverage reports. I've used own solution for that in past and that was really far from working well (somehow I never managed to get coverage fully uploaded to Codecov). Fortunately there exists CMake script called CMake-codecov which does all needed work and works out of the box on GCC and Clang (even on OSX). Well it works on Travis only once you update the compilers and install llvm-cov tool.

The Windows part on AppVeyor was much harder for me. This can be heavily accounted to lack of my experience with Windows and especially development on Windows in past ten years (probably even more). First challenge was to find something what can generate code coverage there.

After lot of googling I've settled down on OpenCppCoverage what seems to be the only free solution I was able to find. The good thing is that it can generate coverage in Cobertura format that Codecov undestands. There are also bad things that I've learned. First of all it's quite hard to integrate this with CTest. There is no support for wrapping test calls in custom commands, so I've misused the memory checks for that purpose. I've written small python script which pretends the valgrind interface and does call OpenCppCoverage in the background.

Now I had around 800 coverage files (one for each test case) and we need to deal with them somehow. The Codeconv command line client doesn't deal wit this out of the box so the obvious choice was to merge them before upload. There even seems to be script doing that, but unfortunately trying that on our coverage data make it nowhere near completion within hour, so that's not really good choice. Second thing I've tried was merging binary coverage in OpenCppCoverage and then exporting to Cobertura format. Obviously Gammu is special project as all I got from this attempt was crashing OpenCppCoverage (it did merge some of the coverages, but it failed in the end without indicating any error).

In the end I've settled down to uploading files in chunks to Codecov. This seems to work quite okay, though is a bit slow, mostly due to way how Codecov bash uploader prepares data to upload (but this will be hopefully fixed soon).

Anyway the goal has been reached, both Windows and Linux code shows in coverage reports.

Over the weekend I've played with continuous integration for Gammu to make it run on more platforms. I had to remember many things from the Windows world on the way and the solution is not yet complete, but the basic build is working, the only problematic part are external dependencies.

First of all we already have Linux builds on Travis CI. These cover compilation with both GCC and Clang compilers, hopefully covering most of the possible problems.

Recently I've added OS X builds on Travis CI, what was pretty much painless and worked out of the box.

The next major architecture to support is Windows. Once I've discovered AppVeyor I thought it might be the way to go. The have free plans for open-source projects (though it has only one parallel build compared to four provided by Travis CI).

As our build system is cross platform based on CMake, it should work pretty much out of the box, right? Well almost, tweaking the basics took some time (unfortunately there is no CMake support on AppVeyor, so you have to script it a bit).

The most painful things on the way:

• finding our correct way to invoke build and testsuite
• our code was broken on Windows, making the testsuite to fail
• how to work with power shell (no, I'm not going to like it)
• how to download and install executable to PATH
• test output integration with AppVeyor - done using XSLT transformation and uploading test results manually
• 32-bit / 64-bit mess, CMake happily finds 32-bit libs during the 64-bit build and vice versa, what makes the build fail later when linking - fixed by trying if code can be built with given library
• 64-bit code crashes in dummy driver, causing testsuite failures (this has to be something Windows specific as the code works fine on 64-bit Linux) - this seems to be caused by too big allocations on stack, moving them to heap will fix this

You can check our current appveyor.yml in case you're going to try something similar. Current build results are on AppVeyor.

As a nice side effect, we now have up to date Windows binaries for Gammu.