Michal Čihař - Archive for 8/2016

Quite on schedule (just one day later), Weblate 2.7 is out today. This release brings Subversion support or improved zen mode.

Full list of changes:

• Documentation improvements.
• Translations.
• Updated bundled javascript libraries.
• Added list_translators management command.
• Django 1.8 is no longer supported.
• Fixed compatibility with Django 1.10.
• Added Subversion support.
• Separated XML validity check from XML mismatched tags.
• Fixed API to honor HIDE_REPO_CREDENTIALS settings.
• Show source change in zen mode.
• Alt+PageUp/PageDown/Home/End now works in zen mode as well.
• Add tooltip showing exact time of changes.
• Add option to select filters and search from translation page.
• Added UI for translation removal.
• Improved behavior when inserting placeables.
• Fixed auto locking issues in zen mode.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Last week was a bit calmer with bigger focus on bug fixing, so that phpMyAdmin 4.6.5 works better thatn 4.6.4, where we managed to screw up some functionality due to too strict hardening fixes.

This includes reporting of too short secret when one was autogenerated, broken browsing of mysql.user or mysql.db tables or breakage when editing ENUM fields. All these are now fixed in QA_4_6.

Another major topic last week was our Docker container, which got heavily improved. To name the most important thing it now uses proper web server (nginx) and PHP FPM instead of PHP built in server, which is not really suitable for production use.

Handled issues:

• #12493 Allow CSP headers to be disabled
• #12480 Tablecolumns Length/Values can't be changed
• #12496 Trying to create enum field gives error and enum is 'enum(0)'
• #12251 Logged out at random
• #12485 Error: The secret passphrase in configuration (blowfish_secret) is too short.
• #12479 Share code for URL validation
• #12483 mysql.user and mysql.db show code instead of usernames

Four months after 1.0 release, motranslator 1.1 is out. If you happen to use it for untrusted data, this might be as well called security release, though this is still not good idea until we remove usage of eval() used to evaluate plural formula.

Full list of changes:

• Improved handling of corrupted mo files
• Minor performance improvements
• Stricter validation of plural expression

The motranslator is a translation library used in current phpMyAdmin master (upcoming 4.7.0) with focus on speed and memory usage. It uses Gettext MO files to load the translations. It also comes with testsuite (100% coverage) and basic documentation.

Recommended way to install it is using composer from Packagist repository:

composer require phpmyadmin/motranslator

The Debian package will be available probably at point phpMyAdmin 4.7.0 will be out, but if you see need to have it earlier, just let me know.

Since I've created the phpMyAdmin container for Docker I've always felt strange about using PHP's built in web server there. It really made it poor choice for any production setup and probably was causing lot of problems users saw with this container. During the weekend, I've changed it to use more complex setup with Supervisor, nginx and PHP FPM.

As building this container is one of my first experiences with Docker (together with Weblate container), it was not as straightforward as I'd hope for, but in the end is seems to be working just fine. While touching the code, I've also improved testing of the Docker container to tests all supported setups and to better report in case of test fails.

The nice side effect of this is that the PHP code is no longer being executed under root in the container, so that should make it more sane for production use as well (honestly I never liked this approach that almost everything is executed as root in Docker containers).

Once I got CI working on multiple platforms the obvious next step was to be able to aggregate coverage reports across them. This should not be that hard, right? Well I've spent couple of hours on that during last few days.

On Linux and OSX it was pretty much straightforward. Both GCC and Clang do support coverage, so it's just matter of configuring them properly and collect the coverage reports. I've used own solution for that in past and that was really far from working well (somehow I never managed to get coverage fully uploaded to Codecov). Fortunately there exists CMake script called CMake-codecov which does all needed work and works out of the box on GCC and Clang (even on OSX). Well it works on Travis only once you update the compilers and install llvm-cov tool.

The Windows part on AppVeyor was much harder for me. This can be heavily accounted to lack of my experience with Windows and especially development on Windows in past ten years (probably even more). First challenge was to find something what can generate code coverage there.

After lot of googling I've settled down on OpenCppCoverage what seems to be the only free solution I was able to find. The good thing is that it can generate coverage in Cobertura format that Codecov undestands. There are also bad things that I've learned. First of all it's quite hard to integrate this with CTest. There is no support for wrapping test calls in custom commands, so I've misused the memory checks for that purpose. I've written small python script which pretends the valgrind interface and does call OpenCppCoverage in the background.

Now I had around 800 coverage files (one for each test case) and we need to deal with them somehow. The Codeconv command line client doesn't deal wit this out of the box so the obvious choice was to merge them before upload. There even seems to be script doing that, but unfortunately trying that on our coverage data make it nowhere near completion within hour, so that's not really good choice. Second thing I've tried was merging binary coverage in OpenCppCoverage and then exporting to Cobertura format. Obviously Gammu is special project as all I got from this attempt was crashing OpenCppCoverage (it did merge some of the coverages, but it failed in the end without indicating any error).

In the end I've settled down to uploading files in chunks to Codecov. This seems to work quite okay, though is a bit slow, mostly due to way how Codecov bash uploader prepares data to upload (but this will be hopefully fixed soon).

Anyway the goal has been reached, both Windows and Linux code shows in coverage reports.

Last week was finally a bit calmer on security issues side so I could look into other issues as well. I finally had time to review some of the pull requests and go through the newly opened issues, but still there is lot of work to do.

The biggest change was that we've launched new website. This was mostly finished some weeks ago, but now it is alive. Hopefully it looks better and cleaner as it was created with all current page content in mind and not continuously adding new things on the way. Anyway it feels a bit strange for me doing designer work when I'm not really good at it. Anyway this is probably fourth version of our website I've done...

One other thing worth mentioning is reintroduction of $cfg['PmaAbsoluteUri'] in upcoming 4.6.5 release. It turns out to be needed in some reverse proxy setups.

Handled issues:

• #12471 use of != for checking if password equal to repeated password
• #12374 Reverse Proxy without PmaAbsoluteUri?
• #12457 Fixes #12380 Refactored http requests into one function "httpRequest" in Util.php
• #12333 Fixes #12330 - Shortcut for closing console
• #12430 Partitions editor: Allow to edit partition names; Add 'COLUMNS Partitioning' support.
• #12458 Translation improvements
• #12459 When editing table records, display readonly inputs for fields without write access.
• #12462 Reload Navigation if no previous entry stored
• #12384 Left panel not expanding when opening PMA DB by direct link
• #12429 git pull origin STABLE - reduce traffic
• #12460 Revert #12274
• #40 PMA_ABSOLUTE_URI not working anymore
• #47 Unable to nagivate
• #19 Upgrade embedded libraries
• #26 Download popup
• #17 Suggesting a donation on download
• #22 Bootstrap version of website

Over the weekend I've played with continuous integration for Gammu to make it run on more platforms. I had to remember many things from the Windows world on the way and the solution is not yet complete, but the basic build is working, the only problematic part are external dependencies.

First of all we already have Linux builds on Travis CI. These cover compilation with both GCC and Clang compilers, hopefully covering most of the possible problems.

Recently I've added OS X builds on Travis CI, what was pretty much painless and worked out of the box.

The next major architecture to support is Windows. Once I've discovered AppVeyor I thought it might be the way to go. The have free plans for open-source projects (though it has only one parallel build compared to four provided by Travis CI).

As our build system is cross platform based on CMake, it should work pretty much out of the box, right? Well almost, tweaking the basics took some time (unfortunately there is no CMake support on AppVeyor, so you have to script it a bit).

The most painful things on the way:

• finding our correct way to invoke build and testsuite
• our code was broken on Windows, making the testsuite to fail
• how to work with power shell (no, I'm not going to like it)
• how to download and install executable to PATH
• test output integration with AppVeyor - done using XSLT transformation and uploading test results manually
• 32-bit / 64-bit mess, CMake happily finds 32-bit libs during the 64-bit build and vice versa, what makes the build fail later when linking - fixed by trying if code can be built with given library
• 64-bit code crashes in dummy driver, causing testsuite failures (this has to be something Windows specific as the code works fine on 64-bit Linux) - this seems to be caused by too big allocations on stack, moving them to heap will fix this

You can check our current appveyor.yml in case you're going to try something similar. Current build results are on AppVeyor.

As a nice side effect, we now have up to date Windows binaries for Gammu.

Yesterday, I've released Wammu 0.42. There are no major updates, more likely it's usual localization and minor bugfixes release.

• Improved AppData metadata.
• Added ttyHS ports to selection.
• Localization updates.

As usual up to date packages are now available in Debian sid, Gammu PPA for Ubuntu or openSUSE buildservice for various RPM based distros.

Want to support further Wammu development? Check our donation options or support Gammu team on BountySource Salt.

Tonight phpMyAdmin 4.0.10.17, 4.4.15.8, and 4.6.4 were released and you can probably see that there are quite some security issues fixed. Most of them are not really exploitable unless your PHP and webserver are poorly configured, but still it's good idea to upgrade.

If you are running Debian unstable, use our phpMyAdmin PPA for Ubuntu or use phpMyAdmin Docker image upgrading should be as simple as pulling new version.

Besides fixing security issues, we're generally hardening our infrastructure. I'm really grateful that Emanuel Bronshtein (@e3amn2l) is doing great review of all of our code and helps us in this area. This will really make our code and infrastructure much better.

Handled issues:

• #12448 Avoid using Short-Keys in GPG
• #12446 Not listing 4th column in table 'dos'
• #43 Verify downloaded sources
• #46 Specify version in variable
• #38 Cannot change configuration options
• #41 PMA DB/Configuration Storage Missing from Docker Container Capability
• #45 Verify downloaded tarball using PGP
• #42 config.userdef.inc.php hook to allow user configured settings.
• #44 User configurattion

It has been already six years since I've moved to using RSA4096 PGP key. For various reasons, the old DSA key was still kept valid till today. This is no longer true and it has been revoked now.

The revoked key is DC3552E836E75604 and new one is 9C27B31342B7511D. In case you've signed the old one and not the new one (quite unlikely if you did not sign it more than six years ago), there has been migration document, where you can verify my new key being signed by the old one.