Michal Čihař - Blog Archives for Debian

Improving phpMyAdmin Docker container

Since I've created the phpMyAdmin container for Docker I've always felt strange about using PHP's built in web server there. It really made it poor choice for any production setup and probably was causing lot of problems users saw with this container. During the weekend, I've changed it to use more complex setup with Supervisor, nginx and PHP FPM.

As building this container is one of my first experiences with Docker (together with Weblate container), it was not as straightforward as I'd hope for, but in the end is seems to be working just fine. While touching the code, I've also improved testing of the Docker container to tests all supported setups and to better report in case of test fails.

The nice side effect of this is that the PHP code is no longer being executed under root in the container, so that should make it more sane for production use as well (honestly I never liked this approach that almost everything is executed as root in Docker containers).

CI coverage from Windows, Linux and OSX

Once I got CI working on multiple platforms the obvious next step was to be able to aggregate coverage reports across them. This should not be that hard, right? Well I've spent couple of hours on that during last few days.

On Linux and OSX it was pretty much straightforward. Both GCC and Clang do support coverage, so it's just matter of configuring them properly and collect the coverage reports. I've used own solution for that in past and that was really far from working well (somehow I never managed to get coverage fully uploaded to Codecov). Fortunately there exists CMake script called CMake-codecov which does all needed work and works out of the box on GCC and Clang (even on OSX). Well it works on Travis only once you update the compilers and install llvm-cov tool.

The Windows part on AppVeyor was much harder for me. This can be heavily accounted to lack of my experience with Windows and especially development on Windows in past ten years (probably even more). First challenge was to find something what can generate code coverage there.

After lot of googling I've settled down on OpenCppCoverage what seems to be the only free solution I was able to find. The good thing is that it can generate coverage in Cobertura format that Codecov undestands. There are also bad things that I've learned. First of all it's quite hard to integrate this with CTest. There is no support for wrapping test calls in custom commands, so I've misused the memory checks for that purpose. I've written small python script which pretends the valgrind interface and does call OpenCppCoverage in the background.

Now I had around 800 coverage files (one for each test case) and we need to deal with them somehow. The Codeconv command line client doesn't deal wit this out of the box so the obvious choice was to merge them before upload. There even seems to be script doing that, but unfortunately trying that on our coverage data make it nowhere near completion within hour, so that's not really good choice. Second thing I've tried was merging binary coverage in OpenCppCoverage and then exporting to Cobertura format. Obviously Gammu is special project as all I got from this attempt was crashing OpenCppCoverage (it did merge some of the coverages, but it failed in the end without indicating any error).

In the end I've settled down to uploading files in chunks to Codecov. This seems to work quite okay, though is a bit slow, mostly due to way how Codecov bash uploader prepares data to upload (but this will be hopefully fixed soon).

Anyway the goal has been reached, both Windows and Linux code shows in coverage reports.

Continuous integration on multiple platforms

Over the weekend I've played with continuous integration for Gammu to make it run on more platforms. I had to remember many things from the Windows world on the way and the solution is not yet complete, but the basic build is working, the only problematic part are external dependencies.

First of all we already have Linux builds on Travis CI. These cover compilation with both GCC and Clang compilers, hopefully covering most of the possible problems.

Recently I've added OS X builds on Travis CI, what was pretty much painless and worked out of the box.

The next major architecture to support is Windows. Once I've discovered AppVeyor I thought it might be the way to go. The have free plans for open-source projects (though it has only one parallel build compared to four provided by Travis CI).

As our build system is cross platform based on CMake, it should work pretty much out of the box, right? Well almost, tweaking the basics took some time (unfortunately there is no CMake support on AppVeyor, so you have to script it a bit).

The most painful things on the way:

  • finding our correct way to invoke build and testsuite
  • our code was broken on Windows, making the testsuite to fail
  • how to work with power shell (no, I'm not going to like it)
  • how to download and install executable to PATH
  • test output integration with AppVeyor - done using XSLT transformation and uploading test results manually
  • 32-bit / 64-bit mess, CMake happily finds 32-bit libs during the 64-bit build and vice versa, what makes the build fail later when linking - fixed by trying if code can be built with given library
  • 64-bit code crashes in dummy driver, causing testsuite failures (this has to be something Windows specific as the code works fine on 64-bit Linux) - this seems to be caused by too big allocations on stack, moving them to heap will fix this

You can check our current appveyor.yml in case you're going to try something similar. Current build results are on AppVeyor.

As a nice side effect, we now have up to date Windows binaries for Gammu.

Wammu 0.42

Yesterday, I've released Wammu 0.42. There are no major updates, more likely it's usual localization and minor bugfixes release.

As usual up to date packages are now available in Debian sid, Gammu PPA for Ubuntu or openSUSE buildservice for various RPM based distros.

Want to support further Wammu development? Check our donation options or support Gammu team on BountySource Salt.

Weekly phpMyAdmin contributions 2016-W32

Tonight phpMyAdmin 4.0.10.17, 4.4.15.8, and 4.6.4 were released and you can probably see that there are quite some security issues fixed. Most of them are not really exploitable unless your PHP and webserver are poorly configured, but still it's good idea to upgrade.

If you are running Debian unstable, use our phpMyAdmin PPA for Ubuntu or use phpMyAdmin Docker image upgrading should be as simple as pulling new version.

Besides fixing security issues, we're generally hardening our infrastructure. I'm really grateful that Emanuel Bronshtein (@e3amn2l) is doing great review of all of our code and helps us in this area. This will really make our code and infrastructure much better.

Handled issues:

Revoking old PGP key

It has been already six years since I've moved to using RSA4096 PGP key. For various reasons, the old DSA key was still kept valid till today. This is no longer true and it has been revoked now.

The revoked key is DC3552E836E75604 and new one is 9C27B31342B7511D. In case you've signed the old one and not the new one (quite unlikely if you did not sign it more than six years ago), there has been migration document, where you can verify my new key being signed by the old one.

Gammu 1.37.4

It has been almost three months since last Gammu release and it's time to push fixes out to users. This time the amount of fixes is quite small, covering Huawei devices and text mode for sending SMS.

Full list of changes in 1.37.4:

  • Improved support for Huawei E3131.
  • Fixed SMS support for MULTIBAND 900E.
  • Fixed SMS created in text mode.

Would you like to see more features in Gammu? You an support further Gammu development at Bountysource salt or by direct donation.

New projects on Hosted Weblate

For almost two months I found very little time to process requests to host free software on Hosted Weblate. Today the queue has been emptied, what means that you can find many new translations there.

To make it short, here is list of new projects:

PS: If you didn't receive reply for your hosting request today, it was probably lost, so don't hesitate to ask again.

Weblate 2.7

Slightly later than on monthly schedule but Weblate 2.7 is out today. This release brings improvements to the API and is first to officially support wlc a command line client for Weblate.

Full list of changes for 2.7:

  • Removed Google web translate machine translation.
  • Improved commit message when adding translation.
  • Fixed Google Translate API for Hebrew language.
  • Compatibility with Mercurial 3.8.
  • Added import_json management command.
  • Correct ordering of listed traslations.
  • Show full suggestion text, not only a diff.
  • Extend API (detailed repository status, statistics, ...).
  • Testsuite no longer requires network access to test repositories.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

wlc 0.4

wlc 0.4, a command line utility for Weblate, has been just released. This release doesn't bring much changes, but still worth announcing.

The most important change is that development repository has been moved under WeblateOrg organization at GitHub, you can now find it at https://github.com/WeblateOrg/wlc. Another important news is that Debian package is currently waiting in NEW queue and will hopefully soon hit unstable.

wlc is built on API introduced in Weblate 2.6 and still being in development. Several commands from wlc will not work properly if executed against Weblate 2.6, first fully supported version will be 2.7 (current git is okay as well, it is now running on both demo and hosting servers). You can usage examples in the wlc documentation.