Michal Čihař - Blog Archives for English

Weekly phpMyAdmin contributions 2016-W36

Last week was a bit calmer on my phpMyAdmin contributions as I've spent more time on other free software projects (namely Gammu).

Anyway there was still some amount of reviewing pull requests and bug screening and I've added PHP 7.1 to the testsuite on Travis while fixing some bugs this has revealed (thanks Deven for helping in this as well). Most notable these were bugs in our testsuite feeding invalid mock data in some situations. Without type warnings in upcoming PHP 7.1 these would be further unnoticed.

Handled issues:

Improved Docker container for Weblate

In last days, the Weblate Docker container got several improvements to make it better fit for production setup. All changes are heavily based on pull request by Fred Cox.

What all does this change bring?

  • It now brings complete environment for Weblate including web server.
  • The preferred database engine is now PostgreSQL.
  • There is also memcached instance for better performance.
  • Much more things can be configured using environment variables (machine translation or social auth).

I think this makes Weblate deployment for you much simpler if you already use Docker. I think this can can be further extended to other services like Kubernetes, so patches are welcome :-).

Weekly phpMyAdmin contributions 2016-W35

Last week had still major focus on our Docker container, which is now better than ever :-). There also was a motranslator release to push some of the bug fixes to users.

Other than that there was usual amount of bug screening and fixing which you can see from list of issues below.

Handled issues:

Enca 1.19

Last version of Enca has been released several months and now it's time for new release. There are various bug fixes which have been committed to the Git repository meanwhile.

If you don't know Enca, it is an Extremely Naive Charset Analyser. It detects character set and encoding of text files and can also convert them to other encodings using either a built-in converter or external libraries and tools like libiconv, librecode, or cstocs.

Full list of changes for 1.19 release:

  • fix possible memory leak
  • make utf-8 detection work even on one character

Still enca is in maintenance mode only and I have no intentions to write new features. However there is no limitation to other contributors, join the project at GitHub :-).

You can download from https://cihar.com/software/enca/.

Weblate 2.8

Quite on schedule (just one day later), Weblate 2.7 is out today. This release brings Subversion support or improved zen mode.

Full list of changes:

  • Documentation improvements.
  • Translations.
  • Updated bundled javascript libraries.
  • Added list_translators management command.
  • Django 1.8 is no longer supported.
  • Fixed compatibility with Django 1.10.
  • Added Subversion support.
  • Separated XML validity check from XML mismatched tags.
  • Fixed API to honor HIDE_REPO_CREDENTIALS settings.
  • Show source change in zen mode.
  • Alt+PageUp/PageDown/Home/End now works in zen mode as well.
  • Add tooltip showing exact time of changes.
  • Add option to select filters and search from translation page.
  • Added UI for translation removal.
  • Improved behavior when inserting placeables.
  • Fixed auto locking issues in zen mode.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Weekly phpMyAdmin contributions 2016-W34

Last week was a bit calmer with bigger focus on bug fixing, so that phpMyAdmin 4.6.5 works better thatn 4.6.4, where we managed to screw up some functionality due to too strict hardening fixes.

This includes reporting of too short secret when one was autogenerated, broken browsing of mysql.user or mysql.db tables or breakage when editing ENUM fields. All these are now fixed in QA_4_6.

Another major topic last week was our Docker container, which got heavily improved. To name the most important thing it now uses proper web server (nginx) and PHP FPM instead of PHP built in server, which is not really suitable for production use.

Handled issues:

motranslator 1.1

Four months after 1.0 release, motranslator 1.1 is out. If you happen to use it for untrusted data, this might be as well called security release, though this is still not good idea until we remove usage of eval() used to evaluate plural formula.

Full list of changes:

  • Improved handling of corrupted mo files
  • Minor performance improvements
  • Stricter validation of plural expression

The motranslator is a translation library used in current phpMyAdmin master (upcoming 4.7.0) with focus on speed and memory usage. It uses Gettext MO files to load the translations. It also comes with testsuite (100% coverage) and basic documentation.

Recommended way to install it is using composer from Packagist repository:

composer require phpmyadmin/motranslator

The Debian package will be available probably at point phpMyAdmin 4.7.0 will be out, but if you see need to have it earlier, just let me know.

Improving phpMyAdmin Docker container

Since I've created the phpMyAdmin container for Docker I've always felt strange about using PHP's built in web server there. It really made it poor choice for any production setup and probably was causing lot of problems users saw with this container. During the weekend, I've changed it to use more complex setup with Supervisor, nginx and PHP FPM.

As building this container is one of my first experiences with Docker (together with Weblate container), it was not as straightforward as I'd hope for, but in the end is seems to be working just fine. While touching the code, I've also improved testing of the Docker container to tests all supported setups and to better report in case of test fails.

The nice side effect of this is that the PHP code is no longer being executed under root in the container, so that should make it more sane for production use as well (honestly I never liked this approach that almost everything is executed as root in Docker containers).

CI coverage from Windows, Linux and OSX

Once I got CI working on multiple platforms the obvious next step was to be able to aggregate coverage reports across them. This should not be that hard, right? Well I've spent couple of hours on that during last few days.

On Linux and OSX it was pretty much straightforward. Both GCC and Clang do support coverage, so it's just matter of configuring them properly and collect the coverage reports. I've used own solution for that in past and that was really far from working well (somehow I never managed to get coverage fully uploaded to Codecov). Fortunately there exists CMake script called CMake-codecov which does all needed work and works out of the box on GCC and Clang (even on OSX). Well it works on Travis only once you update the compilers and install llvm-cov tool.

The Windows part on AppVeyor was much harder for me. This can be heavily accounted to lack of my experience with Windows and especially development on Windows in past ten years (probably even more). First challenge was to find something what can generate code coverage there.

After lot of googling I've settled down on OpenCppCoverage what seems to be the only free solution I was able to find. The good thing is that it can generate coverage in Cobertura format that Codecov undestands. There are also bad things that I've learned. First of all it's quite hard to integrate this with CTest. There is no support for wrapping test calls in custom commands, so I've misused the memory checks for that purpose. I've written small python script which pretends the valgrind interface and does call OpenCppCoverage in the background.

Now I had around 800 coverage files (one for each test case) and we need to deal with them somehow. The Codeconv command line client doesn't deal wit this out of the box so the obvious choice was to merge them before upload. There even seems to be script doing that, but unfortunately trying that on our coverage data make it nowhere near completion within hour, so that's not really good choice. Second thing I've tried was merging binary coverage in OpenCppCoverage and then exporting to Cobertura format. Obviously Gammu is special project as all I got from this attempt was crashing OpenCppCoverage (it did merge some of the coverages, but it failed in the end without indicating any error).

In the end I've settled down to uploading files in chunks to Codecov. This seems to work quite okay, though is a bit slow, mostly due to way how Codecov bash uploader prepares data to upload (but this will be hopefully fixed soon).

Anyway the goal has been reached, both Windows and Linux code shows in coverage reports.

Weekly phpMyAdmin contributions 2016-W33

Last week was finally a bit calmer on security issues side so I could look into other issues as well. I finally had time to review some of the pull requests and go through the newly opened issues, but still there is lot of work to do.

The biggest change was that we've launched new website. This was mostly finished some weeks ago, but now it is alive. Hopefully it looks better and cleaner as it was created with all current page content in mind and not continuously adding new things on the way. Anyway it feels a bit strange for me doing designer work when I'm not really good at it. Anyway this is probably fourth version of our website I've done...

One other thing worth mentioning is reintroduction of $cfg['PmaAbsoluteUri'] in upcoming 4.6.5 release. It turns out to be needed in some reverse proxy setups.

Handled issues: