I'm reporting two weeks at once as I had several days of and there was not that much of work done. Pretty much everything was bug screening, relaxing after previous security fixes and starting with new round of them.
Since quite a long time phpMyAdmin had embedded the bfShapeFiles library for import of geospatial data. Over the time we had to apply fixes to it to stay compatible with newer PHP versions, but there was really no development. Unfortunately, as it seems to be only usable PHP library which can read and write ESRI shapefiles.
With recent switch of phpMyAdmin to dependency handling using Composer I wondered if we should get rid of the last embedded PHP library, which was this one - bfShapeFiles. As I couldn't find alive library which would work well for us, I resisted that for quite long, until pull request to improve it came in. At that point I've realized that it's probably better to separate it and start to improve it outside our codebase.
That's when phpmyadmin/shapefile was started. The code is based on bfShapeFiles, applies all fixes which were used in phpMyAdmin and adds improvements from the pull request. On top of that it has brand new testsuite (the coverage is still much lower than I'd like to have) and while writing the tests several parsing issues have been discovered and fixed. Anyway you can now get the source from GitHub or install using Composer from Packagist.
PS: While fixing parser bugs I've looked at other parsers as well to see how they handle some situations unclear in the specs and I had to fix Python pyshp on the way as well :-).
As you could see from the release news it has been quite busy week in terms of fixing security issues. It has actually started just after announcement of security audit funded by Mozilla SOS Fund. It seems this is best way to attract attention security reviewers and we got really a lot of it.
So most of work in last two weeks was to deal with incoming security reports. Fortunately there is still nothing critical if you are not using ancient unpatched PHP version which is vulnerable to null termination of strings. This was quite hard work as immediately once we started to think about releasing version with fixes, new report came in and the process repeated several times. Fortunately we've made it to do three security releases (one for each supported branch) and it seems that we've not broken anything (at least there is no bug report indicating that).
Let's see what next weeks bring and how much security work will be there, but we definitely should focus on doing some reviews continuously rather than doing such one off actions.
On the other side in terms of handled public issues this week was really low volume:
Last week was again focused on code cleanup. The biggest part is splitting up the shapefile library out of our codebase. It's original upstream is not active for years and people started to use the library from our code instead, so separating it makes perfect sense.
While working on that, the library got some basic tests, but I'm still looking for more complex testcases to cover even situation we do not use in phpMyAdmin.
Besides this, there were some bug fixes in phpMyAdmin itself and it's Docker container. Additionally here was quite some security work after we've published information about passed security audit, but that will be described later.
Last week was a bit more focused on improving our Docker container. It's still not perfect, but it works way better than before. I'm also learning Docker on the way, so the progress is not as fast as it could be.
When speaking about learning I've again learned some new things about PHP - this time it was fact that the debug_backtrace function returns reference to actual interpreters backtrace, so if you change something there, you change the parameters in the code above in the stack. It was quite hard to figure out, but fortunately easy to fix afterwards. Anyway if you have not matching library and PHP MySQL module, you could not connect to MySQL server with phpMyAdmin because of this.
Rest of work was regular bug screening and fixing, nothing really outstanding.
Last week was a bit relaxed for me as I had few days off, so the amount of work was also quite limited.
Quite a lot of time was spent on investigating issue #12243, which in the end turned out to be problem in Fedora packaging as it's using outdated SQL parser library, which contains many bugs which have been fixed meanwhile. This is now reported in their bug tracker and hopefully get fixed soon. Anyway if you're running phpMyAdmin from Fedora / EPEL packages, you might be bitten by various bugs which are already fixed upstream.
Also if you're looking for free software job, you can join me in working on phpMyAdmin, we're looking for second developer!
Last week I found time to dig into some ancient issues and managed to fix them. There was some security work as well as we've managed to issue 3 security announcements (nothing really important, but still worth of fixing).
There was usual amount of bug fixing as well, but I'd say there is nothing noteworthy in the bugs, just there was quite a lot of them :-).
Another week is over and it's time to report contributions from it. Issue wise not much things were fixed, but quite a lot of time was spent on reviewing reported issues.
Another continued effort was migrating content from wiki to our documentation. Unfortunately this is also not yet completed as some of the documents have diverged quite a lot and integrating them back is not as straightforward as I'd like it to be.
It is built on API introduced in Weblate 2.6 and still being in development. Several commands from wlc will not work properly if executed against Weblate 2.6, first fully supported version will be 2.7 (current git is okay as well, it is now running on both demo and hosting servers).
How to use it? First you will probably want to store the credentials, so that your requests are authenticated (you can do unauthenticated requests as well, but obviously only read only and on public objects), so lets create
[weblate] url = https://hosted.weblate.org/api/ [keys] https://hosted.weblate.org/api/ = APIKEY
Now you can do basic commands:
$ wlc show weblate/master/cs ... last_author: Michal Čihař last_change: 2016-05-13T15:59:25 revision: 62f038bb0bfe360494fb8dee30fd9d34133a8663 share_url: https://hosted.weblate.org/engage/weblate/cs/ total: 1361 total_words: 6144 translate_url: https://hosted.weblate.org/translate/weblate/master/cs/ translated: 1361 translated_percent: 100.0 translated_words: 6144 url: https://hosted.weblate.org/api/translations/weblate/master/cs/ web_url: https://hosted.weblate.org/projects/weblate/master/cs/
You can find more examples in wlc documentation.
After week of vacation I got back to work. There was lot of bug screening this week, it seems that people are finally migrating to 4.6 and discovering new problems there. Fortunately 4.6.1 has been released fixing most annoying issues.
Several issues were quite tricky to debug issue happening solely on Windows servers and looked quite tricky from beginning (breaking utf-8 chars). It turned out to be caused by preg_replace calls on the content, which could break utf-8 chars in some cases.