Michal Čihař - Blog Archives for English

Weekly phpMyAdmin contributions 2017-W22

Last week was quite equally spread across our projects. I've spent time on motranslator, sql-parser, localized documentation and of course phpMyAdmin itself.

Generally it was mostly focused on going through issue trackers and pull requests to see what has happened in week I was offline.

Handled issues:

Weekly phpMyAdmin contributions 2017-W21

Last week was a bit shorter for me, but still there was quite some bugfixes done.

Most of the time was spent of fixing handling malformed mo files in motranslator, where the issues were reported by Emanuel Bronshtein. The library is now way more fault tolerant than it used to be if it gets corrupted file.

There was also quite some pull requests on phpMyAdmin to review and merge, but that seems to be usual in last weeks :-).

Handled issues:

Running Bitcoin node on Turris Omnia

For quite some I'm happy user of Turris Omnia router. The router has quite good hardware, so I've decided to try if I can run Bitcoin node on that and ElectrumX server.

To make the things easier to manage, I've decided to use LXC and run all these in separate container. First of all you need LXC on the router. This is the default setup, but in case you've removed it, you can add it back in the Updater settings.

Now we will create Debian container. There is basic information on using in Turris Documentation on how to create the container, in latter documentation I assume it is called debian.

It's also good idea to enable LXC autostart, to do so add your container to cat /etc/config/lxc-auto on :

config container
    option name debian

You might also want to edit lxc container configration to enable clean shutdown:

# Send SIGRTMIN+3 to shutdown systemd (37 on Turris Omnia)
lxc.haltsignal = SIGRTMIN+3

To make the system more recent, I've decided to use Debian Stretch (one of reasons was that ElectrumX needs Python 3.5.3 or newer). Which is anyway probably sane choice right now given that it's already frozen and will be soon stable. As Stretch is not available as a download option in Omnia, I've chosen to use Debian Jessie and upgrate it later:

$ lxc-attach  --name debian
$ sed -i s/jessie/stretch/ /etc/apt/sources.list
$ apt update
$ apt full-upgrade

Now you have up to date system and we can start installing dependencies. First thing to install is Bitcoin Core. Just follow the instructions on their website to do that. Now it's time to set it up and wait for downloading full blockchain:

$ adduser bitcoin
$ su - bitcoin
$ bitcoind -daemon

Depending on your connection speed, the download will take few hours. You can monitor the progress using bitcoin-cli, you're waiting for 450k blocks:

$ bitcoin-cli getinfo
{
  "version": 140000,
  "protocolversion": 70015,
  "walletversion": 130000,
  "balance": 0.00000000,
  "blocks": 301242,
  "timeoffset": -1,
  "connections": 8,
  "proxy": "",
  "difficulty": 8853416309.1278,
  "testnet": false,
  "keypoololdest": 1490267950,
  "keypoolsize": 100,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "errors": ""
}

Depending how much memory you have (mine has 2G) and what all you run on the router, you will have to tweak bitcoind configuration to consume less memory. This can be done by editing .bitcoin/bitcoin.conf, I've ended up with following settings:

par=1
dbcache=150
maxmempool=150

You can also create startup unit for Bitcoin daemon (place that as /etc/systemd/system/bitcoind.service):

[Unit]
Description=Bitcoind
After=network.target

[Service]
ExecStart=/opt/bitcoin/bin/bitcoind
User=bitcoin
TimeoutStopSec=30min
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Now we can enable services to start on container start:

systemctl enable bitcoind.service

Then I wanted to setup ElectrumX as well, but I've quickly realized that it uses way more memory that my router has, so there is no option to run it without using swap, what will probably make it quite slow (I haven't tried that).

Weblate 2.14.1

Weblate 2.14.1 has been released today. It is bugfix release fixing possible migration issues, search results navigation and some minor security issues.

Full list of changes:

  • Fixed possible error when paginating search results.
  • Fixed migrations from older versions in some corner cases.
  • Fixed possible CSRF on project watch and unwatch.
  • The password reset no longer authenticates user.
  • Fixed possible captcha bypass on forgotten password.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Weekly phpMyAdmin contributions 2017-W20

Last week I was again quite active on development side bringing several improvements to master branch.

The biggest news is probably that phpMyAdmin no longer relies on eval() function. We've used it to run advisory rules on server configuration, but that is now done using Symfony ExpressionLanguage (which we anyway need due to motranslator).

When looking at things this does pull in, I've noticed that there is mbstring polyfill, which can be used instead of the one we ship (and was never completed). Thanks to this the mbstring dependency is now optional, but still recommended for performance reasons.

Another quite visible change is adding JSON metadata to our themes. Right now it covers basic things like theme compatibility and authorship, but more can be added later. This is also covered in our documentation.

Handled issues:

HackerOne experience with Weblate

Weblate has started to use HackerOne Community Edition some time ago and I think it's good to share my experience with that. Do you have open source project and want to get more attention of security community? This post will answer how it looks from perspective of pretty small project.

I've applied with Weblate to HackerOne Community Edition by end of March and it was approved early in April. Based on their recommendations I've started in invite only mode, but that really didn't bring much attention (exactly none reports), so I've decided to go public.

I've asked for making the project public just after coming from two weeks vacation, while expecting the approval to take some time where I'll settle down things which have popped up during vacation. In the end that was approved within single day, so I was immediately under fire of incoming reports:

Reports on HackerOne

I was surprised that they didn't lie - you will really get huge amount of issues just after making your project public. Most of them were quite simple and repeating (as you can see from number of duplicates), but it really provided valuable input.

Even more surprisingly there was second peak coming in when I've started to disclose resolved issues (once Weblate 2.14 has been released).

Overall the issues could be divided to few groups:

  • Server configuration such as lack of Content-Security-Policy headers. This is certainly good security practice and we really didn't follow it in all cases. The situation should be way better now.
  • Lack or rate limiting in Weblate. We really didn't try to do that and many reporters (correctly) shown that this is something what should be addressed in important entry points such as authentication. Weblate 2.14 has brought lot of features in this area.
  • Not using https where applicable. Yes, some APIs or web sites did not support https in past, but now they do and I didn't notice.
  • Several pages were vulnerable to CSRF as they were using GET while POST with CSRF protection would be more appropriate.
  • Lack of password strength validation. I've incorporated Django password validation to Weblate hopefully avoiding the weakest passwords.
  • Several issues in authentication using Python Social Auth. I've never really looked at how the authentication works there and there are some questionable decisions or bugs. Some of the bugs were already addressed in current releases, but there are still some to solve.

In the end it was really challenging week to be able to cope with the incoming reports, but I think I've managed it quite well. The HackerOne metrics states that there are 2 hours in average to respond on incoming incidents, what I think will not work in the long term :-).

Anyway thanks to this, you can now enjoy Weblate 2.14 which more secure than any release before, if you have not yet upgraded, you might consider doing that now or look into our support offering for self hosted Weblate.

The downside of this all was that the initial publishing on HackerOne made our website target of lot of automated tools and the web server was not really ready for that. I'm really sorry to all Hosted Weblate users who were affected by this. This has been also addressed now, but the infrastructure really should have been prepared before on this. To share how it looked like, here is number of requests to the nginx server:

nxing requests

I'm really glad I could make Weblate available on HackerOne as it will clearly improve it's security and security of hosted offering we have. I will certainly consider providing swag and/or bounties on further severe reports, but that won't be possible without enough funding for Weblate.

phpMyAdmin available in Packagist

For quite some time, we did provide Composer packages for phpMyAdmin, though they were available only in separate repository and not in the main Packagist repository, but now it's there!

The reason why we didn't do that was that it really doesn't integrate well with our release process - we release ready to use tarballs, while the VCS doesn't contain all things end users expect (eg. byte compiled localization files). Putting generated content to VCS didn't sound right and there is no option of using own tarballs on Packagist repo.

That's why we've ended up providing own channel with release tarballs. However this approach is not good either as that already bundles dependencies installable by composer, possibly causing problems when trying to upgrade these.

Therefore I've decided to generate separate VCS for composer packages. This way it doesn't pollute development VCS, but still Composer gets what it expects. The phpmyadmin/phpmyadmin is now using separate VCS and is updated daily using shell script. There might be some glitches during initial runs, so please report me any problems you see.

You can find more information on installing phpMyAdmin using Composer in our documentation.

Weblate 2.14

Weblate 2.14 has been released today slightly ahead of the schedule. There are quite a lot of security improvements based on reports we got from HackerOne program, API extensions and other minor improvements.

Full list of changes:

  • Add glossary entries using AJAX.
  • The logout now uses POST to avoid CSRF.
  • The API key token reset now uses POST to avoid CSRF.
  • Weblate sets Content-Security-Policy by default.
  • The local editor URL is validated to avoid self-XSS.
  • The password is now validated against common flaws by default.
  • Notify users about imporant activity with their account such as password change.
  • The CSV exports now escape potential formulas.
  • Various minor improvements in security.
  • The authentication attempts are now rate limited.
  • Suggestion content is stored in the history.
  • Store important account activity in audit log.
  • Ask for password confirmation when removing account or adding new associations.
  • Show time when suggestion has been made.
  • There is new quality check for trailing semicolon.
  • Ensure that search links can be shared.
  • Included source string information and screenshots in the API.
  • Allow to overwrite translations through API upload.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Weekly phpMyAdmin contributions 2017-W19

Last week I finally got to doing something else than bug screening and fixing.

First of all the daily snapshots were improved in order to indicate the snapshot detail on our website, so that it's clear when it has been built and from which Git commit.

I've also looked at long outstanding issue of removing eval() usage from our codebase. The last piece where it has been used for Advisor and there is now my pull request to get rid of that.

Second long annoying thing is that we really don't have theme metadata in some easy to read format. Some of the information is set by PHP code and that's not really something you want to use to just get theme name, author or compatibility (actually the last bit is not really there). I've rewritten this to use JSON and there is pull request to implement the changes.

Probably both pull requests will land into master this week.

Handled issues:

New projects on Hosted Weblate

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue was over one month long, so it's time to process it and include new project.

This time, the newly hosted projects include:

We now also host few new Minetest mods:

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do them on Liberapay or Bountysource.