Michal Čihař - Archive for Jan. 1, 2017

For quite some I'm happy user of Turris Omnia router. The router has quite good hardware, so I've decided to try if I can run Bitcoin node on that and ElectrumX server.

To make the things easier to manage, I've decided to use LXC and run all these in separate container. First of all you need LXC on the router. This is the default setup, but in case you've removed it, you can add it back in the Updater settings.

Now we will create Debian container. There is basic information on using in Turris Documentation on how to create the container, in latter documentation I assume it is called debian.

It's also good idea to enable LXC autostart, to do so add your container to cat /etc/config/lxc-auto on :

config container
    option name debian

You might also want to edit lxc container configration to enable clean shutdown:

# Send SIGRTMIN+3 to shutdown systemd (37 on Turris Omnia)
lxc.haltsignal = SIGRTMIN+3

To make the system more recent, I've decided to use Debian Stretch (one of reasons was that ElectrumX needs Python 3.5.3 or newer). Which is anyway probably sane choice right now given that it's already frozen and will be soon stable. As Stretch is not available as a download option in Omnia, I've chosen to use Debian Jessie and upgrate it later:

$ lxc-attach  --name debian
$ sed -i s/jessie/stretch/ /etc/apt/sources.list
$ apt update
$ apt full-upgrade

Now you have up to date system and we can start installing dependencies. First thing to install is Bitcoin Core. Just follow the instructions on their website to do that. Now it's time to set it up and wait for downloading full blockchain:

$ adduser bitcoin
$ su - bitcoin
$ bitcoind -daemon

Depending on your connection speed, the download will take few hours. You can monitor the progress using bitcoin-cli, you're waiting for 450k blocks:

$ bitcoin-cli getinfo
{
  "version": 140000,
  "protocolversion": 70015,
  "walletversion": 130000,
  "balance": 0.00000000,
  "blocks": 301242,
  "timeoffset": -1,
  "connections": 8,
  "proxy": "",
  "difficulty": 8853416309.1278,
  "testnet": false,
  "keypoololdest": 1490267950,
  "keypoolsize": 100,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "errors": ""
}

Depending how much memory you have (mine has 2G) and what all you run on the router, you will have to tweak bitcoind configuration to consume less memory. This can be done by editing .bitcoin/bitcoin.conf, I've ended up with following settings:

par=1
dbcache=150
maxmempool=150

You can also create startup unit for Bitcoin daemon (place that as /etc/systemd/system/bitcoind.service):

[Unit]
Description=Bitcoind
After=network.target

[Service]
ExecStart=/opt/bitcoin/bin/bitcoind
User=bitcoin
TimeoutStopSec=30min
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Now we can enable services to start on container start:

systemctl enable bitcoind.service

Then I wanted to setup ElectrumX as well, but I've quickly realized that it uses way more memory that my router has, so there is no option to run it without using swap, what will probably make it quite slow (I haven't tried that).

Weblate 2.14.1 has been released today. It is bugfix release fixing possible migration issues, search results navigation and some minor security issues.

Full list of changes:

• Fixed possible error when paginating search results.
• Fixed migrations from older versions in some corner cases.
• Fixed possible CSRF on project watch and unwatch.
• The password reset no longer authenticates user.
• Fixed possible captcha bypass on forgotten password.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Last week I was again quite active on development side bringing several improvements to master branch.

The biggest news is probably that phpMyAdmin no longer relies on eval() function. We've used it to run advisory rules on server configuration, but that is now done using Symfony ExpressionLanguage (which we anyway need due to motranslator).

When looking at things this does pull in, I've noticed that there is mbstring polyfill, which can be used instead of the one we ship (and was never completed). Thanks to this the mbstring dependency is now optional, but still recommended for performance reasons.

Another quite visible change is adding JSON metadata to our themes. Right now it covers basic things like theme compatibility and authorship, but more can be added later. This is also covered in our documentation.

Handled issues:

• #13274 Error in query builder - class 'Util' not found
• #13277 Fix unwanted escape
• #13279 Replace our mbstring polyfills with symfony/polyfill-mbstring
• #12386 request to remove mbstring dependency
• #13276 HTML shown when rendering permissions
• #13278 Error: Token mismatch (4.7.0)
• #6363 Eliminate remaining occurences of eval() in phpMyAdmin to make it work on machines where eval() is disabled
• #13260 Replace eval() in Advisor by Symfony\ExpressionLanguage
• #13261 Rewrite theme metadata handling
• #13167 Missing theme compatibility check
• #13275 Add relevant links in repos description
• #13265 Port templates to Twig
• #13273 500 error when trying to view a table with JSON formatted fields
• #13266 Broken link on 'User accounts overview' page
• #13272 Query execution adds a LIMIT clause twice, if I use colons
• #13262 Redirect user to last page that has any tables to display.
• #57 Missing a link to the Author at themes page
• #152 Inline CREATE PROCEURE
• #153 Don't print duplicated cli formatting characters

Weblate has started to use HackerOne Community Edition some time ago and I think it's good to share my experience with that. Do you have open source project and want to get more attention of security community? This post will answer how it looks from perspective of pretty small project.

I've applied with Weblate to HackerOne Community Edition by end of March and it was approved early in April. Based on their recommendations I've started in invite only mode, but that really didn't bring much attention (exactly none reports), so I've decided to go public.

I've asked for making the project public just after coming from two weeks vacation, while expecting the approval to take some time where I'll settle down things which have popped up during vacation. In the end that was approved within single day, so I was immediately under fire of incoming reports:

I was surprised that they didn't lie - you will really get huge amount of issues just after making your project public. Most of them were quite simple and repeating (as you can see from number of duplicates), but it really provided valuable input.

Even more surprisingly there was second peak coming in when I've started to disclose resolved issues (once Weblate 2.14 has been released).

Overall the issues could be divided to few groups:

• Server configuration such as lack of Content-Security-Policy headers. This is certainly good security practice and we really didn't follow it in all cases. The situation should be way better now.
• Lack or rate limiting in Weblate. We really didn't try to do that and many reporters (correctly) shown that this is something what should be addressed in important entry points such as authentication. Weblate 2.14 has brought lot of features in this area.
• Not using https where applicable. Yes, some APIs or web sites did not support https in past, but now they do and I didn't notice.
• Several pages were vulnerable to CSRF as they were using GET while POST with CSRF protection would be more appropriate.
• Lack of password strength validation. I've incorporated Django password validation to Weblate hopefully avoiding the weakest passwords.
• Several issues in authentication using Python Social Auth. I've never really looked at how the authentication works there and there are some questionable decisions or bugs. Some of the bugs were already addressed in current releases, but there are still some to solve.

In the end it was really challenging week to be able to cope with the incoming reports, but I think I've managed it quite well. The HackerOne metrics states that there are 2 hours in average to respond on incoming incidents, what I think will not work in the long term :-).

Anyway thanks to this, you can now enjoy Weblate 2.14 which more secure than any release before, if you have not yet upgraded, you might consider doing that now or look into our support offering for self hosted Weblate.

The downside of this all was that the initial publishing on HackerOne made our website target of lot of automated tools and the web server was not really ready for that. I'm really sorry to all Hosted Weblate users who were affected by this. This has been also addressed now, but the infrastructure really should have been prepared before on this. To share how it looked like, here is number of requests to the nginx server:

I'm really glad I could make Weblate available on HackerOne as it will clearly improve it's security and security of hosted offering we have. I will certainly consider providing swag and/or bounties on further severe reports, but that won't be possible without enough funding for Weblate.

For quite some time, we did provide Composer packages for phpMyAdmin, though they were available only in separate repository and not in the main Packagist repository, but now it's there!

The reason why we didn't do that was that it really doesn't integrate well with our release process - we release ready to use tarballs, while the VCS doesn't contain all things end users expect (eg. byte compiled localization files). Putting generated content to VCS didn't sound right and there is no option of using own tarballs on Packagist repo.

That's why we've ended up providing own channel with release tarballs. However this approach is not good either as that already bundles dependencies installable by composer, possibly causing problems when trying to upgrade these.

Therefore I've decided to generate separate VCS for composer packages. This way it doesn't pollute development VCS, but still Composer gets what it expects. The phpmyadmin/phpmyadmin is now using separate VCS and is updated daily using shell script. There might be some glitches during initial runs, so please report me any problems you see.

You can find more information on installing phpMyAdmin using Composer in our documentation.

Weblate 2.14 has been released today slightly ahead of the schedule. There are quite a lot of security improvements based on reports we got from HackerOne program, API extensions and other minor improvements.

Full list of changes:

• Add glossary entries using AJAX.
• The logout now uses POST to avoid CSRF.
• The API key token reset now uses POST to avoid CSRF.
• Weblate sets Content-Security-Policy by default.
• The local editor URL is validated to avoid self-XSS.
• The password is now validated against common flaws by default.
• Notify users about imporant activity with their account such as password change.
• The CSV exports now escape potential formulas.
• Various minor improvements in security.
• The authentication attempts are now rate limited.
• Suggestion content is stored in the history.
• Store important account activity in audit log.
• Ask for password confirmation when removing account or adding new associations.
• Show time when suggestion has been made.
• There is new quality check for trailing semicolon.
• Ensure that search links can be shared.
• Included source string information and screenshots in the API.
• Allow to overwrite translations through API upload.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Last week I finally got to doing something else than bug screening and fixing.

First of all the daily snapshots were improved in order to indicate the snapshot detail on our website, so that it's clear when it has been built and from which Git commit.

I've also looked at long outstanding issue of removing eval() usage from our codebase. The last piece where it has been used for Advisor and there is now my pull request to get rid of that.

Second long annoying thing is that we really don't have theme metadata in some easy to read format. Some of the information is set by PHP code and that's not really something you want to use to just get theme name, author or compatibility (actually the last bit is not really there). I've rewritten this to use JSON and there is pull request to implement the changes.

Probably both pull requests will land into master this week.

Handled issues:

• #13245 Left Column Search Error
• #13196 phpMyAdmin downgrading to http from https while using CloudFlare's Flexible SSL
• #13248 Fix unwanted escaping
• #13249 Fix dropdown selector
• #12653 Duplicate query execution in inline SQL textarea
• #13066 Can't login or do something else
• #13162 Hidden dependency on ctype
• #13258 Correct the check for Amazon RDS
• #13259 Privacy
• #13254 Export of the results of huge queries does result in a reset without an error message
• #13244 Create createbtable.php
• #13252 Fatal JavaScript error when grid editing "Time" column
• #13255 Fixed tooltip not showing up.
• #13246 Browsing returns an empty result set
• #13257 zxcvbn dependency was missing.
• #64 unneeded /downloads/ in link text of stable download
• #150 Maximum length of a delimiter is 15 characters.

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue was over one month long, so it's time to process it and include new project.

This time, the newly hosted projects include:

• Good Weather - a weather app for Andorid
• Privacy4Kids - teaching privacy to kids
• Libresonic - a media streaming software
• Ionic Super Starter - a batteries-included starter project for Ionic apps
• Android-Open-Radio - a radio app for Android

We now also host few new Minetest mods:

• Game Internationalization Mod for Minetest
• Minetest Mods for Android
• Minetest Mod Storage Drawers
• Minetest Mod Lapis Lazuli

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do them on Liberapay or Bountysource.

Last week I finally got back to work after mostly two weeks of vacation, so there was quite a lot of things to do. I've merged several pull requests, gone through incoming bugs and generally did some cleanup in our issue trackers.

I've also worked on new daily snapshots of our code, which are now available for download of for use from Docker Hub.

Handled issues:

• #12821 Provide regular (daily?) dev builds
• #13198 Change composer package type to project
• #13199 Reorder a few keys in the composer.json file
• #13240 begin and commit keywords cause a 500 error
• #13243 Error "Token mismatch"
• #13238 Bookmark notice shown in error page
• #13192 Fix errors detected by PHP_CodeSniffer
• #13225 Twig error on read only filesystem
• #13227 Run Twig in degraded mode if cache is not available.
• #13237 Share TempDir for Twig and other uses
• #13226 Missing documentation for creating twig cache dir
• #13239 SELECT command denied to user ... for table 'pma__tracking'
• #13236 Port some templates to Twig
• #13175 BIT column displays as 31 ones
• #13234 Can not log in: Undefined index: PMA_token
• #13214 Importing a file into a table gives a "SELECT command denied to user ..." for table pma_tracking error
• #13229 Token Mismatch error when attempting to edit a view
• #13231 Twig error
• #13221 Don't destroy the session if logins are still present. fixes #12301
• #13216 Support MySql 8.0.1 dmr version and new Character Set default.
• #13218 phpMyAdmin logo .png file incorrect transparency
• #13224 Rewite collation naming code
• #13223 Update logo images
• #109 Add latest/master version
• #121 Update to version 3
• #123 Run Travis tests on Trusty
• #122 Unable to export datasets for large queries/large results.
• #62 Adjust latest downloads to new release structure
• #63 Update logo images

We have stopped providing daily snapshots for phpMyAdmin pretty much at time we've moved to GitHub, which allowed to download any branch as zip file. However since introduction of Composer to manage our dependencies, additional steps were required to get working copy of phpMyAdmin out of the snapshots.

Since today the ready to use snapshots are available again. They will be updated every day and are built in exactly same way as our releases, so all you need to do is download them and start using.

These snapshots can be also used from Docker - the phpMyAdmin image now has brand new tags edge-4.7 and edge-4.8 which are updated with every snapshot and contain latest changes from development branches.