Michal Čihař - Archive for 11/2017

Weekly phpMyAdmin contributions 2017-W47

Last week was mostly spent on improving two factor authentication support. It turned out that Firefox 57 behaves differently than Firefox 56 with U2F extension. Also it behaves differently than Chrome (which was broken as well by the way). Anyway all of these should work fine, but there still seem to be some issues with the two factor auth, but those will be certainly addressed in next weeks.

Handled issues:

Weekly phpMyAdmin contributions 2017-W46

Last week was equally spent on refactoring, bugfixing and infrastructure. We're looking for replacement our oldish server and it seems that rented server or virtual hosts seems to be best fit for us these days. Still there are quite some choices to consider.

I've done quite some development as well - I'm most happy with Util::linkOrButton refactoring which helped to cleanup the code quite a lot, but there were other fixes and improvements as well.

Handled issues:

Running Bitcoin node and ElectrumX server

I've been tempted to run own ElectrumX server for quite some. First attempt was to run this on Turris Omnia router, however that turned out to be impossible due to memory requirements both Bitcoind and ElectrumX have.

This time I've dedicated host for this and it runs fine:

Electrum connecting to btc.cihar.com

The server runs Debian sid (probably it would be doable on stretch as well, but I didn't try much) and the setup was pretty simple.

First we need to install some things - Bitcoin daemon and ElectrumX dependencies:

# Bitcoin daemon, not available in stretch
apt install bitcoind

# We will checkout ElectrumX from git
apt install git

# ElectrumX deps
apt install python3-aiohttp

# Build environment for ElectrumX deps
apt install build-essentials python3-pip libleveldb-dev

# ElectrumX deps not packaged in Debian
pip3 install plyvel pylru

# Download ElectrumX sources
su - electrumx -c 'git clone https://github.com/kyuupichan/electrumx.git'

Create users which will run the services:

adduser bitcoind
adduser electrumx

Now it's time to prepare configuration for the services. For Bitcoin it's quite simple - we need to configure RPC interface and enable transaction index in /home/bitcoind/.bitcoin/bitcoin.conf:

txindex=1
listen=1
rpcuser=bitcoin
rpcpassword=somerandompassword

The ElectrumX configuration is quite simple as well and it's pretty well documented. I've decided to place it in /etc/electrumx.conf:

COIN=BitcoinSegwit
DB_DIRECTORY=/home/electrumx/.electrumx
DAEMON_URL=http://bitcoin:somerandompassword@localhost:8332/
TCP_PORT=50001
SSL_PORT=50002
HOST=::

DONATION_ADDRESS=3KPccmPtejpMczeog7dcFdqX4oTebYZ3tF

SSL_CERTFILE=/etc/letsencrypt/live/btc.cihar.com/fullchain.pem
SSL_KEYFILE=/etc/letsencrypt/live/btc.cihar.com/privkey.pem

REPORT_HOST=btc.cihar.com
BANNER_FILE=banner

I've decided to control both services using systemd, so it's matter of creating pretty simple units for that. Actually the Bitcoin one closely matches the one I've used on Turris Omnia and the ElectrumX the one they ship, but there are some minor changes.

Systemd unit for ElectrumX in /etc/systemd/system/electrumx.service:

[Unit]
Description=Electrumx
After=bitcoind.target

[Service]
EnvironmentFile=/etc/electrumx.conf
ExecStart=/home/electrumx/electrumx/electrumx_server.py
User=electrumx
LimitNOFILE=8192
TimeoutStopSec=30min

[Install]
WantedBy=multi-user.target

And finally systemd unit for Bitcoin daemon in /etc/systemd/system/bitcoind.service:

[Unit]
Description=Bitcoind
After=network.target

[Service]
ExecStart=/usr/bin/bitcoind
User=bitcoind
TimeoutStopSec=30min
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Now everything should be configured and it's time to start up the services:

# Enable services so that they start on boot 
systemctl enable electrumx.service bitcoind.service

# Start services
systemctl start electrumx.service bitcoind.service

Now you have few days time until Bitcoin fetches whole blockchain and ElectrumX indexes that. If you happen to have another Bitcoin node running (or was running in past), you can speedup the process by copying blocks from that system (located in ~/.bitcoin/blocks/). Only get blocks from sources you trust absolutely as it might change your view of history, see Bitcoin wiki for more information on the topic. There is also magnet link in the ElectrumX docs to download ElectrumX database to speed up this process. This should be safe to download from untrusted source.

The last think I'd like to mention is resources usage. You should have at least 4 GB of memory to run this, 8 GB is really preferred (both services consume around 4GB). On disk space, Bitcoin currently consumes 170 GB and ElectrumX 25 GB. Ideally all this should be running on the SSD disk.

You can however offload some of the files to slower storage as old blocks are rarely accessed and this can save some space on your storage. Following script will move around 50 GB of blockchain data to /mnt/btc/blocks (use only when Bitcoin daemon is not running):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/sh
set -e

DEST=/mnt/btc/blocks

cd ~/.bitcoin/blocks/

find . -type f \( -name 'blk00[0123]*.dat' -o -name 'rev00[0123]*dat' \) | sed 's@^\./@@' | while read name ; do
        mv $name $DEST/$name
        ln -s $DEST/$name $name
done

Anyway if you would like to use this server, configure btc.cihar.com in your Electrum client.

If you find this howto useful, you can send some Satoshis to 3KPccmPtejpMczeog7dcFdqX4oTebYZ3tF.

New projects on Hosted Weblate

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue has grown too long, so it's time to process it and include new project.

This time, the newly hosted projects include:

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do that easily on Liberapay or Bountysource.

Weekly phpMyAdmin contributions 2017-W44

Most of last week was spent on bringing U2F and 2FA to phpMyAdmin. Besides that it was really just minor bug fixing.

We've had one pull request for the 2FA already open, however it was not really in quality to be merged and the original author really didn't come back to address our concerns. I was thinking to start over from his work, but in the end I've decided to start from scratch and come with code which will make easier to implement additional second authentication factors in the future.

Anyway what is now there is support for HOTP/TOTP authentication (Google Authenticator and similar) and FIDO U2F hardware tokens. The latter has been tested with TREZOR, but should work with any compliant device.

phpMyAdmin with TREZOR

Handled issues: