Michal Čihař - Archive for Jan. 1, 2017

New projects on Hosted Weblate

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue has grown too long, so it's time to process it and include new projects. I hope that gives you have good motivation to spend Christmas break by translating free software.

This time, the newly hosted projects include:

There are also some notable additions to existing projects:

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do that easily on Liberapay or Bountysource.

Weekly phpMyAdmin contributions 2017-W50

Last week contributions were mostly focused on fixing bugs. It's a bit hard to pinpoint something out of these, but most of them were really old, but not easy to trigger.

For example there were some javascript errors which were not easily reproducible, but turned out to be affecting quite a lot of our users. It turned out to be affecting only some variants of TIMESTAMP fields, what none of the reports did mention.

I've also switched all our libraries to use ApiGen 4 for generating docs as ApiGen 5 is way slower and is looking for new maintainer in the same time.

Handled issues:

Weblate 2.18

Weblate 2.18 has been released today. The biggest improvement is probably reviewer based workflow, but there are some other enhancements as well.

Full list of changes:

  • Extended contributor stats.
  • Improved configuration of special chars virtual keyboard.
  • Added support for DTD file format.
  • Changed keyboard shortcuts to less likely collide with browser/system ones.
  • Improved support for approved flag in Xliff files.
  • Added support for not wrapping long strings in Gettext po files.
  • Added button to copy permalink for current translation.
  • Dropped support for Django 1.10 and added support for Django 2.0.
  • Removed locking of translations while translating.
  • Added support for adding new units to monolingual translations.
  • Added support for translation workflows with dedicated reviewers.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Weekly phpMyAdmin contributions 2017-W49

Last week was a bit calmer, the most visible part probably being release of the SQL parser with several fixes with improved SQL context handling.

Handled issues:

Weekly phpMyAdmin contributions 2017-W48

Looking at list of handled issues, last week was extremely productive. Many of that are issues where I've been working on them for long time and I've managed to complete them last week. For example the user preferences cleanup to store less things in cookies or common.inc.php cleanup.

I've also gone through open pull requests and merged the ones which made sense or were basically good to merge, but needed some cleanups.

There was also some fun with phpseclib 2.0.8 which was mistakenly released from master branch instead of 2.0, what lead to API breakage. Fortunately this was really just a mistake and 2.0.9 reverted these changes.

Handled issues:

Weekly phpMyAdmin contributions 2017-W47

Last week was mostly spent on improving two factor authentication support. It turned out that Firefox 57 behaves differently than Firefox 56 with U2F extension. Also it behaves differently than Chrome (which was broken as well by the way). Anyway all of these should work fine, but there still seem to be some issues with the two factor auth, but those will be certainly addressed in next weeks.

Handled issues:

Weekly phpMyAdmin contributions 2017-W46

Last week was equally spent on refactoring, bugfixing and infrastructure. We're looking for replacement our oldish server and it seems that rented server or virtual hosts seems to be best fit for us these days. Still there are quite some choices to consider.

I've done quite some development as well - I'm most happy with Util::linkOrButton refactoring which helped to cleanup the code quite a lot, but there were other fixes and improvements as well.

Handled issues:

Running Bitcoin node and ElectrumX server

I've been tempted to run own ElectrumX server for quite some. First attempt was to run this on Turris Omnia router, however that turned out to be impossible due to memory requirements both Bitcoind and ElectrumX have.

This time I've dedicated host for this and it runs fine:

Electrum connecting to btc.cihar.com

The server runs Debian sid (probably it would be doable on stretch as well, but I didn't try much) and the setup was pretty simple.

First we need to install some things - Bitcoin daemon and ElectrumX dependencies:

# Bitcoin daemon, not available in stretch
apt install bitcoind

# We will checkout ElectrumX from git
apt install git

# ElectrumX deps
apt install python3-aiohttp

# Build environment for ElectrumX deps
apt install build-essentials python3-pip libleveldb-dev

# ElectrumX deps not packaged in Debian
pip3 install plyvel pylru

# Download ElectrumX sources
su - electrumx -c 'git clone https://github.com/kyuupichan/electrumx.git'

Create users which will run the services:

adduser bitcoind
adduser electrumx

Now it's time to prepare configuration for the services. For Bitcoin it's quite simple - we need to configure RPC interface and enable transaction index in /home/bitcoind/.bitcoin/bitcoin.conf:

txindex=1
listen=1
rpcuser=bitcoin
rpcpassword=somerandompassword

The ElectrumX configuration is quite simple as well and it's pretty well documented. I've decided to place it in /etc/electrumx.conf:

COIN=BitcoinSegwit
DB_DIRECTORY=/home/electrumx/.electrumx
DAEMON_URL=http://bitcoin:somerandompassword@localhost:8332/
TCP_PORT=50001
SSL_PORT=50002
HOST=::

DONATION_ADDRESS=3KPccmPtejpMczeog7dcFdqX4oTebYZ3tF

SSL_CERTFILE=/etc/letsencrypt/live/btc.cihar.com/fullchain.pem
SSL_KEYFILE=/etc/letsencrypt/live/btc.cihar.com/privkey.pem

REPORT_HOST=btc.cihar.com
BANNER_FILE=banner

I've decided to control both services using systemd, so it's matter of creating pretty simple units for that. Actually the Bitcoin one closely matches the one I've used on Turris Omnia and the ElectrumX the one they ship, but there are some minor changes.

Systemd unit for ElectrumX in /etc/systemd/system/electrumx.service:

[Unit]
Description=Electrumx
After=bitcoind.target

[Service]
EnvironmentFile=/etc/electrumx.conf
ExecStart=/home/electrumx/electrumx/electrumx_server.py
User=electrumx
LimitNOFILE=8192
TimeoutStopSec=30min

[Install]
WantedBy=multi-user.target

And finally systemd unit for Bitcoin daemon in /etc/systemd/system/bitcoind.service:

[Unit]
Description=Bitcoind
After=network.target

[Service]
ExecStart=/usr/bin/bitcoind
User=bitcoind
TimeoutStopSec=30min
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

Now everything should be configured and it's time to start up the services:

# Enable services so that they start on boot 
systemctl enable electrumx.service bitcoind.service

# Start services
systemctl start electrumx.service bitcoind.service

Now you have few days time until Bitcoin fetches whole blockchain and ElectrumX indexes that. If you happen to have another Bitcoin node running (or was running in past), you can speedup the process by copying blocks from that system (located in ~/.bitcoin/blocks/). Only get blocks from sources you trust absolutely as it might change your view of history, see Bitcoin wiki for more information on the topic. There is also magnet link in the ElectrumX docs to download ElectrumX database to speed up this process. This should be safe to download from untrusted source.

The last think I'd like to mention is resources usage. You should have at least 4 GB of memory to run this, 8 GB is really preferred (both services consume around 4GB). On disk space, Bitcoin currently consumes 170 GB and ElectrumX 25 GB. Ideally all this should be running on the SSD disk.

You can however offload some of the files to slower storage as old blocks are rarely accessed and this can save some space on your storage. Following script will move around 50 GB of blockchain data to /mnt/btc/blocks (use only when Bitcoin daemon is not running):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/sh
set -e

DEST=/mnt/btc/blocks

cd ~/.bitcoin/blocks/

find . -type f \( -name 'blk00[0123]*.dat' -o -name 'rev00[0123]*dat' \) | sed 's@^\./@@' | while read name ; do
        mv $name $DEST/$name
        ln -s $DEST/$name $name
done

Anyway if you would like to use this server, configure btc.cihar.com in your Electrum client.

If you find this howto useful, you can send some Satoshis to 3KPccmPtejpMczeog7dcFdqX4oTebYZ3tF.

New projects on Hosted Weblate

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue has grown too long, so it's time to process it and include new project.

This time, the newly hosted projects include:

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do that easily on Liberapay or Bountysource.