How to set up a GNU/Linux server at Shellit.org ecohosting

Consumerium banner
Consumerium.org – Enhancing Consumer Informedness gets its first ecoserver.

 

 

Trollböle et al gets its gear and moves to Björneborg. Consumerium.org acquires its first ecoserver from a company that uses electricity sourced from renewable sources only.

Trollböle-sign
Going under the granite to work on electricity from the wind above the granite and the sunshine and rainfall. Seems someone stuck a huge fusion reactor in the sky, which is nice.

 

Objective: Set up a stack in efficient and clear manner in order to move the current Consumerium.org development wiki onto the new ecological server paying mind to also future needs of upcoming services. Document process.

Information: Today I purchased a VPS server from https://shellit.org, a Finnish hosting services provider that has been operating since 2007. To the international clientele they cater with the brand https://vpsfinland.com. Price is same but in dollars. Costs a hell lot more than the global price leader as always but the laws, the laws.

Shellit.org logo
Shellit.org uses exclusively renewable sourced electricity for their operations and has been operating since 2007. Of the non-transnational i.e. small and local hosting guys they as of 2016 have the best price in Finland. Their hosting site is also slightly security enhanced since it is located inside the Finnish granite bedrock in Björneborg in a cave system that formerly belonged to the Finnish Defence Forces until Shellit.org bought it.

 

 

 

 

 

Their main selling points were the wind powered computers, their data center which is built into a cave system in Finnish bedrock in Björneborg and naturally being local (not multinational giant with huge deep financial pockets to absorb short term losses in gaining market share) and naturally the Finnish law that protects various parties very well.


Step #1: Purchase and pay in webbank. Server is activated very rapidly upon webshop receiving information from webbank that the funds have been debited and are on their way to their bank account. Default OS is Debian GNU/Linux and others are available on request.

Step #2 Login with the root account details given by the GUI. System language seems to be set to Finnish. Can change the locale later on so not very relevant, I hope.

Step #3 Change root pass. Don’t lose it. Resetting will be terribly expensive.

Step #4 Update installed software ‘apt update && apt upgrade’ will get the base system to latest good version. Running it you one notices they run a Debian mirror at their site. Ecological and fast, good citizen.

( Step #5 install tmux session multiplexer and detacher ‘apt install tmux’ and run tmux (not a necessary step but it is good idea. alternative program is ‘screen’ but I use tmux )

Step #6 Add the command ‘sudo‘ with ‘apt install sudo’. This also creates the ‘/etc/sudoers’-file.

Step #7 Now add a normal user and give sudo rights (command ‘useradd’) and add it to sudoers (edit /etc/sudoers). I used ‘useradd -m -s /bin/bash usernamegoeshere’ and ‘nano /etc/sudoers’ and naturally set a password for the usernamegoeshere (‘passwd usernamegoeshere’)

Step #8 Log-out and log-in as the user you just gave sudo rights to. Check that you can sudo. ‘sudo ls’ will do just fine.

Step #9 (Unless causing something unwanted) Disable root logins. There is no reason to allow anyone to attempt to login as root to the sshd. As the normal user with sudo rights you can always ‘sudo su’ if you need the superuser shell. Do so with ‘sudo nano /etc/ssh/sshd_config’ and edit till it says ‘PermitRootLogin no’. Apply changes with ‘sudo service sshd reload’ and test by opening another shell and attempting to login to sshd as root. It should now complain that you have wrong password, excellent. Now log in as the normal user.

Step #10 Set up a stash for storing backups. You do want it outside of your home directory so you can backup that without complications. Start moving your backups from other machines to the server with ‘scp’ or some other more advanced system like rsync over ssh.

Step #11 Install and use nmap: ‘sudo apt install nmap’. Scan localhost from inside and scan the external server address from an outside machine to quickly see what are the firewall settings. Looks like the hosting guys are showing a filtered SMTP port to the Internet even if ‘nmap localhost’ does not see it.

Step #12 Get your firewall in place. I use a simple setup where 22, 80 and 443 are open for incoming traffic and everything else is blocked. This is straightforward and easy to verify to function correctly. Find out about iptables kernel level firewall at The Debian Wiki.

Step #13 Get some monitoring gear ‘sudo apt install htop atop iotop glances

Step #14 Start ‘glances’ and have some snacks and refreshments.


From hereon one probably wants to install a LAMP and LNMP stack as well. I stop here for now because I must contemplate choices as I am moving some sites to this new server and I do not want to make questionable rushed choices.

To be continued..

Relevant reading:

 

Upgrading MediaWiki

This method is discouraged. Use the latest tarball instead unless good reason to run the live code tree

Objective: Get the freshest MediaWiki installed on http://develop.consumerium.org/wiki/ using git where ever possible.

Instructions followed: The official upgrading MediaWiki guide

Thanks go to: Naturally the MediaWiki devels and documenters. Special thanks to SPF|Cloud @ freenode irc for kind help in figuring out the problems encountered

MediaWiki - Because ideas want to be free.
MediaWiki logo is public domain by User:Anthere

I upgraded from 1.25.6 to 1.28 alpha which is the in-development version. My experience with MediaWiki is that the people responsible for the tech know what they are doing. Prestige software attracts prestige devels so I decided to go with the development version.


First off: Backup the files and the database. Something could go wrong even if usually MediaWiki upgrade procedures are rock-solid.

Then get the latest MediaWiki from git with

git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git

Next get all the skins by removing the skins directory and then issuing in the MediaWiki core directory

git clone --recursive https://github.com/wikimedia/mediawiki-skins skins

Change into the extensions directory and update the extensions with

git pull

Put the images/ and the extensions/ directory into a .tar.gz and move it over to the ../core/ directory and uncompress there

Get the vendor libraries used by WMF clusters

git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git

Now you are ready to do the upgrade. I shut the Apache2 down for the duration of that just to feel more secure that the update.php will work but I don’t think this is required for safety. Change to the maintenance/-directory in core/ and

php5 update.php

move the old w/-directory out of the way and move core/ into w/ and access the Special:Version of your wiki in browser and you should see that it is up and running with the latest software version.


And keep up-to-date

  • ‘git pull’
  • ‘git pull –recurse-submodules’ in skins/ and extension/
  • Got some complains about composer being out-of-date. Fix with ‘sudo composer self-update’ and ‘composer update’
  • ‘php maintenance/update.php’

Protecting GNU MediaGoblin

GNU MediaGoblin
GNU MediaGoblin is sympa but under attack.

Objective: GNU MediaGoblin instances that have open registrations are suffering from botnets registering accounts en masse for spamming purposes and thus forcing instance maintainers to close registrations. Especially annoying thing the botnets are doing is that they do not even check if the email address lists they traded something in exchange for are valid causing massive amounts of mail returned by Mail Delivery Subsystem on the basis that the email box does not exist. Teslas_moustache on freenode irc proposed that we should look into how Fail2ban could be utilized to stop known vandals.

Fail2ban logo
Fail2ban dynamically alters firewall settings to counter vandal activity by denying access to known vandal IPs. Logo used under the clauses CC-BY-SA 3.0 courtesy of WMC user Palosirkka.

Fail2ban wiki on Fail2ban

“Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).”

How to use Fail2ban

When properly configured Fail2ban dynamically modifies the iptables rules when it sees improper behavior.

  • Any IP addresses that can be associated with generating flood of returned mail because they try to register an account with an email address that doesn’t exist should be banned. Stupid, annoying and basic FUD technique employed to discourage MediaGoblin people.


Sharing information on vandal IP and email addresses

Also the issue has been raised that instead of lying down as the firing from the FUD campaign botnets ensues we should try to take their ground. For this it would be beneficial to form a data sharing arrangement between GMG hosters so that we can more effectively combat the FUD campaign.

Installation of Etherpad

Etherpad logo
Etherpad is a free system for collaborative editing of text documents well suited to both working in parallel and serially. It is provided courtesy of the Etherpad Foundation and the developers

Objective: Install a private instance of Etherpad.org secured with TLS encryption and configuring the system to have good level of controll over who gets to see and edit what i.e. to authenticate the users.

Instructions used:


Basic install

Install the dependencies

sudo apt-get install gzip git curl python libssl-dev pkg-config build-essential

You will also need to download and install a working node.js system. The installation manual does recommend against using the version that apt-get installs and go for the downloadable one.

The official Node.js installation guide gives the following instructions for a Debian8:

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -

followed by

sudo apt-get install -y nodejs

Which worked just fine installing the nodejs from deb.nodesource.com

Next create the directory where you want Etherpad to reside and git clone into the source tree

git clone git://github.com/ether/etherpad-lite.git

and change directory to there and run

bin/run.sh

and

lynx http://127.0.0.1:9001

and you should see your Etherpad installation.

TLS encryption with LetEncrypt.org certificates

Let's Encrypt logo
Let’s Encrypt is a free certification authority kindly provided by Internet Security Research Group (ISRG)

Objectives to the accomplished

  1. First I will be getting and installing a new cert for use on pad.byjuho.fi which will host an Etherpad instance to fulfill my secure textual collaboration needs safely.
  2. Second I will be replacing the shortly expiring commercial certificate for *.consumium.org. So far I know that I can have the old cert still in place and insert the new certs under a subjectAltName. This way the free social media that I host can continue operating normally (hopefully) without any downtime.

How I did it

The definitive instructions from readthedocs.io I found only sometime after starting this were very helpful as they almost always are.

https://letsencrypt.org/getting-started/ recommends using

CertBot logo
CertBot is a free cert management solution provided by The Electronic Frontier Foundation (EFF)

CertBot from Electronic Frontier Foundation to automate the installation of LetsEncrypt certificates so I’m doing that.

CertBot takes as arguments your web server and operating system and provides instructions customized by those.

ByJuho.fi is being served by an Apache2.4 on a Debian8.5 so I chose those.

CertBot points to instructions for enabling backports on my system. Which I promptly followed successfully.

Then you naturally need to

sudo apt-get update

before the backports start to work.

After that

sudo apt-get install python-certbot-apache -t jessie-backport

Runs fine and installs a bunch of python candy

Next I ran

sudo certbot --apache

as instructed by CertBot interactive website. That complained that it did not find any ‘ServerName’s in the configuration files which is slightly strange. When answering ‘no’ to the “Do you want to proceed?” question it exited and hinted to specify domain name with the ‘–domains’ switch

sudo certbot --apache --domains byjuho.fi

A blue screen comes up that asks for the “emergency” email address. Put one that you will never lose like an https://iki.fi address which I’ve used for over 20 yrs now and which is valid for a lifetime.

Next the blue screen asks if you want to have all traffic redirected to TLS encrypted. I chose to allow normal http too.

Program exits and gives good advice to check the installed cert with the awesome free test tool by SSLLabs so I proceeded to do so. Certbot apparently knows its stuff since the site got an ‘A’ rating for the things SSL.

SSLTest rating A
Parasta A-ryhmää / TLS protection rated A by QUALLABS SSL LAB’s free awesome SSLTEST service

Automating renewal of certificates

LetsEncrypt.org certificates are valid only for 90 days. Probably due to meticulous planning and execution to maximize security so we want to automate the renewal.

Now CertBot site instructs to test automatic renewal arrangement by issuing command

sudo certbot renew --dry-run

and it reports that everything seems to be in order to automate the renewal so I proceeded to do so with

crontab -e

and inserted instructions to run on quiet the renewal script twice a day 12-hours apart. The command to be run is given as

certbot renew --quiet

But that will fail unless run with sudo because it cannot access certain files so you need to set the cronjob as superuser. Type

sudo su

give password and then run

crontab -e

(See here for practical examples of crontab entry syntax). Exit super user account with ctrl-d and you are done automating the renewal of the certs.

The encrypted URL now leads to the default Apache2 on Debian landing page “It works.. blahblahblah…” so I need to make a new VirtualHost directive for the encrypted site in /etc/apache2/sites-enabled/001-hosts which is where I keep the directives.

So I need to figure where the CertBot put the certificate and the key.

CertBot puts the very secret key and the very public certificate in

‘/etc/letsencrypt/live/domain.tld’ and the automagic from the blue screen creates a VirtualHost entry in ‘/etc/apache2/sites-enabled/000-default-le-ssl.conf’. After I made a normal VirtualHost entry in ‘/etc/apache2/sites-enabled/001-sites.conf’ and commented everything out in the 000-default-le-ssl.conf this blog is now available also in TLS protected https://ByJuho.fi.

Friendly folks at #freenode pointed out that

sudo apachectl -S

is very useful for locating problem points regarding conflicting VirtualHost directives

Next I am going to figure out if the commenting out stuff from 000-default-le-ssl.conf has any adverse effects. It seems the files with lower prefixed number takes precedence.


Next I try to replicate the necessary steps described in this blog post to actually enable https://pad.byjuho.fi

All that was needed to bring up the default Debian/Apache “it works page” over TLS encrypted https was one run of

sudo certbot --apache --domains pad.byjuho.fi

and fix the VirtualHost directive to your liking to actually serve your content.


Getting new certs with nginx.

There doesn’t seem to be quite the same level of automation with Nginx hosted sites than the Apache ones.

sudo certbot certonly --webroot -d d.consumium.org --webroot-path /var/www/diaspora

Is what I used to successfully get the new certificates in place.

Installation of GNU MediaGoblin on Debian GNU/Linux

Installing GNU MediaGoblin 0.9.0 with Py3 support and using Postgreql as RDBMS on GNU/Linux system

Installing GNU MediaGoblin (over and over again)

Here I document how to install the GNU MediaGobin 0.9.0, The Three Gobliners on Debian GNU/Linux.

Why over and over again?

Short answer: The installation instructions given are required to be read to be completely understood. So I’ll be installing again a third time.

  • GNU MediaGoblin 0.8.0 I accidentally set to use SQLite, instead of Postgresql the intended database backend. No migration script exists so reinstall was needed
  • GNU MediaGoblin 0.9.0 I managed to install the 0.9.0 using Py2 instead of Py3.
  • GNU MediaGoblin 0.9.0 with python3 version is what I am aiming at the third time around installing
    UPDATE: Seems installation of GNU MediaGoblin 0.9.0 with python3 support is currently impossible if the idea was to use flup and fcgi. Follow this ticket for updates on the situation.

Since the installation using python3 is impossible at the moment I have installed the py2 version instead at https://media.consumium.org. using py2, Nginx and fcgi for serving content.

Previously installed on the server are Nginx for webserver with TLS security enabled. Services already running on the server are https://hub.consumium.org (Hubzilla), https://d.consumium.org (diaspora), https://social.consumium.org (GNU social) and https://friendica.consumium.org so some of the dependencies are likely there.


I run into some problems which caused that the Postgresql cluster was not created and started. I got good help from StuckMojo @ #postgresql @ freenode irc.

I fixed the situation by running

  • ‘nano /etc/locale.gen’ and uncommented the Finnish and US English locales
  • ‘sudo locale-gen’ generated the locales according to /etc/locale.gen
  • ‘sudo pg_createcluster –locale=en_US.UTF-8 9.4 main’ creates the cluster and
  • ‘sudo pg_ctlcluster 9.4 main start’ starts the cluster
  • check its status with ‘sudo pg_lsclusters’

Now should be ready to create the database user and the database.

 

Migration of various free social media from GNU/Linux server to server

Migration procedure for moving various free social media from a GNU/Linux to another GNU/Linux system and end results

Consumium free social medias and Consumerium consumer empowerment effort
Current logo for Consumium free social media services and Consumerium – Enhancing Consumer Informedness – effort

This is the record for what went well and what didn’t go well in the process of migrating the *.consumium.org sites (except https://c.consumium.org, that’s in Espoo)

This migration was completed on 2016-06-09. I would like to extend a warm  you to https://TransIP.eu for showing compassion in my predicament and offering to credit me some of the costs incurred by requiring 2 servers for a period of a time.

<spam>Their operation is really top-notch and I have never had outages with them that I would not have been responsible. Ever since I started hosting free social media with them in July 2013 the service has been outstanding and their control panel does include ability to take snapshots of system disks and a VNC just in case someone is not comfortable working with cli. The first time I saw the VNC in the control panel and it started to show the Debian GNU/Linux white-on-black bootup in my browser I was impressed.. Then it moved to run level 6 and I was naturally like “Whoa! It can do that!”. TransIP.eu is maybe not the most inexpensive hosting guys out there at the moment but I tell you their service level and its consistency are worth all the extra money. SSD system disks are spaceous and very fast and just as soon as http://maidsafe.net/  starts I’ll be purchasing at least one 10€ unit of 2,000GB big storage (which can be grown to 400,000GB, slightly under 400TB). Scp’ing between 2 servers in the same data center in Netherlands I was able to clock 101 Mbit/s speed. That is almost a gigabit / second, normal HDD couldn’t handle that.</spam>

Debian8 -> Debian8 migration of 4 free social media instances. Debian GNU/Linux, Nginx for web server, MariaDB for RMDBS and ruby, PHP and python as langauges the services run on

Migration of diaspora* to a new server

  • https://d.consumium.org (how to install diaspora* freesome on Debian GNU/Linux)
    diaspora* is the biggest and best known of the free social media. It has innovative features though is somewhat limited due to the creators thinking really hard about protecting the consumer from possible privacy related threats. The software is high quality and reliable. It uses a asymmetric sharing arrangement that is diametrical to twitter’s

    The original raison d’être for the old server called Debian7. The name is not very well chosen and misleading since the machine was dist-upgraded to Debian8 stable without hick-ups. Diaspora* was originally installed in July 2013 which at the time took couple of days

  • Grabbed the database, app/views/home and public/uploads and inserted those into place and the pod looks fine now after the migration.
  • Email was more of an hassle and is covered in a separate paragraph you’ll find down this page.

Migration of GNU social to a new server

  • https://social.consumium.org (how to install GNU social freesome) (How I originally installed GNU social)
    GNU social is a no-nonsense microblogging platform that is simple to grasp. Unfortunately it does not work very at the moment.

    – GNU social is a handy microblogging service.This instance was installed in 2016. Should pose no problems. MySQL was replaced with MariaDB during installation of this with no problems. Update: GNU social migration was the first one to be done. Grabbed the database (which contains the confs) and the ‘avatar’ and ‘files’ directories. Shut down. Put those in place and restart web server and GNU social was up with apparently all the old information from the previous box.

  • If you are getting an Error 400: After the migration the GNU social has been doing the same thing as before.. It often when trying to microblog gives an error “400”. Here one just needs to know to hit ctrl-r, no need to even hit ctrl-a ctrl-c, ctrl-r, ctrl-v as the software preserves what was written into the textbox.

Interesting point about Hubzilla and Friendica

Friendica and Hubzilla leverage the same instructional capital and best-practice which leads to that their installation instructions have many portions in common.


Migration of Hubzilla to a new server

  • https://hub.consumium.org – (how to install Hubzilla freesome)
    Hubzilla is a very high quality software and it has always worked just like the label said. It’s use of channels is intuitive as a way of interacting with other people.

    This will probably not have the old database restored because when I originally installed this I didn’t realize the point is to have many many channels but just one login. Of course it might be possible to restore the database but manipulate it so that the Consum(er)ium relevant channels would be under the same user

  • Well I did restore the old database.
  • Pretty much everything that was needed for installation of Hubzilla was already there. Just needed to run ‘sudo aptitude install mcrypt php5-mcrypt’ and installed the Hubzilla, Stopped Nginx and dropped in the database and the user uploads located in /var/www/hubzilla/store and it seems to work fine.

Migration of Friendica to a new server

https://friendica.consumium.org (how to install Friendica free social media)

Friendica is the least learning curve free social media solution for the people escaping Facebook more and more often.

the freesome of least steep learning curve for the people who want to free themselves of Facebook every now and then.

 

Friendica migration did not require copying over more than just the database as Friendica saves the uploaded files in the database and not flat file system.


Dealing with outgoing and incoming email

Getting email arrangements to work in a safe and reasonable way is by no means as easy as one may think at start. diaspora* email was configured to use SMTP over a TLS encrypted hop over to https://gandi.net‘s SMTP server. Took a while to figure out but I am guessing this will make the email look better to spam filters as the “origin” is under the same domain as the machines given in the MX records in DNS to be the Mail eXchange servers for consumium.org

‘sudo aptitude install sendmail’ installs sendmail, an MTA this is apparently all that is needed for PHP’s mail()-function to work.



The migration plan (and how it went)

(Note: to lazily get all the dependencies and hope there wasn’t old junk you could follow this post http://juboblo.gr/index.php/2015/12/02/original-howto-migrate-gnulinux-to-bigger-disk-with-clean-install-and-grab-all-apt-gettable-software-settings-and-files/)

Migration of system settings

  • Update services to latest version so you get the same exact version when you reinstall each service from latest release [✔]
  • Grab TLS key and cert – Remember to keep the key safe [✔] (note: exposing the server.key usually kept in /etc/ssl/private is very dangerous as it will expose all communications encrypted with that key)
  • Grab firewall settings allowing traffic to 22, 80 and 443 [✔] NMAP security scanner is great copyleft free tool for looking at this. tip: ‘nmap localhost’ inside the firewall and ‘nmap the IP address” from outside the firewall will be very useful scans for verifying firewall settings.
  • Grab confs:
  • /etc/nginx/nginx.conf [✔]
  • /etc/nginx/sites-enabled/nginx.conf [✔]
  • Grab home dir [✔]
  • Grab logs [✔]
  • /var/log/nginx/access.log [✔]
  • /var/log/nginx/error.log [✔]
  • Then decided to grab all of /var/log into a .tar.gz, Is only logs, cannot hurt and  [✔]
  • Mass grab /etc and /var/www for later reference when the old server is recycled and resources returned to cloud.
  • Get new server. [✔] Remember to install an ssh server when installing the software or you’ll be unable to access via ssh. Only if hosting guys provide a Virtual Network Console you can fix this problem there
  • Add self to sudoers [✔]
  • Restore home dir contents [✔]
  • Install Nginx [✔]
  • Put logs, key, cert and nginx.conf in place [✔]

Repeat following steps for each service

  • Install dependencies [✔]
  • Install new service clean [✔]
  • NOTIFY USERS THAT NOW IS FEW HOURS OF DATA LOSS IF YOU POST Better idea: When all is ready with the new installation in place and you are thus ready to start the DNS change propagation tell people that the database will be frozen when the old machine is “unreachable” due to the DNS already pointing to the next machine.
  • Grab databases. Each database separately. [✔]
  • Grab user uploaded content and the custom landing page for d* [✔]
  • Insert grabbed database, confs, landing page, user uploaded content. [✔]

Four free copyleft open source tools for system monitoring

Nice article on #free #copyleft #opensource #tools for #system #monitoring https://opensource.com/life/16/2/open-source-tools-system-monitoring – #top, #atop, #htop and my favourite atm #glances