Category Archives: System Administration

Part I: Raspberry PI, Dynamic DNS, Apache 2 (Virtual Hosts and Reverse Proxy)

In Ersilia, to establish the relationships that sustain the city’s life, inhabitants stretch strings from the corner’s of the houses, white or black or gray or black-and-white according to whether they mark a relationship of blood, of trade, authority, agency. When the treads become so numerous that you can no longer pass among them, the inhabitants leave: the houses are dismantled.

Italo Calvino “Invisible Cities”

This article stems from a number private queries, the questions being mostly twofold: How to ensure that a private web server is reachable per DNS (Domain Name System)? And can a private web server maintain more than one website? Furthermore, these questions where as a rule concerned with Raspbian, Apache2 as well as any cloud or file sharing software had already been installed.

Raspberry Pi are especially useful as private cloud or photo sharing application platforms. They’re cheap, silent and economical, while being relatively easy to install and maintain. Web applications such as OwnCloud, NextCloud, PicApport, Piwigo, RoundCude, DokuWiki and many others lend themselves naturally to Raspbian.

Dynamic DNS
Every machine in the Internet has a public IP address.  Some are fixed and unchanging, others are changed intentionally in regular intervals. Most private Internet connections will be of the latter type (addresses which are changed at regular intervals). This is a problem. How to ensure that a private server is still reachable even though and despite of regular IP changes. The answer is DNS or more precisely dynamic DNS. There are a number of free DNS providers (This article here describes a few) available.  Alternatively, most commercial web service providers also provide dynamic DNS.  So a decision has to be made, free or commercial DNS provider and then there are still two questions open:
Firstly, how to (regularly) retrieve your public IP?
Secondly, how to inform your DNS provider about any changes?

Retrieving your public IP address
Any external (external insofar as that the server is not within the LAN) website knows your public IP address. They have to if they wish to communicate with you. There are also servers whose soul purpose it is to communicate this public IP back to the requester.  https://wtfismyip.com/, https://api.ipify.org/ or http://checkip.dyndns.org/ to name just a few such servers. So retrieving the public IP isn’t a problem. Now the only question is how to automate this request so that the public IP address can be retrieved in regular intervals. There are various possibilities: BASH, PERL, Ruby, C#, Python, Javascript, JQuery, NodeJS to name but a few. Here’s an example in BASH

#!/bin/bash
IP=$(curl --silent 'https://api.ipify.org')
echo $IP

This code snippet will retrieve the public IP address. But it won’t inform the DNS provider of any changes.

Regularly informing the DNS provider of any changes
Each provider will have different ways of automatically updating a DNS record. The web service provider STRATO serves here as an example. STRATO provides a DNS update web page for it’s customers. The following example retrieves the public IP and sets two separate DNS records for two different sites. Both sites being housed on the same PI.

#!/bin/bash

# Get the public IP address
IP=$(curl --silent 'https://api.ipify.org')
# Which Sites get should be reachable over this IP
HOSTS="ownCloud.travers-berlin.de dokuWiki.travers-berlin.de"
# User Credentials
USER="XXXXXXXXXXXX"
PASS="XXXXXXXXXXXX"
# Strato web page
URL="https://dyndns.strato.com/nic/update/nic/update"

# Now set the DNS record for all websites.
for HOST in ${HOSTS}
do
  QUERY="?system=dyndns&hostname=${HOST}&myip=${IP}"
  curl --ssl --user ${USER}:${PASS} ${URL}${QUERY}
done

This script must now run in regular intervals and this can be achieved using the Linux scheduling daemon crond. The script has to be saved. The directory and name are arbitrary, something like /root/ipUpdate.sh is more than sufficient. Crond reads crontabs (cron tables) and executes the programs listed. The criteria for running are listed together with the program in the crontab. Each user has his or her own crontab. To create a crontab for root, use the following command:

pi :~ $ sudo crontab -e -u root

and add the following table entry:

0.15,30,45 * * * * /root/ipUpdate.sh

This will ensure that ipUpdate.sh will run every 15 min.

Public Domain Mark

Creating colorful bash output using the ANSI color codes

“Even the simplest shepherd modestly watching his sheep under the stars would discover, once he understood the part he was playing, that he was more than a servant, was a sentinel. And each sentinel among men is responsible for the whole kingdom.”
Antoine de Saint Exupèry Wind, Sand and Stars

Why bother?
Working with the shell is very satisfying. The closeness to the operating system which a shell allows offers a freedom which cannot be (in my opinion) replicated within a GUI. The shell has many helpful features, auto command completion with tab, command history, shell shortcuts (eg. CTRL+e brings the cursor to the end of a line or CTRL+k  cuts everything after the cursor to the clipboard) , to name but a few and one of these features is the ability to color the screen content. This feature can be used to personalize the shell on a user basis. E.g. red prompt for root, blue for user0, green for user1 etc. Other possibilities include coloring warning banners such as /etc/issue or /etc/motd, making script messages standout (error messages colored red), coloring texts (like todo lists) and there is always just fooling around more for fun than effect. There are a lot of tutorials available on how to color bash output. Some good, some bad. This article concerns itself with the 8 colors (not 8 bit 256 colors) which are available. The reasoning being simple, I don’t use them.

How?
Firstly you need to know how to use ANSI escape sequences. These escape sequences start (funnily enough) with the ESC (ASCII  decimal 27/ hex 0x1B/ octal 033) character, followed by two or more characters. A modern bash is quite capable of interpreting \033 (which is needed if 256 colors are to be used) but generally ANSI/VT100 Terminal Codes are incorporated in scripts (\e being the escape code). If more than two characters follow, these are preceded by a left crotchet ([). The ANSI color codes are grouped thus: text attribute code, text foreground color code, text background color code with each code separated by a semi-colon (;). The sequence is closed with the letter m. Eg. \e[0;0m resets the shell and generally terminates any color changes that have be made.
The text attribute codes are as follows:
0 : No Attribute
1 : Bold
4 : Underscore
5 : Blink
7 : Reverse
8 : Concealed

The text foreground color codes are:
30 : Black
31 : Red
32 : Green
33 : Yellow
34 : Blue
35 : Magenta
36 :  Cyan
37 : White

And finally the text background codes can be seen below :
40 : Black
41 : Red
42 : Green
43 : Yellow
44 : Blue
45 : Magenta
46 :  Cyan
47 : White

As you can see the second character (1..7) defines the color, while the first character (3|4) denotes text or background color.
Here are a few examples to help clarify.

Colorful Prompt:
Edit ~/.bashrc and add the following variable
PS1=’\e[1;32m[\u@\h \W]\$\e[0;0m ‘

The command
$ source .bashrc
makes the changes visible. In this case a bold green prompt containing the login, host and working directory, surrounded by crotchets and followed by a dollar sign.

[user@host ~]$

echo -e
This short script will run through the colors on the prompt. There are many, many similar such scripts to be found all over the web. This example is quick and dirty and can be typed directly into a shell.

$ for i in {3..4}
> do
> for n in {0..7}
> do
> echo -e “${i}${n} \e[${i}${n}mcolor\e[0;0m”
> done
> done

Video

text.txt
It is of course possible to color simple txt files. The file text.txt contains the following text,

This is a test and can be deleted.

and this should be presented in color, lets say red, and in bold print. When using vim to color a file the ANSI/VT100 escape code ^[ replaces \e. This code is created within vim using the following key combination:
ctrl +v crtl + [

The new text.txt file will look like this:

^[[1;31mThis is a test and can be deleted^[[0;0m

Using cat the contains are displayed red and in bold print.

$ cat text.txt

This is a test and can be deleted.

Video

Public Domain Mark

Connecting Kindle Fire HD to Debian 7

“Because of my poor patch of moorland I can never produce crops to compete with my neighbour’s, ought I remain arms folded?”
Paul Sabatier Modernism: the Jewett Lectures 1908
Santa came and left exactly one Kindle Fire HD 7 per child, there were two, went about his merry way, never again to lose a thought about the father, the mother, the two children or the Kindles.
Not the first order of business, but certainly pretty high on the list was connecting a Kindle to a Debian 7 PC. Kindle Fire doesn’t support USB, but instead uses MTP (Media Transfer Protocol) for automatically transferring data to its media. MTP has several drawbacks, some of which are no parallelization, no multi-tasking and it doesn’t support direct modification of data. But the biggest drawback is that MTP devices are not treated as traditional removable disks. The file system is implemented by the device and not by the operating system, which means that most software will not recognize a MTP device.
libmtp (http://libmtp.sourceforge.net/ and thanks very much for that) offers a MTP implementation for POSIX based operating systems. libmtp is really just a C library and it comes with a set of example programs which allow you to do anything imaginable with your MTP device.  A FUSE (Filesystem in User Space) filesystem which supports reading and writing to a MTP device is also required. This comes in the form of MTPfs (http://www.adebenham.com/mtpfs/ and again thanks for that). There is also a GUI (gmtp http://sourceforge.net/projects/gmtp/) for anybody who isn’t console enamored.
All that is left is to run through an installation and mount the device.
  1. Add the Debian backports
    echo “deb http://http.debian.net/debian wheezy-backports main” >> /etc/apt/sources.list
    # apt-get update
  2. Install the software
    # apt-get install libmtp9 mtp-tools jmtpfs gmtp
  3. Connect the kindle
    Sounds silly but the kindle has to be connected and logged on.
  4. Mount the device
    # mtp-detect
    # jmtpfs -o allow_other /mnt

That’s it.  When the device is to be unmounted.

  1. Unmount the Kindle
    # fusermount -u /mnt

Of course, once the software has been installed, there is the GUI option.  Although the GUI does offer an exercise in patience, as it tends to lag and it takes a 30 or 40 seconds before the device becomes visible, it is far easier and more intuitive to use than the console.


Public Domain Mark

A random assortment of helpful shell commands/scripts.

“It lies in the power of every man, either permissively to hasten, or actively to shorten, but not to lengthen or extend the limits of his natural life. He only (if any) hath the art to lengthen his taper, that put it to best advantage.”
Francis Quarles

Here are some nifty command line scripts which come in handy:

  1. Show the lines in a file which are uncommented and not empty
    grep -E -v “#|^$” file
    or alternatively (just to use sed)
    grep -E -v “#” | sed ‘/^$/d’
  2. Create 10 txt files
    touch test{0..9}.txt
  3. Move all txt files to bak files
    for i in *txt
    do
    j=`echo $i | sed ‘s/txt/bak/’`
    mv $i $j
    done
  4. Find out who is using the most disk space
    du -hs /home/* | sort -n
  5. Selecting the most recent file in a directory
    ls -rt | tail -1
  6. Excluding the grep command from ps aux
    Lets say you want to grep the process list for all vi processes. The resulting list  will contain the grep command itself. To avoid this use
    ps aux | grep [v]i
  7. Compare the differences between a remote & local file
    ssh user@server cat remote.txt | diff -y local.txt –
  8. Quickly create a dated backup
    Firstly create an alias in your .bashrc
    echo “alias d=’date +%F” >> .bashrc
    Restart your terminal & try
    mv file.txt{,.$(d)}
  9. Quickly remove all whitespace
    cat file.txt | sed ‘s/\s\+//g’
  10. Columnize output
    column -s : -t /etc/passwd
    would format passwd in columns using : as a delimiter.
  11. Get the total weight of certain files in a directory
    du -hc *txt
    or
    ls -al *txt | awk ‘{ print; total += } END { print total / 1.024e+9 }’
  12. Count any number of lines before or after a search match
    egrep -A 15 test testytest.txt
    Provides the next 15 lines after the match.
    egrep -B 15 test testytest.txt
    Provides the 15 lines before the match.
  13. List services running on a machine
    netstat -nptl | egrep ‘^tcp’ | awk ‘{if($7!=”-“){print $7}}’ | cut -d/ \ -f2 | sort -u
  14. How much RAM does a machine have?
    egrep ‘MemTotal’ /proc/meminfo | awk ‘{print $2/1024, “MB”}’
  15. How many disks does a machine have?
    fdisk -l 2>/dev/null | egrep ‘/dev/(s|xv)d[a-z]:’ | \
    awk ‘{count++}END{print count}’

Public Domain Mark

Standardizing a personal work environment over multiply ssh servers.

“As if the finest and most manly of spectacles were not that of a man who conquers his soul hour after hour, fighting first against himself, against the suggestions of egoism, idleness, discouragement…”
Paul Sabatier “The Life of St. Francis of Assisi”

A system administrator will generally, during a normal day, work on more than one server . It is very helpful if certain settings (a personal working environment or PWE) and changes to these settings travel without any effort from server to server. OpenSSH offers a novel but effective approach of achieving just this. This article is concerned with describing experiences made with Debian Wheezy and its openssh-server (1:6.0p1-4+deb7u2).

The Principle
A client defined shell variable will be filled with the files which make up the personal environment (in this case .vimrc and .bashrc) and then passed to the server, which in turn will per shell script decode the variable and place the files in the appropriate user home directory.

The Preparations (The Client)
To simplify the creation of the shell variable ($PWE) a directory  ~/personal_work_envir is created. The files .bashrc and .vimrc are copied to this directory.

  • client: $ mkdir ~/personal_work_envir
  • client: $ cp -v ~/{.bashrc,.vimrc} ~/personal_work_envir

To create the shell variable $PWE the following line is added to ~/.bashrc

  • export PWE=$(tar -C ~/personal_work_envir -cz .| base64)

Finally the directive SendEnv is added to ~/.ssh/config.

  1. Host server
  2.     SendEnv PWE

The Preparations (The Server)
The shell script /etc/sshrc is invoked by the ssh server for every incoming ssh connection. A similar script namely ~/.ssh/rc can be defined for individual users. If this script exists it overrides /etc/sshrc but like /etc/sshrc,  it is executed before the shell or any remote command requested by the incoming connection. Unlike /etc/sshrc which is always executed by /bin/sh, the ~/.ssh/rc should be executed by the user’s normal login shell. Generally speaking /bin/bash. The ~/.ssh/rc script should look as follows:

  1.   #!/bin/bash
  2.   # unpack the personal work environment
  3.   if test “$PWE” ; then
  4.               echo “$PWE” | base64 -d | tar -xzC ~
  5.   fi

Now it only remains to ensure that the server accepts the client defined variable. This is achieved by adding the following entry to the sshd configuration /etc/ssh/sshd_config

  • AcceptEnv PWE


Public Domain Mark