Adventures in Red Hat Enterprise Linux, CentOS, Fedora, OpenBSD and other open source solutions.

Setting up iSCSI (target/server and initiator/client) on RHEL

It's quite easy to setup an iSCSI environment on Red Hat Enterprise Linux. Try this easy setup to get a better understanding of iSCSI.


  1. Two (virtual) machines, a server and a client
  2. Access to the "RHEL Cluster-Storage" channel on Red Hat Network.

N.B. SELinux must be disabled when using this recipe, iptables tcp port 3260 must be opened on the server.

On the server execute these commands to setup a 100 Mb iSCSI target. This target can later be mounted on the client(s).

# yum install scsi-target-utils
# cat /etc/tgt/targets.conf
backing-store /iscsi1.img
# dd if=/dev/zero of=/iscsi1.img bs=1024 count=102400
# chkconfig tgtd on
# service tgtd start

Now on (all) client(s) follow these steps. (Please pay attention that only one client was give access in the configuration example above;

# yum install iscsi-initiator-utils

Start iscsi daemon.

# service iscsi start

To see what IQNs are available, run:

# iscsiadm -m discovery -t sendtargets -p

The result is a list of IQN(s) available. This discovery is a mandatory step of connecting to the iSCSI target.

Login to the iSCSI target:

# iscsiadm -m node -T -p -l

If that all works, you have new SCSI devices available, check dmesg and start iscsi at boot time:

# chkconfig iscsi on

In this example the iSCSI target does not have a filesystem. Create it on the client and mount it at boot time:

# fdisk /dev/sda
# mkfs.ext3 /dev/sda1
# echo "/dev/sda1 /mnt ext3 defaults,_netdev 0 0" >> /etc/fstab

You are done, but these commands are quite useful when connecting to an unknown iSCSI device.

To see more about the IQN:

# iscsiadm -m node -T -p

Using "recording" in VI instead of being annoyed

Here is a trick you can use in VI if you need to repeat an action multiple times. Imagine you have this file:


You want to change it to read "Hello world!" Here is what you can do in VI:

  1. Go to the first occurrence of what you would like to have changed.
  2. Enter "q" to initiate a recording.
  3. Enter a letter or digit to save the recording under, like "a" or "1".
  4. Do the actions you would like to repeat.
  5. Enter "q" again to stop and save the recording.
  6. Go to the line where you would like to start running the saved recording/macro.
  7. Enter "@" followed by the letter of digit you have saved the recording under, like "a" or "1".

So, now you have learned what this (annoying) "recording" option can be used for!

Debugging an SSL connection

Imagine you would like to debug an SSL connection to a box, for example HTTPS. Using telnet or netcat, you are not able to do so, but openssl makes this possible, here is how:

First; connect to a machine:

$ openssl s_client -connect

You will be given the certificate details:

depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify return:1
depth=0 /C=US/ST=California/L=Mountain View/O=Google Inc/
verify return:1
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/
  i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
  i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
No client certificate CA names sent
SSL handshake has read 1778 bytes and written 343 bytes
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
   Protocol  : TLSv1
   Cipher    : AES256-SHA
   Session-ID: 53530BBF94619E255B7956A18D9B9F26241B2A1BF16F30C18C73C88A60200E5F
   Key-Arg   : None
   Krb5 Principal: None
   Start Time: 1264674178
   Timeout   : 300 (sec)
   Verify return code: 0 (ok)

Now you can enter commands in plain text, just as you would using telnet or netcat:


Here is what you get:

HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Content-Length: 218
Date: Thu, 28 Jan 2010 10:23:05 GMT
Server: GFE/2.0
X-XSS-Protection: 0


An easy trick that can also be used to connect to SSL-ed STMP connections, IMAPS, POP3S, etc.

Making an RPM for a shell script.

So, you have written an enterprise quality shell script and would like to deploy it on serveral Red Hat based machines? Creating an RPM will make this easy to do. Here are the steps required.

1. Install rpmbuild so you may start to build your own RPMs.
2. Package your shell script into a tar.gz file and move that to /usr/src/redhat/SOURCES/

# tar -cvzf shell-script-0.1.tar.gz shell-script-0.1
# mv shell-script-0.1.tar.gz /usr/src/redhat/SOURCES/

3. Create a .spec file that describes where everything is.
# cat /usr/src/redhat/SPECS/shell-script.spec
Summary: The do it all script. (Enterprise quality)
Name: shell-script
Version: 0.1
Release: 1
License: GPL
Group: Applications/Internet
BuildRoot: %{_tmppath}/%{name}-root
Requires: bash
Source0: shell-script-%{version}.tar.gz
BuildArch: noarch

A shell script.



rm -rf ${RPM_BUILD_ROOT}
mkdir -p ${RPM_BUILD_ROOT}/usr/bin
install -m 755 ${RPM_BUILD_ROOT}%{_bindir}

rm -rf ${RPM_BUILD_ROOT}

%attr(755,root,root) %{_bindir}/

* Tue Jan 12 2010 Robert de Bock <[email protected]>
- Uberscript!

3. Build it!
# rpmbuild --bb /usr/src/redhat/SPECS/shell-script.spec

4. Install it!
# rpm -Uvh /usr/src/redhat/RPMS/noarch/shell-script-0.1.1.noarch.rpm

Ranges in shell scripts

I have been admiring the people who know how to use ranges in shell scripts. These people are faster and more fluent on the Linux command line than anybody without knowledge of ranges could be.

Here are some ranges or patters that you could use.

A sequence of characters, in this case 1 to 10, printed on one line.

$ echo "file{1..9}"
file1 file2 file3 file4 file5 file6 file7 file8 file9
$ echo file{s..z}
files filet fileu filev filew filex filey filez

The order of the range can be found in the man-page for "ascii".

A pattern that describes either a range, or a separated pattern.

$ echo file{1,2,4}
file1 file2 file4

Additional information can be found in the man page of bash, under "Brace Expansion".

Restore hidden files with Apple Mac OS X Time Machine

Apple's Time Machine works great, but restoring hidden files (files that start with a dot, like .ssh, .bashrc or .Trash) is difficult, but possible!

Time machine uses the settings as used by the Finder. So first step is to change Finders behaviour, to show hidden files. Execute this command (as a regular user) from within the Terminal.

$ defaults write AppleShowAllFiles TRUE
$ killall Finder

Now you should be able to see extra files in the finder, like this:

Now start Time Machine and scroll back to the date you were sure a file existed.

Restore it and to hide all these (annoying) hidden files, revert to original Finder settings:

$ defaults write AppleShowAllFiles FALSE
$ killall Finder

Nagios time check using SNMP

When you would like to retrieve the remotely configured time using SNMP and compare it to see how accurate the time is, here is a script to help you out.

This setup does not specifically require NTP to be running on the hosts that are checked, it just requires that the time is correct. Virtual machines for example are advised to have the appropriate "tools" installed to synchronize time. NTP is not desirable for virtual machines.

(Parts of the script are borrowed from

This is the graph that is created:

The script:


# Nagios plugin to report time difference as received via SNMP compared to the local time.
# Make sure the machine this script runs on (poller/nagios host) is using NTP.

usage() {
# This function is called when a user enters impossible values.
echo "     The host to check, either IP address or a resolvable hostname."
echo " -C COMMUNITY"
echo "     The SNMP community to use, defaults to public."
echo " -v VERSION"
echo "     The SNMTP version to use, defaults to 2c."
echo " -w WARNING"
echo "     The amount of seconds from where warnings start. Defaults to 60."
echo " -c CRITICAL"
echo "     The amount of seconds from where criticals start. Defaults to 120."
exit 3

readargs() {
# This function reads what options and arguments were given on the
# command line.
while [ "$#" -gt 0 ] ; do
  case "$1" in
    if [ "$2" ] ; then
     shift ; shift
     echo "Missing a value for $1."
    if [ "$2" ] ; then
     shift ; shift
     echo "Missing a value for $1."
    if [ "$2" ] ; then
     shift ; shift
     echo "Missing a value for $1."
    if [ "$2" ] ; then
     shift ; shift
     echo "Missing a value for $1."
    if [ "$2" ] ; then
     shift ; shift
     echo "Missing a value for $1."
    echo "Unknown option $1."

checkvariables() {
# This function checks if all collected input is correct.
if [ ! "$host" ] ; then
  echo "Please specify a hostname or IP address."
if [ ! "$community" ] ; then
  # The public community is used when a user did not enter a community.
if [ ! "$version" ] ; then
  # Version 2c is used when a user did not enter a version.
if [ ! "$critical" ] ; then
if [ ! "$warning" ] ; then

getandprintresults() {
# This converts the date retreived from snmp to a unix time stamp.
rdatestring=$( snmpget -v $version -c $community $host HOST-RESOURCES-MIB::hrSystemDate.0 | gawk '{print $NF}' )

if [ ! "$rdatestring" ] ; then
  echo "Time difference could not be calculated; no time received."
  exit 3

rdate=$( echo $rdatestring | gawk -F',' '{print $1}' )
rtime=$( echo $rdatestring | gawk -F',' '{print $2}' | gawk -F'.' '{print $1}' )
cldate=$( echo $rdate | gawk -F'-' '{printf("%4i",$1)}; {printf("%02i",$2)}; {printf("%02i",$3)};' )
cltime=$( echo $rtime | gawk -F':' '{printf("%02i",$1)}; {printf("%02i",$2)}; {printf(" %02i",$3)};' )
rdate_s=$( date -d "$cldate $cltime sec" +%s )
ldate_s=$(date +'%s')

# If the calculated difference is negative, make it positive again for comparison.
difference=$(($rdate_s - $ldate_s))
if [ "$difference" -lt 0 ] ; then

if [ "$positivedifference" -gt "$critical" ] ; then
  echo "Time difference is more than $critical seconds: $difference|diff=$difference"
  exit 2

if [ "$positivedifference" -gt "$warning" ] ; then
  echo "Time difference is more than $warning seconds: $difference|diff=$difference"
  exit 1

echo "Time difference is less than $warning seconds: $difference|diff=$difference"
exit 0

# The calls to the different functions.
readargs "[email protected]"

To implement it in Nagios, add these sniplets to nagios.cfg. (or any other applicable nagios file.)

The service for a group.

define service{
        hostgroup_name                  Servertype_Linux
        service_description             time
        _SERVICE_ID                     1856
        use                             SNMP-time

The service template.

define service{
        name                            SNMP-time
        service_description             time
        use                             generic-service
        check_command                   check_snmp_time!$_HOSTSNMPCOMMUNITY$!120!60
        max_check_attempts                      30
        normal_check_interval           5
        retry_check_interval            1
        notification_interval           0
        register                                0

The command.

define command{
        command_name                    check_snmp_time
        command_line                    $USER1$/check_snmp_time -H $HOSTADDRESS$ -C $ARG1$ -c $ARG2$ -w $ARG3$

Optimize only fragmented tables in MySQL

When you are using MySQL, you will (likely) have tables that can be fragmented. In MySQL terms this is called "OPTIMIZE".

You could simply OPTIMIZE every table in every database, but during an OPTIMIZE, the tables are locked, so writing is not possible.

To minimize the time that MySQL will be locked (and results cannot be written), here is a script that checks fragmentation of every table of every database. Only if a table is fragmented, the table is OPTIMIZED.


echo -n "MySQL username: " ; read username
echo -n "MySQL password: " ; stty -echo ; read password ; stty echo ; echo

mysql -u $username -p"$password" -NBe "SHOW DATABASES;" | grep -v 'lost+found' | while read database ; do
mysql -u $username -p"$password" -NBe "SHOW TABLE STATUS;" $database | while read name engine version rowformat rows avgrowlength datalength maxdatalength indexlength datafree autoincrement createtime updatetime checktime collation checksum createoptions comment ; do
  if [ "$datafree" -gt 0 ] ; then
   fragmentation=$(($datafree * 100 / $datalength))
   echo "$database.$name is $fragmentation% fragmented."
   mysql -u "$username" -p"$password" -NBe "OPTIMIZE TABLE $name;" "$database"

Result will look something like this:

MySQL username: root
MySQL password:
database.cache_filter is 19% fragmented.
meinit.cache_filter optimize status OK
database.cache_page is 35% fragmented.
meinit.cache_page optimize status OK

You may comment out that line with OPTIMIZE TABLE in it, if you are just interested in seeing the fragmentation.

Failover on OpenBSD is so easy to setup using carp!

I am not the first (and last) to write about carp, the failover/vip/floating-IP solution OpenBSD is using. Many articles describe this topic including a very complete answer to a frequently asked question about carp.

If you are not familiar with IP failover situations; in case of carp/pulse/HSRP/VIP, an IP "floats" between different machines. One machine actually answers request to received packets, so this is an solution that knows of a MASTER of ACTIVE node .

A CARP interface (which is not physical) is bound to a physical interface. The physical interface advertises statuses so other CARP interfaces know about each other.

You can bind almost any service to a CARP interface, some examples are:

  • DNS
  • HTTP
  • NTP
  • Proxy/Squid

Services that store data/stadia locally are not very suitable for a CARP solution. Examples are: DHCP (because leases are stored localy), MySQL/PostgreSQL (because data is stored on a physical local storage) and SSH (because you can never be sure what machine you are connecting to.

Here is how to set it up. On both boxes add a file /etc/hostname.carp0 with this content:

inet 255.255.0 vhid 1 pass SeCrEt carpdev em0

Remember to activate the interface like this: (All your network cards will be (re-) configured!)

# sh /etc/netstart

In this case, is the floating IP address and em0 is the physical device that carp0 is running on. Be aware that the other server's carpdev should be connected to the same LAN.

Now that this is done, you may access services on the newly created CARP device's IP address. You may also specifically bind applications to only the CARP device.

You may check the status using ifconfig: (Please not the "carp: MASTER" part, it tells you this machine is the master, all others are "BACKUP".)

# ifconfig carp0
        lladdr 00:00:5e:00:01:02
        priority: 0
        carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 0
        groups: carp
        inet6 fe80::200:5eff:fe00:102%carp0 prefixlen 64 scopeid 0x5
        inet netmask 0xffffff00 broadcast

One limitation I found; you can not run dhclient on a carp interface, you will need to assign an IP address to the carp device. Please be aware that this would be a very odd setup; DHCP in a failover interface...

Your Soekris OpenBSD as a OpenVPN appliance

I have an existing network at home, but would like to be able to connect to it using a VPN every now and then. This enables me to access the fileserver, printer and so on.

My network contains an Apple Time Capsule as a nat router, an ethernet modem provided by my cable company Ziggo and devices such as laptops, that use the network.

A Soekris box I had lying around meets all requirements perfectly for a VPN-server. Here is how to set it up.

1. Forward UDP port 1194 from your router to your soekris box.

This one is easy enough, on Apple Mac OS X and a Time Capsule (or Airport Express) open AirPort Utility on your Mac, select the Time Capsule, click Manual Setup.
Go to Internet - NAT
Select the box "Enable NAT Port Mapping Protocol" and click on "Configure Port Mappings..."

Click on the "+" to add a portmapping. OpenVPN uses UDP port 1194, so map it from the "Public UDP Port(s)" to the "Private UDP Port(s)" on the "Private IP Address" of your soekris box. Fill in "OpenVPN" in the next "Description" field.

Finish your router configuration by pressing "Update". N.B. The network connection will be gone for a minute or two.

2. Install OpenVPN

I assume OpenBSD is already running on your Soekris box, otherwise check out how to install your soekris box with OpenBSD.
Add the package "openvpn". A dependency "lzo" will be added automatically.

3. Configure OpenVPN

Create a directory /etc/openvpn/keys:

soekris # mkdir -p /etc/openvpn/keys

And create the file /etc/openvpn.conf with this content:
port 1194
proto udp
dev tun0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key                                 
dh /etc/openvpn/keys/dh1024.pem
# This is the network that lives on the tun0 device.
# My regular network uses, so using
# seems pretty logical.
ifconfig-pool-persist ipp.txt
# When clients connect, tell them that can
# be reached through this tunnel. (You may also set this on the,
# client instead of "broadcasting" this...
push "route"
keepalive 10 120
user _openvpn
group _openvpn
status openvpn-status.log
verb 3

4. Create OpenSSL certificates

This is quite an abstract step. It boils down to this: on the server you will create a certificate authority (ca) key and certificate, also you will create a key and certificate for each client connecting and sign them using your newly create certificate authority. The certificate from the certificate authority (ca.crt) and client (client1.crt) and the key for the client (client1.key) will be distributed to all clients. That's a mouth full, but here is how to do it in steps:

soekris # cp -Rip /usr/local/share/example/openvpn/easy-rsa /etc/openvpn
soekris # cd /etc/openvpn/easy-rsa/2.0
soekris # cat vars
export EASY_RSA="`pwd`"
export OPENSSL="openssl"
export PKCS11TOOL="pkcs11-tool"
export GREP="grep"
export KEY_CONFIG="/etc/openvpn/easy-rsa/2.0/openssl.cnf"
export KEY_DIR="/etc/openvpn/keys"
echo NOTE: If you run ./clean-all, I will be doing a rm -rf on $KEY_DIR
export PKCS11_MODULE_PATH="dummy"
export PKCS11_PIN="dummy"
export KEY_SIZE=1024
export CA_EXPIRE=3650
export KEY_EXPIRE=3650
export KEY_CITY="Utrecht"
export KEY_ORG="Me in It Consultancy"
export KEY_EMAIL="[email protected]"

N.b. Please change the KEY_ values to match your personal settings.

Now execute these steps, as stolen from The OpenVPN homepage.

soekris # . vars
soekris # ./clean-all
soekris # ./build-ca
soekris # ./build-key-server server
soekris # ./build-key client1
soekris # ./build-key client2
soekris # ./build-key client3
soekris # ./build-dh

Once again; send the newly created file /etc/openvpn/keys/ca.crt, /etc/openvpn/keys/client1.crt and /etc/openvpn/keys/client1.key to the machine using the vpn connection.

5. Configure the OpenBSD Packet Filter

This step enables client to reach your local network using network address translation. At the bare minimum, add this rule to your pf configuration in /etc/pf.conf

nat pass on sis0 from !(sis0) to any -> (sis0)

sis0 is a physical interface that connects the Soekris box to my local area network.

Also, make sure the packet filter is enabled and is using your pf.cofn

soekris # pfctl -e
soekris # pfclt -f /etc/pf.conf

And finally make sure it works after a reboot:

soekris # echo "ps=yes" >> /etc/rc.conf.local

6. Start OpenVPN on the server

Wow, almost there, let's start the software:

soekris # /usr/local/sbin/openvpn --config /etc/openvpn/server.conf --key /etc/openvpn/keys/server.key

Some debugging information will scroll down your screen.

7. Make sure OpenVPN starts at boot time

Add these lines to your /etc/rc.local.

# Add your local startup actions here.

echo " openvpn"
/usr/local/sbin/openvpn --config /etc/openvpn/server.conf --key /etc/openvpn/keys/server.key >> /var/log/openvpn.output &

echo '.'

8. Configure the client(s)

I use Mac OS X to connect to OpenVPN. You will have to install some extra software, your choices are:

  • Tunnelblick - (Free) Tunnelblick is a ready-to-use graphical OpenVPN client for Mac OS X
  • Viscosity - is an OpenVPN client for Mac, providing a rich Cocoa graphical user interface for creating, editing, and controlling VPN connections.

For now I am using the trail version of Viscosity because it looks great. Check out the screenshots below.

Syndicate content