Adventures in Red Hat Enterprise Linux, CentOS, Fedora, OpenBSD and other open source solutions.

Examples for batch on linux

Linux has a few ways to schedule jobs to be executed. I am sure most are familiar with crontab and at, but batch is lesser known.

"batch" can be used to: (from the man-page on batch)
executes commands when system load levels permit; in other words, when the load average drops below 0.8, or the value specified in the invocation of atrun.


  • crontab is used for periodic scheduling.
  • at is used for executing something once at a specific time.
  • batch can be used to execute commands when your system has resourses.

You can also combine crontab and batch. Imagine you need to run a sequence of command in a specific order every hour. crontab does not guarantee one command is finished when it executs the next command.
batch can be used from crontab like so:

crontab -l
0 * * * * /usr/bin/batch now /usr/local/bin/
1 * * * * /usr/bin/batch now /usr/local/bin/
2 * * * * /usr/bin/batch now /usr/local/bin/

This batches these three commands in a specific order, one after the other, when the systemload is not too high.

One specific situation where I use this; Drupal needs to run a program (cron.php) every hour. crontab would be perfect for that, but when the load is too high, it's not a problem that this program is executed a little later. This is what I have setup:

0 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null
1 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null
2 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null

This ensures that every hour cron.php is ran, but not if the systemload is too high (0.8 or more). One disadvantage of this solution; when your system is overloaded for a long period of time, these batch jobs pile up, then when the load drops below 0.8, all batched commands will be executed. Happily Drupals cron.php will not consume that much resources when it's ran twice.

Release scheme for RPM based Linux distributions

It can be rather confusing what the differences and similarities are on Fedora, Red Hat Enterprise Linux and CentOS. Especially with different versions. This article explains what release schedule and relations the various RPM based Linux distributions have.

Fedora is a Red Hat sponsored community project. Fedora is release approximately every 6 month. Fedora "supports" (supplies updates) for 13 month only. Clearly this is a development distribution.

Red Hat picks up a Fedora version and adds a few patches and call that "Red Hat Enterprise Linux". Red Hat supports that version for quite some time. Red Hat releases more conservatively; every 2 years. Red Hat supports a release for about 5 years after releasing, making this distribution much more "enterprise".

Fedora - Red Hat release relationship
Fedora release Red Hat release
Fedora Core 3 Red Hat Enterprise Linux 4
Fedora Core 6 Red Hat Enterprise Linux 5
Fedora Core 13 Red Hat Enterprise Linux 6

CentOS picks up the source code that Red Hat published for Red Hat Enterprise Linux. The CentOS community patches the artwork and very few other things. CentOS "supports" (provides updates) for as long as Red Hat supplies updates to Red Hat Enterprise Linux.
Interesting to know; once you have choosen to use a certain main version of CentOS, you'll automatically update to the most recent child-version of that release when using "yum update". So; if you install "CentOS 5.0" and run "yum update", you will automatically have "CentOS 5.7". (at the time of this writing.)

So; this makes this image:

Apache Tomcat 7 spec file RPM

I tried to find an RPM for Apache Tomcat version 7, but could not find one. You can use this one, it requires the source code, downloadable from Apache Tomcat's website, under "Source Code Distributions".

This SPEC file creates an RPM "apache-tomcat" that installs in /opt/tomcat, and the default web applications (apache-tomcat-manager, apache-tomcat-ROOT, apache-tomcat-docs, apache-tomcat-examples, apache-tomcat-host-manager). An init-script is included at the bottom, that needs to be available in the SOURCES directory, named "apache-tomcat-iniscript".

Apache Tomcat SRC rpm

Apache Tomcat 86_64 rpm
Apache Tomcat ROOT application 86_64 rpm
Apache Tomcat docs 86_64 rpm
Apache Tomcat example application 86_64 rpm
Apache Tomcat host manager application 86_64 rpm
Apache Tomcat manager application 86_64 rpm

So far it's been fine, but any comments would be appreciated.


Name: apache-tomcat
Version: 7.0.20
Release: 1
Summary: Open source software implementation of the Java Servlet and JavaServer Pages technologies.
Group: Productivity/Networking/Web/Servers
License: Apache Software License.
Source: %{name}-%{version}-src.tar.gz

BuildRoot: %{_tmppath}/%{name}-%{version}-build
BuildRequires: ant
BuildRequires: ant-trax
Requires: java-1.6.0-openjdk
BuildArch: x86_64

Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies. The Java Servlet and JavaServer Pages specifications are developed under the Java Community Process.

%package manager
Summary: The management web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name} = %{version}-%{release}
BuildArch: noarch

%description manager
The management web application of Apache Tomcat.

%package ROOT
Summary: The ROOT web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch

%description ROOT
The ROOT web application of Apache Tomcat.

%package docs
Summary: The docs web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch

%description docs
The docs web application of Apache Tomcat.

%package examples
Summary: The examples web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch

%description examples
The examples web application of Apache Tomcat.

%package host-manager
Summary: The host-manager web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch

%description host-manager
The host-manager web application of Apache Tomcat.


%setup -q -n %{name}-%{version}-src
# This tells ant to install software in a specific directory.
cat << EOF >>


rm -Rf %{buildroot}
mkdir -p %{buildroot}/opt/apache-tomcat
mkdir -p %{buildroot}/opt/apache-tomcat/pid
mkdir -p %{buildroot}/opt/apache-tomcat/webapps
mkdir -p %{buildroot}/etc/init.d/
mkdir -p %{buildroot}/var/run/apache-tomcat
%{__cp} -Rip ./output/build/{bin,conf,lib,logs,temp,webapps} %{buildroot}/opt/apache-tomcat
%{__cp} %{_sourcedir}/apache-tomcat-initscript %{buildroot}/etc/init.d/apache-tomcat

rm -rf %{buildroot}

getent group tomcat > /dev/null || groupadd -r tomcat
getent passwd tomcat > /dev/null || useradd -r -g tomcat tomcat

chkconfig --add %{name}

if [ "$1" = "0" ] ; then
service %{name} stop > /dev/null 2>&1
chkconfig --del %{name}

%dir /opt/apache-tomcat
%config /opt/apache-tomcat/conf/*
%dir /opt/apache-tomcat/webapps
%attr(0755,root,root) /etc/init.d/apache-tomcat

%files manager

%files ROOT

%files docs

%files examples

%files host-manager

* Fri Aug 19 2011 - robert (at)
- Updated to apache tomcat 7.0.20
- Split (example) applications into their own RPM.
* Mon Jul 4 2011 - robert (at)
- Initial release.


# apache-tomcat
# chkconfig: - 85 15
# description: Jakarta Tomcat Java Servlets and JSP server
# processname: java
# pidfile: /var/run/apache-tomcat/pid

. /etc/rc.d/init.d/functions

# Set Tomcat environment.
export BASEDIR=/opt/apache-tomcat
export CATALINA_PID=/var/run/apache-tomcat/pid
export CATALINA_OPTS="-DHOME=$BASEDIR/home -Xmx512m -Djava.awt.headless=true"

case "$1" in
        echo -n "Starting apache-tomcat: "
        status -p $CATALINA_PID apache-tomcat > /dev/null && failure || (su -p -s /bin/sh $USER -c "$TOMCAT_HOME/bin/ start" > /dev/null && (touch $LOCKFILE ; success))
        echo -n "Shutting down apache-tomcat: "
        status -p $CATALINA_PID apache-tomcat > /dev/null && su -p -s /bin/sh $USER -c "$TOMCAT_HOME/bin/ stop" > /dev/null && (rm -f $LOCKFILE ; success) || failure
        $0 stop
        $0 start
       [ -e $LOCKFILE ] && $0 restart
        status -p $CATALINA_PID apache-tomcat
        echo "Usage: $0 {start|stop|restart|condrestart|status}"
        exit 1

service status somedaemon - status: unrecognized service

I created an initscript for a daemon recently it works well for starting and stopping, but "service status somedaemon" would return "status: unrecognized service"

After a bit of googling I found that the top few lines of the initscript are critical. It should be a describe here:

# somedaemon
# chkconfig: - 85 15
# description: Some daemon
# processname: somedaemon
# pidfile: /var/run/

Here is what these lines do.

  1. #!/bin/sh - This is a shell script.
  2. # somedaemon - This is the name of the service used with for "service status somedaemon". If this is missing, you'll see: "status: unrecognized service".
  3. chkconfig: - 85 15 - This tells chkconfig three things, firstly (the dash) at what runlevel to start it. A dash means default. Secondly at what priority to start and lastly at what priority to stop.
  4. description: Some daemon - Not used.
  5. processname: somedaemon - Not used.
  6. pidfile: /var/run/ - Used in /etc/rc.d/init.d/functions to determine if the process is running or not.

Linux interview questions

From time to time you might need to interview somebody for a Linux role. It's hard to think of good questions; you don't want to scare somebody with your questions, but you do want to know if the person is knowledgable.

Questions for a junior.

"People that use your Linux server complain it's slow. What tools would you use to check resource usage?"
top, sar, netstat -an, iostat, free, df.

"You discover a disk is full on your Linux server. What do you use to discover where the bigest files/directories are?"

  • To create a short report on the largest directories: du -sk * | sort -nr | head -n10
  • You might use "find" to find large files: find / -size +100M

"A system needs more disk space. You want to add a partition /var/log. You add a drive to the machine, it becomes /dev/sdb. What actions do you take to use this disk?"
Depending if you use LVM or not. If you don't use LVM:

  1. "fdisk /dev/sdb" to add one partition so /dev/sdb1 becomes available.
  2. "mkfs /dev/sdb1" to create a filesystem on it.
  3. "mount /dev/sdb1 /var/log". Actually the data needs to be copied into the new /var/log.

If you do use LVM:

  1. "fdisk /dev/sdb" to add one partition so /dev/sdb1 becomes available.
  2. "pvcreate /dev/sdb1" to make it an LVM device.
  3. "vgextend SomeVolumeGroup /dev/sdb1" to extend an existing Volume Group.
  4. "lvcreate -L 1G -n SomeLogicalVolumeName SomeVolumeGroup" to create a new Logical Volume.
  5. "mkfs /dev/SomeVolumeGroup/SomeLogicalVolumeName" to create a filesystem on it.
  6. "mount /dev/SomeVolumeGroup/SomeLogicalVolumeName /var/log". Actually the data needs to be copied into the new /var/log.

Questions for a medior.

"People that use your Linux server complain it's slow. You've seen the disk usage is high. What can you do to improve this?"

  • Move away services, devide them over different server.
  • Add more memory so disk caching can be used better.
  • Look into the application, why is it reading/writing so much.
  • User faster disks.

"What happens in relation to DNS, SMTP and IMAP when I send send an email to [email protected]?"
My computer is likely to be configure to send the email to a mail server on port 25, SMTP protocol. That mailserver will query the DNS for the MX records of The mailserver that show up with the lowest priority will be contacted to deliver the email. That mailserver at can accept the email and put in into the imap folder for the user or alias of [email protected]

"You need to setup 100 Red Hat Enterprise Linux systems. If you don't want to walk around and eject and insert a boot CD 100 times, what options would you have?"
Kickstarting would help out. Install one machine as you like it, save /root/anaconda-ks.cfg to a webserver. Setup a PXE (DHCP, TFTP, HTTP, DNS) environment and use that kickstart file to install the rest.

Questions for a senior.

"You have destroyed /etc/pam.d/system-auth and can't login anymore. Another machine has a propper version of /etc/pam.d/system-auth. How would you fix that broken machine?"
The machine needs to be booted in single user mode so you don't get a login prompt. After that here are some option:

  • Start network, use "nc" to get and replace that file.
  • Mount a CD that has the package to fix the broken file.
  • Manually repair it.

"You have installed apache, php and mysql and a webapplication such as Drupal. The webapplication tries to send emails to an external mailserver but fails. What could be the cause when these items have been verified:

  • The web application is correctly configured to use the external mailserver.
  • It's possible to connect to the mailserver on the specified port from the command line.
  • The logs of the mailserver don't record anything when the web application tries to send email."
  • IPtables allows connections out to port 25.

SELinux could be blocking apache from using port 25 on an external system. The logfile /var/log/messages might inform you about it. To fix it issue "setsebool -P httpd_can_network_connect=1".

"What determines the load (w, uptime) of a system?"
The number of processes that are waiting for execution. These processes are in the run queue. Processes could be waiting for io, network or memory allocation.

Drupal on Amazon's Elastic Compute Cloud (EC2)

Hosting Drupal site in the Amazon EC2 cloud is not difficult. Here is a recipe I have used, first attempt was a Fedora 14 EC2 ami, but Fedora 14 comes with php 5.3, which can't be combined with Drupal 5.x. If you only have Drupal 6 (or Drupal 7) sites to host, you can use Fedora 14. If you want to use a "small" instance, please read this bug about readdir64_r. The fix for that bug is easy:

echo "hwcap 1 nosegneg" > /etc/

Let's continue with Drupal on CentOS. Rightscale provides perfect CentOS amis that can be used on Amazons EC2 platform. If you install one, these are the steps I took to make it Drupal 5, Drupal 6 and Drupal 7 ready:

# Update the software.
yum -y update

# Set the timezone for this machine.
cp /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime

# This images came with 10Gb of ESB storage, I added another 64 Gb volume, here it's called "/dev/sdc".
# Use LVM to be ready to grow in the future.
pvcreate /dev/sdc
vgcreate vg0 /dev/sdc
lvcreate vg0 -L 32G -n var-www
lvcreate vg0 -L 1G -n var-lib-mysql
lvcreate vg0 -L 2G -n root

# Put filesystems on the logical volumes.
mke2fs -j /dev/vg0/var-www
mke2fs -j /dev/vg0/var-lib-mysql
mke2fs -j /dev/vg0/root

# Add the mountpoints to fstab.
echo "/dev/vg0/var-www /var/www/virtualhosts ext3 defaults 0 0" >> /etc/fstab
echo "/dev/vg0/var-lib-mysql /var/lib/mysql ext3 defaults 0 0" >> /etc/fstab
echo "/dev/vg0/root /root ext3 defaults 0 0" >> /etc/fstab

# Create the mountpoints.
mkdir /var/www /var/lib/mysql /root

# Mount all mounpoints in /etc/fstab.
mount -a

# Install the webserver.
yum -y install httpd
service httpd start
chkconfig httpd on

# Install the database server.
yum -y install mysql-server
service mysqld start
chkconfig mysqld on
/usr/bin/mysqladmin -u root password 'YourPassWord'

# Install PHP and all required Drupal php modules.
yum -y install php php-mysql php-mcrypt php-xml php-mbstring php-gd

# Add a single new file in the ESB root filesystem, that includes configurations from /var/www/conf.d
# Using this trick allows you to easily remount the volume on another host in case of troubles.
echo 'Include /var/www/conf.d/*.conf' >> /etc/httpd/conf.d/virtualhosts.conf
mkdir /var/www/conf.d

# Rightscale CentOS images comes with postfix and sendmail. Postfix is enabled, but sendmail is fine for me.
# First erase postfix.
yum -y erase postfix

# Now reinstall sendmail to fix a few permissions.
yum -y reinstall sendmail
service sendmail start
chkconfig on sendmail

# Reboot the box to make sure it's working properly.

Ranges in shell scripts

Het is altijd bewonderings waardig om te zien hoe mensen ranges kunnen gebruiken in shell scripts. Mensen die ranges beheersen, zijn sneller op de Linux command line.

Hier zijn wat ranges en patronen die je zou kunnen gebruiken:

Een oplopende reeks van karakters, in dit geval 1 tot 10, getoond op één lijn:

$ echo "file{1..9}"
file1 file2 file3 file4 file5 file6 file7 file8 file9
$ echo file{s..z}
files filet fileu filev filew filex filey filez

De volgorde van ranges kun je vinden in de man-page van "ascii".

Nog een patroon dat je kunt gebruiken:

$ echo file{1,2,4}
file1 file2 file4

Als je meer informatie zoek, Google eens naar "Brace Expansion" of kijk in de man-page van bash.

Een RPM maken van een shell-script

Dus je hebt een enterprise kwaliteit shell script gemaakt en wil het uitrollen over een setje Red Hat-achtige machines? Een RPM maken is makkelijk, hier zijn de stappen die nodig zijn:

1. Installeer rpmbuild zodat je jouw eigen RPMs kunt gaan bouwen.
2. Pak je script in een tar.gz file en verplaats het naar /usr/src/redhat/SOURCES/

# tar -cvzf shell-script-0.1.tar.gz shell-script-0.1
# mv shell-script-0.1.tar.gz /usr/src/redhat/SOURCES/

3. Maak een .spec file dat beschrijft waar alles is.
# cat /usr/src/redhat/SPECS/shell-script.spec
Summary: The do it all script. (Enterprise quality)
Name: shell-script
Version: 0.1
Release: 1
License: GPL
Group: Applications/Internet
BuildRoot: %{_tmppath}/%{name}-root
Requires: bash
Source0: shell-script-%{version}.tar.gz
BuildArch: noarch

A shell script.



rm -rf ${RPM_BUILD_ROOT}
mkdir -p ${RPM_BUILD_ROOT}/usr/bin
install -m 755 ${RPM_BUILD_ROOT}%{_bindir}

rm -rf ${RPM_BUILD_ROOT}

%attr(755,root,root) %{_bindir}/

* Tue Jan 12 2010 Robert de Bock <[email protected]>
- Uberscript!

3. Bouw het!
# rpmbuild --bb /usr/src/redhat/SPECS/shell-script.spec

4. Installeer het!
# rpm -Uvh /usr/src/redhat/RPMS/noarch/shell-script-0.1.1.noarch.rpm

Een loadbalancer maken met CentOS en Linux Virtual Server

Als je nieuw bent in de Red Hat terminology, kan het best wel moeilijk zijn om te begrijpen hoe je een simpele loadbalancer kunt maken met Linux. Hier staat wat informatie om je op weg te helpen.

Lees meer op the CentOS Virtual Server Administration paginas.


  • piranha This is het pakket dat je helpt met opzetten van LVS.
  • ipvs De Linux module die het mogelijk maakt te load balancen.
  • ipvsadm Een pakket en commando om de loadbalancer te beheren. Let op dat je of piranha-gui of /etc/sysconfig/ipvsadm gebruikt om de loadbalancer te configuren, niet beide..
  • LVS Linux Virtual Server - de projectnaam die hier steeds genoemd wordt.
  • pulse Een service (/etc/init.d/pulse) die op de actieve en backup machine draait en een poort opent om het mogelijk te maken checks te doen. Piranha-gui configureert alles, pulse voert alle configuraties door naar andere machines.
  • nanny Een proces gestart door pulse om de echte servers (en services) the checken.
  • nat Network Address Translation. Een gebruikelijke combinatie met LVS. Als NAT gebruikt wordt, accepteert de loadbalancer/director/LVS verkeer op een VIP en stuurt het door naar de echte servers. Let op dat de huidige implementatie van ipvsadm niet goed kan Source NATten, het source adres wordt niet goed herschreven.
  • direct routing Een methode van LVS om te routeren. Verkeer wordt ontvangen op de VIP en doorgestuurd naar de echte server, alsof het verstuurd is naar de VIP van de echte server. Hierdoor moeten de echte servers wel reageren op het VIP adres. Om dit te laten werken, moeten er een aantal "trucs" gedaan worden zodat de arp-cache gezond blijft werken.
  • wlc Weighted Least Connections, een algoritme om verkeer naar de echte servers te balancere.
  • VIP Virtual IP. Het IP-address waarop de service geconfigureerd is.
  • RIP Real (Echte) server IP. Het IP-address van de echte server.
  • realserver/echte server De server die de uiteindelijke dienst draait. Dit kan een Windows, Linux of elk soort machine zijn.

Hier is een overzicht van een mogelijke setup:

Stappen die je moet volgen voor eeuwige glorie:

Installeer piranha-gui

# yum install piranha-gui

(Verander naar het netwerk dat gebruikt wordt om de echte server te bereiken.)

Configureer services

# chkconfig httpd on
# chkconfig piranha-gui on
# chkconfig pulse on
# sed -i 's/net.ipv4.ip_forward = 0/net.ipv4.ip_forward = 1/' /etc/sysctl.conf
# echo "*nat
:OUTPUT ACCEPT [431:32534]
:INPUT ACCEPT [4186:1094786]
:FORWARD ACCEPT [729:111992]
:OUTPUT ACCEPT [4266:388099]
-A FORWARD -i eth1 -j ACCEPT
COMMIT" > /etc/sysconfig/iptables
# chkconfig iptables on
# sed -i 's/SELINUX=enabled/SELINUX=disabled/' /etc/sysconfig/selinux

Configureer via webinterface

Voor deze stappen uit in de piranha-gui web interface:

# service httpd start
# piranha-passwd
# service piranha-gui start

Open een browser en ga naar het IP adres van 1 van de loadbalancers, de URL ziet er ongeveer zo uit:

Synchroniseer /etc/sysconfig/ha/

Je zult de configuratie die piranha maakt in sync moeten houden op beide machines, dit is een suggestie om dat te doen:

# ssh-keygen
# scp .ssh/id_rsa*
# cp .ssh/ .ssh/authorized_keys

Nu moet je kunnen verbinden met beide machines zonder het opgeven van een wachtwoord.

# cat

copiedserialno=$(grep serial_no /tmp/ | awk '{ print $NF }')
runningserialno=$(grep serial_no /etc/sysconfig/ha/ | awk '{ print $NF }')

if [ "$copiedserialno" -gt "$runningserialno" ] ; then
mv /tmp/ /etc/sysconfig/ha/
# crontab -l
* * * * * /usr/bin/scp /etc/sysconfig/ha/ > /dev/null
* * * * * /root/

Reboot de server.

Om alle veranderingen te testen kun je de load-balancer rebooten.

Fedora 14 komt eraan

Het duurt niet zo lang meer voordat Fedora 14 uit komt. Eens kijken welke features Fedora 14 heeft.

  • Een Amazon EC2 Image (AIM) voor Fedorakomt uit! Eindelijk van dat oude Fedora 8 image af.
  • Voor Desktop Virtualisatie kan, Spice gebruikt worden. Dit is een KVM extensie die het mogelijk maakt super snel gebruik te maken van de desktop die virtueel draait, of het nou Windows, Linux of een andere grafische desktop is.
  • Perl, Python en Ruby zijn geupdate, leuk maar voor mij niet van groot belang.
  • OpenSCAP bekijkt of je systeem not security compiant is. Een handige tool voor veel klanten.

Al met al een best wel interessante release, ookal is het moeilijk een goede distributie te verbeteren.

Veel van de veranderingen die gedaan worden in Fedora komen in RHEL terrecht. Het lijkt erop dat RHEL 7 ongeveer zal bestaan uit Fedora 16 (op zijn vroegst) tot Fedora 19. (op zijn laatst)

Syndicate content