If ever you need to determine what the least or most busy time is on an Apache webserver, you can use this set of Linux commands to get a report:
cut -d: -f 2 /var/log/httpd/*access_log* | sort | uniq -cThe output will be something like this:
290873 00
184948 01
115479 02
84129 03
71059 04
67632 05
88071 06
149285 07
275537 08
431069 09
529708 10
586744 11
599993 12
591466 13
565942 14
585796 15
611814 16
639781 17
625244 18
622163 19
574962 20
558504 21
503386 22
412359 23The first column is the number of hits on the webserver, the second column is the time of the day. In this example the 5th hour (05:00 - 05:59) is the least busy hour, the 18th hour (18:00 - 18:59) is the busiest hour.
Lang verhaal kort: Bezoek The Pirate Bay via Me in IT Consultancy in plaats van de directe URL, zodat je weer torrents kunt downloaden. Als je Ziggo of XS4All gebruikt heeft de rechter besloten dat je thepiratebay.org niet meer mag bezoeken. Right, alsof dat helpt...
Kort verhaal lang: Met Apache, mod_proxy en mod_proxy_html kun je andere websites beschikbaar maken met een Location op een website. Als de webserver in een andere locatie staat, is het goed mogelijk dat je de "afgesloten" website weer kunt bezoeken.
Technisch gezien heb je deze ingredienten nodig om het aan de praat te krijgen.
De configuratie ziet er zo uit:
<VirtualHost *.80>
...
ProxyRequests off
ProxyPass /thepiratebay.org/ http://thepiratebay.org/
ProxyPass /static.thepiratebay.org/ http://static.thepiratebay.org/
ProxyPass /rss.thepiratebay.org/ http://rss.thepiratebay.org/
ProxyPass /torrents.thepiratebay.org http://torrents.thepiratebay.org
<Location /thepiratebay.org/>
ProxyPassReverse /
ProxyHTMLEnable On
ProxyHTMLURLMap http://thepiratebay.org /thepiratebay.org
ProxyHTMLURLMap http://static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://rss.thepiratebay.org /rss.thepiratebay.org
ProxyHTMLURLMap //static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://torrents.thepiratebay.org /torrents.thepiratebay.org
ProxyHTMLURLMap / /thepiratebay.org/
RequestHeader unset Accept-Encoding
</Location>
<Location /static.thepiratebay.org/>
ProxyPassReverse /
ProxyHTMLEnable On
ProxyHTMLURLMap http://thepiratebay.org /thepiratebay.org
ProxyHTMLURLMap http://static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://rss.thepiratebay.org /rss.thepiratebay.org
ProxyHTMLURLMap //static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://torrents.thepiratebay.org /torrents.thepiratebay.org
ProxyHTMLURLMap / /static.thepiratebay.org/
RequestHeader unset Accept-Encoding
</Location>
<Location /rss.thepiratebay.org/>
ProxyPassReverse /
ProxyHTMLEnable On
ProxyHTMLURLMap http://thepiratebay.org /thepiratebay.org
ProxyHTMLURLMap http://static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://rss.thepiratebay.org /rss.thepiratebay.org
ProxyHTMLURLMap //static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://torrents.thepiratebay.org /torrents.thepiratebay.org
ProxyHTMLURLMap / /rss.thepiratebay.org/
RequestHeader unset Accept-Encoding
</Location>
<Location /torrents.thepiratebay.org/>
ProxyPassReverse /
ProxyHTMLEnable On
ProxyHTMLURLMap http://thepiratebay.org /thepiratebay.org
ProxyHTMLURLMap http://static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap http://rss.thepiratebay.org /rss.thepiratebay.org
ProxyHTMLURLMap http://torrents.thepiratebay.org /torrents.thepiratebay.org
ProxyHTMLURLMap //static.thepiratebay.org /static.thepiratebay.org
ProxyHTMLURLMap / /rss.thepiratebay.org/
RequestHeader unset Accept-Encoding
</Location>
...
</VirtualHost>Monitoring an environment with some monitoring system gives control, so it's pretty important. But it can be a challenge to setup a monitoring system; it should not alert too fast, but also not too slow.
Nagios uses "flap detection" to prevent many ERROR's and OK's being sent right after each other. Zabbix calls this "hysteresis". Zabbix's hysteresis is rather difficult to understand, so I'd like to share some triggers that I have setup for Zabbix that implement both flap detection/hysteresis and grace.
Grace can be defined like this: "When a value is higher (or lower) then a threshold, make sure it's a little lower (or higher) as the threshold that caused the trigger to alert, before recovering a trigger." I know; it's not easy to understand... Let's look at some examples.
With values that need to be below a threshold, like cpu load, number of users logged in or number of processes running:
({TRIGGER.VALUE}=0&{TEMPLATE:CHECK[ITEM].min(300)>ALERTVALUE)|({TRIGGER.VALUE}=1&{TEMPLATE:CHECK[ITEM].max(300)<RECOVERYVALUE)Just to clarify the different part of the trigger:
For example CPU load with an ALERTVALUE of 5 and a RECOVERYVALUE of 4:
({TRIGGER.VALUE}=0&{Template_Linux:system.cpu.load[,avg1].min(300)}>5)|({TRIGGER.VALUE}=1&{Template_Linux:system.cpu.load[,avg1].max(300)}<4)With values that need to be above a threshold, like percentage diskspace free, number of inodes free or number of httpd processes running:
({TRIGGER.VALUE}=0&{TEMPLATE:CHECK[ITEM].max(300)<ALERTVALUE)|({TRIGGER.VALUE}=1&{TEMPLATE:CHECK[ITEM].min(300)>RECOVERYVALUE)For example disk space of /var free in percent with an ALERTVALUE of 10 and a RECOVERYVALUE of 11:
({TRIGGER.VALUE}=0&{Template_Linux:vfs.fs.size[/var,pfree].max(300)}<10)|({TRIGGER.VALUE}=1&{Template_Linux:vfs.fs.size[/var,pfree].min(300)}>11)These rather complex triggers will prevent spikes of load or diskusage to cause an alert, but the drawback it that you might miss certain interesting spikes too. Overall my opinion is that a monitoring system should not drive people crazy because alerts will be ignored when too many are received.
Linux has a few ways to schedule jobs to be executed. I am sure most are familiar with crontab and at, but batch is lesser known.
"batch" can be used to: (from the man-page on batch)
executes commands when system load levels permit; in other words, when the load average drops below 0.8, or the value specified in the invocation of atrun.
So:
You can also combine crontab and batch. Imagine you need to run a sequence of command in a specific order every hour. crontab does not guarantee one command is finished when it executs the next command.
batch can be used from crontab like so:
crontab -l
0 * * * * /usr/bin/batch now /usr/local/bin/prepare-something.sh
1 * * * * /usr/bin/batch now /usr/local/bin/process-something.sh
2 * * * * /usr/bin/batch now /usr/local/bin/report-something.shThis batches these three commands in a specific order, one after the other, when the systemload is not too high.
One specific situation where I use this; Drupal needs to run a program (cron.php) every hour. crontab would be perfect for that, but when the load is too high, it's not a problem that this program is executed a little later. This is what I have setup:
0 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null http://1.example.com/cron.php
1 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null http://2.example.com/cron.php
2 * * * * /usr/bin/batch now /usr/bin/wget -o /dev/null -O /dev/null http://3.example.com/cron.phpThis ensures that every hour cron.php is ran, but not if the systemload is too high (0.8 or more). One disadvantage of this solution; when your system is overloaded for a long period of time, these batch jobs pile up, then when the load drops below 0.8, all batched commands will be executed. Happily Drupals cron.php will not consume that much resources when it's ran twice.
It can be rather confusing what the differences and similarities are on Fedora, Red Hat Enterprise Linux and CentOS. Especially with different versions. This article explains what release schedule and relations the various RPM based Linux distributions have.
Fedora is a Red Hat sponsored community project. Fedora is release approximately every 6 month. Fedora "supports" (supplies updates) for 13 month only. Clearly this is a development distribution.
Red Hat picks up a Fedora version and adds a few patches and call that "Red Hat Enterprise Linux". Red Hat supports that version for quite some time. Red Hat releases more conservatively; every 2 years. Red Hat supports a release for about 5 years after releasing, making this distribution much more "enterprise".
| Fedora release | Red Hat release |
| Fedora Core 3 | Red Hat Enterprise Linux 4 |
| Fedora Core 6 | Red Hat Enterprise Linux 5 |
| Fedora Core 13 | Red Hat Enterprise Linux 6 |
CentOS picks up the source code that Red Hat published for Red Hat Enterprise Linux. The CentOS community patches the artwork and very few other things. CentOS "supports" (provides updates) for as long as Red Hat supplies updates to Red Hat Enterprise Linux.
Interesting to know; once you have choosen to use a certain main version of CentOS, you'll automatically update to the most recent child-version of that release when using "yum update". So; if you install "CentOS 5.0" and run "yum update", you will automatically have "CentOS 5.7". (at the time of this writing.)
So; this makes this image:
I tried to find an RPM for Apache Tomcat version 7, but could not find one. You can use this one, it requires the source code, downloadable from Apache Tomcat's website, under "Source Code Distributions".
This SPEC file creates an RPM "apache-tomcat" that installs in /opt/tomcat, and the default web applications (apache-tomcat-manager, apache-tomcat-ROOT, apache-tomcat-docs, apache-tomcat-examples, apache-tomcat-host-manager). An init-script is included at the bottom, that needs to be available in the SOURCES directory, named "apache-tomcat-iniscript".
Downloads:
SOURCE:
Apache Tomcat SRC rpm
x86_64:
Apache Tomcat 86_64 rpm
Apache Tomcat ROOT application 86_64 rpm
Apache Tomcat docs 86_64 rpm
Apache Tomcat example application 86_64 rpm
Apache Tomcat host manager application 86_64 rpm
Apache Tomcat manager application 86_64 rpm
So far it's been fine, but any comments would be appreciated.
apache-tomcat.spec:
Name: apache-tomcat
Version: 7.0.20
Release: 1
Summary: Open source software implementation of the Java Servlet and JavaServer Pages technologies.
Group: Productivity/Networking/Web/Servers
License: Apache Software License.
Url: http://tomcat.apache.org
Source: %{name}-%{version}-src.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-build
BuildRequires: ant
BuildRequires: ant-trax
Requires: java-1.6.0-openjdk
BuildArch: x86_64
%description
Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies. The Java Servlet and JavaServer Pages specifications are developed under the Java Community Process.
%package manager
Summary: The management web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name} = %{version}-%{release}
BuildArch: noarch
%description manager
The management web application of Apache Tomcat.
%package ROOT
Summary: The ROOT web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch
%description ROOT
The ROOT web application of Apache Tomcat.
%package docs
Summary: The docs web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch
%description docs
The docs web application of Apache Tomcat.
%package examples
Summary: The examples web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch
%description examples
The examples web application of Apache Tomcat.
%package host-manager
Summary: The host-manager web application of Apache Tomcat.
Group: System Environmnet/Applications
Requires: %{name}-%{version}-%{release}
BuildArch: noarch
%description host-manager
The host-manager web application of Apache Tomcat.
%prep
%setup -q -n %{name}-%{version}-src
# This tells ant to install software in a specific directory.
cat << EOF >> build.properties
base.path=%{buildroot}/opt/apache-tomcat
EOF
%build
ant
%install
rm -Rf %{buildroot}
mkdir -p %{buildroot}/opt/apache-tomcat
mkdir -p %{buildroot}/opt/apache-tomcat/pid
mkdir -p %{buildroot}/opt/apache-tomcat/webapps
mkdir -p %{buildroot}/etc/init.d/
mkdir -p %{buildroot}/var/run/apache-tomcat
%{__cp} -Rip ./output/build/{bin,conf,lib,logs,temp,webapps} %{buildroot}/opt/apache-tomcat
%{__cp} %{_sourcedir}/apache-tomcat-initscript %{buildroot}/etc/init.d/apache-tomcat
%clean
rm -rf %{buildroot}
%pre
getent group tomcat > /dev/null || groupadd -r tomcat
getent passwd tomcat > /dev/null || useradd -r -g tomcat tomcat
%post
chkconfig --add %{name}
%preun
if [ "$1" = "0" ] ; then
service %{name} stop > /dev/null 2>&1
chkconfig --del %{name}
fi
%files
%defattr(-,tomcat,tomcat,-)
%dir /opt/apache-tomcat
%config /opt/apache-tomcat/conf/*
/opt/apache-tomcat/bin
/opt/apache-tomcat/lib
/opt/apache-tomcat/logs
/opt/apache-tomcat/temp
/opt/apache-tomcat/pid
%dir /opt/apache-tomcat/webapps
/var/run/apache-tomcat
%attr(0755,root,root) /etc/init.d/apache-tomcat
%files manager
/opt/apache-tomcat/webapps/manager
%files ROOT
/opt/apache-tomcat/webapps/ROOT
%files docs
/opt/apache-tomcat/webapps/docs
%files examples
/opt/apache-tomcat/webapps/examples
%files host-manager
/opt/apache-tomcat/webapps/host-manager
%changelog
* Fri Aug 19 2011 - robert (at) meinit.nl
- Updated to apache tomcat 7.0.20
- Split (example) applications into their own RPM.
* Mon Jul 4 2011 - robert (at) meinit.nl
- Initial release.apache-tomcat-initscript:
#!/bin/sh
#
# apache-tomcat
#
# chkconfig: - 85 15
# description: Jakarta Tomcat Java Servlets and JSP server
# processname: java
# pidfile: /var/run/apache-tomcat/pid
. /etc/rc.d/init.d/functions
# Set Tomcat environment.
USER=tomcat
LOCKFILE=/var/lock/apache-tomcat
export BASEDIR=/opt/apache-tomcat
export TOMCAT_HOME=$BASEDIR
export CATALINA_PID=/var/run/apache-tomcat/pid
export CATALINA_OPTS="-DHOME=$BASEDIR/home -Xmx512m -Djava.awt.headless=true"
case "$1" in
start)
echo -n "Starting apache-tomcat: "
status -p $CATALINA_PID apache-tomcat > /dev/null && failure || (su -p -s /bin/sh $USER -c "$TOMCAT_HOME/bin/catalina.sh start" > /dev/null && (touch $LOCKFILE ; success))
echo
;;
stop)
echo -n "Shutting down apache-tomcat: "
status -p $CATALINA_PID apache-tomcat > /dev/null && su -p -s /bin/sh $USER -c "$TOMCAT_HOME/bin/catalina.sh stop" > /dev/null && (rm -f $LOCKFILE ; success) || failure
echo
;;
restart)
$0 stop
$0 start
;;
condrestart)
[ -e $LOCKFILE ] && $0 restart
;;
status)
status -p $CATALINA_PID apache-tomcat
;;
*)
echo "Usage: $0 {start|stop|restart|condrestart|status}"
exit 1
;;
esacI created an initscript for a daemon recently it works well for starting and stopping, but "service status somedaemon" would return "status: unrecognized service"
After a bit of googling I found that the top few lines of the initscript are critical. It should be a describe here:
#!/bin/sh
#
# somedaemon
#
# chkconfig: - 85 15
# description: Some daemon
# processname: somedaemon
# pidfile: /var/run/somedaemon.pidHere is what these lines do.
From time to time you might need to interview somebody for a Linux role. It's hard to think of good questions; you don't want to scare somebody with your questions, but you do want to know if the person is knowledgable.
"People that use your Linux server complain it's slow. What tools would you use to check resource usage?"
top, sar, netstat -an, iostat, free, df.
"You discover a disk is full on your Linux server. What do you use to discover where the bigest files/directories are?"
"A system needs more disk space. You want to add a partition /var/log. You add a drive to the machine, it becomes /dev/sdb. What actions do you take to use this disk?"
Depending if you use LVM or not. If you don't use LVM:
If you do use LVM:
"People that use your Linux server complain it's slow. You've seen the disk usage is high. What can you do to improve this?"
"What happens in relation to DNS, SMTP and IMAP when I send send an email to example@example.com?"
My computer is likely to be configure to send the email to a mail server on port 25, SMTP protocol. That mailserver will query the DNS for the MX records of example.com. The mailserver that show up with the lowest priority will be contacted to deliver the email. That mailserver at example.com can accept the email and put in into the imap folder for the user or alias of example@example.com.
"You need to setup 100 Red Hat Enterprise Linux systems. If you don't want to walk around and eject and insert a boot CD 100 times, what options would you have?"
Kickstarting would help out. Install one machine as you like it, save /root/anaconda-ks.cfg to a webserver. Setup a PXE (DHCP, TFTP, HTTP, DNS) environment and use that kickstart file to install the rest.
"You have destroyed /etc/pam.d/system-auth and can't login anymore. Another machine has a propper version of /etc/pam.d/system-auth. How would you fix that broken machine?"
The machine needs to be booted in single user mode so you don't get a login prompt. After that here are some option:
"You have installed apache, php and mysql and a webapplication such as Drupal. The webapplication tries to send emails to an external mailserver but fails. What could be the cause when these items have been verified:
SELinux could be blocking apache from using port 25 on an external system. The logfile /var/log/messages might inform you about it. To fix it issue "setsebool -P httpd_can_network_connect=1".
"What determines the load (w, uptime) of a system?"
The number of processes that are waiting for execution. These processes are in the run queue. Processes could be waiting for io, network or memory allocation.
Hosting Drupal site in the Amazon EC2 cloud is not difficult. Here is a recipe I have used, first attempt was a Fedora 14 EC2 ami, but Fedora 14 comes with php 5.3, which can't be combined with Drupal 5.x. If you only have Drupal 6 (or Drupal 7) sites to host, you can use Fedora 14. If you want to use a "small" instance, please read this bug about readdir64_r. The fix for that bug is easy:
echo "hwcap 1 nosegneg" > /etc/ld.so.conf.d/libc6-xen.confLet's continue with Drupal on CentOS. Rightscale provides perfect CentOS amis that can be used on Amazons EC2 platform. If you install one, these are the steps I took to make it Drupal 5, Drupal 6 and Drupal 7 ready:
# Update the software.
yum -y update
# Set the timezone for this machine.
cp /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime
# This images came with 10Gb of ESB storage, I added another 64 Gb volume, here it's called "/dev/sdc".
# Use LVM to be ready to grow in the future.
pvcreate /dev/sdc
vgcreate vg0 /dev/sdc
lvcreate vg0 -L 32G -n var-www
lvcreate vg0 -L 1G -n var-lib-mysql
lvcreate vg0 -L 2G -n root
# Put filesystems on the logical volumes.
mke2fs -j /dev/vg0/var-www
mke2fs -j /dev/vg0/var-lib-mysql
mke2fs -j /dev/vg0/root
# Add the mountpoints to fstab.
echo "/dev/vg0/var-www /var/www/virtualhosts ext3 defaults 0 0" >> /etc/fstab
echo "/dev/vg0/var-lib-mysql /var/lib/mysql ext3 defaults 0 0" >> /etc/fstab
echo "/dev/vg0/root /root ext3 defaults 0 0" >> /etc/fstab
# Create the mountpoints.
mkdir /var/www /var/lib/mysql /root
# Mount all mounpoints in /etc/fstab.
mount -a
# Install the webserver.
yum -y install httpd
service httpd start
chkconfig httpd on
# Install the database server.
yum -y install mysql-server
service mysqld start
chkconfig mysqld on
/usr/bin/mysqladmin -u root password 'YourPassWord'
# Install PHP and all required Drupal php modules.
yum -y install php php-mysql php-mcrypt php-xml php-mbstring php-gd
# Add a single new file in the ESB root filesystem, that includes configurations from /var/www/conf.d
# Using this trick allows you to easily remount the volume on another host in case of troubles.
echo 'Include /var/www/conf.d/*.conf' >> /etc/httpd/conf.d/virtualhosts.conf
mkdir /var/www/conf.d
# Rightscale CentOS images comes with postfix and sendmail. Postfix is enabled, but sendmail is fine for me.
# First erase postfix.
yum -y erase postfix
# Now reinstall sendmail to fix a few permissions.
yum -y reinstall sendmail
service sendmail start
chkconfig on sendmail
# Reboot the box to make sure it's working properly.
rebootHet is altijd bewonderings waardig om te zien hoe mensen ranges kunnen gebruiken in shell scripts. Mensen die ranges beheersen, zijn sneller op de Linux command line.
Hier zijn wat ranges en patronen die je zou kunnen gebruiken:
Een oplopende reeks van karakters, in dit geval 1 tot 10, getoond op één lijn:
$ echo "file{1..9}"
file1 file2 file3 file4 file5 file6 file7 file8 file9
$ echo file{s..z}
files filet fileu filev filew filex filey filezDe volgorde van ranges kun je vinden in de man-page van "ascii".
Nog een patroon dat je kunt gebruiken:
$ echo file{1,2,4}
file1 file2 file4Als je meer informatie zoek, Google eens naar "Brace Expansion" of kijk in de man-page van bash.