This post is more for future personal reference than anything, but maybe it will help someone else. It assumes nginx and haproxy running on the same front end box with nginx listening on port 80 and forwarding requests to haproxy on port 81. nginx could also be used as an SSL accelerator in this configuration, but the directives are not included in the example configuration shown here.
The basic idea is that nginx sits at the front doing rewrites, SSL decryption, etc… and then forwards the traffic to haproxy which distributes the traffic among the nodes on the backend. I’d love to have used just haproxy or just nginx, but nginx supports url rewriting and SSL accelerator features among others, while haproxy provides better load balancing algorithms and such. I haven’t had the chance to install yet in an environment that is good for benchmarking, but when I do, I’ll post the results here.
All servers were running default “server” installs of CentOS 5.3 for this configuration. The only dependencies I had to install were:
- gcc
- openssl-devel
- pcre-devel
Install HAProxy
cd /tmp
wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.18.tar.gz
tar -xvf haproxy*.gz
cd haproxy*
make TARGET=linux26 CPU=i686
mv haproxy /usr/sbin
Create or modify /etc/haproxy.cfg
global
maxconn 25000
daemon
defaults
mode http
cookie SERVERID insert nocache indirect
clitimeout 60000
srvtimeout 30000
contimeout 4000
option httpclose
maxconn 25000
listen http_proxy 192.168.1.14:81
balance roundrobin
option httpchk
option forwardfor
server server1 192.168.1.15:80 weight 1 maxconn 5000 cookie SERVER1 check
server server2 192.168.1.16:80 weight 1 maxconn 5000 cookie SERVER2 check
Create or modify /etc/init.d/haproxy
#!/bin/sh
#
# chkconfig: 2345 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \
# for high availability environments.
# processname: haproxy
# config: /etc/haproxy.cfg
# pidfile: /var/run/haproxy.pid
# Source function library.
if [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
. /etc/rc.d/init.d/functions
else
exit 0
fi
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
[ -f /etc/haproxy.cfg ] || exit 1
RETVAL=0
start() {
/usr/sbin/haproxy -c -q -f /etc/haproxy.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
echo -n "Starting HAproxy: "
daemon /usr/sbin/haproxy -D -f /etc/haproxy.cfg -p /var/run/haproxy.pid
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/haproxy
return $RETVAL
}
stop() {
echo -n "Shutting down HAproxy: "
killproc haproxy -USR1
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/haproxy
[ $RETVAL -eq 0 ] && rm -f /var/run/haproxy.pid
return $RETVAL
}
restart() {
/usr/sbin/haproxy -c -q -f /etc/haproxy.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with 'haproxy check'."
return 1
fi
stop
start
}
check() {
/usr/sbin/haproxy -c -q -V -f /etc/haproxy.cfg
}
rhstatus() {
status haproxy
}
condrestart() {
[ -e /var/lock/subsys/haproxy ] && restart || :
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
reload)
restart
;;
condrestart)
condrestart
;;
status)
rhstatus
;;
check)
check
;;
*)
echo $"Usage: haproxy {start|stop|restart|reload|condrestart|status|check}"
RETVAL=1
esac
exit $RETVAL
Set haproxy to start on boot (and start now)
chkconfig --add haproxy
chkconfig haproxy on
service haproxy start
Install Nginx
cd /tmp
wget http://sysoev.ru/nginx/nginx-0.7.61.tar.gz
tar -xvf nginx*.gz
cd nginx*
./configure
make
make install
Create or modify /usr/local/nginx/conf/nginx.conf
worker_processes 2;
worker_rlimit_nofile 10000;
pid /var/run/nginx.pid;
events {
worker_connections 4000;
use epoll;
}
http {
include mime.types;
default_type application/octet-stream;
keepalive_timeout 65;
gzip on;
server {
listen 80;
server_name test1.com;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
proxy_pass http://192.168.1.14:81/;
}
}
server {
listen 80 default;
server_name_in_redirect off;
rewrite ^ http://test1.com$request_uri;
}
}
Create or modify /etc/init.d/nginx
#! /bin/sh
# chkconfig: 2345 87 13
# description: A HTTP and mail proxy server licensed under a \
# 2-clause BSD-like license.
# Description: Startup script for nginx webserver on Debian. Place in /etc/init.d and
# run 'sudo update-rc.d nginx defaults', or use the appropriate command on your
# distro.
#
# Author: Ryan Norbauer
# Modified: Geoffrey Grosenbach http://topfunky.com
set -e
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="nginx daemon"
NAME=nginx
DAEMON=/usr/local/nginx/sbin/$NAME
CONFIGFILE=/usr/local/nginx/conf/nginx.conf
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0
d_start() {
$DAEMON -c $CONFIGFILE || echo -n " already running"
}
d_stop() {
kill -QUIT `cat $PIDFILE` || echo -n " not running"
}
d_reload() {
kill -HUP `cat $PIDFILE` || echo -n " can't reload"
}
case "$1" in
start)
echo -n "Starting $DESC: $NAME"
d_start
echo "."
;;
stop)
echo -n "Stopping $DESC: $NAME"
d_stop
echo "."
;;
reload)
echo -n "Reloading $DESC configuration..."
d_reload
echo "reloaded."
;;
restart)
echo -n "Restarting $DESC: $NAME"
d_stop
# One second might not be time enough for a daemon to stop,
# if this happens, d_start will fail (and dpkg will break if
# the package is being upgraded). Change the timeout if needed
# be, or change d_stop to have start-stop-daemon use --retry.
# Notice that using --retry slows down the shutdown process somewhat.
sleep 1
d_start
echo "."
;;
*)
echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
exit 3
;;
esac
exit 0
Set nginx to start on boot (and start now)
chkconfig --add nginx
chkconfig nginx on
service nginx start
If everything works, you should now be able to access your nginx box on port 80. This will proxy the requests through haproxy on port 81 to whatever back end nodes you have configured in haproxy.cfg and return the content. Spiffy, eh?
4 comments