Linux Administration: Deploying HAProxy 1.5 from source

In an older post I showed how to create highly available HAProxy load balancer and front-end it with Pound for SSL termination. With the release of HAProxy 1.5 the SSL termination is now built in, along with a nice set of new features, such as stick tables.

In this post I'll setup HAProxy with SSL offloading and load balance HTTP, MySQL and rabbitmq in an active/passive mode.

First lets install HAProxy from source (you can of course use a package, but I'll be using Debian Squeeze for this deployment):

File: gistfile1.sh ------------------ [root@server ~] apt-get install libpcre3-dev libssl-dev [root@server ~] wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.4.tar.gz [root@server ~] tar zxfv haproxy-1.5.4.tar.gz [root@server ~] cd haproxy-1.5.4/ [root@server haproxy-1.5.4] make TARGET=custom CPU=native USE_PCRE=1 USE_LIBCRYPT=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_OPENSSL=yes [root@server haproxy-1.5.4] make install

Since we are installing from source, we'll need an init script in /etc/init.d and the default options file in /etc/default. You can get one from an older package and make sure that the paths match the installation or just use the one bellow:

File: gistfile1.sh ------------------ [root@server ~] cat /etc/init.d/haproxy #!/bin/sh ### BEGIN INIT INFO # Provides: haproxy # Required-Start: $local_fs $network $remote_fs # Required-Stop: $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: fast and reliable load balancing reverse proxy # Description: This file should be used to start and stop haproxy. ### END INIT INFO # Author: Arnaud Cornet <acornet@debian.org> PATH=/sbin:/usr/local/sbin:/bin:/usr/bin PIDFILE=/var/run/haproxy.pid CONFIG=/etc/haproxy/haproxy.cfg HAPROXY=/usr/local/sbin/haproxy EXTRAOPTS= ENABLED=0 test -x $HAPROXY || exit 0 test -f "$CONFIG" || exit 0 if [ -e /etc/default/haproxy ]; then . /etc/default/haproxy fi test "$ENABLED" != "0" || exit 0 [ -f /etc/default/rcS ] && . /etc/default/rcS . /lib/lsb/init-functions haproxy_start() { start-stop-daemon --start --pidfile "$PIDFILE" \ --exec $HAPROXY -- -f "$CONFIG" -D -p "$PIDFILE" \ $EXTRAOPTS || return 2 return 0 } haproxy_stop() { if [ ! -f $PIDFILE ] ; then # This is a success according to LSB return 0 fi for pid in $(cat $PIDFILE) ; do /bin/kill $pid || return 4 done rm -f $PIDFILE return 0 } haproxy_reload() { $HAPROXY -f "$CONFIG" -p $PIDFILE -D $EXTRAOPTS -sf $(cat $PIDFILE) \ || return 2 return 0 } haproxy_status() { if [ ! -f $PIDFILE ] ; then # program not running return 3 fi for pid in $(cat $PIDFILE) ; do if ! ps --no-headers p "$pid" | grep haproxy > /dev/null ; then # program running, bogus pidfile return 1 fi done return 0 } case "$1" in start) log_daemon_msg "Starting haproxy" "haproxy" haproxy_start ret=$? case "$ret" in 0) log_end_msg 0 ;; 1) log_end_msg 1 echo "pid file '$PIDFILE' found, haproxy not started." ;; 2) log_end_msg 1 ;; esac exit $ret ;; stop) log_daemon_msg "Stopping haproxy" "haproxy" haproxy_stop ret=$? case "$ret" in 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac exit $ret ;; reload|force-reload) log_daemon_msg "Reloading haproxy" "haproxy" haproxy_reload case "$?" in 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac ;; restart) log_daemon_msg "Restarting haproxy" "haproxy" haproxy_stop haproxy_start case "$?" in 0) log_end_msg 0 ;; 1) log_end_msg 1 ;; 2) log_end_msg 1 ;; esac ;; status) haproxy_status ret=$? case "$ret" in 0) echo "haproxy is running." ;; 1) echo "haproxy dead, but $PIDFILE exists." ;; *) echo "haproxy not running." ;; esac exit $ret ;; *) echo "Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}" exit 2 ;; esac : [root@server ~] chmod u+x /etc/init.d/haproxy [root@server ~] cat /etc/default/haproxy # Set ENABLED to 1 if you want the init script to start haproxy. ENABLED=1

Time to configure HAProxy for 4 different services - plain HTTP, HTTPS terminated by the load balancer, and MySQL and RabbitMQ in active/passive mode:

File: gistfile1.sh ------------------ [root@server ~] mkdir /etc/haproxy [root@server ~] cat /etc/haproxy/haproxy.cfg global log 127.0.0.1 local1 log-tag haproxy maxconn 4096 user haproxy group haproxy daemon stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m tune.ssl.default-dh-param 2048 defaults log global mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms option dontlognull option http-server-close peers HAPEERS peer haproxy-n01 10.13.238.51:5555 peer haproxy-n02 10.13.238.51:5555 frontend http bind 10.13.238.21:80 reqadd X-Forwarded-Proto:\ http default_backend http_nodes frontend https-terminated bind 10.13.238.22:443 ssl crt /etc/ssl/certs/api.example.net.pem option httpclose option forwardfor option httplog clf reqadd X-Forwarded-Proto:\ https default_backend https-terminated_nodes frontend mysql mode tcp bind 10.13.238.19:3306 timeout client 168h default_backend mysql_nodes log global option tcplog option logasap frontend rabbitmq mode tcp bind 10.13.238.20:5672 timeout client 168h default_backend rabbitmq_nodes log global option tcplog option logasap backend http_nodes mode http balance leastconn option httpclose option forwardfor cookie JSESSIONID prefix server apt-n01 10.13.238.39:80 check inter 5000 backend https-terminated_nodes mode http balance roundrobin cookie JSESSIONID prefix server api-n01 10.13.238.37:8888 check inter 5000 server api-n02 10.13.238.38:8888 check inter 5000 backend mysql_nodes mode tcp balance roundrobin option mysql-check user haproxy_check stick-table type ip size 20k peers HAPEERS stick on dst timeout server 168h server mysql-n01 10.13.238.53:3306 check server mysql-n02 10.13.238.54:3306 check backup backend rabbitmq_nodes mode tcp balance roundrobin stick-table type ip size 20k peers HAPEERS stick on dst timeout server 168h server rabbitmq-n01 10.13.238.35:5672 check server rabbitmq-n02 10.13.238.36:5672 check backup

All of the options are documented [1], but the most interesting once are the "stats socket", which allows you to connect to a socket and perform simple operations on HAProxy and gather statistics without needing to restart the server. You can do this in either interactive or non-interactive mode using the socat utility:

File: gistfile1.sh ------------------ [root@server ~] apt-get install socat [root@server ~] socat /var/run/haproxy.sock readline prompt > show info Name: HAProxy Version: 1.5.4 Release_date: 2014/09/02 Nbproc: 1 Process_num: 1 Pid: 17066 Uptime: 0d 1h07m55s Uptime_sec: 4075 Memmax_MB: 0 Ulimit-n: 8248 Maxsock: 8248 Maxconn: 4096 Hard_maxconn: 4096 CurrConns: 313 CumConns: 1792 CumReq: 27874 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 537 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 1 ConnRateLimit: 0 MaxConnRate: 68 SessRate: 1 SessRateLimit: 0 MaxSessRate: 68 SslRate: 0 SslRateLimit: 0 MaxSslRate: 5 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 6 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 Tasks: 345 Run_queue: 1 Idle_pct: 100 node: zxtm-n01 description: > quit

To enable the static HTTP status page add the following to the config file:

File: gistfile1.txt ------------------- listen stats :8080 mode http stats enable stats hide-version stats realm Haproxy\ Statistics stats uri / stats auth admin:yourpassword

To have some real time view of what's going on, you can use the ncurses hatop, just install and run:
The SSL termination is configured by specifying the "ssl" option and providing a PEM file, containing the cert and the private key on line 34.

To setup MySQL and RabbitMQ in an active/passive mode, where only one of the servers will be accepting connections, and when the main server comes back up after a failure the connections will still stick to the backup server, you specify the "backup" option and define a stick-table on line 87 and line 83.

One more option worth mentioning is the MySQL health check specified on line 82. The check consists of sending two MySQL packets, one Client Authentication packet, and one QUIT packet, to correctly close the MySQL session, using the specified user - in this case haproxy_check - so make sure you create it first:

File: gistfile1.sql ------------------- mysql> USE mysql; mysql> INSERT INTO user (Host,User) values ('10.13.238.51','haproxy_check'); mysql> FLUSH PRIVILEGES;

Another useful option is the tcp-check. With this it's easy to send and receive certain messages to the back-ends and act on them. This is particularly useful in managing Redis master and slave instances controlled by Sentinel:

File: gistfile1.txt ------------------- backend redis_nodes mode tcp option tcplog option tcp-check tcp-check send AUTH\ somepassword\r\n tcp-check expect +OK tcp-check send PING\r\n tcp-check expect +PONG tcp-check send info\ replication\r\n tcp-check expect string role:master tcp-check send QUIT\r\n tcp-check expect string +OK server redis_node1 192.168.1.10:6379 maxconn 4096 check inter 5s server redis_node2 192.168.1.20:6379 maxconn 4096 check inter 5s server redis_node3 192.168.1.30:6379 maxconn 4096 check inter 5s

In the example above HAProxy queries Redis nodes about their role, and it will fail-over only if a back-end became the new master. If Sentinel is controlling the cluster and a new master is elected HAProxy will fail-over traffic to it.

To allow HAProxy to send separate logs to rsyslog you can use a file similar to this:

File: gistfile1.txt ------------------- [root@server ~] cat /etc/rsyslog.d/haproxy.conf [root@server ~] $ModLoad imudp [root@server ~] $UDPServerAddress 127.0.0.1 [root@server ~] $UDPServerRun 514 [root@server ~] if $syslogfacility-text == 'local1' and $msg contains 'rabbitmq' then /var/log/haproxy_rabbitmq.log [root@server ~] if $syslogfacility-text == 'local1' and $msg contains 'http' then /var/log/haproxy_api.log [root@server ~] if $syslogfacility-text == 'local1' and $msg contains 'mysql' then /var/log/haproxy_mysql.log local1.* -/var/log/haproxy.log & ~

And finally start the service:
Resources:
[1]. http://cbonte.github.io/haproxy-dconv/configuration-1.5.html

Pages

Deploying HAProxy 1.5 from source