[OpenWrt-Devel] [PATCH 3/3] IP Virtual Server - Load Balancer - OpenWRT configuration - IGNORE PREVIOUS

Mauro Mozzarelli mauro at ezplanet.net
Tue Feb 23 15:19:26 EST 2016


All,

Please ignore the previous patch 3/3 as I have pasted 2/3 again by mistake.

This is the CORRECT PATCH 3/3

Part 3/3 includes /etc/init.d/ipvsadm script, uci configuration and cron job to check for real servers' availability and
re-configure the ipvs tables accordingly (using ipvsadm).

This patch depends on the ipvs kernel modules patch 1/3 and ipvsadm tool 2/3

This last patch is not necessary like the previous 2 to add load balancing capability to OpenWRT. One could write his own
scripts and configuration files, however I thought it would be helpful to provide an OpenWRT standard configuration using uci.
This can be further extended to support more or all ipvs options.

I tested the configuration file and scripts on a router with a static subnet on the wan side. I run both internet facing and
intranet facing load balanced servers and it works perfectly. It does not include NAT Masquerade configuration. This can be
easily added, however since personally I do not use ipvs NAT (and I would not advise to do so because the real servers would
not be able to track the original client's source IP) I have not implemented it.

Should you find issues with the scripts please feel free just not to push this last patch into the mainstream trunk.

Mauro


Author: Mauro Mozzarelli <mauro at ezplanet.net>
Date:   Tue Feb 23 19:30:59 2016 +0000

    This ipvs package adds Linux IP Virtual Server capability to OpenWRT
    IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching.
IPVS running on a host acts as a load balancer at the front of a cluster of real servers, it can direct requests for
TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual service on a
single IP address.

    This ipvs package contains:
    - uci ipvs sample configuration file
    - ipvsadm service startup/shutdown script
    - scheduler checking real servers' availability and re-configuring ipvs tables accordingly

    Dependencies:
    - ipvs kernel modules
    - ipvsadm tool

    Signed-off-by: Mauro Mozzarelli <mauro at ezplanet.net>

diff --git a/package/network/utils/ipvsadm/Makefile b/package/network/utils/ipvsadm/Makefile
index 8abde77..6ce06a1 100644
--- a/package/network/utils/ipvsadm/Makefile
+++ b/package/network/utils/ipvsadm/Makefile
@@ -54,6 +54,7 @@ define Package/ipvsadm/install
 	$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm $(1)/usr/sbin/
 	$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-save $(1)/usr/sbin/
 	$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-restore $(1)/usr/sbin/
+	$(CP) ./files/* $(1)/
 endef

 $(eval $(call BuildPackage,$(PKG_NAME)))
diff --git a/package/network/utils/ipvsadm/files/etc/config/ipvs b/package/network/utils/ipvsadm/files/etc/config/ipvs
new file mode 100644
index 0000000..c392e39
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/config/ipvs
@@ -0,0 +1,44 @@
+# Sample Configuration
+# please see the documentation at www.linuxvirtualserver.org to learn how to
+# configure real servers.
+# This configuration is used by /etc/init.d/ipvsadm to setup and load balance
+# virtual servers to available real servers.
+# A cron job is used to activate/deactivate real servers based on their availability (ping)
+#
+# Currently this configuration defaults to persistent gateway mode
+# Currently NAT masquerading and other options provided by ipvsadm are not implemented by this configuration
+# If virtual and real servers have internet static IPs, the wan interface must be part of a static IP Subnet
+# ipvsadm is fully functional for manual configuration
+
+config ipvs globals
+	option enabled 0
+# role master|slave
+	option role 'master'
+	option interface 'lan'
+# valid schedulers are included in the kernel schedulers list
+	list scheduler wlc
+
+# for each vip an alias ip is initialized on the wan/lan router interface
+# the vip must not be the same as the router's ip
+config virtual
+	option enabled 0
+	option name admin
+	option vip '192.168.10.10'
+	option interface lan
+	option alias 10
+	option scheduler 'wlc'
+	list real '192.168.10.100'
+	list real '192.168.10.101'
+
+# lists protocols/ports
+config admin
+	option protocol tcp
+	option src_port 80
+	option dest_port 80
+
+config admin
+	option protocol udp
+# if only 'port' is specified then source and destination ports are the same
+# port 0 means all ports are forwarded to the real servers
+	option port 0
+
diff --git a/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
new file mode 100755
index 0000000..7c8c378
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
@@ -0,0 +1,187 @@
+#!/bin/sh /etc/rc.common
+
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: builds ipvs configuration from uci openwrt parameters
+#              load/unload ip_vs kernel modules
+#              start/stop ipvs scheduler
+#              start/stop cron job scheduler
+# Dependencies: /etc/config/ipvs; ipvsadm; ip_vs Kernel modules; cron
+# Notes: Firewall must be configured separately
+#	Configuration files in /tmp/ipvsadm.d:
+#	virt.*  = virtual servers
+#   *.stop  = real server in down state ipvs table
+#   *.start = real server in up state ipvs table
+#   vserv.* = virtual servers ipvs table
+################################################################################
+
+START=70
+STOP=10
+
+USE_PROCD=1
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+KERNEL_RELEASE=`uname -r`
+MODULES=/lib/modules/$KERNEL_RELEASE
+IPVSMOD=ip_vs
+# Cron job used to check if real servers are available and update ipvs
+# scheduler accordingly
+CRON_SCHEDULER='*		*	*	*	*	/usr/sbin/checkRealServers		# IPVS_SCHEDULER'
+
+# Check if real server is in the list and add if not present
+update_real_server () {
+	realServer=$1
+	gotit=0
+	for s in `cat $BASE_DIR/$REAL_SERVERS`; do
+		if [ $r == $s ]; then
+			gotit=1
+			break
+		fi
+	done
+	if [ $gotit -eq 0 ]; then
+		echo $r >> $BASE_DIR/$REAL_SERVERS
+	fi
+}
+
+start_service () {
+	enabled=`uci -q get ipvs.globals.enabled`
+	case $enabled in
+		0)
+			exit
+			;;
+		1)
+			;;
+		*)
+			echo Invalid initialization parameter: enabled=$enabled
+			exit
+			;;
+	esac
+	if [ -d $BASE_DIR ]; then
+		cd $BASE_DIR
+	else
+		mkdir -p $BASE_DIR
+		cd $BASE_DIR
+	fi
+	rm -f $BASE_DIR/*
+	rm -f $TMP_DIR/*.down
+	touch $BASE_DIR/$REAL_SERVERS
+	modprobe $IPVSMOD
+	cm=0
+	for m in `uci -q get ipvs.globals.scheduler`; do
+			cm=$((cm+1))
+			modprobe ${IPVSMOD}_$m
+	done
+	# Default to wlc if no schedulers
+	if [ $cm -eq 0 ]; then
+		modprobe ${IPVSMOD}_wlc
+	fi
+	cv=0
+	while [ x`uci -q get ipvs. at virtual[$cv]` == x'virtual' ]; do
+		enabled=`uci get ipvs. at virtual[$cv].enabled`
+		if [ $enabled -eq 0 ]; then
+			cv=$((cv+1))
+			continue
+		fi
+		name=`uci get ipvs. at virtual[$cv].name`
+		vip=`uci get ipvs. at virtual[$cv].vip`
+		interface=`uci get ipvs. at virtual[$cv].interface`
+		device=`uci -P /var/state get network.${interface}.ifname`
+		alias=`uci get ipvs. at virtual[$cv].alias`
+		scheduler=`uci get ipvs. at virtual[$cv].scheduler`
+		# ifconfig
+		ifconfig $device:$alias $vip netmask 255.255.255.255 up
+		#
+		real=`uci get ipvs. at virtual[$cv].real`
+		if [ ! -z "$real" -a "$real" != " " ]; then
+			vip_done=no
+			for r in $real; do
+				update_real_server $r
+				cd=0
+				while [ x`uci -q get ipvs.@$name[$cd]` == x$name ]; do
+					protocol=`uci -q get ipvs.@$name[$cd].protocol`
+					src_port=`uci -q get ipvs.@$name[$cd].src_port`
+					dest_port=`uci -q get ipvs.@$name[$cd].dest_port`
+					case $protocol in
+						tcp)
+							protocol="-t"
+							;;
+						udp)
+							protocol="-u"
+							;;
+						*)
+							# Default to tcp protocol
+							protocol="-t"
+							;;
+					esac
+					if [ -z $src_port ]; then
+						port=`uci -q get ipvs.@$name[$cd].port`
+						if [ -z $port ]; then
+							port=0
+						fi
+						src_port=$port
+					fi
+					if [ -z $dest_port ]; then
+						dest_port=$src_port
+					fi
+					if [ x$vip_done == x'no' ];then
+						echo "-A $protocol $vip:$src_port -s $scheduler -p" >> $BASE_DIR/vserv.start.$interface
+						echo "-D $protocol $vip:$src_port" >> $BASE_DIR/vserv.stop.$interface
+					fi
+					echo "-a $protocol $vip:$src_port -r $r:$dest_port -g -w 1" >> $BASE_DIR/$r.start
+					echo "-d $protocol $vip:$src_port -r $r" >> $BASE_DIR/$r.stop
+					cd=$((cd+1))
+				done
+				vip_done=yes
+			done
+		fi
+		cv=$((cv+1))
+	done
+	for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+		date > $TMP_DIR/$i.down
+	done
+	for i in ${BASE_DIR}/vserv.start.* ; do
+		ipvsadm -R < $i
+	done
+
+	grole=`uci get ipvs.globals.role`
+	gintf=`uci get ipvs.globals.interface`
+	ipvsadm --start-daemon $grole --mcast-interface `uci -P /var/state get network.$gintf.ifname`
+	if [ $? -gt 0 ]; then
+		logger -p info "ERROR: IPVSADM daemon failed to start"
+		exit
+	fi
+	(crontab -l ; echo "$CRON_SCHEDULER")|crontab -
+	logger -p info "IPVSADM daemon started"
+}
+
+stop_service () {
+	for i in ${BASE_DIR}/*.stop ; do
+		ipvsadm -R < $i
+	done
+	for i in ${BASE_DIR}/vserv.stop.* ; do
+		ipvsadm -R < $i
+	done
+	n=0
+	while [ x`uci -q get ipvs. at virtual[$n]` == x'virtual' ]; do
+		interface=`uci get ipvs. at virtual[$n].interface`
+		device=`uci -P /var/state get network.${interface}.ifname`
+		alias=`uci get ipvs. at virtual[$n].alias`
+		ifconfig $device:$alias down
+		n=$((n+1))
+	done
+	ipvsadm --stop-daemon `uci get ipvs.globals.role`
+	rm -f $TMP_DIR/*.down
+	rm -rf $BASE_DIR
+	for m in `uci -q get ipvs.globals.scheduler`; do
+			cm=$((cm+1))
+			rmmod ${IPVSMOD}_$m
+	done
+	# Default to wlc if no schedulers
+	if [ $m -eq 0 ]; then
+		rmmod ${IPVSMOD}_wlc
+	fi
+	rmmod $IPVSMOD
+	crontab -l |grep -v IPVS_SCHEDULER|crontab -
+}
diff --git a/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
new file mode 100755
index 0000000..f801c4b
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
@@ -0,0 +1,36 @@
+#!/bin/sh
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: checks whether a real server is up and running and if it is
+#              then it starts the services
+# Dependencies: /tmp/ipvsadm.d
+#				configuration files created by /etc/init.d/ipvsadm
+#	virt.*  = virtual servers
+#   *.stop  = real server in down state ipvs table
+#   *.start = real server in up state ipvs table
+#   *.down  = real server down status
+################################################################################
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+
+if [ -f $BASE_DIR/$REAL_SERVERS ]; then
+	for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+		ping -qc 3 $i > /dev/null
+		RESULT=$?
+		if [ $RESULT -gt 0 ]; then
+			if [ ! -f $TMP_DIR/$i.down ]; then
+				ipvsadm -R < $BASE_DIR/$i.stop
+				date > $TMP_DIR/$i.down
+				logger -p info "IPVS Server $i is down"
+			fi
+		else
+			if [ -f $TMP_DIR/$i.down ]; then
+				ipvsadm -R < $BASE_DIR/$i.start
+				rm $TMP_DIR/$i.down
+				logger -p info "IPVS Server $i is on-line"
+			fi
+		fi
+	done
+fi
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel



More information about the openwrt-devel mailing list