LVS高可用（六）LVS+keepalived主从

在之前的篇幅中对LVS及keepalived都分别做了介绍，本篇开始总结下LVS+keepalived组合方案。这里以LVS的DR模式为例，在LB层再实现HA的功能 。具体加架如下图：

一、IP及规划

两台realserver安装apache httpd（yum -y install httpd）

两台director主机安装ipvsadm 、keepalived（yum -y install ipvsadm keepalived）

VIP需要配置在两台director上的eth0网口上，同时需要将该IP配置在两台realserver的lo回环口上。

在之前的篇幅中也介绍过，keepalived底层有关于IPVS的功能模块，可以直接在其配置文件中实现LVS的配置，不需要通过ipvsadm命令再单独配置。

二、director主机配置

MASTER节点的keepalived配置文件如下：

# cat /etc/keepalived/keepalived.conf
global_defs {
router_id LVS_T1
}
vrrp_sync_group bl_group {
group {
  bl_one
}
}
vrrp_instance bl_one {
    state MASTER
    interface eth0
    lvs_sync_daemon_interface eth0
    virtual_router_id 38
    priority 150
    advert_int 3
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
      192.168.122.100
    }
}
virtual_server 192.168.122.100 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    persistence_timeout 1
    protocol TCP
    real_server 192.168.122.10 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.122.20 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}                   

BACKUP director主机的配置文件如下：

# cat /etc/keepalived/keepalived.conf
global_defs {
router_id LVS_T1
}
vrrp_sync_group bl_group {
group {
  bl_one
}
}
vrrp_instance bl_one {
    state MASTER
    interface eth0
    lvs_sync_daemon_interface eth0
    virtual_router_id 38
    priority 150
    advert_int 3
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
      192.168.122.100
    }
}
virtual_server 192.168.122.100 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    persistence_timeout 1
    protocol TCP
    real_server 192.168.122.10 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.122.20 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}       

从上面的配置可以看到，这里配置的健康检测方式是基于IP端口的，我们也可憎修改成基于URL的。这点可以参考 keepalived健康检查方式 。主机在启动keepalived服务时，可以从message中看到如下日志：

三、realserver主机配置

两台realserver上使用的脚本一样，内容如下：

# cat dr_client.sh
#!/bin/bash
VIP=192.168.122.100
BROADCAST=192.168.122.255  #vip's broadcast
. /etc/rc.d/init.d/functions
case "$1" in
 start)
  echo "reparing for Real Server"
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $BROADCAST up
     /sbin/route add -host $VIP dev lo:0
     ;;
 stop)
     ifconfig lo:0 down
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
     ;;
 *)
     echo "Usage: lvs {start|stop}"
     exit 1
esac 

四、测试

分别在两台主机上执行如下命令启动：

# /etc/init.d/keepalived start
# sh dr_client.sh start

在两台realserver上启动httpd服务，关闭防火墙（或放行80端口）。可以使用如下脚本进行访问测试：

#!/bin/sh
for((i=1;i<=100;i++));do
curl http://192.168.122.100 >> /tmp/q;
done

1、简单测试

在任一台director主机上使用ipvsadm命令观察：

[root@lvs-dr ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.122.100:80 rr persistent 1
  -> 192.168.122.10:80            Route   1      0          0
  -> 192.168.122.20:80            Route   1      0          100     

注：这里会现一个问题，发现使用轮询算法，使用一台client得到的结果是不均衡的（在一定的时间内其一直走其中一台realserver） 。当使用多台client去访问时，发现访问的结果是均衡的。

2、realserver故障测试

将其中一台realserver主机关闭，通过curl 查看发现其会每次请求获取的都是正常的主机的页面，message中也能看到如下日志：

Oct  7 22:35:10 lvs-dr Keepalived_vrrp[32634]: IPVS: Daemon has already run
Oct  7 22:35:10 lvs-dr Keepalived_healthcheckers[32633]: Netlink reflector reports IP 192.168.122.100 added
Oct  7 22:35:15 lvs-dr Keepalived_vrrp[32634]: VRRP_Instance(bl_one) Sending gratuitous ARPs on eth0 for 192.168.122.100
Oct  7 22:38:16 lvs-dr Keepalived_healthcheckers[32633]: TCP connection to [192.168.122.10]:80 failed !!!
Oct  7 22:38:16 lvs-dr Keepalived_healthcheckers[32633]: Removing service [192.168.122.10]:80 from VS [192.168.122.100]:80

这台realserver恢复后，message中的日志如下：

Oct  7 22:39:19 lvs-dr Keepalived_healthcheckers[32633]: TCP connection to [192.168.122.10]:80 success.
Oct  7 22:39:19 lvs-dr Keepalived_healthcheckers[32633]: Adding service [192.168.122.10]:80 to VS [192.168.122.100]:80

3、director主机故障测试

通过关闭master director主机的keepalived服务，在backup director主机上发现日志如下：

Oct  7 22:40:14 lvs-dr Keepalived_vrrp[1601]: VRRP_Instance(bl_one) Transition to MASTER STATE
Oct  7 22:40:14 lvs-dr Keepalived_vrrp[1601]: VRRP_Group(bl_group) Syncing instances to MASTER state
Oct  7 22:40:17 lvs-dr Keepalived_vrrp[1601]: VRRP_Instance(bl_one) Entering MASTER STATE
Oct  7 22:40:17 lvs-dr Keepalived_vrrp[1601]: VRRP_Instance(bl_one) setting protocol VIPs.
Oct  7 22:40:17 lvs-dr Keepalived_vrrp[1601]: VRRP_Instance(bl_one) Sending gratuitous ARPs on eth0 for 192.168.122.100
Oct  7 22:40:17 lvs-dr Keepalived_healthcheckers[1600]: Netlink reflector reports IP 192.168.122.100 added

此时通过122.100访问服务不受影响 。