标签归档:LVS

Keepalived LVS-DR Nginx单网络双活双主配置模式(实战)

何为LVS/DR模式?

LVS是Linux Virtual Server的简写,意即Linux虚拟服务器,是一个虚拟的服务器集群系统。LVS目前有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR)、十种调度算法(rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq)。

LVS在Unix-like系统中是作为一个前端(Director)存在的,又称为调度器,它本身不提供任何的服务,只是将通过互联网进来的请求接受后再转发给后台运行的真正的服务器(RealServer)进行处理,然后响应给客户端。

LVS集群采用IP负载均衡技术和基于内容请求分发技术。调度器具有很好的吞吐率,将请求均衡地转移到不同的服务器上执行,且调度器自动屏蔽掉服务器的故障,从而将一组服务器构成一个高性能的、高可用的虚拟服务器。整个服务器集群的结构对客户是透明的,而且无需修改客户端和服务器端的程序。为此,在设计时需要考虑系统的透明性、可伸缩性、高可用性和易管理性。

LVS有两个重要的组件:一个是IPVS,一个是IPVSADM。ipvs是LVS的核心组件,它本身只是一个框架,类似于iptables,工作于内核空间中。ipvsadm 是用来定义LVS的转发规则的,工作于用户空间中。

LVS有三种转发类型:

LVS-NAT模式:

称为网络地址转换,实现起来比较简单,所有的RealServer集群节点和前端调度器Director都要在同一个子网中,这种模型可以实现端口映射,RealServer的操作系统可以是任意操作系统,前端的Director既要处理客户端发起的请求,又要处理后台RealServer的响应信息,将RealServer响应的信息再转发给客户端,前端Director很容易成为整个集群系统性能的瓶颈。通常情况下RealServer的IP地址(以下简成RIP)为私有地址,便于RealServer集群节点之间进行通信通常情况下前端的Director有两个IP地址,一个为VIP,是虚拟的IP地址,客户端向此IP地址发起请求。一个是DIP,是真正的Director的IP地址,RIP的网关要指向Director的DIP。

LVS-DR模式:

DR:直接路由(direct routing)模式,此种模式通过MAC地址转发工作,所有的RealServer集群节点和前端调度器Director都要在同一个物理网络中,此种模式不支持端口映射,此种模式的性能要优于LVS-NAT,RIP可以使用公网的IP,RIP的网关不能指向DIP。

优点:

相对LVS/NAT模式,DR模式不需要把返回的数据通过负载均衡转发,想要他发挥优势,那么就要相应的数据包的数量和长度远远大于请求数据包,幸运的是,大部分WEB服务都具备这样的特点,响应和请求并不对称,因此常用的WEB服务,都可以使用这种模式。

这种方式,负载均衡器不再是系统的瓶颈。如果你的负载均衡器只拥有100M的全双工网卡和带宽的话,通过集群的横向扩展也可以让整个系统达到1G的流量。

来自LVS官方站点的测试结果也告诉我们,LVS-DR可以容纳100台以上的实际应用服务器,对一般的服务而已,这样的表现足够了。

不足:

DR模式下不能跨网段转发数据,如果必须要跨网段进行负载,那么就必须使用LVS/TUN模式。

LVS-TUN模式:

称为隧道模型RealServer服务器与前端的Director可以在不同的网络中,此种模型也不支持端口映射,RealServer只能使用哪些支持IP隧道的操作系统,前端的Director只处理客户端的请求,然后将请求转发给RealServer,由后台的RealServer直接响应客户端,不再经过Director,RIP一定不能是私有IP,在DR、TUN模式中,数据包是直接返回给用户的,所以,在Director Server上以及集群的每个节点上都需要设置这个地址。此IP在Real Server上一般绑定在回环地址上,例如lo:0,同样,在Director Server上,虚拟IP绑定在真实的网络接口设备上,例如eth0:0。

开始部署:

准备四台服务器或虚拟机:

Web Nginx:10.16.8.8/10.16.8.9
Keepalived:10.16.8.10/10.16.8.11
Keepalived VIP:10.16.8.100/10.16.8.101
OS:CentOS Linux release 7.4.1708 (Core)

先决条件:

安装keepalived。
时间同步。
设置SELinux和防火墙。
互相之间/etc/hosts文件添加对方主机名(可选)。
确认网络接口支持多播(组播)新网卡默认支持。

以上部署请参阅:《keepalived 安装及配置文件讲解》。

1.ka67配置文件

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 60
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"        
}
vrrp_instance VI_2 {
    state BACKUP
    interface eth1
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         
}

2.ka68配置文件

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 60
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 191
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         
}
vrrp_instance VI_2 {
    state MASTER
    interface eth1
    virtual_router_id 192
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"        
}

3.创建通用notify.sh检测脚本

分别创建此脚本:

$ vim /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

4.启动keepalived服务

$ systemctl start keepalived
$ systemctl enable keepalived

5.查看组播状态

我们还可以在任意一台keepalived节点,通过tcpdump命令查看组播心跳状态,例如:

$ tcpdump -nn -i eth1 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
00:32:31.714987 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:31.715739 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20
00:32:32.716150 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:32.716292 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20
00:32:33.717327 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:33.721361 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20

如果提示报错:-bash: tcpdump: command not found.

安装tcpdump即可:

$ yum install tcpdump -y

6.配置LVS

分别安装lvs。CentOS7已经集成了LVS的核心,所以只需要安装LVS的管理工具就可以了:

$ yum -y install ipvsadm

分别停止ka67/ka68的keepalived服务:

$ systemctl stop keepalived

分别在ka67/ka68配置文件最后添加Virtual Server配置:

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
virtual_server 10.16.8.100 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    # sorry_server 127.0.0.1 80
    real_server 10.16.8.8 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.16.8.9 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}
virtual_server 10.16.8.101 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    # sorry_server 127.0.0.1 80
    real_server 10.16.8.8 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.16.8.9 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}

7.配置RS(Real Server)Web服务

分别在Web服务器安装Apache Httpd或Nginx作为Web服务,这里安装Nginx。

关于Nginx请参阅:《Centos 7 源码编译安装 Nginx》。

或通过以下方式安装Nginx,简单快速:

$ yum install epel-release -y
$ yum install nginx -y

测试环境为区分机器的不同,故将显示页面设置成服务器IP地址,但在生产环境中获取的内容是一致的。

分别在web8/web9执行如下命令:

$ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html
$ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

测试是否访问正常:

$ curl //127.0.0.1
Server 10.16.8.8

8.添加RS脚本

由于该脚本部分命令,在Centos7 最小化安装中没有,所以请先安装网络工具包:

$ yum install net-tools -y

分别在web服务器上添加rs.sh脚本:

$ vim /tmp/rs.sh
#!/bin/bash
vip1=10.16.8.100
vip2=10.16.8.101
dev1=lo:1
dev2=lo:2
case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev1 $vip1 netmask 255.255.255.255 broadcast $vip1 up
    ifconfig $dev2 $vip2 netmask 255.255.255.255 broadcast $vip2 up
    echo "VS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "VS Server is Cancel!"
    ;;
*)
    echo "Usage `basename $0` start|stop"
    exit 1
    ;;
esac

再分别启动该脚本:

$ /tmp/rs.sh start

如果需要停止,请执行如下命令:

$ /tmp/rs.sh stop

9.测试

在另一台服务器测试是否能够访问

[root@localhost ~]# for i in `seq 5`; do
>     curl 10.16.8.100
>     curl 10.16.8.101
> done
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.8

根据测试结果判断,已经实现了Keepalived+LVS-DR+Nginx高可用故障切换模式。

Keepalived双网络(内外网)故障同步漂移双活双主模式

前言:

在生产环境中,公网与内网都是独立分开的,所以称之为双网络。公网和内网在故障时实现同步漂移,例如:Keepalived+LVS-NAT 模式,这时就需要用到vrrp_sync_group设置同步漂移组。如果做双主双活,需要分别在两端加2个VIP,以达到互为主备的效果。

1.示意图:

  • 多播IP是:224.0.0.111。
  • 内网VIP1与内网VIP2互为主备。
  • 公网VIP1与公网VIP2互为主备。
  • 内网VIP1和公网VIP1是为一个同步组。
  • 内网VIP2和公网VIP2是为一个同步组。
                        +------+
			|Client|
			+------+
                           /\
		       +--------+   
                       |Internet|
		       +--------+
                           /\
		       +--------+  
                       |NAT 网络|
		       +--------+
                           /\
                +----------------------+
                | 内网VIP1:10.16.8.100 |
	        | 内网VIP2:10.16.8.101 |
                +----------------------+
                   /                \
+-----------------------+      +-----------------------+
|      KA+Lvs-NAT       |      |       KA+Lvs-NAT      |
|内网VIP1:Master (eth1) |      |内网VIP1:BACKUP (eth1) |
|内网VIP2:BACKUP (eth1) |      |内网VIP2:Master (eth1) |
|内网:10.16.8.10 (eth1) |<---->|内网:10.16.8.11 (eth1) |
|-----------------------|多播IP|-----------------------|
|公网VIP1:Master (eth2) |<---->|公网VIP1:BACKUP (eth2) |
|公网VIP2:BACKUP (eth2) |      |公网VIP2:Master (eth2) |
|公网:172.16.8.10(eth2) |      |公网:172.16.8.11(eth2) |                 
+-----------------------+      +-----------------------+
                   \                /
		+-----------------------+	 
                | 公网VIP1:172.16.8.100 |
		| 公网VIP2:172.16.8.101 |
		+-----------------------+
		           \/
			+------+
			|资源池|
		        +------+

2.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka67
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_sync_group VG_2 {
    group {
        External_2
        Internal_2
    }
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state BACKUP
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state BACKUP
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

3.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka68
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_sync_group VG_2 {
    group {
        External_2
        Internal_2
    }
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state MASTER
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state BACKUP
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state MASTER
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

Keepalived双网络(内外网)故障非同步漂移双活双主模式

前言:

在生产环境中,公网与内网都是独立分开的,所以称之为双网络。下面配置将要实现内网和公网故障时不必同步漂移,例如:Keepalived+LVS-DRKeepalived+NginxKeepalived+HAProxy 这些都无需同步漂移的。另外Keepalived+LVS-NAT则需要同步漂移。

1.示意图:

  • 多播IP是:224.0.0.111。
  • 一台机器的VIP内外网互为主备。
                        +------+
			|Client|
			+------+
                           /\
		       +--------+   
                       |Internet|
		       +--------+
                           /\
		       +--------+  
                       |NAT 网络|
		       +--------+
                           /\
                +----------------------+
                | 内网VIP1:10.16.8.100 |
		| 内网VIP2:10.16.8.101 |
                +----------------------+
                   /                \
+-----------------------+      +-----------------------+
|KA+Lvs-DR/Nginx/HAProxy|      |KA+Lvs-DR/Nginx/HAProxy|
|内网VIP1:Master (eth1) |      |内网VIP1:BACKUP (eth1) |
|内网VIP2:BACKUP (eth1) |      |内网VIP2:Master (eth1) |
|内网:10.16.8.10 (eth1) |<---->|内网:10.16.8.11 (eth1) |
|-----------------------|多播IP|-----------------------|
|公网VIP1:Master (eth2) |<---->|公网VIP1:BACKUP (eth2) |
|公网VIP2:BACKUP (eth2) |      |公网VIP2:Master (eth2) |
|公网:172.16.8.10(eth2) |      |公网:172.16.8.11(eth2) |                 
+-----------------------+      +-----------------------+
                   \                /
	        +-----------------------+	 
                | 公网VIP1:172.16.8.100 |
		| 公网VIP2:172.16.8.101 |
	        +-----------------------+
		           \/
			+------+
			|资源池|
			+------+

2.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka67
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state BACKUP
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state BACKUP
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

3.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka68
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state MASTER
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state BACKUP
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state MASTER
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

Keepalived双网络(内外网)故障同步漂移主备单主模式

前言:

在生产环境当中,内网与公网都是分开的,要实现内网和公网同步漂移,比如:Keepalived+LVS-NAT模式,那么就需要设置vrrp_sync_group同步组,不同的是下面配置只是主备模式,而不是主主模式。

1.示意图:

  • 多播IP是:224.0.0.111。
  • Master内网和公网VIP属同组。
  • BACKUP内网与公网VIP属同组。
                        +------+
			|Client|
			+------+
                           /\
		       +--------+   
                       |Internet|
		       +--------+
                           /\
		       +--------+  
                       |NAT 网络|
		       +--------+
                           /\
                +---------------------+
                | 内网VIP:10.16.8.100 |
                +---------------------+
                  /                \
+-----------------------+      +-----------------------+
|KA+Lvs/Nginx/HAProxy   |      |KA+Lvs/Nginx/HAProxy   |
|内网VIP:Master  (eth1) |      |内网VIP:BACKUP  (eth1) |
|内网:10.16.8.10 (eth1) |<---->|内网:10.16.8.11 (eth1) |
|-----------------------|多播IP|-----------------------|
|公网VIP:Master  (eth2) |<---->|公网VIP:BACKUP  (eth2) |
|公网:172.16.8.10(eth2) |      |公网:172.16.8.11(eth2) |                  
+-----------------------+      +-----------------------+
                   \                /
		+----------------------+	 
                | 公网VIP:172.16.8.100 |
		+----------------------+
		           \/
			+------+
			|资源池|
		        +------+

2.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka67
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

3.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka68
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state BACKUP
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

Keepalived双网络(内外网)故障非同步漂移主备单主模式

前言:

在生产环境当中,内网与公网都是独立分开的,因此内网和公网不用同步漂移,例如:Keepalived+LVS-DR、Keepalived+Nginx、Keepalived+HAProxy 都无需同步漂移。

注:Keepalived+LVS-NAT模式除外。

1.示意图:

多播IP是:224.0.0.111。

                        +------+
			|Client|
			+------+
                           /\
		       +--------+   
                       |Internet|
		       +--------+
                           /\
		       +--------+  
                       |NAT 网络|
		       +--------+
                           /\
                +---------------------+
                | 内网VIP:10.16.8.100 |
                +---------------------+
                  /                \
+-----------------------+      +-----------------------+
|KA+Lvs/Nginx/HAProxy   |      |KA+Lvs/Nginx/HAProxy   |
|内网VIP:Master  (eth1) |      |内网VIP:BACKUP  (eth1) |
|内网:10.16.8.10 (eth1) |<---->|内网:10.16.8.11 (eth1) |
|-----------------------|多播IP|-----------------------| 
|公网VIP:Master  (eth2) |<---->|公网VIP:BACKUP  (eth2) |
|公网:172.16.8.10(eth2) |      |公网:172.16.8.11(eth2) |                  
+-----------------------+      +-----------------------+
                   \                /
	        +----------------------+	 
                | 公网VIP:172.16.8.100 |
	        +----------------------+
		           \/
			+------+
			|资源池|
		        +------+

2.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka67
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

3.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka68
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state BACKUP
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

keepalived 源代码编译安装及配置文件讲解

何为Keepalived?

Keepalived是一个基于vrrp协议来实现的服务器高可用解决方案,可以利用其实现避免IP单点故障,类似的工具还有heartbeat、corosync。不过其不会单独出现,而是搭配着 LVS、Nginx、HAproxy,一起协同工作达到高可用的目的。

何为VRRP协议?

VRRP全称Vritual Router Redundancy Protocol,虚拟路由冗余协议。通过把几台提供路由功能的设备组成一个虚拟路由设备,使用一定的机制保证虚拟路由的高可用,从而达到保持业务的连续性与可靠性。

在配置组成的一个虚拟路由器中,有MASTER和BACKUP之分。MASTER是主节点,在一个虚拟路由器中,只能有一个MASTER,但可以有多个BACKUP,BACKUP是备用节点,也就是当master挂掉之后,BACKUP接管MASTER节点的所有资源,当有多个BACKUP节点时,根据其priority(优先级)的值的大小,来选举谁作为MASTER的替代者。当BACKUP节点的优先级值相同时,根据其IP地址的大小,来决定。

先决条件:

  • 节点之间的时间必须同步。
  • 确保FirewalldSELinux不会成为阻碍。
  • 各节点用于集群服务的网络接口必须支持MULTICAST(多播)通信。采用D类地址(224-239)。多播地址建议手动定义,因为若有多个集群服务都使用默认的,虽有认证机制,但仍会互发信息,可能会影响性能,更会产生无用日志信息。

1.时间同步

请参阅《Centos 7 Chrony 设置服务器集群系统时间同步》。

2.Firewalld防火墙配置

$ firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT
$ firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.111 --protocol vrrp -j ACCEPT
$ firewall-cmd --reload

3.开启路由转发

$ echo "net.ipv4.ip_forward=1" >>/etc/sysctl.conf
$ echo 1 > /proc/sys/net/ipv4/ip_forward

4.安装keepalived

$ cd /tmp
$ wget //www.keepalived.org/software/keepalived-1.3.9.tar.gz
$ tar xvf keepalived-1.3.9.tar.gz
$ cd keepalived-1.3.9
$ ./configure --prefix=/usr/local/keepalived
$ make && make install
$ ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin
$ mkdir /etc/keepalived/
$ ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

在安装过程中可能出现以下错误:

可能报错1:

checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/tmp/keepalived-1.3.9':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

解决方案:

$ yum install gcc -y

可能报错2:

configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.            !!!

解决方案:

$ yum install openssl-devel -y

可能报错3:

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

解决方案:

$ yum install libnl-devel

可能报错4:

checking libnfnetlink/libnfnetlink.h usability... no
checking libnfnetlink/libnfnetlink.h presence... no
checking for libnfnetlink/libnfnetlink.h... no
configure: error: libnfnetlink headers missing

解决方案:

$ yum install libnfnetlink-devel -y

安装总结:

一个最小化安装的Centos 7系统,在源代码编译安装keepalived时,首先安装以下依赖包,再编译安装keepalived就不会出现以上错误了。

安装依赖包:

$ yum install gcc openssl-devel libnl-devel libnfnetlink-devel ipvsadm -y

5.keepalived配置文件讲解

# 全局配置,包含两个子配置块:全局定义和静态地址和路由。

! Configuration File for keepalived
global_defs {               # 全局定义
   notification_email {     # 通知邮件相关设置
    acassen@firewall.loc    # 邮件发送目标地址
   }
   notification_email_from rwl@renwole.com  # 发件人
   smtp_server 127.0.0.1    # 使用本机邮件服务
   smtp_connect_timeout 30  # 设置连接smtp server的超时时间
   router_id LVS_DEVEL      # 标识当前节点唯一,节点间不能相同
   # 检查vrrp报文中的所有地址比较耗时
   # 设置此标志的意思是如果接收的到报文和上一个报文来至同一个路由器,则不执行检查。默认是跳过检查
   vrrp_skip_check_adv_addr
   vrrp_strict              # 严格执行VRRP协议规范,此模式不支持节点单播
   # 小数类型,单位秒。
   # 在一个网卡上每组gratuitous arp消息之间的延迟时间。
   # 默认为0,一个发送的消息=n组 arp报文
   vrrp_garp_interval 0
   # 小数类型,单位秒
   # 在一个网卡上每组na消息之间的延迟时间,默认为0
   vrrp_gna_interval 0
}

# 此区域是VRRP配置,含两个子配置块:vrrp_sync_group/vrrp_instance,主要对外提供服务VIP区域及其相关属性

vrrp_instance VI_1 {     # VRRP实例
    state MASTER         # 只能有一个是MASTER,余下的都应该为BACKUP;
    interface eth1	 # 对外提供网络的接口
    virtual_router_id 65 # 虚拟路由id标识,数字,必须和backup里相同
    priority 100         # 优先级,数字必须比backup大
    advert_int 1         # 组播信息发送间隔,两个节点设置必须一样,秒
    authentication {	 # 设置验证信息,两个节点必须一致(明文)
        auth_type PASS
        auth_pass 1111
    }
	# 虚拟地址,即Floating IP
    virtual_ipaddress {  # 可简写为单个地址,系统会默认计算掩码和设备
	
        172.16.28.65
	# 172.16.28.65/24
        # 172.16.28.65/24 dev eth1
    }
	# 定义通知脚本(另添加)
        notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
        notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
        notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

# 此区域是LVS配置。若用Keepalived+LVS,需要这段配置,若用其他,例如:Keepalived+Nginx,则无需配置。
# LVS包含两个子配置块:virtual_server_group/virtual_server
# virtual_server:虚拟服务器。每个虚拟服务器里面包含多个真实服务器real_server。

virtual_server 172.16.28.65 80 {  # 虚拟IP 监听80端口
    delay_loop 6		  # 健康检查时间间隔,秒
    lb_algo rr		          # 负载调度算法,常见使用wlc或rr
    lb_kind NAT		          # LVS负载转发规则,DR,NAT,TUN等
    persistence_timeout 50        # 会话保持时间,秒
    protocol TCP		  # 转发协议一般有tcp和udp两种

    real_server 172.16.28.65 80 { # 配置真实服务器的地址与端口
        weight 1		  # 权重
        SSL_GET {
            url {
              path /		  # 健康检查的页面
              digest ff20ad2481f97b1754ef3e12ecd3a9cc # 计算出的MD5值
            }
            url {
              path /mrtg/
              digest 9b3a0c85a887a256d6939da88aabd8cd
            }
            connect_timeout 3     # 连接超时时间,秒
            nb_get_retry 3	  # 失败重试次数,超过后移除
            delay_before_retry 3  # 失败重试间隔,秒
        }
    }
}
...

6.启动keepalived

在正确配置完keepalived.conf后,就可以启动keepalived,并且加入开机自启动服务。

$ systemctl start keepalived
$ systemctl enable keepalived

keepalived源码编译安装完毕。

说明:此文章主要介绍何为Keepalived及如何安装keepalived与注解keepalived的配置文件作用,无具体实验或生产配置。

欲了解生产环境中的keepalived各种模式配置,请参阅以下文章:

Keepalived单网络主备单主配置模式(实战)
Keepalived单网络双活双主配置模式
Keepalived双网络(内外网)故障非同步漂移主备单主模式
Keepalived双网络(内外网)故障同步漂移主备单主模式
Keepalived双网络(内外网)故障非同步漂移双活双主模式
Keepalived双网络(内外网)故障同步漂移双活双主模式