Tag Archives: 组播

Keepalived Nginx双网络（内外网）故障非同步漂移双活双主模式（实战）

1 Reply

介绍：

有了keepalived+Lvs这样的高性能组合，为什么还需keepalived+Nginx呢。keepalived是为了Lvs而设计。Lvs是一个四层的负载均衡设备，虽然有着高性能的优势，但它无后端服务器的健康检查机制。keepalived为lvs提供一系列的健康检查机制，例如：TCP_CHECK,UDP_CHECK,HTTP_GET等。同时lvs也可以自己写健康检查脚脚本。或者结合ldirectory来实现后端健康检测。但LVS始终无法摆脱它是一个四层设备，无法对上层协议进行解析。而Nginx就不一样了，Nginx是一个七层的设备可以对七层协议进行解析，可以对一些请求进行过滤，还可以对请求结果进行缓存。这些都是Nginx独有的优势。但是keepalived并没有为Nginx提供健康检测。需要自己去写一些脚步来进行健康检测。

下面主要讲解Keepalived+Nginx的模式，不包含lvs。如果不是大型负载，一般用不到LVS，当然你也可以参阅：《Keepalived LVS-DR Nginx单网络双活双主配置模式（实战）》篇。

准备四台服务器或虚拟机：

Web Nginx 内网：10.16.8.8/10.16.8.9

Keepalived 内网：10.16.8.10（ka67）/10.16.8.11（ka68）
Keepalived 公网：172.16.8.10/172.16.8.11

Keepalived 内网VIP：10.16.8.100/10.16.8.101
Keepalived 公网VIP：172.16.8.100/172.16.8.101

OS：CentOS Linux release 7.4.1708 (Core)

先决条件：

安装keepalived。
时间同步。
设置SELinux和防火墙。
互相之间/etc/hosts文件添加对方主机名（可选）。
确认网络接口支持多播（组播）新网卡默认支持。

以上部署请参阅：《keepalived 安装及配置文件讲解》。

1.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"  
}
vrrp_instance External_2 {
    state BACKUP
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"  
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth0
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}
vrrp_instance Internal_2 {
    state BACKUP
    interface eth0
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}

2.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
 }
 
vrrp_instance External_2 {
    state MASTER
    interface eth1
    virtual_router_id 172
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
   }
   
vrrp_instance Internal_1 {
    state BACKUP
    interface eth0
    virtual_router_id 191
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}
vrrp_instance Internal_2 {
    state MASTER
    interface eth0
    virtual_router_id 192
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}

3.创建检测通用脚本

$ vim /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup
    systemctl start nginx   # 此处配置后，Nginx服务挂了能自动启动   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

4.启动keepalived服务并测试

启动ka67后查看其网卡状态：

[root@ka67 ~]# systemctl start keepalived
[root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.101/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::436e:b837:43b:797c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.100/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.101/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::1261:7633:b595:7719/64 scope link
       valid_lft forever preferred_lft forever

在ka68没有启动时，ka67添加了4个VIP，分别是：

公网eth0：

172.16.8.100/32
172.16.8.101/32

内网eth1：

10.16.8.100/32
10.16.8.101/32

启动ka68后查看其网卡状态：

[root@ka68 ~]# systemctl start keepalived
[root@ka68 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:79 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.11/24 brd 103.28.204.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.101/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3d2c:ecdc:5e6d:70ba/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:82 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.11/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.101/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::4fb3:d0a8:f08c:4536/64 scope link
       valid_lft forever preferred_lft forever

ka68添加了2个VIP，分别是：

公网eth0：

172.16.8.101/32

内网eth1：

10.16.8.101/32

再次查看ka67的网卡状态信息：

[root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::436e:b837:43b:797c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.100/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::1261:7633:b595:7719/64 scope link
       valid_lft forever preferred_lft forever

注意到 172.16.8.101/10.16.8.101 已经被移除了，此时无论停掉任意一台服务器，4个VIP都不会停止通信。

另外可以在ka67/ka68通过如下命令查看组播地址的心跳状态：

[root@ka67 ~]# tcpdump -nn -i eth1 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
02:00:15.690389 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
02:00:15.692654 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20
02:00:16.691552 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
02:00:16.693814 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20
02:00:17.692710 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20

到目前为止，vrrp的高可用配置&测试已完成，接下来我们继续配置Web Nginx服务。

5.安装并配置Nginx

分别在后端服务器 10.16.8.8/10.16.8.9 安装Nginx：

关于Nginx请参阅：《Centos 7源码编译安装 Nginx》。

或通过以下方式yum安装Nginx；简单快速：

$ yum install epel-release -y
$ yum install nginx -y

测试环境为区分机器的不同，故将web页面设置服务器IP地址，但在生产环境中获取的内容是一致的。

分别在10.16.8.8/10.16.8.9执行如下命令：

$ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html
$ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

测试是否访问正常：

$ curl //10.16.8.8
Server 10.16.8.8

分别在ka67/ka68上安装Nginx，我这里用yum安装：

$ yum install nginx psmisc -y

说明：psmisc包含了：fuser,killall,pstree等命令。

在ka67/ka68上配置Nginx：

备份默认配置文件：

$ mv /etc/nginx/conf.d/default.conf{,.bak}
$ mv /etc/nginx/nginx.conf{,.bak}

分别在ka67/ka68将nginx主配置文件中添加如下内容：

$ vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    include /etc/nginx/conf.d/*.conf;
    upstream webserverapps {
    server 10.16.8.8:80;
    server 10.16.8.9:80;
    #server 127.0.0.1:8080 backup;
   }

server {
        listen 80;
        server_name _;
location / {
     proxy_pass //webserverapps;
     proxy_redirect off;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     client_max_body_size 10m;
     client_body_buffer_size 128k;
     proxy_connect_timeout 90;
     proxy_send_timeout 90;
     proxy_read_timeout 90;
     proxy_buffer_size 4k;
     proxy_buffers 4 32k;
     proxy_busy_buffers_size 64k;
     proxy_temp_file_write_size 64k;
     add_header Access-Control-Allow-Origin *;
       }
    }

}

注意：以上配置主要添加了蓝色部分，其他默认，仅为测试使用。生产环境请根据自己需求调整配置。

在ka67/ka68重启Nginx服务：

$ systemctl restart nginx

分别在ka67/ka68上测试：

[root@ka67 ~]# for i in `seq 10`; do curl 10.16.8.10; done
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.9

到目前为止，Nginx反代功能也已实现，下面我们将把Nginx与Keepalived结合起来，使Nginx支持高可用。

6.配置Keepalived Nginx高可用

分别在ka67/ka68配置文件/usr/local/keepalived/etc/keepalived/keepalived.conf的全局配置块global_defs下方添加vrrp_script配置块：

vrrp_script chk_nginx {
    script "killall -0 nginx"
    interval 2
    weight -10
    fall 2
    rise 2
}

在所有vrrp_instance实例块里，添加track_script块：

track_script {
    chk_nginx
}

例如：

...
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    track_script {
    chk_nginx
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_"/usr/local/keepalived/etc/keepalived/notify.sh fault"
 }
...

配置完以后，重启ka67/ka68的keepalived服务：

$ systemctl stop keepalived
$ systemctl start keepalived

总结：

在配置过程中出现了无法漂移的情况，跨网段问题。解决通道，还是要多看日志，多分析判断，最终还是能解决问题的。无论在何种情况下，既然选择了keepalived，就要坚信自己的初心。
如你在配置过程中出现任何问题，欢迎留言，共同解决问题。