月度归档:2018年04月

Centos 7利用内存优化磁盘缓存读写速度

Linux中的/dev/shm目录不属于磁盘,而是内存,如果使用/dev/shm/目录作为Linux中的磁盘文件读写缓存,可想效率是非常惊人的。

默认/dev/shm目录没有挂载,需要手动进行挂载。

在以下文件尾端添加以下内容:

$ vim /etc/fstab
tmps /dev/shm tmpfs defaults,size=1G 0 0

请根据自己的物理内存大小添加,一般是物理内存的10-50%左右。

挂载/dev/shm/目录:

$ mount -o remount /dev/shm/
$ mkdir /dev/shm/tmp
$ chmod 755 /dev/shm/tmp
$ mount -B /dev/shm/tmp /tmp

注意:

/dev/shm/tmp在系统重启后会丢失挂载,需要重新设置挂载,下面有个shell脚本,你加入开机自启即可:

$ vim /etc/init.d/shmtmp.sh
#!/bin/bash
mkdir /dev/shm/tmp
chmod 755 /dev/shm/tmp
mount -B /dev/shm/tmp/ /tmp

之后再以下文件末尾添加如下内容:

$ vim /etc/rc.local
sh /etc/init.d/shmtmp.sh

这样就实现了重启自动挂载。你就可以利用内存提高读写性能,例如:将php的session,以及其他的缓存放在/tmp目录下,速度效率成倍提升。

Let’s Encrypt SSL证书续期失败 ascii编解码器不能编码

今天复查服务器SSL证书,发现Let’s Encrypt证书就快要到期了,查看crontab定时任务计划日志,也是正常执行的。例如:

$ cat /var/log/cron
...
CROND[31471]: (root) CMD (/usr/bin/certbot renew --quiet && /bin/systemctl restart nginx)
CROND[31470]: (root) MAIL (mailed 375 bytes of output but got status 0x004b#012)
CROND[31482]: (root) CMD (run-parts /etc/cron.hourly)
...

奇怪的是,证书并没有正常续期,什么个原因呢,后来手动执行证书更新:

$ /usr/bin/certbot renew --quiet
Attempting to renew cert from /etc/letsencrypt/renewal/renwole.com.conf produced an unexpected error: 'ascii' codec can't encode characters in position 247-248: ordinal not in range(128). Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/renwole.com.conf/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

更新失败,提示“ascii”编解码器不能编码字符。

经过分析研究发现,开发人员修改了网站根目录,导致LetsEncrypt找不到相关配置文件。
PS:唉,出了问题,都是运维的锅哈。

解决方案

修改以下配置文件中的网站根目录:

$ vim /etc/letsencrypt/renewal/renwole.com.conf
...
# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = a07a7160ea489g586aeaada1368ce0d6
[[webroot_map]]
renwole.com = /apps/data/www/renwolecom
...

修改蓝色部分为Nginx指定的根目录,其他默认,然后保存即可。

再次更新证书成功。

使用以下命令查看续期状态:

$ certbot certificates

运行环境正常 织梦DedeCMS验证码无法显示

在环境一切OK的情况下,DedeCMS的验证码无法显示。

解决方案一:

include目录下打开vdimgck.php文件,查找 if(function_exists(“imagejpeg”)) 在上方添加 ob_clean(); 问题解决。

这个问题很奇怪,有时候不加也没问题,当然我的是运行有问题了加上就没事,之前没事,突然就不行了。

添加过后的具体代码如下:

ob_clean();
if(function_exists(“imagejpeg”))
{
    header(“content-type:image/jpeg\r\n”);
    imagejpeg($im);
}
else
{
    header(“content-type:image/png\r\n”);
    imagepng($im);
}

解决方案二:

将织梦后台登录验证码取消。

具体操作如下:

找到以下文件:

/data/safe/inc_safe_config.php

在该文件中查找 $safe_gdopen = ’1,2,3,5,6,7′; 删除6数字即可取消验证码功能。

Centos 7如何备份和还原Redis数据

何为Redis?

Redis是内存中的键值缓存和存储(即数据库),也可以永久保存到磁盘。在本文中,你将了解,如何在Centos 7上备份还原您的redis数据库。

备份还原说明

默认情况下,Redis数据将保存到磁盘中的.rdb文件中,该文件是Redis数据集的时间点快照。快照是按照指定的时间间隔进行的,所以对于备份来说是完美的。

1.数据备份篇

在Centos 7和其他Linux发行版中,Redis数据库目录默认是/var/lib/redis。但是,如果您更改了redis存储位置,则可以通过键入以下命令查找:

[root@renwolecom ~]# find / -name *rdb

使用redis-cli管理工具进入数据库:

[root@renwolecom ~]# redis-cli

由于大部分数据都运行在内存中,redis只会每隔一段时间保存一次,为获取最新副本,请执行以下命令:

10.10.204.64:6379> save
OK
(1.02s)

另外,如果Redis设置了用户认证,就需要先验证,再保存,例如:

10.10.204.64:6379> auth RenwoleQxl5qpKHrh9khuTW
10.10.204.64:6379> save

之后进行备份即可,例如:

[root@renwolecom ~]# cp /var/lib/redis/dump.rdb /apps/redis-backup-20180129

2.数据还原篇

恢复备份需要您使用恢复文件替换现有的Redis数据库文件。为保证原始数据文件不被破坏,我们建议尽可能恢复到新的Redis服务器。

停止Redis数据库,一旦停止,Redis数据库处于脱机状态。

[root@renwolecom ~]# systemctl stop redis

如果恢复到原始Redis服务器,请重命名当前数据文件,之后再进行恢复:

[root@renwolecom ~]# mv /var/lib/redis/dump.rdb /var/lib/redis/dump.rdb.old
[root@renwolecom ~]# cp -p /apps/redis-backup-20180129/dump.rdb /var/lib/redis/dump.rdb

设置dump.rdb文件权限,拷贝过来的数据文件,可能不具备Redis用户及读取权限,需要手动赋予:

[root@renwolecom ~]# chown redis:redis /var/lib/redis/dump.rdb
[root@renwolecom ~]# chmod 660 /var/lib/redis/dump.rdb

启动redis

[root@renwolecom ~]# systemctl start redis

大功告成!现在你可以登录redis核实数据了。

注意说明:

根据需求,关闭AOF,AOF跟踪每个写操作到Redis数据库。由于我们试图从时间点备份中恢复,所以我们不希望Redis重新创建存储在其AOF文件中的操作。

是否开启AOF可以通过查看文件获悉:

[root@renwolecom ~]# ls /var/lib/redis/

如果看到.aof后缀的文件,说明你启用了AOF。

重命名.aof文件,

[root@renwolecom ~]# mv /var/lib/redis/*.aof /var/lib/redis/appendonly.aof.old

如果有多个.aof文件,请单独命名。

编辑你的Redis配置文件暂时关闭AOF:

[root@renwolecom ~]# vim /etc/redis/redis.conf
appendonly no

如果备份期间有任何疑问,欢迎留言。

Centos 7添加删除Swap交换分区

Swap介绍:

Linux 将物理内存分为内存段,叫做页面。交换是指内存页面被复制到预先设定好的硬盘空间(叫做交换空间)的过程,目的是释放对于页面的内存。物理内存和交换空间的总大小是可用的虚拟内存的总量。

Swap即:交换分区,类似于Windows的虚拟内存,但物理内存不足时,把部分硬盘空间当成虚拟内存使用,从而解决了物理内存容量不足。

优点:节省成本。
缺点:性能不足。

此方法不局限于Centos 7,Linux系统均可使用。

操作用户:root。

1.添加swap交换分区空间

使用dd命令创建swap交换分区文件/dev/mapper/centos-swap,大小为2G:

$ dd if=/dev/zero of=/dev/mapper/centos-swap bs=1024 count=2048000

格式化swap分区:

$ mkswap /dev/mapper/centos-swap

设置交换分区:

$ mkswap -f /dev/mapper/centos-swap

激活swap分区:

$ swapon /dev/mapper/centos-swap

设为开机自动启用:

$ vim /etc/fstab

在该文件底部添加如下内容:

/dev/mapper/centos-swap swap swap default 0 0

2.删除swap交换分区

停止正在使用的swap分区:

$ swapoff /dev/mapper/centos-swap

删除swap分区文件:

$ rm /dev/mapper/centos-swap

删除或注释在/etc/fstab文件中的以下开机自动挂载内容:

/dev/mapper/centos-swap swap swap default 0 0

大功告成!

Keepalived Nginx双网络(内外网)故障非同步漂移双活双主模式(实战)

介绍:

有了keepalived+Lvs这样的高性能组合,为什么还需keepalived+Nginx呢。keepalived是为了Lvs而设计。Lvs是一个四层的负载均衡设备,虽然有着高性能的优势,但它无后端服务器的健康检查机制。keepalived为lvs提供一系列的健康检查机制,例如:TCP_CHECK,UDP_CHECK,HTTP_GET等。同时lvs也可以自己写健康检查脚脚本。或者结合ldirectory来实现后端健康检测。但LVS始终无法摆脱它是一个四层设备,无法对上层协议进行解析。而Nginx就不一样了,Nginx是一个七层的设备可以对七层协议进行解析,可以对一些请求进行过滤,还可以对请求结果进行缓存。这些都是Nginx独有的优势。但是keepalived并没有为Nginx提供健康检测。需要自己去写一些脚步来进行健康检测。

下面主要讲解Keepalived+Nginx的模式,不包含lvs。如果不是大型负载,一般用不到LVS,当然你也可以参阅:《Keepalived LVS-DR Nginx单网络双活双主配置模式(实战)》篇。

准备四台服务器或虚拟机:

Web Nginx 内网:10.16.8.8/10.16.8.9

Keepalived 内网:10.16.8.10(ka67)/10.16.8.11(ka68)
Keepalived 公网:172.16.8.10/172.16.8.11

Keepalived 内网VIP:10.16.8.100/10.16.8.101
Keepalived 公网VIP:172.16.8.100/172.16.8.101

OS:CentOS Linux release 7.4.1708 (Core)

先决条件:

安装keepalived。
时间同步。
设置SELinux和防火墙。
互相之间/etc/hosts文件添加对方主机名(可选)。
确认网络接口支持多播(组播)新网卡默认支持。

以上部署请参阅:《keepalived 安装及配置文件讲解》。

1.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"  
}
vrrp_instance External_2 {
    state BACKUP
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"  
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth0
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}
vrrp_instance Internal_2 {
    state BACKUP
    interface eth0
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}

2.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
 }
 
vrrp_instance External_2 {
    state MASTER
    interface eth1
    virtual_router_id 172
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
   }
   
vrrp_instance Internal_1 {
    state BACKUP
    interface eth0
    virtual_router_id 191
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}
vrrp_instance Internal_2 {
    state MASTER
    interface eth0
    virtual_router_id 192
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"          
}

3.创建检测通用脚本

$ vim /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup
    systemctl start nginx   # 此处配置后,Nginx服务挂了能自动启动   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

4.启动keepalived服务并测试

启动ka67后查看其网卡状态:

[root@ka67 ~]# systemctl start keepalived
[root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.101/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::436e:b837:43b:797c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.100/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.101/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::1261:7633:b595:7719/64 scope link
       valid_lft forever preferred_lft forever

在ka68没有启动时,ka67添加了4个VIP,分别是:

公网eth0:

172.16.8.100/32
172.16.8.101/32

内网eth1:

10.16.8.100/32
10.16.8.101/32

启动ka68后查看其网卡状态:

[root@ka68 ~]# systemctl start keepalived
[root@ka68 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:79 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.11/24 brd 103.28.204.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.101/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3d2c:ecdc:5e6d:70ba/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:82 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.11/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.101/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::4fb3:d0a8:f08c:4536/64 scope link
       valid_lft forever preferred_lft forever

ka68添加了2个VIP,分别是:

公网eth0:

172.16.8.101/32

内网eth1:

10.16.8.101/32

再次查看ka67的网卡状态信息:

[root@ka67 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:78 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.10/24 brd 172.16.8.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.16.8.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::436e:b837:43b:797c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:15:5d:ae:02:84 brd ff:ff:ff:ff:ff:ff
    inet 10.16.8.10/24 brd 10.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.16.8.100/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::1261:7633:b595:7719/64 scope link
       valid_lft forever preferred_lft forever

注意到 172.16.8.101/10.16.8.101 已经被移除了,此时无论停掉任意一台服务器,4个VIP都不会停止通信。

另外可以在ka67/ka68通过如下命令查看组播地址的心跳状态:

[root@ka67 ~]# tcpdump -nn -i eth1 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
02:00:15.690389 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
02:00:15.692654 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20
02:00:16.691552 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20
02:00:16.693814 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 172, prio 100, authtype simple, intvl 1s, length 20
02:00:17.692710 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 171, prio 100, authtype simple, intvl 1s, length 20

到目前为止,vrrp的高可用配置&测试已完成,接下来我们继续配置Web Nginx服务。

5.安装并配置Nginx

分别在后端服务器 10.16.8.8/10.16.8.9 安装Nginx:

关于Nginx请参阅:《Centos 7源码编译安装 Nginx》。

或通过以下方式yum安装Nginx;简单快速:

$ yum install epel-release -y
$ yum install nginx -y

测试环境为区分机器的不同,故将web页面设置服务器IP地址,但在生产环境中获取的内容是一致的。

分别在10.16.8.8/10.16.8.9执行如下命令:

$ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html
$ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

测试是否访问正常:

$ curl //10.16.8.8
Server 10.16.8.8

分别在ka67/ka68上安装Nginx,我这里用yum安装:

$ yum install nginx psmisc -y

说明:psmisc包含了:fuser,killall,pstree等命令。

ka67/ka68上配置Nginx:

备份默认配置文件:

$ mv /etc/nginx/conf.d/default.conf{,.bak}
$ mv /etc/nginx/nginx.conf{,.bak}

分别在ka67/ka68将nginx主配置文件中添加如下内容:

$ vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    include /etc/nginx/conf.d/*.conf;
    upstream webserverapps {
    server 10.16.8.8:80;
    server 10.16.8.9:80;
    #server 127.0.0.1:8080 backup;
   }

server {
        listen 80;
        server_name _;
location / {
     proxy_pass //webserverapps;
     proxy_redirect off;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     client_max_body_size 10m;
     client_body_buffer_size 128k;
     proxy_connect_timeout 90;
     proxy_send_timeout 90;
     proxy_read_timeout 90;
     proxy_buffer_size 4k;
     proxy_buffers 4 32k;
     proxy_busy_buffers_size 64k;
     proxy_temp_file_write_size 64k;
     add_header Access-Control-Allow-Origin *;
       }
    }

}

注意:以上配置主要添加了蓝色部分,其他默认,仅为测试使用。生产环境请根据自己需求调整配置。

ka67/ka68重启Nginx服务:

$ systemctl restart nginx

分别在ka67/ka68上测试:

[root@ka67 ~]# for i in `seq 10`; do curl 10.16.8.10; done
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.9

到目前为止,Nginx反代功能也已实现,下面我们将把Nginx与Keepalived结合起来,使Nginx支持高可用。

6.配置Keepalived Nginx高可用

分别在ka67/ka68配置文件/usr/local/keepalived/etc/keepalived/keepalived.conf的全局配置块global_defs下方添加vrrp_script配置块:

vrrp_script chk_nginx {
    script "killall -0 nginx"
    interval 2
    weight -10
    fall 2
    rise 2
}

在所有vrrp_instance实例块里,添加track_script块:

track_script {
    chk_nginx
}

例如:

...
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    track_script {
    chk_nginx
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_"/usr/local/keepalived/etc/keepalived/notify.sh fault"
 }
...

配置完以后,重启ka67/ka68的keepalived服务:

$ systemctl stop keepalived
$ systemctl start keepalived

总结:

在配置过程中出现了无法漂移的情况,跨网段问题。解决通道,还是要多看日志,多分析判断,最终还是能解决问题的。无论在何种情况下,既然选择了keepalived,就要坚信自己的初心。
如你在配置过程中出现任何问题,欢迎留言,共同解决问题。

Keepalived LVS-DR Nginx单网络双活双主配置模式(实战)

何为LVS/DR模式?

LVS是Linux Virtual Server的简写,意即Linux虚拟服务器,是一个虚拟的服务器集群系统。LVS目前有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR)、十种调度算法(rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq)。

LVS在Unix-like系统中是作为一个前端(Director)存在的,又称为调度器,它本身不提供任何的服务,只是将通过互联网进来的请求接受后再转发给后台运行的真正的服务器(RealServer)进行处理,然后响应给客户端。

LVS集群采用IP负载均衡技术和基于内容请求分发技术。调度器具有很好的吞吐率,将请求均衡地转移到不同的服务器上执行,且调度器自动屏蔽掉服务器的故障,从而将一组服务器构成一个高性能的、高可用的虚拟服务器。整个服务器集群的结构对客户是透明的,而且无需修改客户端和服务器端的程序。为此,在设计时需要考虑系统的透明性、可伸缩性、高可用性和易管理性。

LVS有两个重要的组件:一个是IPVS,一个是IPVSADM。ipvs是LVS的核心组件,它本身只是一个框架,类似于iptables,工作于内核空间中。ipvsadm 是用来定义LVS的转发规则的,工作于用户空间中。

LVS有三种转发类型:

LVS-NAT模式:

称为网络地址转换,实现起来比较简单,所有的RealServer集群节点和前端调度器Director都要在同一个子网中,这种模型可以实现端口映射,RealServer的操作系统可以是任意操作系统,前端的Director既要处理客户端发起的请求,又要处理后台RealServer的响应信息,将RealServer响应的信息再转发给客户端,前端Director很容易成为整个集群系统性能的瓶颈。通常情况下RealServer的IP地址(以下简成RIP)为私有地址,便于RealServer集群节点之间进行通信通常情况下前端的Director有两个IP地址,一个为VIP,是虚拟的IP地址,客户端向此IP地址发起请求。一个是DIP,是真正的Director的IP地址,RIP的网关要指向Director的DIP。

LVS-DR模式:

DR:直接路由(direct routing)模式,此种模式通过MAC地址转发工作,所有的RealServer集群节点和前端调度器Director都要在同一个物理网络中,此种模式不支持端口映射,此种模式的性能要优于LVS-NAT,RIP可以使用公网的IP,RIP的网关不能指向DIP。

优点:

相对LVS/NAT模式,DR模式不需要把返回的数据通过负载均衡转发,想要他发挥优势,那么就要相应的数据包的数量和长度远远大于请求数据包,幸运的是,大部分WEB服务都具备这样的特点,响应和请求并不对称,因此常用的WEB服务,都可以使用这种模式。

这种方式,负载均衡器不再是系统的瓶颈。如果你的负载均衡器只拥有100M的全双工网卡和带宽的话,通过集群的横向扩展也可以让整个系统达到1G的流量。

来自LVS官方站点的测试结果也告诉我们,LVS-DR可以容纳100台以上的实际应用服务器,对一般的服务而已,这样的表现足够了。

不足:

DR模式下不能跨网段转发数据,如果必须要跨网段进行负载,那么就必须使用LVS/TUN模式。

LVS-TUN模式:

称为隧道模型RealServer服务器与前端的Director可以在不同的网络中,此种模型也不支持端口映射,RealServer只能使用哪些支持IP隧道的操作系统,前端的Director只处理客户端的请求,然后将请求转发给RealServer,由后台的RealServer直接响应客户端,不再经过Director,RIP一定不能是私有IP,在DR、TUN模式中,数据包是直接返回给用户的,所以,在Director Server上以及集群的每个节点上都需要设置这个地址。此IP在Real Server上一般绑定在回环地址上,例如lo:0,同样,在Director Server上,虚拟IP绑定在真实的网络接口设备上,例如eth0:0。

开始部署:

准备四台服务器或虚拟机:

Web Nginx:10.16.8.8/10.16.8.9
Keepalived:10.16.8.10/10.16.8.11
Keepalived VIP:10.16.8.100/10.16.8.101
OS:CentOS Linux release 7.4.1708 (Core)

先决条件:

安装keepalived。
时间同步。
设置SELinux和防火墙。
互相之间/etc/hosts文件添加对方主机名(可选)。
确认网络接口支持多播(组播)新网卡默认支持。

以上部署请参阅:《keepalived 安装及配置文件讲解》。

1.ka67配置文件

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 60
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"        
}
vrrp_instance VI_2 {
    state BACKUP
    interface eth1
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         
}

2.ka68配置文件

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 60
   vrrp_mcast_group4 224.0.0.111
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 191
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"         
}
vrrp_instance VI_2 {
    state MASTER
    interface eth1
    virtual_router_id 192
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"        
}

3.创建通用notify.sh检测脚本

分别创建此脚本:

$ vim /usr/local/keepalived/etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@localhost'
                
notify() {
    local mailsubject="$(hostname) to be $1, vip floating"
    local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
                
case $1 in
master)
    notify master   
    ;;
backup)
    notify backup   
    ;;
fault)
    notify fault    
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

4.启动keepalived服务

$ systemctl start keepalived
$ systemctl enable keepalived

5.查看组播状态

我们还可以在任意一台keepalived节点,通过tcpdump命令查看组播心跳状态,例如:

$ tcpdump -nn -i eth1 host 224.0.0.111
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
00:32:31.714987 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:31.715739 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20
00:32:32.716150 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:32.716292 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20
00:32:33.717327 IP 10.16.8.10 > 224.0.0.111: VRRPv2, Advertisement, vrid 191, prio 100, authtype simple, intvl 1s, length 20
00:32:33.721361 IP 10.16.8.11 > 224.0.0.111: VRRPv2, Advertisement, vrid 192, prio 100, authtype simple, intvl 1s, length 20

如果提示报错:-bash: tcpdump: command not found.

安装tcpdump即可:

$ yum install tcpdump -y

6.配置LVS

分别安装lvs。CentOS7已经集成了LVS的核心,所以只需要安装LVS的管理工具就可以了:

$ yum -y install ipvsadm

分别停止ka67/ka68的keepalived服务:

$ systemctl stop keepalived

分别在ka67/ka68配置文件最后添加Virtual Server配置:

$ vim /usr/local/keepalived/etc/keepalived/keepalived.conf
virtual_server 10.16.8.100 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    # sorry_server 127.0.0.1 80
    real_server 10.16.8.8 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.16.8.9 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}
virtual_server 10.16.8.101 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    # sorry_server 127.0.0.1 80
    real_server 10.16.8.8 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
    real_server 10.16.8.9 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}

7.配置RS(Real Server)Web服务

分别在Web服务器安装Apache Httpd或Nginx作为Web服务,这里安装Nginx。

关于Nginx请参阅:《Centos 7 源码编译安装 Nginx》。

或通过以下方式安装Nginx,简单快速:

$ yum install epel-release -y
$ yum install nginx -y

测试环境为区分机器的不同,故将显示页面设置成服务器IP地址,但在生产环境中获取的内容是一致的。

分别在web8/web9执行如下命令:

$ echo "Server 10.16.8.8" > /usr/share/nginx/html/index.html
$ echo "Server 10.16.8.9" > /usr/share/nginx/html/index.html

测试是否访问正常:

$ curl //127.0.0.1
Server 10.16.8.8

8.添加RS脚本

由于该脚本部分命令,在Centos7 最小化安装中没有,所以请先安装网络工具包:

$ yum install net-tools -y

分别在web服务器上添加rs.sh脚本:

$ vim /tmp/rs.sh
#!/bin/bash
vip1=10.16.8.100
vip2=10.16.8.101
dev1=lo:1
dev2=lo:2
case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev1 $vip1 netmask 255.255.255.255 broadcast $vip1 up
    ifconfig $dev2 $vip2 netmask 255.255.255.255 broadcast $vip2 up
    echo "VS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "VS Server is Cancel!"
    ;;
*)
    echo "Usage `basename $0` start|stop"
    exit 1
    ;;
esac

再分别启动该脚本:

$ /tmp/rs.sh start

如果需要停止,请执行如下命令:

$ /tmp/rs.sh stop

9.测试

在另一台服务器测试是否能够访问

[root@localhost ~]# for i in `seq 5`; do
>     curl 10.16.8.100
>     curl 10.16.8.101
> done
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.8
Server 10.16.8.8
Server 10.16.8.9
Server 10.16.8.9
Server 10.16.8.8

根据测试结果判断,已经实现了Keepalived+LVS-DR+Nginx高可用故障切换模式。

Keepalived双网络(内外网)故障同步漂移双活双主模式

前言:

在生产环境中,公网与内网都是独立分开的,所以称之为双网络。公网和内网在故障时实现同步漂移,例如:Keepalived+LVS-NAT 模式,这时就需要用到vrrp_sync_group设置同步漂移组。如果做双主双活,需要分别在两端加2个VIP,以达到互为主备的效果。

1.示意图:

  • 多播IP是:224.0.0.111。
  • 内网VIP1与内网VIP2互为主备。
  • 公网VIP1与公网VIP2互为主备。
  • 内网VIP1和公网VIP1是为一个同步组。
  • 内网VIP2和公网VIP2是为一个同步组。
                        +------+
			|Client|
			+------+
                           /\
		       +--------+   
                       |Internet|
		       +--------+
                           /\
		       +--------+  
                       |NAT 网络|
		       +--------+
                           /\
                +----------------------+
                | 内网VIP1:10.16.8.100 |
	        | 内网VIP2:10.16.8.101 |
                +----------------------+
                   /                \
+-----------------------+      +-----------------------+
|      KA+Lvs-NAT       |      |       KA+Lvs-NAT      |
|内网VIP1:Master (eth1) |      |内网VIP1:BACKUP (eth1) |
|内网VIP2:BACKUP (eth1) |      |内网VIP2:Master (eth1) |
|内网:10.16.8.10 (eth1) |<---->|内网:10.16.8.11 (eth1) |
|-----------------------|多播IP|-----------------------|
|公网VIP1:Master (eth2) |<---->|公网VIP1:BACKUP (eth2) |
|公网VIP2:BACKUP (eth2) |      |公网VIP2:Master (eth2) |
|公网:172.16.8.10(eth2) |      |公网:172.16.8.11(eth2) |                 
+-----------------------+      +-----------------------+
                   \                /
		+-----------------------+	 
                | 公网VIP1:172.16.8.100 |
		| 公网VIP2:172.16.8.101 |
		+-----------------------+
		           \/
			+------+
			|资源池|
		        +------+

2.ka67配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka67
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_sync_group VG_2 {
    group {
        External_2
        Internal_2
    }
}
vrrp_instance External_1 {
    state MASTER
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state BACKUP
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state MASTER
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state BACKUP
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}

3.ka68配置文件

global_defs {
   notification_email {
     root@localhost
   }
   notification_email_from ka@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka68
   vrrp_mcast_group4 224.0.0.111
}
vrrp_sync_group VG_1 {
    group {
        External_1
        Internal_1
    }
}
vrrp_sync_group VG_2 {
    group {
        External_2
        Internal_2
    }
}
vrrp_instance External_1 {
    state BACKUP
    interface eth1
    virtual_router_id 171
    priority 100
    advert_int 1    
    authentication {
        auth_type PASS
        auth_pass renwole0
    }
    virtual_ipaddress {
        10.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance External_2 {
    state MASTER
    interface eth1
    virtual_router_id 172
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole1
    }
    virtual_ipaddress {
        10.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_1 {
    state BACKUP
    interface eth2
    virtual_router_id 191
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole2
    }
    virtual_ipaddress {
        172.16.8.100
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}
vrrp_instance Internal_2 {
    state MASTER
    interface eth2
    virtual_router_id 192
    priority 95
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass renwole3
    }
    virtual_ipaddress {
        172.16.8.101
    }
    notify_master "/usr/local/keepalived/etc/keepalived/notify.sh master"
    notify_backup "/usr/local/keepalived/etc/keepalived/notify.sh backup"
    notify_fault "/usr/local/keepalived/etc/keepalived/notify.sh fault"
}