Centos Keepalived-Ha集群+Lvs-DR+Nginx集群配置

背景介绍

我们都知道一台服务器能提供的资源是有限的，如果想打破这个限制，只有两个办法，一个是购买硬件，提升服务器性能，但是性价比过高，而第二种方法这则是，使用集群软件让两台服务器处理一个服务的不同请求，分带单台服务器的负载，从而提升性能，实现这一的软件开源著名的有Nginx或者Lvs，这里我们使用的是LVS，但是lvs需要一个调度服务器，这样调度服务器就是一个集训的单点故障，一旦调度服务器出现故障，整个集群便崩溃，所以我们需要两台调度节点，一旦一个调度节点损坏，那么另一台调度节点开始工作，这里我们使用Keepalived开源软件实现调度器的高可用。

逻辑图介绍

1.用户在访问Nginx的时候，先通过VIP访问Master1，由Master指定那个real-server提供服务

2.在Master通过lvs-Dr指定的算法选定由Nginx1来提供服务，那么数据在通过VIP发送到Nginx1上

3.Nginx处理完请求后，在通过自身的VIP将处理后的请求直接发送给User

注明：

如果Master出现故障，那么Baskup将替代Master工作

如果Nginx1出现故障，那么Nginx2将提点Nginx1工作

如果Nginx1没有出现故障，那么LVS服务使用算法会将请求平均到Nginx1和Nginx2上

如果两台Nginx同时出现故障，那么Master会提供Sorry Server服务

物理拓扑图及环境介绍

服务器：使用虚拟机虚拟4套服务器

操作系统：CentOS7.3

HA软件/LVs软件：keepalived

Web软件：Nginx

IP及主机名配置

VIP：172.18.17.30

Master：

主机名：shiyan7.one

IP地址：172.18.17.31

Backup

主机名：shiyan7.fore

IP地址：172.18.17.34

Nginx1

主机名：shiyan7.two

IP地址：172.18.17.34

Nginx2

主机名：shiyan7.thy

IP地址：172.18.17.34

4台服务器初始化配置

同步时间服务

    #Master配置NTP服务器     [root@shiyan7 ~ ]# vim /etc/ntp.conf       restrict 172.18.17.0 mask 255.255.0.0 nomodify notrap       server 127.127.1.0       fudge 127.1.1.0 stratum 8    [root@shiyan7 ~ ]# ntpdate 172.18.17.31

关闭Selinux

     [root@shiyan7 ~ ]# vim /etc/selinux/config       SELINUX=disabled

关闭防火墙

     [root@shiyan7 ~ ]# systemctl stop firewalld     [root@shiyan7 ~ ]# systemctl disable firewalld      [root@shiyan7 ~ ]# iptables -F

主机名解析配置

      [root@shiyan7 ~ ]# cat /etc/hosts            127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4            ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6            172.18.17.34 shiyan.fore            172.18.17.31 shiyan.one            172.18.17.32 shiyan7.two            172.18.17.33 shiyan7.thy

Master/Backup初始化配置

安装KeepAlived、Nginx软件

      [root@shiyan7 ~ ]# yum install keepalived nginx ipvsadm

Nginx1/2初始化配置

安装Nginx软件

      [root@shiyan7 ~ ]# yum install nginx      #这里配置一个配置VIP及更改内核参数的脚本。            [root@shiyan7 ~ ]# touch roush.sh            [root@shiyan7 ~ ]# chmod +x roush.sh            [root@shiyan7 ~ ]# vim roush.sh                #!/bin/bash                    #                                vip="172.18.17.30"                netmask="255.255.255.255"                iface="lo:0"                                case $1 in                start)                        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore                        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore                        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce                        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce                        ifconfig $iface $vip netmask $netmask broadcast $vip up                        route add -host $vip dev $iface                        ;;                stop)                        ifconfig $iface down                        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore                        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore                        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce                        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce                        ;;                esac

Ngxin1配置

     [root@shiyan7 ~ ]# ./roush.sh start    [root@shiyan7 ~ ]# mkdir /app/nginx/html    [root@shiyan7 ~ ]#  echo "
RS1:172.18.17.32
" > /app/nginx/html/index.html    [root@shiyan7 ~ ]#  cp /etc/nginx/nginx/conf{,.bak}#备份原文件    [root@shiyan7 ~ ]#  vim /etc/nginx/nginx/conf            server {            listen       80 default_server;            listen       [::]:80 default_server;            server_name  _;            root         /app/nginx/html;#修改root这里的路径，其他不需要            # Load configuration files for the default server block.            include /etc/nginx/default.d/*.conf;                location / {            }                error_page 404 /404.html;                location = /40x.html {            }                error_page 500 502 503 504 /50x.html;                            }    [root@shiyan7 ~ ]# systemctl start nginx    #测试Nginx服务    [root@yum ~ ]# curl http://172.18.17.33    
RS1:172.18.17.33

Ngxin2配置

     [root@shiyan7 ~ ]# ./roush.sh start    [root@shiyan7 ~ ]# mkdir /app/nginx/html    [root@shiyan7 ~ ]#  echo "
RS1:172.18.17.33
" > /app/nginx/html/index.html    [root@shiyan7 ~ ]#  cp /etc/nginx/nginx/conf{,.bak}#备份原文件    [root@shiyan7 ~ ]#  vim /etc/nginx/nginx/conf            server {            listen       80 default_server;            listen       [::]:80 default_server;            server_name  _;            root         /app/nginx/html;#修改root这里的路径，其他不需要            # Load configuration files for the default server block.            include /etc/nginx/default.d/*.conf;                location / {            }                error_page 404 /404.html;                location = /40x.html {            }                error_page 500 502 503 504 /50x.html;                            }    [root@shiyan7 ~ ]# systemctl start nginx    #测试Nginx服务    [root@yum ~ ]# curl http://172.18.17.34    
RS1:172.18.17.34

Master上的配置

    #配置Nginx1-Sorry Server    [root@shiyan7 ~ ]# mkdir /app/nginx/html    [root@shiyan7 ~ ]#  echo "Sorry server-1" > /app/nginx/html/index.html    [root@shiyan7 ~ ]#  cp /etc/nginx/nginx/conf{,.bak}#备份原文件    [root@shiyan7 ~ ]#  vim /etc/nginx/nginx/conf        server {            listen       80 default_server;            listen       [::]:80 default_server;            server_name  _;            root         /app/nginx/html;#修改root这里的路径，其他不需要            # Load configuration files for the default server block.            include /etc/nginx/default.d/*.conf;                location / {            }                error_page 404 /404.html;                location = /40x.html {            }                error_page 500 502 503 504 /50x.html;                            }    [root@shiyan7 ~ ]# systemctl start nginx    #测试Nginx1-Sorry Server    [root@yum ~ ]# curl http://172.18.17.31        Sorry server-1            #配置Keepalived服务    [root@shiyan7 ~ ]# cp /etc/keepalived/keepalived.conf{,.bak}    [root@shiyan7 ~ ]# vim /etc/keepalived/keepalived.conf            vrrp_instance Ha_1 {            state MASTER            interface ens33            virtual_router_id 51            priority 100            advert_int 1            authentication {                auth_type PASS                auth_pass 1111            }            virtual_ipaddress {                172.18.17.30/16 dev ens33            }        }                virtual_server 172.18.17.30 80 {            delay_loop 6            lb_algo lc            lb_kind DR            persistence_timeout 50            protocol TCP            sorry_server 172.18.17.31 80                        real_server 172.18.17.32 80 {               weight 1                HTTP_GET {                    url {                      path /                        status_code 200                    }                    connect_timeout 3                    nb_get_retry 3                    delay_before_retry 3                }        }            real_server 172.18.17.33 80 {                weight 1                HTTP_GET {                    url {                      path /                        status_code 200                    }                    connect_timeout 3                    nb_get_retry 3                    delay_before_retry 3                }            }        }    [root@shiyan7 ~ ]#systemctl start keepalived    #查看配置是否成功    [root@shiyan7 ~ ]# ip a l        2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000        link/ether 00:0c:29:1b:f5:ae brd ff:ff:ff:ff:ff:ff        inet 172.18.17.31/16 brd 172.18.255.255 scope global ens33           valid_lft forever preferred_lft forever        inet 172.18.17.30/16 scope global secondary ens33           valid_lft forever preferred_lft forever        [root@shiyan7 ~ ]# ipvsadm -ln            IP Virtual Server version 1.2.1 (size=4096)            Prot LocalAddress:Port Scheduler Flags              -> RemoteAddress:Port           Forward Weight ActiveConn InActConn            TCP  172.18.17.30:80 lc persistent 50              -> 172.18.17.32:80              Route   1      0          0                       -> 172.18.17.33:80              Route   1      0          0

Backup上的配置

    #配置Nginx1-Sorry Server    [root@shiyan7 ~ ]# mkdir /app/nginx/html    [root@shiyan7 ~ ]#  echo "Sorry server-2" > /app/nginx/html/index.html    [root@shiyan7 ~ ]#  cp /etc/nginx/nginx/conf{,.bak}#备份原文件    [root@shiyan7 ~ ]#  vim /etc/nginx/nginx/conf        server {            listen       80 default_server;            listen       [::]:80 default_server;            server_name  _;            root         /app/nginx/html;#修改root这里的路径，其他不需要            # Load configuration files for the default server block.            include /etc/nginx/default.d/*.conf;                location / {            }                error_page 404 /404.html;                location = /40x.html {            }                error_page 500 502 503 504 /50x.html;                            }    [root@shiyan7 ~ ]# systemctl start nginx    #测试Nginx1-Sorry Server    [root@yum ~ ]# curl http://172.18.17.34        Sorry server-2            #配置Keepalived服务    [root@shiyan7 ~ ]# cp /etc/keepalived/keepalived.conf{,.bak}    [root@shiyan7 ~ ]# vim /etc/keepalived/keepalived.conf            vrrp_instance Ha_1 {            state MASTER            interface ens33            virtual_router_id 51            priority 100            advert_int 1            authentication {                auth_type PASS                auth_pass 1111            }            virtual_ipaddress {                172.18.17.30/16 dev ens33            }        }                virtual_server 172.18.17.30 80 {            delay_loop 6            lb_algo lc            lb_kind DR            persistence_timeout 50            protocol TCP            sorry_server 172.18.17.32 80                        real_server 172.18.17.32 80 {               weight 1                HTTP_GET {                    url {                      path /                        status_code 200                    }                    connect_timeout 3                    nb_get_retry 3                    delay_before_retry 3                }        }            real_server 172.18.17.33 80 {                weight 1                HTTP_GET {                    url {                      path /                        status_code 200                    }                    connect_timeout 3                    nb_get_retry 3                    delay_before_retry 3                }            }        }    [root@shiyan7 ~ ]#systemctl start keepalived        [root@shiyan7 ~ ]# ip a l                2: ens33:  mtu 1500 qdisc pfifo_fast state UP qlen 1000            link/ether 00:0c:29:1b:f5:ae brd ff:ff:ff:ff:ff:ff            inet 172.18.17.34/16 brd 172.18.255.255 scope global ens33               valid_lft forever preferred_lft forever               #这里是Baskup服务所以不会有VIP地址        [root@shiyan7 ~ ]# ipvsadm -ln            IP Virtual Server version 1.2.1 (size=4096)            Prot LocalAddress:Port Scheduler Flags              -> RemoteAddress:Port           Forward Weight ActiveConn InActConn            TCP  172.18.17.30:80 lc persistent 50              -> 172.18.17.32:80              Route   1      0          0                       -> 172.18.17.33:80              Route   1      0          0

测试服务

#正常测试[root@yum ~ ]# curl http://172.18.17.30    
RS1:172.18.17.32
    [root@yum ~ ]# curl http://172.18.17.30    
RS1:172.18.17.34
    #关闭Nginx1测试[root@yum ~ ]# curl http://172.18.17.30    
RS1:172.18.17.33
    #关闭所有Nginx服务测试[root@yum ~ ]# curl http://172.18.17.30Sorry server-1#关闭Master测试[root@yum ~ ]# curl http://172.18.17.30Sorry server-2