K8s集群部署高可用架构

这篇文章主要介绍"K8s集群部署高可用架构"，在日常操作中，相信很多人在K8s集群部署高可用架构问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答"K8s集群部署高可用架构"的疑惑有所帮助！接下来，请跟着小编一起来学习吧！

环境

系统                                    角色                                  IPcentos7.4                           master-1                                 10.10.25.149centos7.4                           master-2                                 10.10.25.112centos7.4                           node-1                                   10.10.25.150centos7.4                           node-2                                   10.10.25.151centos7.4                           lb-1 backup                              10.10.25.111centos7.4                           lb-2 master                              10.10.25.110 centos7.4                           VIP                                      10.10.25.113

部署master02 节点

拷贝master01上面的 /opt/kubernetes/目录scp -r /opt/kubernetes/ root@10.10.25.112:/opt拷贝master01上的相关服务scp /usr/lib/systemd/system/{kube-apiserver,kube-scheduler,kube-controller-manager}.service root@10.10.25.112:/usr/lib/systemd/systemvim /usr/lib/systemd/system/kube-apiserver.service[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]ExecStart=/opt/kubernetes/bin/kube-apiserver \  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \  --bind-address=10.10.25.112 \  --insecure-bind-address=127.0.0.1 \  --authorization-mode=Node,RBAC \  --runtime-config=rbac.authorization.k8s.io/v1 \  --kubelet-https=true \  --anonymous-auth=false \  --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \  --enable-bootstrap-token-auth \  --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \  --service-cluster-ip-range=10.1.0.0/16 \  --service-node-port-range=20000-40000 \  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \  --client-ca-file=/opt/kubernetes/ssl/ca.pem \  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \  --etcd-servers=https://10.10.25.149:2379,https://10.10.25.150:2379,https://10.10.25.151:2379 \  --enable-swagger-ui=true \  --allow-privileged=true \  --audit-log-maxage=30 \  --audit-log-maxbackup=3 \  --audit-log-maxsize=100 \  --audit-log-path=/opt/kubernetes/log/api-audit.log \  --event-ttl=1h \  --v=2 \  --logtostderr=false \  --log-dir=/opt/kubernetes/logRestart=on-failureRestartSec=5Type=notifyLimitNOFILE=65536[Install]WantedBy=multi-user.target启动apiserversystemctl start kube-apiserver# ps -aux | grep kubesystemctl start  kube-scheduler  kube-controller-manager加入系统pathvim /root/.bash_profile 添加PATH=$PATH:$HOME/bin:/opt/kubernetes/binsource .bash_profile

查看组件状态

# /opt/kubernetes/bin/kubectl get csNAME                 STATUS    MESSAGE             ERRORscheduler            Healthy   ok                  controller-manager   Healthy   ok                  etcd-1               Healthy   {"health":"true"}   etcd-2               Healthy   {"health":"true"}   etcd-0               Healthy   {"health":"true"}   此时已经说明可以连接到ETCD集群

查看node状态

# /opt/kubernetes/bin/kubectl get nodeNAME           STATUS     ROLES     AGE       VERSION10.10.25.150   NotReady       14d       v1.10.310.10.25.151   NotReady       14d       v1.10.3说明master02 还无法与node通信

配置单节点LB负载均衡
注:做高可用集群时间上需要同步

lb02节点配置

配置nginx yum源,使用4层代理做

vim /etc/yum.repos.d/nginx.repo[nginx]name=nginx repobaseurl=https://nginx.org/packages/centos/7/$basearch/gpgcheck=0enabled=1yum install -y nginx

修改Nginx配置文件

vim /etc/nginx/nginx.conf stream {      log_format main "remote_addr $upstream_addr  $time_local $status";      access_log /var/log/nginx/k8s-access.log main;      upstream k8s-apiserver {          server 10.10.25.149:6443;          server 10.10.25.112:6443;      }      server {          listen 10.10.25.110:6443;          proxy_pass k8s-apiserver;     }}

修改node节点

cd /opt/kubernetes/cfg/vim bootstrap.kubeconfig 修改  server: https://10.10.25.149:6443 为     server: https://10.10.25.110:6443vim kubelet.kubeconfig 修改  server: https://10.10.25.149:6443 为     server: https://10.10.25.110:6443vim kube-proxy.kubeconfig 修改  server: https://10.10.25.149:6443 为     server: https://10.10.25.110:6443systemctl restart kubeletsystemctl restart kube-proxy

此时启动以后会发现master01 master02 都无法与node节点通讯,查看node日志发现,提示证书错误,大致意思是kube-proxy证书是master01节点的而不是LB节点的.所以接下来我们需要重新生成Kube-proxy证书

master01重新生成api-server证书

编辑证书json文件[root@master ssl]# cat kubernetes-csr.json {  "CN": "kubernetes",  "hosts": [    "127.0.0.1",    "10.10.25.149",    "10.10.25.112",    "10.10.25.110",    "10.10.25.111",    "10.10.25.113",    "10.1.0.1",    "kubernetes",    "kubernetes.default",    "kubernetes.default.svc",    "kubernetes.default.svc.cluster",    "kubernetes.default.svc.cluster.local"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "BeiJing",      "L": "BeiJing",      "O": "k8s",      "OU": "System"    }  ]}说明:json文件中的IP包括master01 master02节点IP,所有LB节点IP和VIP 的地址,因为我们最终需要实现 Nginx + Keepalive 0单节点的负载均衡架构生成证书cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \   -ca-key=/opt/kubernetes/ssl/ca-key.pem \   -config=/opt/kubernetes/ssl/ca-config.json \   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes拷贝到相应节点cp kubernetes*.pem /opt/kubernetes/ssl/scp kubernetes*.pem 10.10.25.112:/opt/kubernetes/ssl/scp kubernetes*.pem 10.10.25.150:/opt/kubernetes/ssl/scp kubernetes*.pem 10.10.25.151:/opt/kubernetes/ssl/

重启master节点的服务

systemctl start  kube-scheduler  kube-controller-manager kube-apiserver

重启node节点服务

systemctl restart kube-proxy kubelet

验证

# kubectl get nodeNAME           STATUS    ROLES     AGE       VERSION10.10.25.150   Ready         15d       v1.10.310.10.25.151   Ready         15d       v1.10.3说明已经实现了单节点负载均衡.这里有个地方需要注意,在以上配置都完成并且没有错误的情况下,有可能出现获取到的node状态是notready,有可能出现此问题的有原因是在日志里面发现node无法注册,此时我们需要手动注册,在master01上执行一下命令即可kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve

lb01节点配置

同样安装nginx,过程不赘述，nginx配置也相同。需要改变的只是绑定的IP

vim /etc/nginx/nginx.conf stream {      log_format main "remote_addr $upstream_addr  $time_local $status";      access_log /var/log/nginx/k8s-access.log main;      upstream k8s-apiserver {          server 10.10.25.149:6443;          server 10.10.25.112:6443;      }      server {          listen 10.10.25.111:6443;          proxy_pass k8s-apiserver;     }}

使用Keepalive实现LB节点的高可用

安装keepalive两个节点都需要

yum install keepalived -y

设置lb02为keepalived为master节点

修改lb02keepalived配置文件

vim /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {   notification_email {     acassen@firewall.loc     failover@firewall.loc     sysadmin@firewall.loc   }   notification_email_from Alexandre.Cassen@firewall.loc   smtp_server 192.168.200.1   smtp_connect_timeout 30   router_id LVS_DEVEL   vrrp_skip_check_adv_addr   #vrrp_strict   vrrp_garp_interval 0   vrrp_gna_interval 0}vrrp_script check_nginx {     script "/etc/keepalived/check_nginx.sh"  #脚本检查ngixn状态}vrrp_instance VI_1 {    state MASTER    interface ens192    virtual_router_id 51    priority 100    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.10.25.113/24    }    track_script {        check_nginx   }}

编写nginx状态检测脚本

cat /etc/keepalived/check_nginx.sh #!/bin/shcount=$(ps -ef | grep nginx | egrep -cv "grep|$$")#获取nginx进程数if [ "$count" -eq 0 ];then     systemctl stop keepalivedfi授予脚本执行权限chmod +x /etc/keepalived/check_nginx.sh

启动keepalived

systemctl start keepalived

查看是VIP是否生效

ip addr2: ens192:  mtu 1500 qdisc mq state UP qlen 1000    link/ether 00:0c:29:2e:86:82 brd ff:ff:ff:ff:ff:ff    inet 10.10.25.110/24 brd 10.10.25.255 scope global dynamic ens192       valid_lft 71256sec preferred_lft 71256sec    inet 10.10.25.113/32 scope global ens192       valid_lft forever preferred_lft forever    inet6 fe80::58b8:49be:54a7:4c43/64 scope link        valid_lft forever preferred_lft forever

配置lb01keepalived

修改为backup的keepalived配置文件

 cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs {   notification_email {     acassen@firewall.loc     failover@firewall.loc     sysadmin@firewall.loc   }   notification_email_from Alexandre.Cassen@firewall.loc   smtp_server 127.0.0.1    smtp_connect_timeout 30   router_id LVS_DEVEL   vrrp_skip_check_adv_addr   #vrrp_strict   vrrp_garp_interval 0   vrrp_gna_interval 0}vrrp_script check_nginx {     script "/etc/keepalived/check_nginx.sh"}vrrp_instance VI_1 {    state BACKUP    interface ens192    virtual_router_id 51    priority 90    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.10.25.113    }    track_script {        check_nginx      }}

编写nginx状态检测脚本

cat /etc/keepalived/check_nginx.sh #!/bin/shcount=$(ps -ef | grep nginx | egrep -cv "grep|$$")#获取nginx进程数if [ "$count" -eq 0 ];then     systemctl stop keepalivedfi授予脚本执行权限chmod +x /etc/keepalived/check_nginx.sh

启动lb01keepalived

systemctl start keepalived

Keepalive故障切换

做keepalived故障切换，测试方法1 打开一个窗口一直ping VIP2 kill master节点nginx3 观察VIP是否迁移到备份和VIP的丢包情况4 启动master节点的nginx 和keepalive5 观察VIP时候漂移回到master节点

接入K8s集群

将node节点的接入VIP

cd /opt/kubernetes/cfg/vim bootstrap.kubeconfig 修改  server: https://10.10.25.110:6443 为     server: https://10.10.25.113:6443vim kubelet.kubeconfig 修改  server: https://10.10.25.110:6443 为     server: https://10.10.25.113:6443vim kube-proxy.kubeconfig 修改  server: https://10.10.25.110:6443 为     server: https://10.10.25.113:6443systemctl restart kubeletsystemctl restart kube-proxy

重启服务

systemctl restart kubeletsystemctl restart kube-proxy

修改nginx配置文件（两个节点都需要）

cat /etc/nginx/nginx.confuser  nginx;worker_processes  2;error_log  /var/log/nginx/error.log warn;pid        /var/run/nginx.pid;events {    worker_connections  1024;}stream {      log_format main "remote_addr $upstream_addr  $time_local $status";      access_log /var/log/nginx/k8s-access.log main;      upstream k8s-apiserver {          server 10.10.25.149:6443;          server 10.10.25.112:6443;      }      server {          listen 0.0.0.0:6443;          proxy_pass k8s-apiserver;     }}http {    include       /etc/nginx/mime.types;    default_type  application/octet-stream;    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                      '$status $body_bytes_sent "$http_referer" '                      '"$http_user_agent" "$http_x_forwarded_for"';    access_log  /var/log/nginx/access.log  main;    sendfile        on;    #tcp_nopush     on;    keepalive_timeout  65;    #gzip  on;    include /etc/nginx/conf.d/*.conf;}

重启Nginx

systemctl restart nginx

验证VIP接入

kubectl get nodeNAME           STATUS    ROLES     AGE       VERSION10.10.25.150   Ready         15d       v1.10.310.10.25.151   Ready         15d       v1.10.3此时说明接入VIP成功

到此，关于"K8s集群部署高可用架构"的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注网站，小编会继续努力为大家带来更多实用的文章！