如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群
本篇文章为大家展示了如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群，内容简明扼要并且容易理解，绝对能使你眼前一亮，通过这篇文章的详细介绍希望你能有所收获。
服务器规划：
至少三台服务器组建高可用集群，配置2C4G以上
master，node节点均已安装docker，版本18以上
VIP 172.30.2.60
172.30.0.109 k8s-master1 nginx keepalived
172.30.0.89 k8s-master2 nginx keepalived
172.30.0.81 k8s-node1
二进制Kubernetes安装路径：/opt/kubernetes/{ssl,cfg,bin,logs}分别存放密钥，配置，可执行，日志文件
二进制etcd安装路径：/opt/etcd/{ssl,cfg,bin}分别存放密钥，配置，可执行文件
1、系统初始化
1.关闭防火墙：
# systemctl stop firewalld
# systemctl disable firewalld
2.关闭selinux：
# setenforce 0 # 临时
# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
3.关闭swap：
# swapoff -a # 临时
# vim /etc/fstab # 永久
4.同步系统时间：
# ntpdate time.windows.com
5.添加hosts：
# vim /etc/hosts
172.30.0.109 k8s-master1
172.30.0.81 k8s-master2
172.30.0.89 k8s-node1
6.修改主机名：
hostnamectl set-hostname k8s-master1
2、Etcd集群
1.安装cfssl工具
# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
2.生成etcd证书
①生成server.csr.json 修改请求文件中hosts字段包含所有etcd节点IP：
# vi server-csr.json
{
"CN": "etcd",
"hosts": [
"etcd01节点IP",
"etcd02节点IP",
"etcd03节点IP"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
②生成自定义CA，初始化配置文件
# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
# vim ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
etcd密钥初始化文件准备好后，即可生成密钥
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
生成的密钥放置在指定的etcd/ssl目录下
etcd二进制可执行文件etcd，etcdctl放置在etcd/bin下
etcd配置文件放置在etcd/cfg下，配置文件如下
# cat /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.30.0.81:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.30.0.81:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.30.0.81:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.30.0.81:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.30.2.10:2380,etcd02=https://172.30.0.81:2380,etcd03=https://172.30.0.89:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
将etcd.service放置在/usr/lib/systemd/system/下
# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
分别拷贝到Etcd三个节点：
# scp -r /opt/etcd root@etcd节点IP:/opt
# scp /usr/lib/systemd/system/etcd.service root@etcd节点IP:/usr/lib/systemd/system
同步修改etcd节点配置文件的配置名称，使其与主机相对应，如
启动etcd节点，etcd安装完成
# systemctl start etcd && systemctl enable etcd
检查etcd节点健康状态
# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem -- endpoints="https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379" cluster-health
member 37f20611ff3d9209 is healthy: got healthy result from https://172.30.2.10:2379
member b10f0bac3883a232 is healthy: got healthy result from https://172.30.0.81:2379
member b46624837acedac9 is healthy: got healthy result from https://172.30.0.89:2379
cluster is healthy
3、部署K8S集群--master节点
①重新生成自定义CA，与etcd进行区分，生成apiserver，kube-proxy，kubectl-admin证书
# vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
# vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
②生成kube-proxy密钥初始化文件
cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
③生成apiserver密钥初始化文件，注意需要写入所有master节点IP，以及需要访问apiserver的IP地址，包括nginx，nginx vip，否则需要重新生成密钥
# vim server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"172.30.2.60",
"172.30.0.109",
"172.30.0.81",
"172.30.2.10",
"172.30.0.89"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
④admin密钥，供kubectl远程客户端运行
# vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
⑤生成密钥
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
同步*.pem密钥至所有节点的/opt/kubernetes/ssl（也可以针对性同步密钥），kubelet密钥是后面部署kubelet自动生成的
生成admin的kubeconfig文件，用于客户端访问k8s集群
# kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://172.30.0.109:6443 --kubeconfig=/root/.kube/kubectl.kubeconfig
# kubectl config set-credentials kube-admin --client-certificate=/opt/kubernetes/ssl/admin.pem --client-key=/opt/kubernetes/ssl/admin-key.pem --embed-certs=true --kubeconfig=/root/.kube/kubectl.kubeconfig
# kubectl config set-context kube-admin@kubernetes --cluster=kubernetes --user=kube-admin --kubeconfig=/root/.kube/kubectl.kubeconfig
# kubectl config use-context kube-admin@kubernetes --kubeconfig=/root/.kube/kubectl.kubeconfig
# mv /root/.kube/{kubectl.kubeconfig,config}
⑥下载master二进制包放置在/opt/kubernetes/bin下
⑦配置配置文件，service启动文件
kube-apiserve 注意：部署另一台master时，需要更换IP地址
# vim /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://172.30.2.10:2379,https://172.30.0.81:2379,https://172.30.0.89:2379 \
--bind-address=172.30.0.109 \
--secure-port=6443 \
--advertise-address=172.30.0.109 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
kube-controller-manager
# vim /opt/kubernetes/cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
kube-scheduler
# vim /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--address=127.0.0.1"
# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
生成token文件，供node节点与apiserver进行通信，注意token值，master需要与node保持一致
# cat /opt/kubernetes/cfg/token.csv
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
格式：token,用户,uid,用户组
token也可自行生成替换：
# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
授权kubelet-bootstrap，使node节点kubelet能够正常访问到apiserver
# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
启动master节点，观察日志/opt/kubernetes/logs
# systemctl start kube-controller-manager
# systemctl start kube-scheduler
# systemctl enable kube-apiserver
# systemctl enable kube-controller-manager
# systemctl enable kube-scheduler
原样部署第二个master节点172.30.0.81，不同之处仅在于apiserver配置文件当中的IP地址
4、部署K8S集群--node节点
①node节点密钥准备
已提前同步至node节点的/opt/kubernetes/ssl下
②配置node节点kubelet，kube-proxy配置文件
kubelet 注意：不同节点的kubelet需要修改hostname配置
# vim kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-node1 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
bootstrap.kubeconfig 这个文件是kubelet与apiserver的通信认证文件，内部token需要与master节点的token文件保持一致
#vim bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /opt/kubernetes/ssl/ca.pem
server: https://172.30.0.109:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet-bootstrap
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: c47ffb939f5ca36231d9e3121a252940
# vim kubelet-config.yml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Before=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
kube-proxy
# vim kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
# vim kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-node1
clusterCIDR: 10.0.0.0/24
mode: ipvs
ipvs:
scheduler: "rr"
iptables:
masqueradeAll: true
kube-proxy.kubeconfig kube-proxy通信认证文件
# vim kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /opt/kubernetes/ssl/ca.pem
server: https://172.30.0.109:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
user:
client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
client-key: /opt/kubernetes/ssl/kube-proxy-key.pem
# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
/opt/kubernetes/cfg配置如下kubelet.kubeconfig是启动后自动生成的
/opt/kubernetes/bin
③启动node节点
# systemctl start kubelet
# systemctl start kube-proxy
# systemctl enable kubelet
# systemctl enable kube-proxy
④允许给node颁发证书
# kubectl get csr
# kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI
# kubectl get node
5、部署CNI网络
二进制包下载地址：https://github.com/containernetworking/plugins/releases
# mkdir /opt/cni/bin /etc/cni/net.d
# tar zxvf cni-plugins-linux-amd64-v0.8.2.tgz -C /opt/cni/bin
确保kubelet启用CNI：
# cat /opt/kubernetes/cfg/kubelet.conf
--network-plugin=cni
在Master执行：
# kubectl apply -f kube-flannel.yaml
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-amd64-5xmhh 1/1 Running 6 171m
kube-flannel-ds-amd64-ps5fx 1/1 Running 0 150m
6、授权apiserver访问kubelet
为提供安全性，kubelet禁止匿名访问，必须授权才可以。
# vim apiserver-to-kubelet-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
# kubectl apply -f apiserver-to-kubelet-rbac.yaml
这样就能通过kubectl操作pod，查看日志
7、部署coredns
# kubectl apply -f coredns.yaml
8、K8S高可用配置
kube-controller-manager以及kube-scheduler高可用已经在部署集群的时候在配置中体现
选举进行负载均衡，所以我们只需要关注apiserver高可用即可
kube-apiserver高可用
①首先配置好两台master节点
②在两台master节点上各自部署nginx，keepalived
keepalived监听nginx健康状态
nginx配置为四层监听两台master的6443端口，四层转发
nginx配置如下
# rpm -vih http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.0-1.el7.ngx.x86_64.rpm
# vim /etc/nginx/nginx.conf
……
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 172.30.0.109:6443;
server 172.30.0.81:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
……
启动nginx
# systemctl start nginx
# systemctl enable nginx
keepliaved vip配置为172.30.2.60
③配置node节点访问master apiserver经由172.30.2.60进行四层转发
修改node节点配置，将单点的master的节点，更改为VIP 172.30.2.60即可
批量修改：
# sed -i 's#172.30.0.109#172.30.2.60#g' *
上述内容就是如何在CentOS7.3下二进制安装Kubernetes 1.16.0高可用集群，你们学到知识或技能了吗？如果还想学到更多技能或者丰富自己的知识储备，欢迎关注行业资讯频道。
