Kubernetes集群部署记录
本文记录在五台Ubuntu 16.04上，搭建Kubernetes 1.12高可用集群 + IPVS集群网络的完整步骤。
准备工作
Ansible配置
|
1 2 3 4 5 6 7 8 9 10 |
[k8s] boron.gmem.cc carbon.gmem.cc radon.gmem.cc neon.gmem.cc xenon.gmem.cc [k8s-no-master] boron.gmem.cc carbon.gmem.cc |
根证书
采用自签名证书：
|
1 2 3 |
openssl genrsa -out ca.key 2048 openssl req -x509 -newkey rsa:2048 -keyout ca.key -out ca.crt -days 3650 \ -subj "/C=CN/ST=BeiJing/L=BeiJing/O=Gmem Studio/OU=IT Support/CN=Gmem SHA256 CA" |
拷贝到Ansible主节点：
|
1 |
cp ca.crt /usr/share/ca-certificates/GmemCA.crt |
分发到所有节点：
|
1 2 3 4 5 6 7 8 9 10 11 |
--- - hosts: k8s tasks: - name: sync GmemCA.crt copy: src: /usr/share/ca-certificates/GmemCA.crt dest: /usr/share/ca-certificates/GmemCA.crt backup: no - name: reconfigure CA shell: | update-ca-certificates |
调整内核参数
|
1 2 3 4 5 6 7 8 |
echo "8192" > /sys/block/sda/queue/read_ahead_kb echo "8192" > /sys/block/sdb/queue/read_ahead_kb echo "noop" > /sys/block/sda/queue/scheduler echo "cfq" > /sys/block/sdb/queue/scheduler echo "vm.swappiness = 0" | tee -a /etc/sysctl.conf echo "kernel.pid_max = 4194303" | tee -a /etc/sysctl.conf |
加载内核模块
|
1 2 3 4 5 |
ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 |
安装Etcd集群
下载软件
下载地址：https://github.com/etcd-io/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz
解压到/opt/etcd目录。
准备证书
本集群采用完全的TLS连接，需要为每个节点准备数字证书。
节点Xenon：
|
1 2 3 4 5 6 7 8 |
openssl genrsa -out xenon.key 2048 export SAN_CFG=$(printf "\n[SAN]\nsubjectAltName=IP:127.0.0.1,IP:10.0.5.1,DNS:xenon.gmem.cc") openssl req -new -sha256 -key xenon.key -out xenon.csr \ -subj "/C=CN/ST=BeiJing/O=Gmem Studio/CN=Server Xenon" \ -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(echo $SAN_CFG)) openssl x509 -req -sha256 -in xenon.csr -out xenon.crt -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -days 3650 \ -extensions SAN -extfile <(echo "${SAN_CFG}") |
节点Radon：
|
1 2 3 4 5 6 7 8 |
openssl genrsa -out radon.key 2048 export SAN_CFG=$(printf "\n[SAN]\nsubjectAltName=IP:127.0.0.1,IP:10.0.2.1,DNS:radon.gmem.cc") openssl req -new -sha256 -key radon.key -out radon.csr \ -subj "/C=CN/ST=BeiJing/O=Gmem Studio/CN=Server Radon" \ -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(echo $SAN_CFG)) openssl x509 -req -sha256 -in radon.csr -out radon.crt -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -days 3650 \ -extensions SAN -extfile <(echo "${SAN_CFG}") |
节点Neon：
|
1 2 3 4 5 6 7 8 |
openssl genrsa -out neon.key 2048 export SAN_CFG=$(printf "\n[SAN]\nsubjectAltName=IP:127.0.0.1,IP:10.0.3.1,DNS:neon.gmem.cc") openssl req -new -sha256 -key neon.key -out neon.csr \ -subj "/C=CN/ST=BeiJing/O=Gmem Studio/CN=Server Neon" \ -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(echo $SAN_CFG)) openssl x509 -req -sha256 -in neon.csr -out neon.crt -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -days 3650 \ -extensions SAN -extfile <(echo "${SAN_CFG}") |
私钥、证书拷贝到对应节点的/opt/etcd/cert目录下。
启动脚本
节点Xenon：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
#!/bin/bash /opt/etcd/etcd --name xenon --initial-advertise-peer-urls https://10.0.5.1:2380 \ --listen-peer-urls https://10.0.5.1:2380 \ --listen-client-urls https://10.0.5.1:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://10.0.5.1:2380 \ --initial-cluster-token etcd.gmem.cc \ --initial-cluster xenon=https://10.0.5.1:2380,radon=https://10.0.2.1:2380,neon=https://10.0.3.1:2380 \ --initial-cluster-state new \ --client-cert-auth \ --trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --cert-file=/opt/etcd/cert/xenon.crt \ --key-file=/opt/etcd/cert/xenon.key \ --peer-client-cert-auth \ --peer-trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --peer-cert-file=/opt/etcd/cert/xenon.crt \ --peer-key-file=/opt/etcd/cert/xenon.key \ --data-dir=/opt/etcd/data |
节点Radon：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
#!/bin/bash /opt/etcd/etcd --name radon --initial-advertise-peer-urls https://10.0.2.1:2380 \ --listen-peer-urls https://10.0.2.1:2380 \ --listen-client-urls https://10.0.2.1:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://10.0.2.1:2380 \ --initial-cluster-token etcd.gmem.cc \ --initial-cluster xenon=https://10.0.5.1:2380,radon=https://10.0.2.1:2380,neon=https://10.0.3.1:2380 \ --initial-cluster-state new \ --client-cert-auth \ --trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --cert-file=/opt/etcd/cert/radon.crt \ --key-file=/opt/etcd/cert/radon.key \ --peer-client-cert-auth \ --peer-trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --peer-cert-file=/opt/etcd/cert/radon.crt \ --peer-key-file=/opt/etcd/cert/radon.key \ --data-dir=/opt/etcd/data |
节点Neon：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
#!/bin/bash /opt/etcd/etcd --name neon --initial-advertise-peer-urls https://10.0.3.1:2380 \ --listen-peer-urls https://10.0.3.1:2380 \ --listen-client-urls https://10.0.3.1:2379,https://127.0.0.1:2379 \ --advertise-client-urls https://10.0.3.1:2380 \ --initial-cluster-token etcd.gmem.cc \ --initial-cluster xenon=https://10.0.5.1:2380,radon=https://10.0.2.1:2380,neon=https://10.0.3.1:2380 \ --initial-cluster-state new \ --client-cert-auth \ --trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --cert-file=/opt/etcd/cert/neon.crt \ --key-file=/opt/etcd/cert/neon.key \ --peer-client-cert-auth \ --peer-trusted-ca-file=/usr/share/ca-certificates/GmemCA.crt \ --peer-cert-file=/opt/etcd/cert/neon.crt \ --peer-key-file=/opt/etcd/cert/neon.key \ --data-dir=/opt/etcd/data |
开机启动：
|
1 2 |
# Startup Etcd nohup /opt/etcd/startup.sh > /var/log/etcd.log 2>&1 & |
安装Docker
安装
|
1 |
apt install -y docker.io |
配置
|
1 2 3 4 |
http_proxy="http://10.0.0.1:8088" https_proxy="http://10.0.0.1:8088" DOCKER_OPTS="--registry-mirror https://prpr233y.mirror.aliyuncs.com" |
安装K8S软件
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
apt install -y apt-transport-https # 官方源 export http_proxy="http://10.0.0.1:8088" export https_proxy="http://10.0.0.1:8088" curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list # 如果无法访问官方源，可以使用阿里云的 curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF apt update apt install -y kubelet kubeadm kubectl ipvsadm # apt install -y kubelet=1.12.1-00 kubeadm=1.12.1-00 kubectl=1.12.1-00 kubernetes-cni=0.6.0-00 ipvsadm |
配置APIServer负载均衡器
Keepalived配置
采用DR模式进行转发。使用两个LVS Director节点Oxygen、Hydrogen。
LB主节点
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
global_defs { router_id oxygen } vrrp_instance apiserver { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass gmem } virtual_ipaddress { 10.0.10.1/32 brd 10.0.10.1 dev eth0 label eth0:k8s } } virtual_server 10.0.10.1 6443 { delay_loop 5 lb_algo wlc lb_kind DR protocol TCP real_server 10.0.5.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 10.0.2.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 10.0.3.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } |
LB从节点
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
global_defs { router_id hydrogen } vrrp_instance apiserver { state BACKUP interface eth0 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass gmem } virtual_ipaddress { 10.0.10.1/32 brd 10.0.10.1 dev eth0 label eth0:k8s } } virtual_server 10.0.10.1 6443 { delay_loop 5 lb_algo wlc lb_kind DR protocol TCP real_server 10.0.5.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 10.0.2.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 10.0.3.1 6443 { weight 10 SSL_GET { url { path /healthz status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } |
K8S主节点配置
|
1 2 3 4 5 |
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce ifconfig lo:k8s 10.0.10.1 netmask 255.255.255.255 broadcast 10.0.10.1 |
初始化K8S主节点
清空Etcd
|
1 |
etcdctl del --prefix=true "" |
准备Etcd客户端证书
|
1 2 3 4 5 6 7 |
mkdir apiserver-etcd-client cd apiserver-etcd-client/ openssl genrsa -out apiserver-etcd-client.key 2048 openssl req -new -sha256 -key apiserver-etcd-client.key -out apiserver-etcd-client.csr -subj "/C=CN/ST=BeiJing/L=BeiJing/O=Gmem Studio/OU=IT Support/CN=Kubernetes API Server" openssl x509 -req -days 3650 -in apiserver-etcd-client.csr -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -out apiserver-etcd-client.crt |
将上述证书拷贝到第一个主节点上：
|
1 2 3 |
cd /etc/kubernetes/ mkdir pki scp -r alex@zircon.gmem.cc:/home/alex/Documents/puTTY/apiserver-etcd-client/* pki |
kubeadm配置
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
apiVersion: kubeadm.k8s.io/v1alpha3 kind: ClusterConfiguration kubernetesVersion: stable clusterName: gmem imageRepository: docker.gmem.cc/k8s apiServerCertSANs: - "k8s.gmem.cc" - "10.0.10.1" - "127.0.0.1" - "10.0.5.1" - "10.0.2.1" - "10.0.3.1" api: controlPlaneEndpoint: "k8s.gmem.cc:6443" etcd: external: endpoints: - https://10.0.5.1:2379 - https://10.0.2.1:2379 - https://10.0.3.1:2379 caFile: /usr/share/ca-certificates/GmemCA.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key networking: dnsDomain: k8s.gmem.cc podSubnet: "172.27.0.0/16" serviceSubnet: 10.96.0.0/12 controllerManagerExtraArgs: node-monitor-grace-period: 10s pod-eviction-timeout: 10s kubeProxy: config: mode: ipvs ipvs: minSyncPeriod: 0s scheduler: "lc" syncPeriod: 30s |
镜像预拉取
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# 列出控制平面镜像 kubeadm config images list # 手工拉取 docker pull k8s.gcr.io/kube-apiserver:v1.12.1 docker pull k8s.gcr.io/kube-controller-manager:v1.12.1 docker pull k8s.gcr.io/kube-scheduler:v1.12.1 docker pull k8s.gcr.io/kube-proxy:v1.12.1 docker pull k8s.gcr.io/pause:3.1 docker pull k8s.gcr.io/etcd:3.2.24 docker pull k8s.gcr.io/coredns:1.2.2 # 重新打标签 docker tag k8s.gcr.io/kube-apiserver:v1.12.1 docker.gmem.cc/k8s/kube-apiserver:v1.12.1 docker tag k8s.gcr.io/kube-controller-manager:v1.12.1 docker.gmem.cc/k8s/kube-controller-manager:v1.12.1 docker tag k8s.gcr.io/kube-scheduler:v1.12.1 docker.gmem.cc/k8s/kube-scheduler:v1.12.1 docker tag k8s.gcr.io/kube-proxy:v1.12.1 docker.gmem.cc/k8s/kube-proxy:v1.12.1 docker tag k8s.gcr.io/pause:3.1 docker.gmem.cc/k8s/pause:3.1 docker tag k8s.gcr.io/etcd:3.2.24 docker.gmem.cc/k8s/etcd:3.2.24 docker tag k8s.gcr.io/coredns:1.2.2 docker.gmem.cc/k8s/coredns:1.2.2 # 推送到私服 docker push docker.gmem.cc/k8s/kube-apiserver:v1.12.1 docker push docker.gmem.cc/k8s/kube-controller-manager:v1.12.1 docker push docker.gmem.cc/k8s/kube-scheduler:v1.12.1 docker push docker.gmem.cc/k8s/kube-proxy:v1.12.1 docker push docker.gmem.cc/k8s/pause:3.1 docker push docker.gmem.cc/k8s/etcd:3.2.24 docker push docker.gmem.cc/k8s/coredns:1.2.2 # 在所有节点执行 docker pull docker.gmem.cc/k8s/pause:3.1 docker tag docker.gmem.cc/k8s/pause:3.1 k8s.gcr.io/pause:3.1 |
初始化第一主节点
临时禁用虚IP
|
1 |
ifconfig lo:k8s down |
初始化
|
1 2 3 4 |
kubeadm init --config /etc/kubernetes/kubeadm-config.yaml # 完成后提示： # kubeadm join 10.0.5.1:6443 --token itpbsl.d7rhkt4hskkbz8oh --discovery-token-ca-cert-hash sha256:46fe290b71f353272cd7b32250ff0253d34cc3f317810325fd5caeb3546ca6e5 |
修改配置kubeadm-config
kubeadm目前可能存在一个缺陷，生成的kubeadm-config的controlPlaneEndpoint为空，需要手工修改一下：
|
1 2 3 4 |
# kubectl -n kube-system edit cm kubeadm-config # 修改控制平面端点 controlPlaneEndpoint: "k8s.gmem.cc:6443" |
修改配置kube-proxy
kubeadm目前可能存在一个缺陷，设置ipvs模式没有效果，需要手工修改：
|
1 2 3 4 5 6 7 8 |
# kubectl -n kube-system edit cm kube-proxy # 修改负载均衡算法 scheduler: "lc" # 修改代理模式 mode: "ipvs" |
完毕后删除Pod：
|
1 |
kubectl -n kube-system delete pod -l k8s-app=kube-proxy |
Pod重新创建后，应该可以在主节点上看到虚拟服务：
|
1 2 3 4 5 6 7 8 |
ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.0.1:443 lc -> 10.0.5.1:6443 Masq 1 0 0 TCP 10.96.0.10:53 lc UDP 10.96.0.10:53 lc |
修改apiserver配置
编辑文件/etc/kubernetes/manifests/kube-apiserver.yaml，添加参数--service-node-port-range=80-32767，放开NodePort的范围。
修改coredns配置
执行 kubectl -n kube-system edit configmap coredns，设置上游DNS服务器：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
apiVersion: v1 data: Corefile: | .:53 { errors health hosts { 10.0.10.1 k8s.gmem.cc 10.0.10.4 dashboard.k8s.gmem.cc 10.0.10.4 media-api.k8s.gmem.cc 10.0.10.4 chartmuseum.k8s.gmem.cc 10.0.10.4 monocular.k8s.gmem.cc 10.0.10.4 grafana.k8s.gmem.cc 10.0.10.4 alertmanager.k8s.gmem.cc 10.0.10.4 pushgateway.k8s.gmem.cc 10.0.10.4 prometheus.k8s.gmem.cc 10.0.10.4 kibana.k8s.gmem.cc 10.0.10.4 chartmuseum.k8s.gmem.cc 10.0.10.4 jenkins.k8s.gmem.cc 10.0.10.4 nginx.k8s.gmem.cc 10.0.10.4 ceph-mon.ceph.svc.k8s.gmem.cc 10.0.10.4 api.k8s.gmem.cc 10.0.10.4 rabbitmq.k8s.gmem.cc 10.0.10.4 elastichq.k8s.gmem.cc 10.0.10.4 dbadm.k8s.gmem.cc # 如果当前插件(hosts)无法解析域名，将请求传递给下一个插件 fallthrough } # 负责解析K8S集群内域名 kubernetes k8s.gmem.cc # 上游DNS服务器 proxy . 223.5.5.5:53 1.1.1.1:53 { except gmem.cc } proxy gmem.cc 10.0.0.1:53 { except svc.k8s.gmem.cc } prometheus :9153 cache 30 loop reload loadbalance } kind: ConfigMap |
注意：修改configmap后不需要重启coredns实例，很快就会自动生效。
恢复虚IP
必须在控制平面的Pod正常运行后，再恢复虚IP：
|
1 |
ifconfig lo:k8s 10.0.10.1 netmask 255.255.255.255 broadcast 10.0.10.1 |
复制kubeconfig
复制到当前主节点：
|
1 2 3 |
mkdir -p $HOME/.kube cp /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config |
复制到其它管理客户机：
|
1 |
scp -i ~/Documents/puTTY/gmem.key root@xenon.gmem.cc:/etc/kubernetes/admin.conf ~/.kube/config |
注意：手工提供根CA（而不是由K8S自动生成）时，重新创建集群后，不需要复制kubeconfig，原先的仍然有效。
初始化其它主节点
复制配置
从第一主节点复制一些配置文件：
|
1 2 |
cd /etc/kubernetes/ scp -r root@xenon.gmem.cc:/etc/kubernetes/pki/* pki |
临时禁用虚IP
|
1 |
ifconfig lo:k8s down |
加入集群
以--experimental-control-plane参数将节点加入集群：
|
1 2 |
# --experimental-control-plane表示在此节点上创建控制平面 kubeadm join k8s.gmem.cc:6443 --token itpbsl.d7rhkt4hskkbz8oh --discovery-token-ca-cert-hash sha256:46fe290b71f353272cd7b32250ff0253d34cc3f317810325fd5caeb3546ca6e5 --experimental-control-plane |
恢复虚IP
必须在控制平面的Pod正常运行后，再恢复虚IP：
|
1 |
ifconfig lo:k8s 10.0.10.1 netmask 255.255.255.255 broadcast 10.0.10.1 |
添加K8S从节点
|
1 |
kubeadm join k8s.gmem.cc:6443 --token itpbsl.d7rhkt4hskkbz8oh --discovery-token-ca-cert-hash sha256:46fe290b71f353272cd7b32250ff0253d34cc3f317810325fd5caeb3546ca6e5 |
安装CNI
Calico
镜像预拉取
|
1 2 3 4 5 6 7 8 9 10 11 |
docker pull quay.io/calico/node:v3.2.3 docker tag quay.io/calico/node:v3.2.3 docker.gmem.cc/calico/node:v3.2.3 docker push docker.gmem.cc/calico/node:v3.2.3 docker pull quay.io/calico/cni:v3.2.3 docker tag quay.io/calico/cni:v3.2.3 docker.gmem.cc/calico/cni:v3.2.3 docker push docker.gmem.cc/calico/cni:v3.2.3 docker pull quay.io/calico/kube-controllers:v3.2.3 docker tag quay.io/calico/kube-controllers:v3.2.3 docker.gmem.cc/calico/kube-controllers:v3.2.3 docker push docker.gmem.cc/calico/kube-controllers:v3.2.3 |
etcd客户端证书
|
1 2 3 4 5 |
mkdir calico-etcd-client cd calico-etcd-client openssl genrsa -out calico-etcd-client.key 2048 openssl req -new -sha256 -key calico-etcd-client.key -out calico-etcd-client.csr -subj "/C=CN/ST=BeiJing/L=BeiJing/O=Gmem Studio/OU=IT Support/CN=Calico" openssl x509 -req -days 3650 -in calico-etcd-client.csr -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -out calico-etcd-client.crt |
创建对应的Secrets：
|
1 2 3 4 5 |
cp /usr/share/ca-certificates/GmemCA.crt etcd-ca cp ../calico-etcd-client/calico-etcd-client.key etcd-key cp ../calico-etcd-client/calico-etcd-client.crt etcd-cert kubectl -n kube-system create secret generic calico-etcd-secrets --from-file=. |
RBAC资源
|
1 2 3 |
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/rbac.yaml kubectl -n kube-system patch serviceaccount calico-node -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
其它资源
|
1 |
wget https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/calico.yaml |
参考下面的例子修改：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 |
kind: ConfigMap apiVersion: v1 metadata: name: calico-config namespace: kube-system data: # Etcd端点和访问凭证 etcd_endpoints: "https://10.0.5.1:2379,https://10.0.2.1:2379,https://10.0.3.1:2379" etcd_ca: "/calico-secrets/etcd-ca" etcd_cert: "/calico-secrets/etcd-cert" etcd_key: "/calico-secrets/etcd-key" calico_backend: "bird" # MTU，可以根据实际情况调整 veth_mtu: "1440" cni_network_config: |- { "name": "k8s-pod-network", "cniVersion": "0.3.0", "plugins": [ { "type": "calico", "log_level": "info", "etcd_endpoints": "__ETCD_ENDPOINTS__", "etcd_key_file": "__ETCD_KEY_FILE__", "etcd_cert_file": "__ETCD_CERT_FILE__", "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__", "mtu": __CNI_MTU__, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "__KUBECONFIG_FILEPATH__" }kubectl -n kube-system patch serviceaccount coredns -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} } ] } --- kind: DaemonSet apiVersion: extensions/v1beta1 metadata: name: calico-node namespace: kube-system labels: tier: infrastructure app: calico comp: node spec: selector: matchLabels: tier: infrastructure app: calico comp: node updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 template: metadata: labels: tier: infrastructure app: calico comp: node annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: nodeSelector: beta.kubernetes.io/os: linux hostNetwork: true tolerations: - effect: NoSchedule operator: Exists - key: CriticalAddonsOnly operator: Exists - effect: NoExecute operator: Exists serviceAccountName: calico-node terminationGracePeriodSeconds: 0 containers: - name: calico-node image: docker.gmem.cc/calico/node:v3.2.3 env: - name: ETCD_ENDPOINTS valueFrom: configMapKeyRef: name: calico-config key: etcd_endpoints - name: ETCD_CA_CERT_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_ca - name: ETCD_KEY_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_key - name: ETCD_CERT_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_cert - name: CALICO_K8S_NODE_REF valueFrom: fieldRef: fieldPath: spec.nodeName - name: CALICO_NETWORKING_BACKEND valueFrom: configMapKeyRef: name: calico-config key: calico_backend - name: CLUSTER_TYPE value: "k8s,bgp" - name: IP value: "autodetect" # 自动检测节点IP地址的方法，如果具有多网卡注意合理配置 - name: IP_AUTODETECTION_METHOD value: interface=eth.* - name: CALICO_IPV4POOL_IPIP value: "CrossSubnet" - name: FELIX_IPINIPMTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu # Pod网络CIDR - name: CALICO_IPV4POOL_CIDR value: "172.27.0.0/16" - name: CALICO_DISABLE_FILE_LOGGING value: "true" - name: FELIX_DEFAULTENDPOINTTOHOSTACTION value: "ACCEPT" - name: FELIX_IPV6SUPPORT value: "false" - name: FELIX_LOGSEVERITYSCREEN value: "info" - name: FELIX_HEALTHENABLED value: "true" securityContext: privileged: true resources: requests: cpu: 250m livenessProbe: httpGet: path: /liveness port: 9099 host: localhost periodSeconds: 10 initialDelaySeconds: 10 failureThreshold: 6 readinessProbe: exec: command: - /bin/calico-node - -bird-ready - -felix-ready periodSeconds: 10 volumeMounts: - mountPath: /lib/modules name: lib-modules readOnly: true - mountPath: /var/run/calico name: var-run-calico readOnly: false - mountPath: /var/lib/calico name: var-lib-calico readOnly: false - mountPath: /calico-secrets name: etcd-certs - name: install-cni image: docker.gmem.cc/calico/cni:v3.2.3 command: ["/install-cni.sh"] env: - name: CNI_CONF_NAME value: "10-calico.conflist" - name: ETCD_ENDPOINTS valueFrom: configMapKeyRef: name: calico-config key: etcd_endpoints - name: CNI_NETWORK_CONFIG valueFrom: configMapKeyRef: name: calico-config key: cni_network_config - name: CNI_MTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu volumeMounts: - mountPath: /host/opt/cni/bin name: cni-bin-dir - mountPath: /host/etc/cni/net.d name: cni-net-dir - mountPath: /calico-secrets name: etcd-certs volumes: - name: lib-modules hostPath: path: /lib/modules - name: var-run-calico hostPath: path: /var/run/calico - name: var-lib-calico hostPath: path: /var/lib/calico - name: cni-bin-dir hostPath: path: /opt/cni/bin - name: cni-net-dir hostPath: path: /etc/cni/net.d - name: etcd-certs secret: secretName: calico-etcd-secrets defaultMode: 0400 --- apiVersion: v1 kind: ServiceAccount metadata: name: calico-node namespace: kube-system --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: calico-kube-controllers namespace: kube-system labels: tier: infrastructure app: calico comp: kube-controllers annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: replicas: 1 strategy: type: Recreate template: metadata: name: calico-kube-controllers namespace: kube-system labels: tier: infrastructure app: calico comp: kube-controllers spec: nodeSelector: beta.kubernetes.io/os: linux hostNetwork: true tolerations: - key: CriticalAddonsOnly operator: Exists - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: calico-kube-controllers containers: - name: calico-kube-controllers image: docker.gmem.cc/calico/kube-controllers:v3.2.3 env: - name: ETCD_ENDPOINTS valueFrom: configMapKeyRef: name: calico-config key: etcd_endpoints - name: ETCD_CA_CERT_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_ca - name: ETCD_KEY_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_key - name: ETCD_CERT_FILE valueFrom: configMapKeyRef: name: calico-config key: etcd_cert - name: ENABLED_CONTROLLERS value: policy,profile,workloadendpoint,node volumeMounts: - mountPath: /calico-secrets name: etcd-certs readinessProbe: exec: command: - /usr/bin/check-status - -r volumes: - name: etcd-certs secret: secretName: calico-etcd-secrets defaultMode: 0400 --- apiVersion: v1 kind: ServiceAccount metadata: name: calico-kube-controllers namespace: kube-system |
创建资源：
|
1 |
kubectl apply -f calico.yaml |
配置calicoctl
安装calicoctl：
|
1 2 3 |
cd /usr/local/bin curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.2.3/calicoctl chmod +x calicoctl |
拷贝证书：
|
1 2 3 |
mkdir /etc/calico cd /etc/calico scp -r alex@zircon.gmem.cc:/home/alex/Documents/puTTY/calico-etcd-client/* . |
修改配置文件：
|
1 2 3 4 5 6 7 8 9 |
apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: datastoreType: "etcdv3" etcdEndpoints: "https://10.0.5.1:2379,https://10.0.2.1:2379,https://10.0.3.1:2379" etcdKeyFile: /etc/calico/calico-etcd-client.key etcdCertFile: /etc/calico/calico-etcd-client.crt etcdCACertFile: /usr/share/ca-certificates/GmemCA.crt |
配置Secrets
imagepullsecrets
镜像仓库docker.gmem.cc需要身份验证，因此所有命名空间都需要创建对应的Secret，并给所有ServiceAccount打补丁：
|
1 2 3 4 5 6 7 8 9 |
NS=`kubectl get namespace --no-headers | awk '{print $1}'` # 创建用于拉取镜像的Secret for ns in $NS ; do kubectl -n $ns create secret docker-registry gmemregsecret --docker-server=docker.gmem.cc --docker-username=alex --docker-password=*** --docker-email=k8s@gmem.cc ; done # 给命名空间默认SA打补丁 for ns in $NS ; do kubectl -n $ns patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' ; done # 其它需要打补丁的控制平面SA kubectl -n kube-system patch serviceaccount coredns -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' kubectl -n kube-system patch serviceaccount kube-proxy -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
如果你创建了新的ServiceAccount，可能需要手工patch它。
SSL数字证书
本次部署的Ingress全部启用HTTPS，使用Lets Encrypt提供的免费数字证书：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
kubectl delete -n kube-system secret gmemk8scert-dashboard cd /etc/letsencrypt/live/dashboard.k8s.gmem.cc kubectl create -n kube-system secret generic gmemk8scert-dashboard --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-grafana cd /etc/letsencrypt/live/grafana.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-grafana --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-monocular cd /etc/letsencrypt/live/monocular.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-monocular --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-prometheus cd /etc/letsencrypt/live/prometheus.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-prometheus --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-alertmanager cd /etc/letsencrypt/live/alertmanager.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-alertmanager --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-pushgateway cd /etc/letsencrypt/live/pushgateway.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-pushgateway --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-es cd /etc/letsencrypt/live/es.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-es --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-kibana cd /etc/letsencrypt/live/kibana.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-kibana --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-chartmuseum > /dev/null cd /etc/letsencrypt/live/chartmuseum.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-chartmuseum --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-jenkins > /dev/null cd /etc/letsencrypt/live/jenkins.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-jenkins --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-nginx > /dev/null cd /etc/letsencrypt/live/nginx.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-nginx --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem kubectl delete -n devops secret gmemk8scert-elastichq > /dev/null cd /etc/letsencrypt/live/elastichq.k8s.gmem.cc kubectl create -n devops secret generic gmemk8scert-elastichq --from-file=tls.crt=fullchain.pem,tls.key=privkey.pem |
安装其它基础组件
Helm
安装客户端
|
1 2 3 |
wget https://kubernetes-helm.storage.googleapis.com/helm-v2.11.0-linux-amd64.tar.gz tar xzf helm-v2.11.0-linux-amd64.tar.gz sudo cp linux-amd64/helm /usr/local/bin |
安装服务器
镜像预拉取：
|
1 2 3 |
docker pull gcr.io/kubernetes-helm/tiller:v2.11.0 docker tag gcr.io/kubernetes-helm/tiller:v2.11.0 docker.gmem.cc/kubernetes-helm/tiller:v2.11.0 docker push docker.gmem.cc/kubernetes-helm/tiller:v2.11.0 |
创建专用的serviceaccount：
|
1 2 3 |
kubectl -n kube-system create serviceaccount tiller kubectl create clusterrolebinding kube-system-tiller-role-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller kubectl -n kube-system patch serviceaccount tiller -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
创建deployment：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: tier: devops app: helm comp: tiller name: tiller-deploy namespace: kube-system spec: replicas: 1 selector: matchLabels: tier: devops app: helm comp: tiller strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: tier: devops app: helm comp: tiller app: helm name: tiller spec: serviceAccountName: tiller containers: - env: - name: TILLER_NAMESPACE value: kube-system - name: TILLER_HISTORY_MAX value: "0" image: docker.gmem.cc/kubernetes-helm/tiller:v2.11.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /liveness port: 44135 scheme: HTTP initialDelaySeconds: 1 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: tiller ports: - containerPort: 44134 name: tiller protocol: TCP - containerPort: 44135 name: http protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readiness port: 44135 scheme: HTTP initialDelaySeconds: 1 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 |
Ingress控制器
|
1 2 3 4 5 6 7 8 9 10 |
helm update helm delete ngress --purge rm -rf nginx-ingress/ helm fetch gmem/nginx-ingress --untar kubectl label nodes xenon radon neon beta.kubernetes.io/nginx-ingress=true helm install nginx-ingress --name=ngress --namespace=kube-system -f nginx-ingress/overrides/gmem.yaml --set controller.loadBalanceAlgorithm=ewma,controller.kind=DaemonSet,controller.service.type=NodePort,controller.hostNetwork=true,controller.verboseLevel=4 kubectl -n kube-system patch serviceaccount ngress -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
ceph provisioners
访问密钥
|
1 2 3 4 |
ceph auth get client.admin 2>&1 | grep "key = " | awk '{print $3'} | xargs echo -n > pvc-ceph-key kubectl get namespace --no-headers | awk '{ system("kubectl -n "$1" delete secret pvc-ceph-key") }' kubectl get namespace --no-headers | awk '{ system("kubectl -n "$1" create secret generic pvc-ceph-key --from-file=pvc-ceph-key") }' |
安装provisioners
|
1 2 3 4 5 6 |
helm delete ceph-provisioners --purge rm -rf ceph-provisioners/ helm update helm fetch gmem/ceph-provisioners --untar helm install --name ceph-provisioners --namespace kube-system ceph-provisioners -f ceph-provisioners/overrides/gmem.yaml kubectl -n kube-system patch serviceaccount ceph-provisioners -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
安装Devops工具
所有Chart的代码放置于https://git.gmem.cc/alex/oss-charts.git
dashboard
|
1 2 3 4 5 6 7 |
helm update helm delete dashboard --purge rm -rf kubernetes-dashboard helm fetch gmem/kubernetes-dashboard --untar helm install kubernetes-dashboard --name=dashboard --namespace=kube-system -f kubernetes-dashboard/overrides/gmem.yaml kubectl -n kube-system patch serviceaccount dashboard -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
使用下面的命令可以提取访问Dashboard需要的Token：
|
1 |
kubectl -n kube-system get secrets `kubectl -n kube-system get sa dashboard-admin -o jsonpath="{.secrets[0].name}"` -o jsonpath="{.data.token}" | base64 -d |
prometheus
|
1 2 3 4 5 6 7 8 9 10 |
helm delete prometheus --purge rm -rf prometheus helm fetch gmem/prometheus --untar helm install prometheus --name=prometheus --namespace=devops -f prometheus/overrides/gmem.yaml kubectl -n devops patch serviceaccount prometheus-node-exporter -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' kubectl -n devops patch serviceaccount prometheus-kube-state-metrics -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' kubectl -n devops patch serviceaccount prometheus-server -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' kubectl -n devops patch serviceaccount prometheus-pushgateway -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' kubectl -n devops patch serviceaccount prometheus-alertmanager -p '{"imagePullSecrets": [{"name": "gmemregsecret"}]}' |
grafana
安装
|
1 2 3 4 |
helm delete grafana --purge rm -rf grafana helm fetch gmem/grafana --untar helm install grafana --name=grafana --namespace=devops -f grafana/overrides/gmem.yaml |
数据备份
|
1 2 3 |
GRAFANA_POD=`kubectl -n devops get pod -l app=grafana --no-headers | awk '{print $1}'` rm -rf /home/alex/Go/projects/k8s-init/assets/grafana/* kubectl -n devops cp $GRAFANA_POD:/var/lib /home/alex/Go/projects/k8s-init/assets/grafana |
数据还原
|
1 2 |
kubectl -n devops exec $GRAFANA_POD -- rm -rf /var/lib/grafana/* kubectl -n devops cp /home/alex/Go/projects/k8s-init/assets/grafana $GRAFANA_POD:/var/lib |
首页仪表盘
当前使用的首页仪表盘不支持编辑，可以修改ConfigMap：
|
1 |
kubectl -n devops edit configmap grafana |
然后重启Grafana的Pod
EFK
安装ES+Kibana
|
1 2 3 4 5 6 7 8 |
PSWD=*** helm delete es --purge kubectl -n devops get pvc -l release=es,tier=devops --no-headers | awk '{ system( "kubectl -n devops delete pvc " $1) }' rm -rf elasticsearch helm fetch gmem/elasticsearch --untar helm install elasticsearch --name=es --namespace=devops -f elasticsearch/overrides/gmem.yaml --set kibana.password=$PSWD,curator.password=$PSWD |
设置密码
等到ES的Client节点可用后，执行：
|
1 2 |
ES_CLIENT_POD=`kubectl -n devops get pod -l app=elasticsearch,comp=client --no-headers | head -1 | awk '{print $1}'` kubectl -n devops exec -it $ES_CLIENT_POD bin/x-pack/setup-passwords interactive |
然后根据提示输入密码。执行此设置密码操作后，Kibana才可以使用。建议登陆到Kibana后创建自己的用户。
导入XPack授权文件
|
1 2 |
ES_CLIENT_SRV=`kubectl -n devops get service es-elasticsearch --no-headers | awk '{print $3}'` curl -XPUT -u elastic "http://$ES_CLIENT_SRV:9200/_xpack/license" -H "Content-Type: application/json" -d @eslic.json |
安装Fluentd
|
1 2 3 4 |
helm delete fluentd --purge rm -rf fluentd helm fetch gmem/fluentd --untar helm install fluentd --name=fluentd --namespace=devops -f fluentd/overrides/gmem.yaml --set es.password=$PSWD |
为需要采集日志的节点打标签：
|
1 |
kubectl label node --all beta.kubernetes.io/fluentd-ds-ready=true |
chartmuseum
|
1 2 3 4 |
helm delete chartmuseum --purge rm -rf chartmuseum helm fetch gmem/chartmuseum --untar helm install chartmuseum --name=chartmuseum --namespace=devops -f chartmuseum/overrides/gmem.yaml |
monocular
|
1 2 3 4 |
helm delete monocular --purge rm -rf monocular helm fetch gmem/monocular --untar helm install monocular --name=monocular --namespace=devops -f monocular/overrides/gmem.yaml |
jenkins
|
1 2 3 4 |
helm delete jenkins --purge rm -rf jenkins helm fetch gmem/jenkins --untar helm install jenkins --name=jenkins --namespace=kube-system -f jenkins/overrides/gmem.yaml |
执行下面的命令获取Jenkins的登陆密码：
|
1 |
kubectl get secret --namespace kube-system jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode |
升级
从1.12到1.13
第一个主节点
安装新版本软件
|
1 |
apt install -y kubeadm=1.13.12-00 kubelet=1.13.12-00 kubectl=1.13.12-00 |
查看升级计划：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
kubeadm upgrade plan # 外部Etcd版本建议 External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE Etcd 3.3.9 3.2.24 # 需要手工在所有节点上升级kubelet Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE Kubelet 6 x v1.12.1 v1.13.12 1 x v1.13.12 v1.13.12 # 控制平面组件列表 Upgrade to the latest version in the v1.12 series: COMPONENT CURRENT AVAILABLE API Server v1.12.1 v1.13.12 Controller Manager v1.12.1 v1.13.12 Scheduler v1.12.1 v1.13.12 Kube Proxy v1.12.1 v1.13.12 CoreDNS 1.2.6 |
镜像预拉取：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
kubeadm config images list # k8s.gcr.io/kube-apiserver:v1.13.12 # k8s.gcr.io/kube-controller-manager:v1.13.12 # k8s.gcr.io/kube-scheduler:v1.13.12 # k8s.gcr.io/kube-proxy:v1.13.12 # k8s.gcr.io/pause:3.1 # k8s.gcr.io/etcd:3.2.24 # k8s.gcr.io/coredns:1.2.6 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.13.12 docker.gmem.cc/k8s/kube-apiserver:v1.13.12 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.13.12 docker.gmem.cc/k8s/kube-controller-manager:v1.13.12 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.13.12 docker.gmem.cc/k8s/kube-scheduler:v1.13.12 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.13.12 docker.gmem.cc/k8s/kube-proxy:v1.13.12 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 docker.gmem.cc/k8s/pause:3.1 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.2.24 docker.gmem.cc/k8s/etcd:3.2.24 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.2.6 docker.gmem.cc/k8s/coredns:1.2.6 docker push docker.gmem.cc/k8s/kube-apiserver:v1.13.12 docker push docker.gmem.cc/k8s/kube-controller-manager:v1.13.12 docker push docker.gmem.cc/k8s/kube-scheduler:v1.13.12 docker push docker.gmem.cc/k8s/kube-proxy:v1.13.12 docker push docker.gmem.cc/k8s/pause:3.1 docker push docker.gmem.cc/k8s/etcd:3.2.24 docker push docker.gmem.cc/k8s/coredns:1.2.6 |
执行升级：
|
1 |
kubeadm upgrade apply v1.13.12 |
在我的环境下，由于coredns、kube-proxy的标签被修改，因此kubeadm无法识别。手工修改这两种资源的镜像地址即可。
其它主节点
|
1 |
kubeadm upgrade node experimental-control-plane |
从节点
安装新版的kubelet： apt install -y kubelet=1.13.12-00
其它版本
1.19
Kubeadm配置
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: z9oq5i.uehbf6ms1qpmlvxk ttl: 24h0m0s usages: - signing - authentication localAPIEndpoint: advertiseAddress: 10.0.0.21 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: xenial-21 --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration certificatesDir: /etc/kubernetes/pki clusterName: gmem imageRepository: docker.gmem.cc/k8s kubernetesVersion: v1.19.0 dns: type: CoreDNS etcd: external: caFile: "" certFile: "" endpoints: - http://etcd.gmem.cc:2379 keyFile: "" networking: dnsDomain: k8s.gmem.cc podSubnet: 172.27.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: extraArgs: authorization-mode: "Node,RBAC" service-node-port-range: 80-32767 certSANs: - "k8s.gmem.cc" - "127.0.0.1" - "10.0.0.21" - "10.0.0.22" - "10.0.0.23" timeoutForControlPlane: 4m0s controllerManager: extraArgs: scheduler: extraArgs: |
常见问题
坏掉的主节点无法重新加入
Unable to fetch the kubeadm-config ConfigMap
报错信息：Unable to fetch the kubeadm-config ConfigMap: Get https://10.0.5.1:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 10.0.5.1:6443: connect: connection refused
解决办法：修改下面的配置文件，填写可用的server字段：
|
1 |
kubectl -n kube-public edit cm cluster-info |
etcdmain: member ... has already been bootstrapped
以下面的参数启动Etcd：
|
1 |
--initial-cluster-state existing |
Leave a Reply