k8s实践10:docker网桥和pod连接网络结构

1.1
使用到的组件

network namespacebridgeveth pair

注意事项:所有命令都是在集群master节点操作测试.

1.2

network namespace

执行命令ip netns会显示创建的network namespace
如果显示空白,请做link,命令见下:

[root@k8s-master2 ~]# ln -s /var/run/docker/netns/ /var/run/netns

[root@k8s-master1 ~]# ip netnsdefault[root@k8s-master1 ~]#

[root@k8s-master2 ~]# ip netnsnetns290d0c6ad2ab (id: 3)49bfcc08baa8 (id: 2)aeb6eaef384f (id: 1)8d103e06202b (id: 0)default[root@k8s-master2 ~]#

[root@k8s-master3 ~]# ip netns789ea14c2216 (id: 4)a4dbfec964e5 (id: 3)4bdb1409b6e3 (id: 2)daea28f1e8b2 (id: 1)0566deb27814 (id: 0)default[root@k8s-master3 ~]#

可以看到每个节点显示的内容不同,默认只有一个default.

1.3
为什么每个节点显示的不同？

以master3做示例,见下

[root@k8s-master3 ~]# ip a |grep veth8: veth3a37cd9@if7:  mtu 1450 qdisc noqueue master docker0 state UP10: veth37e9e7e@if9:  mtu 1450 qdisc noqueue master docker0 state UP12: veth4ed5bcb@if11:  mtu 1450 qdisc noqueue master docker0 state UP14: vethc4d2905@if13:  mtu 1450 qdisc noqueue master docker0 state UP16: vethd67ea28@if15:  mtu 1450 qdisc noqueue master docker0 state UP[root@k8s-master3 ~]#

[root@k8s-master3 ~]# ip netns789ea14c2216 (id: 4)a4dbfec964e5 (id: 3)4bdb1409b6e3 (id: 2)daea28f1e8b2 (id: 1)0566deb27814 (id: 0)default[root@k8s-master3 ~]#

注意,一个netns对应了一个虚拟的网卡.怎么对应的呢？
示例,见下:

[root@k8s-master3 ~]# ip netns exec  0566deb27814 ip a1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever7: eth0@if8:  mtu 1450 qdisc noqueue state UP    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0    inet 172.30.91.2/24 scope global eth0       valid_lft forever preferred_lft forever    inet6 fe80::42:acff:fe1e:5b02/64 scope link       valid_lft forever preferred_lft forever

请看这条:

eth0@if8:  mtu 1450 qdisc noqueue state UP

对应的是:

8: veth3a37cd9@if7:  mtu 1450 qdisc noqueue master docker0 state UP

这是一个veth pair对.
可用命令

[root@k8s-master3 ~]# ethtool -i veth3a37cd9driver: vethversion: 1.0firmware-version:expansion-rom-version:bus-info:supports-statistics: yessupports-test: nosupports-eeprom-access: nosupports-register-dump: nosupports-priv-flags: no[root@k8s-master3 ~]# ethtool -S veth3a37cd9NIC statistics:     peer_ifindex: 7[root@k8s-master3 ~]#

回到开始的问题,为什么每个节点显示的不同呢？
network namespace是怎么来的呢？

先敲命令做显示对比.

pod

[root@k8s-master3 ~]# kubectl get pod -n kube-system -o wideNAME                                          READY     STATUS    RESTARTS   AGE       IP               NODEcoredns-779ffd89bd-5znxf                      1/1       Running   2          18h       172.30.91.3      k8s-master3kubernetes-dashboard-65c76f6c97-4kg2r         1/1       Running   3          18h       172.30.91.5      k8s-master3tiller-deploy-58d57fd669-w8wsz                1/1       Running   3          18h       172.30.91.2      k8s-master3traefik-ingress-controller-77b549b5d9-4xd58   1/1       Running   4          4d        192.168.32.130   k8s-master3traefik-ingress-controller-77b549b5d9-dj65f   1/1       Running   10         34d       192.168.32.129   k8s-master2traefik-ingress-controller-77b549b5d9-hpt4z   1/1       Running   11         34d       192.168.32.128   k8s-master1[root@k8s-master3 ~]#

pod的ip

[root@k8s-master3 ~]# kubectl exec -it tiller-deploy-58d57fd669-w8wsz /bin/sh -n kube-system~ $ ip a1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever7: eth0@if8:  mtu 1450 qdisc noqueue state UP    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff    inet 172.30.91.2/24 scope global eth0       valid_lft forever preferred_lft forever    inet6 fe80::42:acff:fe1e:5b02/64 scope link       valid_lft forever preferred_lft forever

eth0@if8,看看这个network namespace的ip情况

[root@k8s-master3 ~]# ip netns exec  0566deb27814 ip a1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever7: eth0@if8:  mtu 1450 qdisc noqueue state UP    link/ether 02:42:ac:1e:5b:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0    inet 172.30.91.2/24 scope global eth0       valid_lft forever preferred_lft forever    inet6 fe80::42:acff:fe1e:5b02/64 scope link       valid_lft forever preferred_lft forever

一个pod对应了一个network namespace,pod使用的网络连接口就是network namespace的网络口.

1.4

bridge

执行命令,见下:

[root@k8s-master3 ~]# brctl showbridge name        bridge id                STP enabled        interfacesdocker0                8000.0242c5980278        no                veth37e9e7e                                                        veth3a37cd9                                                        veth4ed5bcb                                                        vethc4d2905                                                        vethd67ea28

[root@k8s-master3 ~]# ip a |grep veth8: veth3a37cd9@if7:  mtu 1450 qdisc noqueue master docker0 state UP10: veth37e9e7e@if9:  mtu 1450 qdisc noqueue master docker0 state UP12: veth4ed5bcb@if11:  mtu 1450 qdisc noqueue master docker0 state UP14: vethc4d2905@if13:  mtu 1450 qdisc noqueue master docker0 state UP16: vethd67ea28@if15:  mtu 1450 qdisc noqueue master docker0 state UP

结论:
这些虚拟网卡每个对应的eth pair一端(另一端在namaspace里其实也就是在pod里,参考上面的分析),同时这些虚拟网卡桥接到了网桥docker上.

1.5
网桥docker和pod的逻辑拓扑关系图

拓扑见下: