k8s的flannel安装报错Failed to create SubnetManager: error retrieving pod spec解决

花了一个上午来追踪问题,k8s都反复新建了十多次,docker都重启了几次。(一次显示不有获取磁盘空间,重启docker,清空存储解决)

在用kubeadm安装容器化的几个组件时,flannel组件死活不能启动,报如下问题:

Failed to create SubnetManager: error retrieving pod spec for ‘kube-system/kube-flannel-ds-xxx’: the server does not allow access to the requested resource.

在如下Url找到解决办法:

http://blog.csdn.net/ximenghappy/article/details/70157361

明天搞DNS和节点加入….

================================================

Kubernetes一共提供五种网络组件,可以根据自己的需要选择。我使用的Flannel网络,此处1.5.5和1.6.1也是不一样的,1.6.1加了RBAC。需要执行一下两个命令:

kubectl create -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel-rbac.yml

clusterrole “flannel” configured
clusterrolebinding “flannel” configured

kubectl create -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

serviceaccount “flannel” created
configmap “kube-flannel-cfg” created
daemonset “kube-flannel-ds” created

  解决此问题参考:
https://github.com/kubernetes/kubernetes/issues/44029
https://github.com/kubernetes/kubeadm/issues/212#issuecomment-290908868

kube-flannel-rbac.yaml文件内容:

# Create the clusterrole and clusterrolebinding:
# $ kubectl create -f kube-flannel-rbac.yml
# Create the pod using the same namespace used by the flannel serviceaccount:
# $ kubectl create --namespace kube-system -f kube-flannel.yml
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system

kube-flannel.yaml内容:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel-amd64:v0.7.1
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel-amd64:v0.7.1
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

未分类

未分类

k8s与flannel网络原理

我们这里假设flannel使用VXLAN协议。每台主机都安装有flannel。k8s定义的flannel网络为10.0.0.0/16,各主机的flannel从这个网络申请一个子网。pod1所在的主机的flannel子网为10.0.14.1/24,pod2所在主机的flannel子网为10.0.5.1/24。每台主机有cni0和flannel.1虚拟网卡。cni0为在同一主机pod共用的网桥,当kubelet创建容器时,将为此容器创建虚拟网卡vethxxx,并桥接到cni0网桥。flannel.1是一个tun虚拟网卡,接收不在同一主机的POD的数据,然后将收到的数据转发给flanneld进程。原理图:
系统管理

pod1到pod2的网络

pod1路由表:

default via 10.0.14.1 dev eth0 
10.0.0.0/16 via 10.0.14.1 dev eth0 
10.0.14.0/24 dev eth0  proto kernel  scope link  src 10.0.14.15 

host1路由表:

default via 192.168.93.254 dev eno16777984  proto static  metric 100 
10.0.0.0/16 dev flannel.1 
10.0.14.0/24 dev cni0  proto kernel  scope link  src 10.0.14.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.93.0/24 dev eno16777984  proto kernel  scope link  src 192.168.93.212  metric 100 

pod1 IP地址:10.0.14.15
pod2 IP地址:10.0.5.150

pod1与pod2不在同一台主机

下面是从pod1 ping pod2的数据包流向
1. pod1(10.0.14.15)向pod2(10.0.5.150)发送ping,查找pod1路由表,把数据包发送到cni0(10.0.14.1)
2. cni0查找host1路由,把数据包转发到flannel.1
3. flannel.1虚拟网卡再把数据包转发到它的驱动程序flannel
4. flannel程序使用VXLAN协议封装这个数据包,向api-server查询目的IP所在的主机IP,称为host2(不清楚什么时候查询)
5. flannel向查找到的host2 IP的UDP端口8472传输数据包
6. host2的flannel收到数据包后,解包,然后转发给flannel.1虚拟网卡
7. flannel.1虚拟网卡查找host2路由表,把数据包转发给cni0网桥,cni0网桥再把数据包转发给pod2
8. pod2响应给pod1的数据包与1-7步类似
下面是这次ping数据包的wireshark解析出的协议数据:
pod1 ping请求:
系统管理
pod2响应:
系统管理

pod1与pod2在同一台主机

pod1和pod2在同一台主机的话,由cni0网桥直接转发请求到pod2,不需要经过flannel。

pod到service的网络

创建一个service时,相应会创建一个指向这个service的域名,域名规则为{服务名}.{namespace}.svc.{集群名称}。之前service ip的转发由iptables和kube-proxy负责,目前基于性能考虑,全部为iptables维护和转发。iptables则由kubelet维护。service仅支持udp和tcp协议,所以像ping的icmp协议是用不了的,所以无法ping通service ip。
现在我们尝试看看在pod1向kube-dns的service ip 10.16.0.10:53发送udp请求是如何转发的。
我们先找出与此IP相关的iptables规则:

【PREROUTING链】
-m comment --comment "kubernetes service portals" -j KUBE-SERVICES

【KUBE-SERVICES链】
-d 10.16.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU

【KUBE-SVC-TCOU7JCQXEZGVUNU链】
-m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-L5MHPWJPDKD7XIFG

【KUBE-SEP-L5MHPWJPDKD7XIFG链】
-p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.0.0.46:53
  1. pod1向service ip 10.16.0.10:53发送udp请求,查找路由表,把数据包转发给网桥cni0(10.0.14.1)
  2. 在数据包进入cnio网桥时,数据包经过PREROUTING链,然后跳至KUBE-SERVICES链
  3. KUBE-SERVICES链中一条匹配此数据包的规则,跳至KUBE-SVC-TCOU7JCQXEZGVUNU链
  4. KUBE-SVC-TCOU7JCQXEZGVUNU不做任何操作,跳至KUBE-SEP-L5MHPWJPDKD7XIFG链
  5. KUBE-SEP-L5MHPWJPDKD7XIFG里对此数据包作了DNAT到10.0.0.46:53,其中10.0.0.46即为kube-dns的pod ip
  6. 查找与10.0.0.46匹配的路由,转发数据包到flannel.1
  7. 之后的数据包流向就与上面的pod1到pod2的网络一样了

pod到外网

  1. pod向qq.com发送请求
  2. 查找路由表,转发数据包到宿主的网卡
  3. 宿主网卡完成qq.com路由选择后,iptables执行MASQUERADE,把源IP更改为宿主网卡的IP
  4. 向qq.com服务器发送请求