У нас работает небольшой кластер kubernetes с 4 узлами. После зависания мастер-узла нам пришлось его перезапустить.
Теперь kube-proxy и weave-net на главном узле продолжают зависать в состоянии ContainerCreating , в то же время нормально функционируя на всех остальных узлах:
kubectl get pod -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE
coredns-78fcdf6894-67wkh 1/1 Running 0 12h 10.42.0.40 i101v182
coredns-78fcdf6894-8xwkq 1/1 Running 3 122d 10.44.0.65 i101v181
etcd-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
heapster-74db55987-sthnp 1/1 Running 1 40d 10.44.0.60 i101v181
kube-apiserver-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kube-controller-manager-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kube-proxy-6dqzf 0/1 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
kube-proxy-nvghr 1/1 Running 9 187d 10.1.101.182 i101v182
kube-proxy-spchz 1/1 Running 8 187d 10.1.101.181 i101v181
kube-proxy-xhg77 1/1 Running 7 187d 10.1.101.183 i101v183
kube-scheduler-i101v180.intra.graz.at 1/1 Running 2 11h 10.1.101.180 i101v180.intra.graz.at
kubernetes-dashboard-767dc7d4d-ws79h 1/1 Running 0 12h 10.42.0.41 i101v182
metrics-server-696868464d-b8njh 1/1 Running 0 12h 10.42.0.42 i101v182
monitoring-influxdb-848b9b66f6-gwbf5 1/1 Running 1 46d 10.44.0.61 i101v181
weave-net-465np 2/2 Running 3 64d 10.1.101.182 i101v182
weave-net-5mgdp 0/2 ContainerCreating 0 1h 10.1.101.180 i101v180.intra.graz.at
weave-net-chsqv 2/2 Running 2 40d 10.1.101.183 i101v183
weave-net-mcn77 2/2 Running 18 187d 10.1.101.181 i101v181
В журнале нет ни ошибок, ни событий, которые могли бы послужить причиной для этого:
kubectl describe pod kube-proxy-6dqzf -n kube-system
Name: kube-proxy-6dqzf
Namespace: kube-system
Node: i101v180.intra.graz.at/10.1.101.180
Start Time: Wed, 16 Jan 2019 14:32:51 +0100
Labels: controller-revision-hash=1151982146
k8s-app=kube-proxy
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.1.101.180
Controlled By: DaemonSet/kube-proxy
Containers:
kube-proxy:
Container ID:
Image: k8s.gcr.io/kube-proxy-amd64:v1.11.0
Image ID:
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/kube-proxy
--config=/var/lib/kube-proxy/config.conf
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/kube-proxy from kube-proxy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-9t8cp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
kube-proxy-token-9t8cp:
Type: Secret (a volume populated by a Secret)
SecretName: kube-proxy-token-9t8cp
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
kubectl describe pod weave-net-5mgdp -n kube-system
Name: weave-net-5mgdp
Namespace: kube-system
Node: i101v180.intra.graz.at/10.1.101.180
Start Time: Wed, 16 Jan 2019 14:21:57 +0100
Labels: controller-revision-hash=2963469815
name=weave-net
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.1.101.180
Controlled By: DaemonSet/weave-net
Containers:
weave:
Container ID:
Image: weaveworks/weave-kube:2.3.0
Image ID:
Port: <none>
Host Port: <none>
Command:
/home/weave/launch.sh
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 10m
Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/host/etc from cni-conf (rw)
/host/home from cni-bin2 (rw)
/host/opt from cni-bin (rw)
/host/var/lib/dbus from dbus (rw)
/lib/modules from lib-modules (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
/weavedb from weavedb (rw)
weave-npc:
Container ID:
Image: weaveworks/weave-npc:2.3.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 10m
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-dcjzm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
weavedb:
Type: HostPath (bare host directory volume)
Path: /var/lib/weave
HostPathType:
cni-bin:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
cni-bin2:
Type: HostPath (bare host directory volume)
Path: /home
HostPathType:
cni-conf:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
dbus:
Type: HostPath (bare host directory volume)
Path: /var/lib/dbus
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType:
weave-net-token-dcjzm:
Type: Secret (a volume populated by a Secret)
SecretName: weave-net-token-dcjzm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
Есть идеи, что может быть причиной этого?