Error getting docker stats: context deadline exceeded

I’m using the telegraf-ds chart 1.3 running on Kubernetes 1.6.2 (GKE) via the tick-charts repo.

Logs show continuous errors like:

2017-05-17T16:44:44Z E! Error in plugin [inputs.docker]: E! Error gathering container [/k8s_chronograf_chronograf-prod-chronograf-604308575-ldmn6_infra_8963a62f-3a53-11e7-9643-42010a8001a5_0] stats: Error getting docker stats: context deadline exceeded

The above is the log for the chronograf pod, but this error is logged for 18 out of my 45 pods.

I tried increasing the docker input timeout to 10s and then to 20s with the same result.

@devth I’ve seen this issue on every cluster I’ve spun up but it seems intermittent and slightly random as to which pods are affected. I initially thought this was isolated to the kube-system namespace but it appears it is not. Looking at the docker logs on the host the following errors are repeated:

May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.551558050Z" level=error msg="collecting stats for 5ac5af5db592267d33ea53d3f3eb53757bf
May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.556143500Z" level=error msg="collecting stats for fc54907131a326049c1e0e6ef498e31d426
May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.557884430Z" level=error msg="collecting stats for b4325f3fdf93ac516fa1b80bd62e4e1592e
May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.561983229Z" level=error msg="collecting stats for 86290e7ba07c3d146848a5795f6cdb4436a
May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.563776749Z" level=error msg="collecting stats for c51d92e5dcb0ec3f271e28f00c004b3bf02
May 17 02:06:02 gke-influx-kube-default-pool-1202464b-3g6m docker[1290]: time="2017-05-17T02:06:02.565495838Z" level=error msg="collecting stats for 8feadbb8ddacfc7fb37d99065c0e3c01632

Docker Version information on gke hosts:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.7.4
 Git commit:   4dc5990
 Built:        
 OS/Arch:      linux/amd64
Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.7.4
 Git commit:   4dc5990
 Built:        
 OS/Arch:      linux/amd64

and related docker issues: