Grafana Loki is a great monitoring solution for services inside a Kubernetes cluster. In this article I’ll show you how to handle the error 500 “no space left on device”.

We have Grafana running inside an Kubernetes cluster and everything was working fine. From one day to the next, there was a problem with a data source.

The error message says “no space left on device” and it seems that Grafana is not able to create new folders for the cache.

Object
  status:500
  statusText:"Internal Server Error"
    data:Object
      message:"mkdir /data/loki/boltdb-shipper-cache/index_19297: no space left on device "
      error:"Internal Server Error"
      response:"mkdir /data/loki/boltdb-shipper-cache/index_19297: no space left on device "
    config:Object
      url:"api/datasources/proxy/2/loki/api/v1/query_range?direction=BACKWARD&limit=1000&query=%7Bapp%3D%22eco-document-classifier%22%2Cnamespace%3D%22eco-production%22%7D&start=1667267724333000000&end=1667289324333000000&step=10"
      retry:0
      headers:Object
      hideFromInspector:false
    message:"mkdir /data/loki/boltdb-shipper-cache/index_19297: no space left on device "

To identify the volume causing the error, I look at the yaml definition of the Grafana pod in the Kubernetes cluster. To do that you just need to find the right pod in the Kubernetes cluster. With the following kubectl command the yaml definition can be shown.