If the file location for a Grafana database is located at a network drive (e.g. persistent storage in Microsoft Azure), it can happen that the file gets corrupted during an unsupported lock action.

"SQLite depends on the underlying filesystem to do locking as the documentation says it will. But some filesystems contain bugs in their locking logic such that the locks do not always behave as advertised. This is especially true of network filesystems and NFS in particular. If SQLite is used on a filesystem where the locking primitives contain bugs, and if two or more threads or processes try to access the same database at the same time, then database corruption might result." Source: [1]

As Grafana cleans up expired authentication tokens on every successful login, it tries to write data to the database file. If the file is locked or corrupted, this behavior prevents Grafana from discarding invalid tokens as well as creating a valid access token. That results in a login error.

When such a problem occurs, error logs may look similar to the following print.

t=2020-01-13T06:52:43+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/login status=500 remote_addr=0.0.0.0 time_ms=159 size=1717 referer=https://<url>/login
t=2020-01-13T06:58:39+0000 lvl=eror msg="failed to lock and execute cleanup of old login attempts" logger=cleanup error="unable to open database file"
t=2020-01-13T06:58:39+0000 lvl=eror msg="failed to create auth token" logger=http.server error="unable to open database file"
t=2020-01-13T06:58:39+0000 lvl=info msg="Successful Login" logger=http.server User=user
t=2020-01-13T06:58:39+0000 lvl=eror msg="Request error" logger=context userId=0 orgId=0 uname= error="runtime error: invalid memory address or nil pointer dereference" stack="/usr/local/go/src/runtime/panic.go" <... STACK TRACE REDUCTED ...>
t=2020-01-13T06:58:39+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/login status=500 remote_addr=0.0.0.0 time_ms=142 size=1717 referer=https://<url>/login
t=2020-01-13T07:08:39+0000 lvl=eror msg="failed to lock and execute cleanup of expired auth token" logger=auth error="unable to open database file"
t=2020-01-13T07:08:39+0000 lvl=eror msg="failed to lock and execute cleanup of old login attempts" logger=cleanup error="unable to open database file"
t=2020-01-13T07:18:39+0000 lvl=eror msg="failed to lock and execute cleanup of old login attempts" logger=cleanup error="unable to open database file"
t=2020-01-13T07:28:39+0000 lvl=eror msg="failed to lock and execute cleanup of old login attempts" logger=cleanup error="unable to open database file"
t=2020-01-13T07:38:39+0000 lvl=eror msg="failed to lock and execute cleanup of old login attempts" logger=cleanup error="unable to open database file"
t=2020-01-13T07:40:01+0000 lvl=eror msg="failed to create auth token" logger=http.server error="unable to open database file"
t=2020-01-13T07:40:01+0000 lvl=info msg="Successful Login" logger=http.server User=user

The relevant messages in the print are unable to open database file and failed to lock and execute cleanup of expired auth token. If these messages appears, there are multiple possible reasons:

  • the database file or the containing folder doesn't exist on the specified path
  • the database file exists, but the user under which Grafana is executed doesn't have sufficient access-rights for using the file
  • the database file exists, the user has sufficient access, but the file is locked or corrupted

For the first two issues the fixes are easy and solutions can be found by searching the net. For the third issue, the following fix can work, if the corruption of the database file isn't severe. Source: [2]

First, create a database dump for the original database file. The dump should not include any messages marked ERROR as this would mean data-loss.

sqlite3 grafana.db ".dump" > dump.sql

Then, create a new database file using the generated dump.sql file.

sqlite3 newgrafana.db < dump.sql

Finally, replace the old database file using the new created file. If everything went well you will now have regained access to your Grafana instance.