See all blog posts

Building a Grafana Backend Plugin

Grafana is a great observability platform. One of its key strengths comes from its plugin system. (To give you an idea of how popular the extension of Grafana through plugins is, you can find the entire list of available plugins here.)

In this post I will explain how to build a simple data source plugin for Grafana 7, using Scylla as an example. I’ll focus on how to create a new data source plugin and specifically on backend plugins.

When building your own dashboard you would usually use two types or plugins: datasource and panels.

  • Data source plugins allow grafana to connect to a variety of sources and display the results. There are already many plugins for known databases.
  • Panel plugins make it possible to extend how data is displayed.

Creating a Backend Plugin

In this article I will show you how I created a backend plugin for Scylla that would also work with Apache Cassandra. To make it easier to follow the instructions I have created the following open source repository:

https://github.com/amnonh/backend-grafana-blogpost

You can look at the commit-list to follow the steps in this post. If you are looking for the actual plugin it is in:

https://github.com/scylladb/scylla-grafana-datasource

All of this work is open source, available under the Apache 2.0 license.

Prerequisites

Grafana has frontend and backend parts. Typically, a user is exposed to the frontend. For example, in the combination of running Grafana with Prometheus as a metric server, the data is collected in the user browser and then displayed.

While we call it a backend plugin, it actually has a frontend and a backend. The backend will be written in Go and the frontend uses react. You don’t need much experience in either to follow along. If you don’t have these already, you’ll need to install the following:

And optionally:

Creating the Plugin

This guide follows the Grafana data source guide and customizes it. In general these are the steps we’ll take:

  • Setup your environment
  • Create a plugin
  • Implement data queries
  • Add support for health checks
  • Enable alerts in Grafana

Setting Up Your Environment

We will need some configuration and I like to do my Grafana configuration from files so I suggest that you create a directory for all Grafana development.

mkdir grafana-dev
cd grafana-dev

Running Grafana in Docker

This is optional. In development I use Docker for running Grafana (I’m also going to use Docker for Scylla). I suggest you would add two scripts to start and stop Grafana in Docker.

This the script I’m going to use to start grafana:

$ cat run-grafana.sh
docker run -d -p 3000:3000 \
    -e "GF_AUTH_BASIC_ENABLED=false" \
    -e "GF_AUTH_ANONYMOUS_ENABLED=true" \
    -e "GF_AUTH_ANONYMOUS_ORG_ROLE=Admin" \
    -v $PWD/grafana/dashboard:/var/lib/grafana/dashboards:z \
    -v $PWD/plugins:/var/lib/grafana/plugins:z \
    -v $PWD/provisioning:/var/lib/grafana/provisioning:z \
    -e "GF_PATHS_PROVISIONING=/var/lib/grafana/provisioning" \
    -e "GF_SECURITY_ADMIN_PASSWORD=admin" \
    -e "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=Your-Plugin-ID" \
    --name grafana grafana/grafana:7.1.5

This is the script I’m going to use to kill it:

$ cat kill-grafana.sh
docker kill grafana
docker rm -v grafana

If you are familiar with Grafana, note the following:

  • The GF_AUTH_ANONYMOUS_ENABLED flag that supports the new unsigned plugin.
  • The plugins directory, it would allow grafana to find the plugin.
  • The provisioning directory that would allow us to configure the plugin from file.

Creating the base for the plugin

mkdir -p provisioning/datasources/

mkdir plugins

cd plugins

I’m using scylla-datasource as the plugin name.

Follow the instructions to create the plugin

npx @grafana/[email protected] plugin:create scylla-datasource

cd scylla-datasource

After you have updated the plugin SDK and compile, stop.

Before we do anything else, let’s use git:

git init
git add .
git commit -m "initial commit"

First Test

If you haven’t done so, run the Grafana server, if you are using docker, you can use docker restart grafana.

Check the logs with:

docker logs grafana

And look for msg="Running an unsigned backend plugin".

If you see a warning about not running an unsigned plugin, make sure you get the plugin name right with the GF_AUTH_ANONYMOUS_ENABLED flag.

You can now follow the rest of the tutorial to add your datasource from the dashboard.
I suggest you do that and get back to this post when you’re done.

Post the basic tutorial

Welcome back! In the rest of the post, I will cover multiple subjects that were not clear to me while developing the backend plugin. I tried to break each subject into its own independent section. Besides the explanations you can follow the commits in the repository here.

Data source configuration

Adding a datasource comes with its own configuration.
If you followed the guide, when you added the plugin, you saw a screen such as:

Which is a basic configuration that can help you get started.

Before explaining how to modify it, you should note the two kinds of configurations that share the screen. There is a regular one and backend only.

When an admin configures a data source (in our case, a Scylla database) to be able connect to it securely, we need a username and a password. Once we configure the username and password, we don’t want other users to be exposed to it. Other parts of configurations can be public and we would like users to be able to see them (for example the host it will be connected to).

The configuration has secure and non-secure parts. The secure part can be set or reset, but once configured it will not be sent back to the frontend. The non secure part is shown and can be changed. Grafana supports those two types.

For our example we are going to have a secure username/password and non-secure host.

There are two files we need to change:

  1. src/types.ts that holds the types definitions.
  2. src/ConfigEditor.tsx that holds the code for showing and changing the configuration.

Non-Secure

In types.ts look for MyDataSourceOptions this is the interface definition of the non-secure part of the configuration. For our example, we will change the path? to host?.

export interface MyDataSourceOptions extends DataSourceJsonData {
    host?: string;
}

Now edit ConfigEditor.tsx and change the path to host. Compile your code with yarn build, now when you adds a new plugin it looks like this:

I’ve also changed the host description, you can see the full changes in the commit.

Secure

Now let’s add the username and password. Edit the types.ts and update MySecureJsonData.

export interface MySecureJsonData {
    user?: string;
    password?: string;
}

Edit configEditor.tst, look at how the apiKey is configured and change it to password, note the use of the secretFormField, once configured secret configuration can only be reset and re-assigned.

After you made the changes compile and reload the data source configuration, it should now looks like this:

Note that I’ve changed the user and password text, look at the git commit for full change.

Congratulations, you have seen how to change the backend configuration!

Configure the data source from a datasource.yml file

As I’ve mentioned before, I like to use files for grafana configurations. You can read more about my take on dashboards from files here.

Just like the start script that I’ve mentioned to start the grafana container you can put your plugin configuration in a file, if you use the suggested run script, you already have everything you need for that.

Go back up to the directory that holds the run-grafana.sh script, under the provisioning directory we already created a datasource directory.

Create a datasource.yml file:

printf "apiVersion: 1\ndatasources:\n- name: scylla-datasource\n type: your-datasource\n orgId: 1\n isDefault:\n jsonData:\n host: ''\n secureJsonData:\n user: 'Scylla-User'\n password: 'Scylla-Password'\n" > provisioning/datasources/datasource.yaml

Make sure you change your-datasource to the data source id you are using, in my case this is scylla-scylla-datasource.

Restart Grafana and make sure that now you have the plugin configured.

Query Configuration

The second frontend configuration, is the query configuration. This is the user’s way to control what will be returned from the data source.

Create a new dashboard, add a panel and set its data source to the newly configured one.
You should see something like:

We will change it to CQL and an optional host.

Basic

Edit the types.ts file and change the MyQuery interface to:

export interface MyQuery extends DataQuery {
    queryText?: string;
    host?: string;
}

 

And remove the use of constans in the defaultQuery part.
Note that defaultQuery is the place to add default values for your input.

Edit QueryEditor.tsx. This is the file that controls how the query editor looks and behaves.
You should replace constant with host, make sure to remove the number formatting.
This is how it looks like, note that I’ve set the input width.

Template variables

Grafana supports template variables, those variables can be set explicitly or taken from a metric. You can read more about adding variables to a plugin here.

In general, you need to explicitly do the variable replacement before sending the query to the backend plugin.

Todo that you will need to edit src/DataSource.ts.

The latest version of grafana runtime holds the TemplateSrv that can be used to fetch available template variables.

You need the getTemplateSrv to get it.

Edit DataSource.ts and add the following:

import { DataSourceInstanceSettings } from '@grafana/data';
import { DataSourceWithBackend } from '@grafana/runtime';
import { MyDataSourceOptions, MyQuery } from './types';
import { getTemplateSrv } from '@grafana/runtime';
 
export class DataSource extends DataSourceWithBackend<MyQuery, MyDataSourceOptions> {
  constructor(instanceSettings: DataSourceInstanceSettings) {
    super(instanceSettings);
  }
  applyTemplateVariables(query: MyQuery) {
    const templateSrv = getTemplateSrv();
    return {
      ...query,
      queryText: query.queryText ? templateSrv.replace(query.queryText) : '',
      host: query.host ? templateSrv.replace(query.host) : '',
    };
  }
}

As you can see, applyTemplateVariables allows you to modify the values before they are sent.
So far we took care of the frontend side, it’s time to move to the backend with go.

Writing the Backend part

Getting the data from the request

Our request holds two parameters, the queryText and the host.
Update the queryModel accordingly:

type queryModel struct {
    Format string `json:"format"`
    QueryText string `json:"queryText"`
    Host string `json:"host"`
}

Compile with mage and restart Grafana for the change would take effect.
I have added a log line in the code to print the results so I can see that data is transferred between the frontend to the backend.

Connection Management

The base template suggests to use an instance manager for the life cycle management of data source instances. In practice, the im variable inside SampleDatasource can generate new connections when needed.

ScyllaDB and Apache Cassandra use CQL to query the database. We will use the Go driver for that.

In the import part, replace the http with
"github.com/gocql/gocql"

Change instanceSettings to:

type instanceSettings struct {
    cluster *gocql.ClusterConfig
    session *gocql.Session
}

The newDataSourceInstance needs to use the host from the backend configuration.

func newDataSourceInstance(setting backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
    type editModel struct {
        Host string `json:"host"`
    }
    var hosts editModel
    err := json.Unmarshal(setting.JSONData, &hosts)
    if err != nil {
        log.DefaultLogger.Warn("error marshalling", "err", err)
        return nil, err
    }
    log.DefaultLogger.Info("looking for host", "host", hosts.Host)
    var newCluster *gocql.ClusterConfig = nil
    newCluster = gocql.NewCluster(hosts.Host)
    session, _ := gocql.NewSession(*newCluster)
    return &instanceSettings{
        cluster: newCluster,
        session: session,
    }, nil
}

You can see in the code, how to get information from the setting JSONData, remember that there are two parts to the backend data source configuration, this is the unsecured part.
Another thing is how a session is created once and is kept for future use.

Dataframe Response

If you look at the query method, you see the use of Dataframe to to return the result. Dataframe are column oriented, Each data frame has multiple fields you can read more about fields here.

In the generated example, fields are created with their type and results. A Database plugin is an example for a situation where both are created based on the returned results.

We are going to add two helper methods getTypeArray that will use the column type return from Scylla to create the fields and toValue that would be responsible for type casting.

We will use the following import:

"math/big"
"gopkg.in/inf.v0"
"strconv"

Add the following helper methods.

func getTypeArray(typ string) interface{} {
    switch t := typ; t {
        case "timestamp":
            return []time.Time{}
        case "bigint", "int":
            return []int64{}
        case "smallint":
            return []int16{}
        case "boolean":
            return []bool{}
        case "double", "varint", "decimal":
            return []float64{}
        case "float":
            return []float32{}
        case "tinyint":
            return []int8{}
        default:
            return []string{}
        }
}
func toValue(val interface{}, typ string) interface{} {
    if (val == nil) {
        return nil
    }
    switch t := val.(type) {
        case float32, time.Time, string, int64, float64, bool, int16, int8:
            return t
        case gocql.UUID:
            return t.String()
        case int:
            return int64(t)
        case *inf.Dec:
            if s, err := strconv.ParseFloat(t.String(), 64); err == nil {
                return s
            }
            return 0
        case *big.Int:
            if s, err := strconv.ParseFloat(t.String(), 64); err == nil {
                return s
            }
            return 0
        default:
            r, err := json.Marshal(val)
            if (err != nil) {
                log.DefaultLogger.Info("Marshalling failed ", "err", err)
            }
            return string(r)
    }
}

Now we need to change the query method to read the results from the CQL connection.

func (td *SampleDatasource) query(ctx context.Context, query backend.DataQuery, instSetting *instanceSettings) backend.DataResponse {
    // Unmarshal the json into our queryModel
    var qm queryModel

    response := backend.DataResponse{}

    response.Error = json.Unmarshal(query.JSON, &qm)
    if response.Error != nil {
        return response
    }
  log.DefaultLogger.Info("Getting query information", "query", qm.QueryText, "host", qm.Host)
    // Log a warning if `Format` is empty.
vif qm.Format == "" {
        log.DefaultLogger.Warn("format is empty. defaulting to time series")
    }

    // create data frame response
    frame := data.NewFrame("response")
  iter := instSetting.session.Query(qm.QueryText).Iter()

  for _, c := range iter.Columns() {
    log.DefaultLogger.Info("Adding Column", "name", c.Name, "type", c.TypeInfo.Type().String())
    frame.Fields = append(frame.Fields,
        data.NewField(c.Name, nil, getTypeArray(c.TypeInfo.Type().String())),
    )
  }
  for {
      // New map each iteration
      row := make(map[string]interface{})
      if !iter.MapScan(row) {
         break
      }
      vals := make([]interface{}, len(iter.Columns()))
      for i, c := range iter.Columns() {
          vals[i] = toValue(row[c.Name], c.TypeInfo.Type().String())
      }
      frame.AppendRow(vals...)
  }
  if err := iter.Close(); err != nil {
      log.DefaultLogger.Warn(err.Error())
  }
    // add the frames to the response
    response.Frames = append(response.Frames, frame)

    return response
}

The CQL results are row based, while the DataFrame is column based.

You can see that we first loop over the columns and add each of the fields without values.
We then iterate over the CQL results and use the AppendRow method to add the information to the dataFrame.

A working plugin

With all the pieces in place after compiling the code I have pointed the data source to a Scylla node. This is what it looks like:

Getting the SecureData

As mentioned earlier, the data source configuration holds a Secure part that once set, is only available to the backend.

We are going to add an optional username/password for the database connection.
In sample-plugin.go we will modify newDataSourceInstance

You can get the secure part from the setting variables.

Here we check if a user and password were set, if they are, we use them to create a CQL authenticator.

    var secureData = setting.DecryptedSecureJSONData
    password, hasPassword := secureData["password"]
    user, hasUser := secureData["user"]
    var authenticator *gocql.PasswordAuthenticator = nil
    if hasPassword && hasUser {
        log.DefaultLogger.Info("Adding authenticator for user", "user", user)
        authenticator = &gocql.PasswordAuthenticator{
            Username: user,
            Password: password,
        }
    }

After the CQL cluster connection was created, if an authenticator was created we use it to authenticate the connection:

    if authenticator != nil {
        newCluster.Authenticator = *authenticator
    }

Summary and our next steps

We demonstrated building a Grafana data source with Scylla as an example. Possible next steps we are planning:

  • Pushing the new plugin to the Grafana plug-in repository
  • Update the data source to provide time-series information
  • Presented the time-series information as a graph

Your next steps

We’ve given you all the tools you need to try this yourself! If you run into any issues, make sure to bring them up with our user community in Slack.

CHECK OUT THE CODE ON GITHUB

Amnon Heiman

About Amnon Heiman

Amnon has 15 years of experience in software development of large scale systems. Previously he worked at Convergin, which was acquired by Oracle. Amnon holds a BA and MSc in Computer Science from the Technion-Machon Technologi Le' Israel and an MBA from Tel Aviv University.