« Visualize your data with Grafana
At niolabs we advocate for “doing something” with the data that comes out of your distributed systems. That means turning on a motor, closing a valve, publishing to a web service, notifying a key stakeholder, and more. While dashboards and reports shouldn’t be the ultimate destination for the data in your systems, they certainly can play a key part in having confidence that your system is performing the way it should be. The flexibility of the nio platform allows you to use pretty much any visualization tool that you like. In this post, we will describe a solution using the open-source analytics platform Grafana.
Grafana is a product that lets you visualize, report, and alert based on time-series data. It can connect to a large selection of data sources such as Prometheus, InfluxDB, and Elasticsearch. It does not actually store any data on its own but rather connects to existing data sources. With nio we can store our data using one of the many created database connection blocks, and then hook that data store to Grafana. For this post we will use Elasticsearch and the Elastic Insert block from nio.
While this post will outline the basic steps needed to start visualizing your data with Grafana, there is also a GitHub repository available that contains a packaged nio instance, Elasticsearch database, and Grafana dashboard that you can use to get started. If you want to jump in and get a dashboard up and running quickly, follow the instructions in the GitHub README file.
For this example, we will visualize some real-time and historical weather data in a Grafana dashboard. We will use the free, HTTP-based OpenWeatherMap API to fetch our weather data. If you want to follow along you’ll need to set up the following prerequisites:
1. A nio account and a cloud nio instance - sign up for a trial here
2. A running Elasticsearch database - host your own or used a managed service like Compose
3. An API key for the OpenWeatherMap API - sign up for free here
4. A running version of Grafana - run it locally yourself for free or use a managed service like GrafanaCloud
nio Services and Blocks
Once you’ve got the prerequisites up and running (or you’ve run the docker containers in the pre-packaged GitHub repo), it’s time to build the nio service to populate our Elasticsearch database. This will only require one nio service that only contains 4 blocks. While the service itself is simple, it does demonstrate some important concepts of nio service building that will be crucial to understand when designing larger systems.
Here is a description of what each of these 4 blocks are doing:
Drive Weather Poll
This is an IdentityIntervalSimulator block that is solely responsible for determining how often we poll the HTTP weather API. Services in nio are streaming workflows. This means that if signals are not flowing through the service, the blocks are not doing anything. We use Simulator blocks to “kick off” the service periodically. The identity simulator will emit empty signals at a configurable interval. For this example, we will notify a signal once every 5 minutes.
Fetch Weather
This is an HTTPRequests block that will make HTTP calls to the OpenWeatherMap API. Configure this block with the relevant endpoint (suggest the Current Weather endpoint) and your API key. An example configuration is also provided in the example GitHub repository here. Notice that the HTTP block doesn’t determine how often to make requests. Instead, it makes requests when signals come into the block. This allows us to use Simulators to “drive” the function of the block. This is a concept that you will often find throughout nio systems. Read more about this service design pattern in the nio best practices documentation.
Add Timestamp
The signal that comes out of the HTTP block will contain information about the current weather conditions. However, our database needs time-series data to visualize with Grafana. That means we’ll need to include a timestamp in the data that we save. Since signals don’t contain timestamp information by default, this is a perfect use case for the Modifier block. We can configure it to add a timestamp field to our signal before saving.
Save to Elastic
The terminating block of our service, this ESInsert block will insert the incoming data into our Elasticsearch database. Configure this block with your Elasticsearch credentials (host, index, username/password, etc). Note that if we wanted to switch our data store out and use a different database we would just have to switch this block out, the rest of the service remains the same. Hooray for modularity with nio blocks!
Once you start your nio service you should see weather data being populated into your Elasticsearch database. If you’re having trouble, check out the Logger Panel to see if your service is logging any errors, double check your block configs, or reach out to the nio support team. We have application engineers standing by ready to help with problems just like this!
Grafana Dashboard
Now that we have a database with some time-series weather data in it, it’s time to configure Grafana to connect to that database. Fire up the admin panel for your Grafana server and, if you haven’t already, configure a Grafana Data Source to connect to Elasticsearch. You can configure your data sources by clicking the Settings gear on the left, and then select Data Sources.
If you’re using the pre-packaged Grafana nio project with Docker, then the data source and dashboard has already been configured for you! Just point to your running Grafana instance and check it out! If not, you’ll want to point your Grafana Data Source to your running Elasticsearch instance. Be sure that your “index name” matches the index you configured the ESInsert block to save to and that your “time field name” matches the attribute name you provided in the Modifier block. You may also want to set a “Min Interval” to the same interval you configured your simulator to emit at. This will prevent our charts from showing empty data if we’re too zoomed in.
Now let’s build a visual! Create a new Dashboard to hold our different visualizations from the Dashboard home screen. Then click the “Add Panel” button at the top and add a new “Graph” to the dashboard. Click the title of the new graph (it probably says “Panel Title”) and then click Edit to configure it. Let’s see if we can create a graph of historical temperatures by configuring the panel with settings that look something like this:
- Data Source - Elasticsearch (or whatever you named your Grafana Data Source)
- Metric
- Aggregation - Average
- Field - main.temp
- Group by
- Type - Date Histogram
- Timestamp field - @timestamp (or however you configured your Modifier block)
You may need to set the Min Doc Count to 1 if you’re seeing an empty graph, this will ensure that we show data only for the points that have data reported. You can control the time range of the graph with the date selector in the top right, but it’s likely that right now you don’t have a lot of data saved up yet, so the Last 3 Hours default setting is probably sufficient.
If it all goes well, you should be seeing a temperature graph that looks something like this:
We can also add some “Singlestat” panels that can show us the current temperature and other single data points like high/low temp. Add a new Panel to your Dashboard, this time with type “Singlestat.” To show the current temperature configure the panel with the same Metric settings as our chart above. However, that will only configure the panel to show the average of the temperatures for the provided time window. We really want the latest value to appear in this panel. To do that, configure the Options > Value > Stat of your panel to be “Current” instead of “Average.” This will select the latest value for display. You can even get fancy and set a custom range for your temperatures to render different colors depending on whether it is hot, medium, or cold out. The accuracy of these range values will likely depend on where you are pulling weather from.
A couple more panels like these and you will have a dashboard that looks something like this (I cheated a bit and let mine build up some data for about a week).
Grafana has a ton more configuration options, all of which are well described in detail on their documentation pages. Play around, connect other data sources and let your data build up for a while to build some even more interesting visuals. And, of course, with nio you have an essentially infinite number of data sets to bring in and visualize.
Happy graphing!
About the Author
