At Crimson Macaw , we recently took on an interesting project with Manchester Airport Group (MAG). We were to create and implement real-time dashboards for Airport Security Operations at Stansted Airport.
This would give the control room and security staff a better view of the passenger flow and security performance, allowing them to quickly make decisions based on real-time data. Data needed to be ingested from a number of on-premises systems and external data sources and then visualised on a number of large screens.
Data Ingestion ChallengesAfter deciding on Elasticsearch as the data storage layer, we needed to determine what data would be ingested and how we would do it. The information was available from a variety of sources. On-premises database systems, files frequently put into AWS S3 Buckets, as well from external API data sources. An example being National Rail data, which we loaded into Elasticsearch using the STOMP (Streaming Text Oriented Messaging Protocol) interfaces.
This in itself presented some initial challenges that we had to overcome:
Polling for data from databases had to be more frequent than 1 minute Incoming data from STOMP was gzip compressed Polling Databases More Frequently than Once per MinuteWe addressed the first problem with a simple patch to the existinglogstash-input-jdbc plugin. Before the patch, schedules could only be expressed in cron format. It used a Job Scheduler known as rufus-scheduler, which supported supplying schedule expressions as either cron or number of seconds. The only change was single line of code to use the repeat method instead of cron:
if @schedule @scheduler = Rufus :: Scheduler . new (: max_work_threads => 1 ) @scheduler . cron @schedule do @scheduler . repeat @schedule do execute_query ( queue ) end
To be able to make use of this, our patch is available on GitHub .
Handling Gzip Compressed MessagesIn order to handle compressed messages arriving from STOMP interfaces, we had to decompress them to filter the data using Logstash. Although there is an existing codec for reading lines of data from a gzip compressed message, in our use case the gzip decompressed messages were multi-line XML. To overcome this, we created our own plugin ― logstash-codec-gzip ― which we’ve made available on GitHub.
Visualising the Data using CanvasWe started to use Kibana to visualise the data that was available in Elasticsearch hosted in Elastic Cloud, but we couldn’t quite do what we wanted. It felt like a more granular level of control was needed. We came across Canvas by chance, when the Head of BI from Manchester Airport Group and I were at the AWS Summit at London May 2018. We spoke to the folks at the Elastic stand who mentioned Canvas and how they felt it was a better fit for what we wanted to achieve.
“We’re having difficulty in trying to visualise our data in Kibana as it doesn’t give the low-level control that we need.”
“Have you seen Canvas?”
“No, what’s that?”
“Our new visualisation tool; it’s in technology preview at the moment, but it sounds like it might give you what you want”
Installing Canvas and First ImpressionsCanvas was available as a plugin for Kibana and installed using the normal process of adding any Kibana plugin. Since Canvas was not available on Elastic Cloud during the technology preview phase,we hosted Kibana in the Manchester Airport Group AWS environment ― fully scripted, using Terraform.
Canvas has a nice simple expression language for controlling how each element is visualised. It reminds me of shell programming where the output of one command is piped into the next and sub-expressions are achieved by declaring an expression inside braces.
My first attempt at a Canvas Dashboard for trains arrivals (see below for the current version)
After a couple of days of familiarising ourselves with Canvas, we were able to visualise the train arrivals into Stansted Airport. This dashboard was created using the Markdown element for the text, Image Repeat for the icons and images in the footer.
Things were looking great and we had control of the individual elements, but there was still something missing. For our use case, we needed more control to provide:
Format timestamps based on a time zone data was being stored as UTC, but the time needed to adjust based on daylight saving Change the colour of the text and/or images based on known thresholds Expanding CanvasWriting plugins for Canvas is very similar to writing plugins for Kibana. They are written in NodeJS and utilise registries which allow you to add additional functions and even new elements that can be selected inside the Canvas UI. We created 3 plugins to achieve the level of control needed.
Using Time Zones when Formatting TimestampsThis was the first plugin we wrote, it’s a very simple expansion of the built-in formatdate :
import moment from 'moment'; import 'moment-timezone/builds/moment-timezone-with-data'; export const formatdatetz = () => ({ name: 'formatdatetz', type: 'string', help: 'Output a ms since epoch number as a formatted string according to a given timezone', context: { types: ['number'], }, args: { _: { types: ['string'], help: 'MomentJS Format with which to bucket (See https://momentjs.com/docs/#/displaying/)', required: true }, timezone: { types: ['string'], help: 'The timezone', required: true, default: 'UTC' } }, fn: (context, args) => { if (!args._) return moment.utc(new Date(context)).tz(args.timezone).toISOString(); return moment.utc(new Date(context)).tz(args.timezone).format(args._); }, });The formatdatetz plugin is available in GitHub for you to install on top of Canvas. Installation is as simple as:
./bin/kibana-plugin install https://github.com/crimsonmacaw/nodejs-canvas-plugin-formatdatetz/releases/download/v1.0.2/canvas-plugin-formatdatetz-1.0.2.zip Controlling Colour of Text and ImagesWe couldn’t use Markdown to control the style of text and images (though you can provide cascade style sheets within canvas, Markdown syntax does not have support for setting HTML class attributes for which style to apply).
The simplest approach was for us to create a plugin that would allow direct coding of HTML and in a similar way to the Markdown element, allow handlebars expressions for data binding. As SVG images can be embedded directly inside HTML, we could apply the same level of control to dynamically change the images based on data retrieved from Elasticsearch.
Creating the VisualisationsOnce we had the plugins we wanted, having the control to create something simple as a table with different colours on the rows was now possible.
The first pass design for the secur
