How to Use Keen to Analyze IoT Devices Over Time

This is a repost, originally published for the Keen.io company blog, Radical Transparency.

In IoT applications, a common scenario is trying to understand the state of your things over time. For example, you might ask questions like:

How long was this parking space unoccupied last week?
Which truck in my fleet was in service the longest?
How long was this machine in power-saving mode?
What are the 5 best and worst performers in this group?

If you are only pushing event data to Keen based on the changes in state of the thing (as is the common model for asset tracking/management-type scenarios), you won’t have enough information to ask these types of questions since you need to know how long the thing has been in each state. This takes a little bit of shifting of how you might normally architect your service since you normally create Keen events as events are happening. But in this case, when an event comes in:

you need to cache the timestamp and state the thing is going into, and
create an event based on the previous cached state that was just transitioned out of, which must include the “duration” of that state.

Once this is done, Keen really shines at the rest! You can simply do a “sum” query on the durations of events, filtering by groups of devices and timeframes.

The below snippet will tell you how long a parking space was occupied:

var timeOccupied = new Keen.Query("sum", {
  event_collection: "deviceUpdates",
  target_property: "duration",
  timeframe: "this_7_days",
  filters: [ 
    { operator: "eq",
      property_name: "hardwareId",
      property_value: hardwareId
    },
    { 
      operator: "eq",
      property_name: "deviceState",
      property_value: "occupied"
    }
  ]
});

If you want to sum all of the parking spots on the street, give each event a “streetId” and filter by that instead of “hardwareId”.

The below snippet will tell you how many parking spaces were occupied longer than an hour (because street parking is limited to one hour and you want to know where the most violations are occurring):

var violationsOccurred = new Keen.Query("count", {
  event_collection: "deviceUpdates",
  target_property: "duration",
  timeframe: "this_7_days",
  filters: [ 
    { operator: "gt",
      property_name: "duration",
      property_value: 60
    },
    { 
      operator: "eq",
      property_name: "deviceState",
      property_value: "occupied"
    }
  ]
});

I could do this all day! That’s because once you have this sort of infrastructure in place, the sky really is the limit on the types of high-value information you can extract. And you did this all without managing any database infrastructure or API surface of your own?!

So, how do we implement the complete system? Here Keen can use a little help from Scriptr.io. Using Scriptr.io’s fast local storage module and Keen connector, we can do some caching and light processing on that ‘in-flight’ datastream in a simple and expressive way that ALSO requires no devops/infrastructure! A match made in #NoDevOps heaven. It would look like this:

var eventData = JSON.parse(request.rawBody); // Any POST body to the above URL can be access with the 'request' object
var lastEventData = storage.local[eventData.hardwareId]; // The 'storage' object is a key/value store which we access with the current device's ID

var eventDuration = eventData.timestamp - lastEventData.timestamp; // In this example, we'll assume these are epoch times, otherwise we'd convert
lastEventData.eventDuration = eventDuration; // Add the duration to the last event data object which we'll push to Keen

var keenModule = require('../../modules/keenio/keenioclient'); // This is the Scriptr.io -> Keen.io connector
var keen = new keenModule.Keenio("my_Keen_credentials");

// Next, record the Keen event
keen.recordEvent({
  collection: "deviceUpdates",
  data: lastEventData
});

storage.local[eventData.hardwareId] = eventData; // cache the current event by the device's ID