Connect AWS ElasticSearch as PDaaS application

In this tutorial we will see how to create an Elasticsearch instance on AWS and configure it as an external application in our PDaaS instance. Events from our devices will be sent to Elasticsearch for indexing and we can use Kibana to create dashboards and visualized our data.

NOTE: PDaaS is the short for IoT-Connector (aka Product) As-a-Service

pdaas-aws-app-flow

Table of Contents

  1. Setup AWS application
  2. Connect External Application
  3. Create Kibana visualization

Requirements:


Setup AWS application

To simplify the setup of the required AWS services we have prepared an AWS CloudFormation template.

  1. Copy and save the CloudFormation Template 'api-lambda-es.yaml' on your computer

  2. Sign in to your AWS account at https://console.aws.amazon.com

  3. Go to the CloudFormation service page

aws-menu

  1. Click Create stack

aws-cf-create

  1. Choose Upload a template file and upload api-lambda-es.yaml from step 1

aws-cf-create-opts

  1. Configure template parameters
    1. Give a cloudformation stack name, ex: pdaas-demo-app
    2. Go to https://www.google.com/search?q=what+is+my+ip to get your current public IP and add it to AllowedIPs
    3. Replace the Token field with random value and remember it as we will use it later.
    4. (Optional) Configure elasticsearch setup
    5. Click Next

aws-cf-create-opts-2

  1. No need for custom stack options so can click Next again

aws-cf-create-opts-3

  1. We have to acknowledge that the template will create an IAM role (used to give Lambda permission to Elasticsearch) and then click Create stack

aws-cf-create-btn

  1. AWS CloudFormation will create and configure our services, it will take ~15 minutes to complete.

  2. Once complete we can go to the Outputs tab to see the relevant URLs created

aws-cf-outputs


Connect External Application

  1. In the PDaaS Instance, we can click the "hamburger" menu button in the top left corner to access the application page.

pdaas-devices

pdaas-menu-apps

  1. Once in the application page we an click CONNECT APP to configure our Elasticsearch application.

pdaas-apps-create-btn

  1. We will use the following configuration
    1. Give an appropriate name, ex: AWS Elasticsearch
    2. Select External HTTPS application type
    3. For HTTPS callback URL we can copy the LambdaAPI URL from our CloudFormation Outputs tab
    4. Choose token as Authentication type
    5. Leave Token prefix empty, and input the token used to create the AWS Elasticsearch stack
    6. Click CONNECT APP to finalize the setup

pdaas-apps-new-ext

After adding an application we will see a popup with a token. This token can be used for external applications to interface with some of the available PDaaS API's such as list devices.

Our Elasticsearch instance does not use such functionality so we can just close this popup.

pdaas-apps-new-token

If there are devices connected and sending events to our PDaaS application we should now see a green-mark indicating that our Elasticsearch app has received and successfully acknowledged device events.

pdaas-apps-ext-ok

In application details, we can also see the last callback status.

pdaas-apps-details-ext-ok


Create Kibana visualization

We can now create visualizations for our device events using Kibana. In the application of this example there are two devices connected, device001 and device002. These two devices are sending a number to a resource called temp. We will create a bar graph for this resource grouped by device identity.

  1. Go to the Kibana UI by clicking the Kibana URL in the AWS CloudFormation Outputs tab

  2. We are then greeted by the Kibana setup page, here we will click Explore on my own

kibana-welcome

  1. On the Kibana main page we can click Connect to your Elasticsearch Index

kibana-add-data

  1. Indexes are automatically created when device events are successfully reaching our Elasticsearch. So if we have devices sending data we should see a pdaas-yyy-mm-dd index created. We will add a wildcard pattern for pdaas*

kibana-create-index

  1. Next we need to select our timestamp field @timestamp and finally click Create index pattern

kibana-create-index-2

  1. We can now see our index pattern and all available fields

kibana-index

  1. Next we can click the Visualize tab

kibana-menu-visualize

  1. Click Create a visualization and select Visual Builder

kibana-new-visualization

  1. Change Aggregation to average and field to our data.resources.temp and set Group by to Terms and By to data.identity.keyword

kibana-visualization-metrics

  1. Go to Options tab next to Metrics and change Chart type to bar, Stacked to Stacked and Split color theme to Rainbow

kibana-visualization-options

  1. We now have our stacked bar graph over temp split by device identity with live data from our devices kibana-visualization