Time Series Storage Service

Write, record, and query device data in our Timeseries database optimized for IoT data.

TSDB (time series database) service provides a storage layer dedicated for metric data produced and evolving over time. For example, TSDB can be used to store periodic sensor data sent from your devices.

Data is associated with a timestamp and optional tags which can be used later for query purposes. Finally, the TSDB service provides a robust aggregation system enabling data-mining capabilities.

Operations

Storage

Events

Operations

Storage


deleteAll

delete all data of a given solution

Responses

All data deleted

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Delete all data of a given solution
local out = Tsdb.deleteAll()
response.message = out

export

Start an export job.
An event of TSDB export will be triggered once the job finished in any state.
You can define a 'tsdb' event handler in Lua for this 'exportJob' event in your solution.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
query object Only normal query arguments are allowed, which means sampling_size and aggregate arguments will be rejected.
If limit argument is not specified, it assumes to export all resulting data.
If start_time/relative_start is not specified, it assumes to export from the oldest data.
Note that timestamp is always returned in numeric format.
format object The data format rules. Property name should be the field name which are the metrics name, "timestamp" and "tags".
filename string File name of export CSV file. The ".csv" extension is not required inside the name. (Space is not allowed in filename)

Responses

Job successfully started

Content: object The information of job ID

Name Type Description
job_id string Job ID
Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Query constraints for export
local metrics = {
  "temperature",
  "humidity",
  "switch",
  "host"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local query = {
  metrics = metrics,
  tags = tags
}
local format = {
  temperature = {
    {round=3},
    {rename="Temp"}
  },
  timestamp = {
    {datetime=-5}
  },
  tags = {
    {normalize={"city", "region"}}
  },
  host = {
    {replace={
      match="/(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}):(\\d{1,5})/",
      to="Port: \\2 and IP: \\1"
    }}
  }
} -- Start a new export job
local job_id = Tsdb.export({
  query = query,
  filename = "export_mlps_20170321",
  format = format
})
response.message = job_id

exportJobInfo

Query the information of an export job, including status.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
job_id ^[a-zA-Z0-9]+$ Job id

Responses

Job info successfully returned

Content: object The information for export job

Name Type Description
error string Error message if job failed
query object Query arguments
state string State of the job (enqueued, expired, in-progress, completed or failed)
format object The data format rules. Property name should be the field name which are the metrics name, "timestamp" and "tags".
job_id string Job ID
length string The total length of export file in bytes
filename string File name of the exported CSV file
content_id string Content ID of the job to Content service
context_id string Solution id
start_time string Start time of the job
update_time string Last updated time of the job
Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

local job_info = Tsdb.exportJobInfo({
  job_id = "xxyyzz"
})
response.message = job_info

exportJobList

List export job records of a given solution in descending timestamp order

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
limit integer Limit the number of results to return (default: 100, maximum allowed: 1000)

Responses

List of export job

Content: [ object ]

Name Type Description
state string State of the job (enqueued, expired, in-progress, completed or failed)
job_id string Job ID
start_time string Start time of the job
Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

local job_list = Tsdb.exportJobList()
response.message = job_list

import

Start an import job with a header row defined in the CSV file
Header line contains many column definitions.
A column is defined by column_name|column_type|data_type
More details are given below.
Here are column types used to annotate a column:

  • t means tag
  • m means metric
  • ts means timestamp
  • mn means metric name in a pair
  • mv means metric value in a pair
    Here are data types supported by different columns:
  • str: tag, metric, metric name, metric value
  • int: metric, metric value
  • sec: timestamp in second as integer
  • ms: timestamp in millisecond as integer
  • us: timestamp in microsecond as integer
  • float: metric, metric value
    Combined the two, you can start to write your column definitions in the
    first line of CSV. The formula is:
    header line := column_def_1,column_def_2,column_def_3,....,column_def_n
    column_def := column_name|column_type|data_type
    The default data type for some kind of columns are listed below. For those columns,
    the data type can be ommitted in it's column definition.
  • timestamp: sec
  • tag: str
  • metric name: str
    Finally, there is an living example of how to use this kind of annotation to represent
    a data set in csv format.
    Given the CSV file:
    timestamp|ts,weather|m|str,temprature|m|float,city|t,max_or_min_pair|mn,max_or_min_pair|mv|float
    12345,cold,15.4,Taipei,lowest,12.4
    12344,warm,23.7,Tainan,highest,25.3
    That will be transformed to two write to Tsdb.write:
    Tsdb.write({
    metrics = {
    weather = "cold",
    temprature = 15.4,
    lowest = 12.4
    }},
    tags = {
    city = "Taipei"
    }},
    ts = "12345000000"
    })
    Tsdb.write({
    metrics = {
    weather = "warm",
    temprature = 23.7,
    highest = 25.3
    }},
    tags = {
    city = "Tainan"
    }},
    ts = "12344000000"
    })

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
url string The url for a CSV file

Responses

Job successfully started

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

local job_id = Tsdb.import({
  url = "http://example.com/a_sample_file"
})
response.message = job_id

importJobInfo

query the status of an import job

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
job_id ^[a-zA-Z0-9]+$ Job id

Responses

Job info successfully returned

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

local job_info = Tsdb.importJobInfo({
  job_id = "2345"
})
response.message = job_id

importJobList

list all job ids started by a given solution id

Responses

Job successfully listed

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

local job_list = Tsdb.importJobList()
response.message = job_list

listMetrics

list metrics of a given solution

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
limit integer {..1000} Limit the number of results to return (default: maximum allowed, maximum allowed: 1000)
Default: 1000
next string Optional, the cursor to get next page if still having more data

Responses

Metrics information retrieved

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Get a list of created metrics for a given solution
local out = Tsdb.listMetrics({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listMetrics({limit = 10, next = "switch_2"})
response.message = out

listTags

list tags of a given solution

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
limit integer {..1000} Limit the number of results to return (default: maximum allowed, maximum allowed: 1000)
Default: 1000
next string Optional, the cursor to get next page if still having more data

Responses

Tags information retrieved

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Get a list of created tags for a given solution
local out = Tsdb.listTags({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listTags({limit = 10, next = "city_taipei_1_1_true"})
response.message = out

multiWrite

Write data points to one or many metrics with an optional set of tags and a timestamp down to microsecond precision.

Note that if multiple data points are written with exactly the same timestamp, only the last one will be kept and it overwrites the others.

Each metric value has a limited size which depends on the number of tags. (Number of tags + 1) multiplies the size of metric value can't over 480KB. A write request will be rejected without partial writes if exceeding the limit. If any of data points are invalid, request will be rejected without partial writes.

To prevent from a synchronized service call take too long time to response, there are some limitations. Total number of data entries in multiple write at most 2,000 (Reference to the limit of single write). Number of data entries per datapoint: "Number of metrics * (Number of tag pairs) + 1"

If succeeds, it turns a list string of write timestamp in microseconds.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
return_ts boolean Whether to return write timestamp in the response
datapoints [ object ] List of data points
datapoints[].ts integer, string Unix timestamp in microseconds used as the write time for given data point.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546
Optional, if not provided, it will use the received time in microseconds from server side
datapoints[].tags object Pairs of tag and its tag value (only text supported).
Maximum number of tags in a single write: 20
datapoints[].metrics object Pairs of metric name and its value.
Maximum number of metrics in a single write: 100

Responses

Data successfully inserted

Content: [ object ] List of result of each data point.

Name Type Description
write_timestamp string The timestamp of data point written to TSDB (in microseconds)
Data successfully inserted

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Write multiple datapoints of metrics with tags
-- If timestamp is not provided, it will use the received time in microseconds from server side
local metrics1 = {
  temperature = 37.2,
  humidity = 73,
  switch = "on"
}
local metrics2 = {
  temperature = 31.2,
  humidity = 55,
  switch = "off"
}
local tags1 = {
  pid = "pzomp8vn4twklnmi",
  identity = "000001",
  region = "us",
  city = "minneapolis"
}
local tags2 = {
  pid = "lvwpoj19hp7k0000",
  identity = "000002",
  region = "tw",
  city = "taipei"
}
local out = Tsdb.multiWrite({
  datapoints = {
    {
      metrics = metrics1,
      tags = tags1
    },
    {
      metrics = metrics2,
      tags = tags2
    }
  },
  return_ts = true
})
response.message = out

query

Query data points by using any metrics and tags. Support absolute (start_time, end_time) or relative (relative_start, relative_unit) time parameters, which cannot be used at the same time.
The first element in returned data point array is always the timestamp.
You can use fill arguement to indicate the imputation of missing values.
The metric names of columns property in the response will always be in the order specified in the query, except for the timestamp column which is always the first one.

If no time constraints are specified, it will return recent data points up to the maximum limit in recent one week.

Note that only unique timestamped data will be returned, which means if multiple data points were written with exactly the same timestamp in the response, only the last one will be kept.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
fill integer, string Value to fill for time slots with no data points exist.
If the query without sampling, it only works in merge mode.
Supported fill types:
- "": fill the slot with empty string "".
- "null": fill the slot with JSON null.
- "previous": fill the slot with previous value.
- "none": fill the slot with "none" string.
- any integer: fill the slot with the specified integer value.
- "~s:CUSTOM_STRING": fill the slot with the specified "CUSTOM_STRING" string.
Default: none
mode string Indicate whether to merge or split the result of each metric.
Supported options: merge, split
Default: merge
tags object One or many tags
epoch string Change returned timestamp of data points to unix epoch format.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
Optional, if not provided, timestamps are returned in RFC3339 UTC with microsecond precision.
limit integer {..1000} Limit the number of data points to return per metric (default: maximum allowed, maximum allowed: 1000)
Default: 1000
metrics [ string ] One or many metrics
end_time integer, string Exclusive UTC ending time range to query, also accept RFC3339 UTC string.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546, 2016-08-30T08:59:06+00:00
Optional, if not provided, it will use current timestamp in microseonds from server side
order_by string Return results in ascending or descending time order.
Supported options: desc, asc
Default: "desc"
aggregate [ string ] One to many aggregation functions to apply.
Supported functions: avg, min, max, count, sum
String type value can only use count function
start_time integer, string Inclusive UTC starting time range to query, also accept RFC3339 UTC string.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546, 2016-08-30T08:59:06+00:00
Optional, if not provided, it will be 7 days earlier than end_time.
relative_end integer, string A negative integer with time unit to indicate relative end time before now.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
sampling_size string The size of time slots used for downsampling. Must be a positive integer greater than zero.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
Optional, used together with fill arguments.
relative_start integer, string A negative integer with time unit to indicate relative start time before now.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
Default: -7d (last 7 days)

Responses

Query results retrieved

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Query by Absolute Time Constraint --
-- Get temperature and humidity data points between 2016-08-01 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "swtich"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00+00:00",
  end_time = "2016-09-01T00:00:00+00:00",
  fill = "null",
  limit = 50
})
response.message = out

-- Query by Relative Time Constraint --
-- Get temperature data points in recent 3 hours from devices in Taipei city and Asia region (with timestamp in milliseconds format)
local metrics = {"temperature"}
local tags = {
  region = "asia",
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-3d",
  epoch = "ms",
  fill = "null",
  limit = 50
})
response.message = out

-- Query without Time Constraint --
-- Get most recent 5 temperature and humidity data points from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "humidity"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  limit = 5
})
response.message = out

-- Query by Downsampling --
-- Get humidity data points in recent two days from devices in Taipei city, downsampled by 4-hours time slots
local metrics = {
  "humidity"
}
local tags = {
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-2d",
  sampling_size = "4h",
  fill = "none",
  epoch = "ms"
})
response.message = out

-- Aggregation by Downsampling --
-- Get average and count of tire pressure data between 2016-08-15 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city, downsampled by 30-minutes time slots
local metrics = {
  "tire_pressure"
}
local tags = {
  city = "minneapolis"
}
local aggregate = {"avg", "count"}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00+00:00",
  end_time = "2016-09-01T00:00:00+00:00",
  aggregate = aggregate,
  sampling_size = "30m",
  fill = "none"
})
response.message = out
-- Query by Fill in Custom String --
-- To fill in the empty time slots with "Empty" string
local metrics = {
  "temperature",
  "swtich"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  fill = "~s:Empty",
})
response.message = out

recent

Get the most recent data point of a particular set of metrics and tag values.

If you want to use advanced metric query, you need to specify an inner Lua table as an element
inside the Lua table of metrics parameter, i.e. metrics = {"m1","m2",{"m3","m4"}}
where {"m3","m4"} is an advanced metric query that query the most recent data
of m3 and also report the value of m4 that written together with m3.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
metrics [ object ] One or many metrics
tag_name string Tag name
tag_values [ integer, string ] One or many tag values

Responses

Operation successfully returned

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Get latest data of metric vibration and humidity from devices with tag sn=123 or sn=456
local out = Tsdb.recent({
  metrics = {"vibration","humidity"},
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

-- Get latest data of metric warning and critical from devices with tag sn=123 or sn=456
-- Together with the corresponding text for warning and critical metrics
local out = Tsdb.recent({
  metrics = {{"warning", "text"}, {"critical", "text"}},
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

write

Write data point to one or many metrics with an optional set of tags and a timestamp down to microsecond precision.

Note that if multiple data points are written with exactly the same timestamp, only the last one will be kept and it overwrites the others.

Each metric value has a limited size which depends on the number of tags. (Number of tags + 1) multiplies the size of metric value can't over 480KB. A write request will be rejected without partial writes if exceeding the limit.

If succeeds, it returns a json of write timestamp in microseconds.

Arguments

parameters - object - Object containing service call parameters.
Name Type Description
ts integer, string Unix timestamp in microseconds used as the write time for given data point.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546
Optional, if not provided, it will use the received time in microseconds from server side
tags object Pairs of tag and its tag value (only text supported).
Maximum number of tags in a single write: 20
metrics object Pairs of metric name and its value.
Maximum number of metrics in a single write: 100
return_ts boolean Whether to return write timestamp in the response

Responses

Data successfully inserted

Content: object The response of write

Name Type Description
write_timestamp string The timestamp of data point written to TSDB (in microseconds)
Data successfully inserted

Content: nil

Error

Content: object The response to the caller

Name Type Description
error string Error Message in case of failure
result object Result message

Example

-- Write data point of metrics with tags
-- If timestamp is not provided, it will use the received time in microseconds from server side
local metrics = {
  temperature = 37.2,
  humidity = 73,
  switch = "on",
  host = "8.168.1.24:443"
}
local tags = {
  pid = "pzomp8vn4twklnmi",
  identity = "000001",
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags
})
response.message = out

-- Write data points of metrics with tags and timestamp
local metrics = {
  temperature = 37.2,
  humidity = 73
}
local tags = {
  identity = "000002"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags,
  ts = "1476243965s"
})
response.message = out

Events


exportJob

An event message containing the export task result.

Arguments

job - object - The information for export job
Name Type Description
error string Error message if job failed
query object Query arguments
state string State of the job (enqueued, expired, in-progress, completed or failed)
format object The data format rules. Property name should be the field name which are the metrics name, "timestamp" and "tags".
job_id string Job ID
length string The total length of export file in bytes
filename string File name of the exported CSV file
content_id string Content ID of the job to Content service
context_id string Solution id
start_time string Start time of the job
update_time string Last updated time of the job

Example

function handle_tsdb_exportJob (job)

 -- Your logic comes here 

end