Time Series Storage Service

This service provides a storage layer dedicated for metric data produced and evolving over time. As example, Tsdb is the storage of choice for tracking devices or end-user clients state.

Metrics are associated with a timestamp and optional tags which can be later used later query purpose. Moreover, the Tsdb service provides a robust aggregation system enabling data-mining capabilities.

Operations

Storage

Events

Operations

Storage


deleteAll

delete all data of a given solution

Responses
All data deleted - nil

All data deleted

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Delete all data of a given solution
local out = Tsdb.deleteAll()
response.message = out

export

Start an export job.
An event of TSDB export will be triggered once the job finished in any state.
You can define a 'tsdb' event handler in Lua for this 'exportJob' event in your solution.

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
query object Only normal query arguments are allowed, which means sampling_size and aggregate arguments will be rejected.
If limit argument is not specified, it assumes to export all resulting data.
Note that timestamp is always returned in numeric format.
filename string File name of export CSV file. The ".csv" extension is not required inside the name.
Responses
Job successfully started - nil

Job successfully started

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Query constraints for export
local metrics = {
  "temperature",
  "humidity",
  "switch"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local query = {
  metrics = metrics,
  tags = tags
}

-- Start a new export job
local job_id = Tsdb.export({
  query = query,
  filename = "export_mlps_20170321"
})
response.message = job_id

exportJobInfo

Query the information of an export job, including status.

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
job_id ^[a-zA-Z0-9]+$ Job id
Responses
Job info successfully returned - nil

Job info successfully returned

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
local job_info = Tsdb.exportJobInfo({
  job_id = "xxyyzz"
})
response.message = job_info

exportJobList

List export job records of a given solution in descending timestamp order

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
limit integer Limit the number of results to return (default: 100, maximum allowed: 1000)
Responses
Job successfully listed - nil

Job successfully listed

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
local job_list = Tsdb.exportJobList()
response.message = job_list

import

Start an import job with a header row defined in the CSV file
Header line contains many column definitions.
A column is defined by column_name|column_type|data_type
More details are given below.
Here are column types used to annotate a column:

  • t means tag
  • m means metric
  • ts means timestamp
  • mn means metric name in a pair
  • mv means metric value in a pair
    Here are data types supported by different columns:
  • str: tag, metric, metric name, metric value
  • int: metric, metric value
  • sec: timestamp in second as integer
  • ms: timestamp in millisecond as integer
  • us: timestamp in microsecond as integer
  • float: metric, metric value
    Combined the two, you can start to write your column definitions in the
    first line of CSV. The formula is:
    header line := column_def_1,column_def_2,column_def_3,....,column_def_n
    column_def := column_name|column_type|data_type
    The default data type for some kind of columns are listed below. For those columns,
    the data type can be ommitted in it's column definition.
  • timestamp: sec
  • tag: str
  • metric name: str
    Finally, there is an living example of how to use this kind of annotation to represent
    a data set in csv format.
    Given the CSV file:
    timestamp|ts,weather|m|str,temprature|m|float,city|t,max_or_min_pair|mn,max_or_min_pair|mv|float
    12345,cold,15.4,Taipei,lowest,12.4
    12344,warm,23.7,Tainan,highest,25.3
    That will be transformed to two write to Tsdb.write:
    Tsdb.write({
    metrics = {
    weather = "cold",
    temprature = 15.4,
    lowest = 12.4
    }},
    tags = {
    city = "Taipei"
    }},
    ts = "12345000000"
    })
    Tsdb.write({
    metrics = {
    weather = "warm",
    temprature = 23.7,
    highest = 25.3
    }},
    tags = {
    city = "Tainan"
    }},
    ts = "12344000000"
    })

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
url string The url for a CSV file
Responses
Job successfully started - nil

Job successfully started

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
local job_id = Tsdb.import({
  url = "http://example.com/a_sample_file"
})
response.message = job_id

importJobInfo

query the status of an import job

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
job_id ^[a-zA-Z0-9]+$ Job id
Responses
Job info successfully returned - nil

Job info successfully returned

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
local job_info = Tsdb.importJobInfo({
  job_id = "2345"
})
response.message = job_id

importJobList

list all job ids started by a given solution id

Responses
Job successfully listed - nil

Job successfully listed

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
local job_list = Tsdb.importJobList()
response.message = job_list

listMetrics

list metrics of a given solution

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
limit integer {..1000} Limit the number of results to return (default: maximum allowed, maximum allowed: 1000)
Default: 1000
next string Optional, the cursor to get next page if still having more data
Responses
Metrics information retrieved - nil

Metrics information retrieved

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Get a list of created metrics for a given solution
local out = Tsdb.listMetrics({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listMetrics({limit = 10, next = "switch_2"})
response.message = out

listTags

list tags of a given solution

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
limit integer {..1000} Limit the number of results to return (default: maximum allowed, maximum allowed: 1000)
Default: 1000
next string Optional, the cursor to get next page if still having more data
Responses
Tags information retrieved - nil

Tags information retrieved

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Get a list of created tags for a given solution
local out = Tsdb.listTags({limit = 10})
response.message = out

-- Use next cursor to fetch next page if found in the result of previous query
local out = Tsdb.listTags({limit = 10, next = "city_taipei_1_1_true"})
response.message = out

query

Query data points by using any metrics and tags. Support absolute (start_time, end_time) or relative (relative_start, relative_unit) time parameters, which cannot be used at the same time.
The first element in returned data point array is always the timestamp.
You can use fill arguement to indicate the imputation of missing values.
The metric names of columns property in the response will always be in the order specified in the query, except for the timestamp column which is always the first one.

If no time constraints are specified, it will return recent data points up to the maximum limit in recent one week.

Note that only unique timestamped data will be returned, which means if multiple data points were written with exactly the same timestamp in the response, only the last one will be kept.

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
fill integer, string Value to fill for time slots with no data points exist in merge mode.
Supported fill types: null, none, any integer, previous
Default: none
mode string Indicate whether to merge or split the result of each metric.
Supported options: merge, split
Default: merge
tags object One or many tags
epoch string Change returned timestamp of data points to unix epoch format.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
Optional, if not provided, timestamps are returned in RFC3339 UTC with microsecond precision.
limit integer {..1000} Limit the number of data points to return per metric (default: maximum allowed, maximum allowed: 1000)
Default: 1000
metrics [ string ] One or many metrics
end_time integer, string Exclusive UTC ending time range to query, also accept RFC3339 UTC string.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546, 2016-08-30T08:59:06+00:00
Optional, if not provided, it will use current timestamp in microseonds from server side
order_by string Return results in ascending or descending time order.
Supported options: desc, asc
Default: "desc"
aggregate [ string ] One to many aggregation functions to apply.
Supported functions: avg, min, max, count, sum
String type value can only use count function
start_time integer, string Inclusive UTC starting time range to query, also accept RFC3339 UTC string.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546, 2016-08-30T08:59:06+00:00
relative_end integer, string A negative integer with time unit to indicate relative end time before now.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
sampling_size string The size of time slots used for downsampling. Must be a positive integer greater than zero.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
Optional, used together with fill arguments.
relative_start integer, string A negative integer with time unit to indicate relative start time before now.
Supported units: u (microseconds), ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks)
Default: -7d (last 7 days)
Responses
Query results retrieved - nil

Query results retrieved

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Query by Absolute Time Constraint --
-- Get temperature and humidity data points between 2016-08-01 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "swtich"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00+00:00",
  end_time = "2016-09-01T00:00:00+00:00",
  fill = "null",
  limit = 50
})
response.message = out

-- Query by Relative Time Constraint --
-- Get temperature data points in recent 3 hours from devices in Taipei city and Asia region (with timestamp in milliseconds format)
local metrics = {"temperature"}
local tags = {
  region = "asia",
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-3d",
  epoch = "ms",
  fill = "null",
  limit = 50
})
response.message = out

-- Query without Time Constraint --
-- Get most recent 5 temperature and humidity data points from devices in Minneapolis city and US region
local metrics = {
  "temperature",
  "humidity"
}
local tags = {
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  limit = 5
})
response.message = out

-- Query by Downsampling --
-- Get humidity data points in recent two days from devices in Taipei city, downsampled by 4-hours time slots
local metrics = {
  "humidity"
}
local tags = {
  city = "taipei"
}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  relative_start = "-2d",
  sampling_size = "4h",
  fill = "none",
  epoch = "ms"
})
response.message = out

-- Aggregation by Downsampling --
-- Get average and count of tire pressure data between 2016-08-15 (inclusive) and 2016-09-01 (exclusive) from devices in Minneapolis city, downsampled by 30-minutes time slots
local metrics = {
  "tire_pressure"
}
local tags = {
  city = "minneapolis"
}
local aggregate = {"avg", "count"}
local out = Tsdb.query({
  metrics = metrics,
  tags = tags,
  start_time = "2016-08-01T00:00:00+00:00",
  end_time = "2016-09-01T00:00:00+00:00",
  aggregate = aggregate,
  sampling_size = "30m",
  fill = "none"
})
response.message = out

recent

Get the most recent data point of a particular set of metrics and tag values.

If you want to use advanced metric query, you need to specify an inner Lua table as an element
inside the Lua table of metrics parameter, i.e. metrics = {"m1","m2",{"m3","m4"}}
where {"m3","m4"} is an advanced metric query that query the most recent data
of m3 and also report the value of m4 that written together with m3.

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
metrics [ object ] One or many metrics
tag_name string Tag name
tag_values [ integer, string ] One or many tag values
Responses
Operation successfully returned - nil

Operation successfully returned

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Get latest data of metric vibration and humidity from devices with tag sn=123 or sn=456
local out = Tsdb.recent({
  metrics = {"vibration","humidity"},
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

-- Get latest data of metric warning and critical from devices with tag sn=123 or sn=456
-- Together with the corresponding text for warning and critical metrics
local out = Tsdb.recent({
  metrics = {{"warning", "text"}, {"critical", "text"}},
  tag_name = "sn",
  tag_values = {"123", "456"}
})
response.message = out

write

Write data points to one or many metrics with an optional set of tags and a timestamp down to microsecond precision.

Note that if multiple data points are written with exactly the same timestamp, only the last one will be kept and it overwrites the others.

Each metric value has a limited size of 500KB, a write request will be rejected without partial writes if exceeding the limit.

If succeeds, it returns a json of write timestamp in microseconds.

Arguments
parameters - object - Object containing service call parameters.
Name Type Description
ts integer, string Unix timestamp in microseconds used as the write time for given data point.
Supported units: u (microseconds), ms (milliseconds), s (seconds)
e.g. 1472547546000000u, 1472547546000ms, 1472547546s, 1472547546
Optional, if not provided, it will use the received time in microseconds from server side
tags object Pairs of tag and its tag value (only text supported).
Maximum number of tags in a single write: 20
metrics [ string ] List of metrics.
Maximum number of metrics in a single write: 100
return_ts boolean Whether to return write timestamp in the response
Responses
The response of write - object
Name Type Description
write_timestamp string The timestamp of data point written to TSDB (in microseconds)
Data successfully inserted - nil

Data successfully inserted

The response to the caller - object
Name Type Description
error string Error Message in case of failure
result object Result message
Example
-- Write data points of metrics with tags
-- If timestamp is not provided, it will use the received time in microseconds from server side
local metrics = {
  temperature = 37.2,
  humidity = 73,
  switch = "on"
}
local tags = {
  pid = "pzomp8vn4twklnmi",
  identity = "000001",
  region = "us",
  city = "minneapolis"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags
})
response.message = out

-- Write data points of metrics with tags and timestamp
local metrics = {
  temperature = 37.2,
  humidity = 73
}
local tags = {
  identity = "000002"
}
local out = Tsdb.write({
  metrics = metrics,
  tags = tags,
  ts = "1476243965s"
})
response.message = out

Events


exportJob

An event message sent by TSDB for the callback of TSDB jobs (for example, export jobs).

Arguments
job - object - The argument for export job event
Name Type Description
error string Error message if job failed
query object Query arguments
state string State of the job (in-progress, completed or failed)
job_id string Job ID
length string The total length of export file in bytes
filename string File name of the exported CSV file
content_id string Content ID of the job to Content service
start_time string Start time of the job
update_time string Last updated time of the job
Example
function handle_tsdb_exportJob (job)

 -- Your logic comes here 

end