The Policy Language

A policy is a short text file. It names the streams of events you have and the answers to keep ready, and says how long each level of detail survives.

The compiler turns it into plain SQLite, and the Engine runs it as it is. This post walks through a policy line by line, and shows what each line becomes in the file.

A Stream

A stream is a sequence of events that share the same fields. Every event has a time, ts, in seconds since 1970, plus the keys it is grouped by and the numbers it measures:

stream trades {
  key    symbol text
  value  price real
  value  size  integer
  derive notional = price * size
}

key fields group events, as text or integers. value fields are measured numbers. derive computes a number from other values of the same event, and it is summarised like any value. To send an event, insert it into the stream’s name as if it were a table:

INSERT INTO trades (ts, symbol, price, size) VALUES (1790602200, 'SIM1', 187.41, 100);

Levels of Detail

The lines inside a stream say what to keep and for how long. Each one becomes a table or a few columns in the file.

  raw     keep 5m
  rollup  10s keep 24h
  rollup  1m  keep 30d  quantiles ms
  rollup  1h  keep 1y   quantiles ms

raw keep 5m keeps whole events for five minutes. Leave it out to keep none. Each rollup keeps one summary per key and window of that length, for as long as keep says; keep forever never deletes. A summary holds, for each value, the count, the sum, the sum of squares, the minimum, the maximum, the first and the last, and all of them merge exactly. quantiles ms adds a sketch of ms to every window of that rollup, so percentiles survive too, within 1% of the exact value unless the policy asks for more with quantile accuracy.

Kept Examples

Summaries say how much and how fast. Sometimes you want to see the events themselves:

  samples 3 per 1m
  anomalies ms log z > 4 keep 20 per 1m

samples 3 per 1m keeps three whole events from every minute, chosen so every event of the minute has the same chance. anomalies keeps unusual events whole. Each event is compared with a running baseline of its own key, a mean and a variance, and an event more than z standard deviations away is kept with its z-score. log judges the logarithm of the value, which suits latencies, prices and sizes that spread by ratios. keep 20 per 1m caps what a storm can write, while every unusual event is still counted in its window.

Two refinements make the baseline useful in practice. Unusual events do not move it, and no event moves it by more than 1.5 standard deviations, so an outage stays unusual while it lasts. And change judges each event by its step from the previous one of the same key, which is the right test for prices:

  anomalies price log change z > 6 memory 500 warmup 200 keep 5 per 1m

A price that jumps 8% in one trade is one unusual step, and the trades after it are judged from the new price.

Ready Answers

Precomputes are the answers kept ready. Each one is a line outside the stream:

precompute requests   = count(latency) by endpoint
precompute p99_ms     = p99(latency.ms) by endpoint
precompute day_volume = sum(trades.size) by symbol per day

The functions are count, sum, avg, min, max, first, last and any percentile, p50, p99 or p999. by groups the answer by keys, and per hour, per day or per month starts it again every period, in UTC. Each precompute becomes a view with its own name, so SELECT * FROM p99_ms returns one row per endpoint. Precomputes that share a stream, by and per share their stored state, so adding one costs little.

Exact Streams

Some numbers have to be right to the unit, such as tokens billed. An exact stream keeps every event until a deadline has passed, and adds four rules:

stream usage exact {
  id     request_id refuse repeats 7d
  key    customer text
  value  tokens integer

  late   48h
  period month close 24h
  raw    until closed + 90d
  rollup 1h keep 400d
}

id request_id refuse repeats 7d refuses an event whose id was counted in the last seven days, so a retried request counts once. late 48h refuses an event more than 48 hours older than the newest one counted. period month close 24h closes each month a day after it ends; from then on its totals cannot change, so an invoice sent on the 2nd stays true. raw until closed + 90d keeps every event whole until its month has been closed for 90 days, so any total can be checked event by event while it can still be disputed.

A refused event changes one thing only: a counter in usage_refused, by reason and key. So whatever was not counted is known exactly. “Newest” and “closed” go by the times on the events and never by the clock on the wall, so sending the same events again always gives the same file. Percentiles are approximate, so an exact stream does not allow them.

Quotas

precompute tokens_month = sum(usage.tokens) by customer per month
quota monthly_tokens = tokens_month

A quota puts a limit beside a precompute. The compiler adds a table for the limits and a view with used, lim, remaining and reached for every key and period. A gateway reads the view before it serves a request, one lookup by primary key, and turns the request away when reached is 1.

Streams from Logs

A stream can take its events from log lines instead of inserts:

logs {
  format "<timestamp> <level> <service> <message>"
}

stream web from logs where service = "web" {
  key    route text
  value  ms real
  rollup 1m keep 30d quantiles ms
}

The logs block says how a line is read. A line’s fields are the format’s fields, every key=value pair in the message, and template, the number of the line’s template, learned as lines arrive. The stream takes the lines whose fields match its where and that have every key and value it declares. The whole line is kept with raw events, samples and anomalies. The Logs demo makes such a policy from a dashboard.

What the Compiler Refuses

The compiler checks the whole policy before it writes any SQL, and it names the line and the column of every mistake:

usage.precompute:7:3: an exact stream keeps exact answers; quantile sketches are approximate
latency.precompute:4:3: samples per 1m needs a rollup 1m in the same stream
latency.precompute:2:7: key name "select" is a reserved word; pick another name
latency.precompute:6:1: stream "latency" has no value "ns"

Letting Detail Fade

Old detail is removed by the distill statements, which the compiler writes beside the SQL and also stores inside the file. Run them every few minutes, or let the Engine run them after each checkpoint. They delete raw events and windows that have outlived their keep. Anomalies, precomputed answers and refusal counts are never deleted.

Try It

Every live demo has a Try the compiler tab: change the policy, press Compile, and read the SQL the real compiler writes. On the command line the same compile is one line:

precomputing compile latency.precompute | sqlite3 latency.db

The How It Works post shows where the language sits in the platform.