Roadmap

Version 0.1 shows the idea working end to end on simulated data: one language, one file, two runtimes and two products, each with a live demo. What comes next takes it to real traffic, first by closing the gaps 0.1 left open and then through pilots with teams who try it, before a 1.0 that freezes the language and the file. Durations are estimates.

Two lanes. The platform: 0.1, done, four live demos on simulated data; 0.2, next, Logs upstream, JSON lines, smaller sketches and releases, about 10 weeks; pilots on real traffic with teams who try it; 1.0, planned, the language and the file frozen. Later: files from many machines merged into one answer, and more dashboard importers.

Stage Status What it covers
0.1 Done The language, the SQL runtime, the Engine, the Meter and Logs, with four live demos
0.2 Next Logs sends its own batches upstream; an OpenTelemetry Collector plug-in; JSON log lines; packed sketches; authentication; releases for three systems
Pilots With 0.2 Teams running Precomputing on their own data, with every result published only with their consent
1.0 Planned The policy language and the file format frozen, so files and policies from 1.0 on stay readable

0.2: The Gaps 0.1 Left Open

Each item comes from a limit written down in the 0.1 documentation.

  1. Logs sends its own batches. In 0.1 the batches meant for upstream are read from the file with SQL, as Demo 4 does. A forwarder sends them to an HTTP endpoint on a schedule, and retries until each batch is acknowledged, with the same sequence numbers the Engine already uses.
  2. An OpenTelemetry Collector plug-in. Many log pipelines already run through the Collector. The plug-in needs Go modules that the workspace 0.1 was built in could not reach, so it waits for 0.2.
  3. JSON log lines. 0.1 reads one text format per policy and treats a JSON line as text. 0.2 reads the fields of a JSON line directly.
  4. Packed sketches. A sketch is stored as one row per bucket. In Demo 1 the sketch buckets are more than half of the file. Packing each window’s buckets into one value should make files with percentiles much smaller, with the same answers.
  5. Authentication for the HTTP server. precomputing serve has none in 0.1. It gets tokens and TLS, so it can run outside localhost.
  6. Builds for three systems. Linux, macOS and Windows, each built and tested, ready for the first public release once the license is chosen.

When 0.2 Is Done

  1. Demo 4 sends its batches through the forwarder, and the dashboard drawn from what arrived still matches the raw lines.
  2. The Collector plug-in runs the Demo 4 policy inside an OpenTelemetry Collector with the same results.
  3. Demo 1’s file is at least a third smaller with packed sketches, and every answer is unchanged.
  4. Every test and the crash lab pass on all three systems.

The work fits about 10 weeks for one engineer (estimate).

Pilots: Real Traffic

Simulated data proves the mechanics. Only real data shows where a policy is awkward to write and how files behave over months. The project would like to run pilots with teams in the four areas the demos cover:

  • SQLite applications that keep counters, rollups or percentiles by hand today.
  • Services that need fast answers from a stream of events, such as an API sidecar or market data.
  • AI and API products that bill by usage and want the meter’s totals exact and in their own database.
  • Teams that pay by log volume and would rather send a dashboard’s answers than every line.

A pilot runs on the pilot’s own machines, and its results are published only with the pilot’s consent. See Contact.

1.0: A Frozen Format

A first stable release should follow the pilots, about 4 to 6 months after 0.2 (estimate). Its promise is stability:

  • The file format frozen. Format 1 already changes only by adding tables and columns. From 1.0 that is a promise, so a file written by 1.0 stays readable by every later version.
  • The policy language frozen, with new features added only in ways that leave existing policies meaning the same.
  • Policies that grow. 0.1 keeps a file’s policy for its whole life. 1.0 lets a file gain new streams and precomputes without starting again.
  • Measured on named hardware. Every published figure measured again on stated machines, with the commands to reproduce it.

Later: Many Machines, One Answer

Every summary the language keeps merges. Two 30-second windows add up to exactly one 1-minute window, and sketches add bucket by bucket. The same rule lets files from many machines add up to one answer: one Engine per server or per region, and a merged file for the whole fleet, with the same accuracy as one Engine that saw everything. Merging files is the natural step after 1.0. So are more dashboard importers, starting with the dashboard exports of the popular log tools.

Open Questions

  • The license. It will be chosen before the first public release. The plan is to keep the engines proprietary, with an open-source branch under consideration.
  • Where Logs should run first. In the OpenTelemetry Collector, as a separate process beside it, or both. Pilots should decide.
  • What the Meter should hand to billing systems. The meter’s totals are exact and in SQLite. Which billing tools to export them to, and in what form, depends on who uses it.