tickdistill-learn

How We Validate Market Data Before It Becomes a Signal

By TickDistill — order-flow microstructure signals. Educational content about our data engineering.

A signal is only as trustworthy as the data underneath it

We are, fundamentally, a data company. A clever signal computed on a silently corrupted day of trades is worse than no signal — it looks authoritative and is wrong. So before any market data is allowed to become a signal you can query or backtest, it passes through a set of quality gates. This page explains how, because for a data vendor, correctness is the product.

The failure modes we refuse to ship

Raw market data fails in quiet, dangerous ways:

• Gaps — a missing hour or day that silently biases every statistic computed over it.
• Corrupt or partial downloads — a truncated file that parses “fine” but is incomplete.
• Schema drift — a source changes a column, a unit, or a timestamp format (milliseconds vs microseconds) without warning.
• Out-of-order or duplicated records — which break any time-dependent calculation.
• Impossible values — negative or zero prices/sizes, NaNs.
• Volume anomalies — far too few rows (a data hole) or far too many (a mis-set threshold inflating the dataset).

Any one of these, shipped silently, poisons both the live signal and the backtest a quant would trust their capital to.

The gate: deterministic, fail-closed, never silent

Every day of data runs through a deterministic QA gate before it is allowed downstream. The rule is simple and strict: at the first violation, stop and report — never continue silently. A fail-closed circuit breaker beats a pipeline that limps on with bad data and discovers it weeks later.

We keep these checks deterministic (explicit assertions, not a model guessing) because data validation must be exact and reproducible. Representative checks:

• Ingestion: download present and intact; schema and units as expected; timestamps strictly ordered; no impossible prices or sizes; no missing days inside the requested range.
• Processing: row counts within an expected band (too few ⇒ a gap; too many ⇒ a mis-calibration); output file size within budget; partitions written; reprocessing a day yields identical output.

When a check fails, the gate halts and emits a precise report — which day, which package, which metric — so the problem is fixed at the source rather than papered over.

Reproducible by construction: idempotent, resumable, point-in-time

Quality is not only “is this byte correct” — it is “can I trust the whole history.”

• Idempotent: reprocessing a given day overwrites its slice cleanly, producing the exact same result. No duplicates, no drift.
• Resumable: the pipeline checkpoints its progress, so an interruption resumes rather than restarting — and never half-writes a day.
• Point-in-time: every normalized statistic is computed using only past data. This rules out look-ahead bias — the single most common way a backtest flatters itself. (More in our sigma-normalization note.)

We process the raw tick data and then discard it — on purpose

We stream each day of raw tick data, compute the derived values, and discard the raw. Because our sources are free, re-deriving later costs time, not money — and it keeps us from hoarding terabytes we do not need. The quality gates run before the discard, so nothing is thrown away until it has been validated and the derived outputs are written and checked.

Why we tell you this

You cannot independently audit every byte we ingest, so we earn that trust by being explicit about how we handle data: fail-closed gates, deterministic checks, reproducible and point-in-time pipelines. We sell shovels, not gold — and a shovel that bends the first time you use it is worthless. The reliability of the data underneath is the part we will never cut.

---

TickDistill sells clean, computed order-flow inputs — not trading advice or guaranteed alpha. Backtests are illustrative and not a promise of future results.

This site is open source. Improve this page.