Store performance at portfolio scale

The Parquet store (camber.store.ParquetStore) is designed so a portfolio of hundreds of buildings over years of interval data lives under one root and a query touches only the data it needs. This note explains the three mechanisms that keep reads fast, how to measure them, and the one cost that does grow with portfolio size.

flowchart TD
  q["query (facility_id, start/end, roles)"] --> flt["_build_filter"]
  flt -- facility_id + year/month bounds --> prune["partition pruning"]
  flt -- needed columns --> proj["column projection"]
  prune --> part["facility_id=.../year=.../month=... partitions"]
  proj --> part
  part --> pivot["fast-path pivot"]
  pivot --> frame["role-named frame (flat per-equip read)"]
  write["write_long / write_role_frame"] -- invalidate --> cat["_catalog.json (cached)"]
  cat --> points["points() enumeration"]

A per-equipment read opens one building's one-year partition regardless of portfolio size.

Layout

One tidy long-form dataset, hive-partitioned by facility_id, then year, then month (since 0.95):

<root>/facility_id=fox-lodge-9f3a1c/year=2024/month=7/part-*.parquet
<root>/_facilities.json     {"fox-lodge-9f3a1c": {"name": "Fox Lodge", ...}}

A query for one facility reads only that facility's directory; a query for one month reads only that month's subdirectory. Stores written before 0.95 have year= partitions only (year=2024/part-*.parquet). They are read unchanged, a store may hold both layouts (a legacy file reads with month null and is never skipped by a month bound), and camber store migrate-partitions STORE converts them (a dry run unless --apply --yes; crash-safe and idempotent). Month partitions are what month-level retention prunes: see PORTFOLIO.md.

Reproducible part files

Since 0.103 every part the store writes is in one canonical layout, so the same rows give the same bytes. That holds for write_long (and write_role_frame, write_rollup and every dataset ingest), migrate-partitions and the retention rollups. Each part:

has its rows sorted by (ts, equip, role, equip_class, value), whatever order the source frame, the archive members or the wide frame's columns came in;
is written single-threaded as one row group (up to 1,048,576 rows), with fixed writer options: snappy, dictionary encoding, statistics, format 2.6, data page v1;
carries no pandas metadata, which records the pandas and pyarrow versions, and instead the key-value marker camber.layout = "1". A retention rollup also keeps its camber.rollup.covers key.

Part names were already deterministic (part-<seq>-<i>.parquet, with <seq> counting the facility's earlier writes).

What is guaranteed. The same inputs with the same CAMBER version and the exact same pyarrow version give byte-identical part files, with the same names. Across pyarrow (or pandas) versions, only the content is guaranteed: the same rows in the same order with the same types. The bytes may differ, because the Parquet footer's created_by names the full pyarrow version and the encoder can change. CAMBER does not overwrite created_by. The tests check both: a content digest that holds on every pyarrow version, and a byte golden pinned to one pyarrow release.

Older stores. Parts written before 0.103 (pandas metadata, a large part in about ten row groups, rows in thread order) read back exactly like canonical ones, and a store may mix both. Nothing needs re-ingesting. The ingest content hash is unchanged, so an already-ingested dataset is still skipped; camber datasets ingest <id> --force rewrites it in the canonical layout, and so does camber store normalize for any store. Sorted rows in one row group also make large ingests smaller on disk: 2 to 3.4 times smaller for the LBNL simulation archives, irish-ahu and nuig-ahu101, and 1.8 times smaller across all the workbook's dataset stores (537 MB to 291 MB).

Checking and converting a store (0.104). camber store verify STORE reports which parts are canonical, how many are legacy (no camber.layout marker), edge or rollup parts, and which fail to read. It takes no lock and changes nothing; --strict exits 1 on any non-canonical part. camber store normalize STORE rewrites the rest:

camber store verify <ws>/store [--facility ID] [--json]
camber store normalize <ws>/store                                     # dry run: what and why
camber store normalize <ws>/store --apply --yes --reason "0.103 layout"

Content-identical, same names. Each part is rewritten under its own name with exactly its rows (a rollup part keeps its camber.rollup.covers key). Names stay because a rollup's covers lists raw part names: renamed raw parts would be rolled up a second time.
Crash-safe. Per partition, the new parts are written to a _trash-normalize-* stage inside the partition, which readers do not see, fsynced and read back; they must equal the original rows, sorted, with the same types. Then each is moved over its original with one atomic rename, and the stage is removed. Old and new hold the same rows, so a reader or a crash at any point sees every row exactly once. The next run removes a leftover stage and rewrites what is left.
Idempotent, under the store lock (the workspace's lock in a workspace, where it is audited as store.normalize and also covers the rollup stores), and it invalidates the catalog. It reports the bytes of the rewritten parts before and after.
Left alone, each with its reason: canonical parts; edge parts (see below); year-only parts (migrate-partitions rewrites them, recording their sha256 for the edge landing); unreadable parts; and facilities that do not accept data (suspended, offboarding, archived, retired ids) or are under a legal hold, whose bytes stay as they are, so an offboarding facility's export bundle keeps matching its data.

The edge forwarder's parts are content-addressed but not canonicalised, and normalize leaves them so: see EDGE-DEPLOY.md.

Facility identity (why an id, not a name)

Each facility is keyed by a stable, path-safe facility_id, decoupled from its human display name — which, with any metadata, lives in a sibling _facilities.json registry (camber.store.FacilityRegistry). This is what makes a portfolio scale safely: a raw name used as the partition directory would collide when two facilities share a name, orphan history on a rename, and — worst — a space/unicode name would URL-encode the written directory while the writer's part counter looked for the un-encoded path, silently overwriting earlier data. write_long/ write_role_frame validate the id up front (require_facility_id), so those failures can't happen.

Supply your own stable id (an external building id), or derive one from a name with camber.store.make_facility_id("Fox Lodge") -> "fox-lodge-9f3a1c" (a slug + short hash; pass a more-specific seed when display names repeat). An older site=<name> store converts in place with camber.store.migrate_site_to_facility(root). The read API addresses facilities by facility_id (the legacy site= argument and /sites endpoint remain as deprecated aliases).

Equipment ids, equipment classes and roles are free text (spaces, slashes, Unicode and punctuation are fine), but since 0.111 (#291) write_long / write_role_frame refuse one that holds a control character (newline, tab, ESC, NUL, C1), a bidi override or isolate, a zero-width or other invisible format character, or a line separator, with a ValueError naming the id and the character. Such an id looks like another one on screen and can forge lines in an assistant's tool results; it is refused rather than rewritten, because rewriting could merge two ids or split one already stored. Ids already in a store are read as they are (the capabilities show them one-line and quoted, see CAPABILITY-REGISTRY.md); rename them in the mapping before re-ingesting.

Since 0.86 each registry entry also carries a lifecycle state, dates, an editable display_name, an owner and tags (registry v2; older entries read as active). Removed ids are tombstoned in _tombstones.json and never reused, and a new id that differs from a known one only by letter case is refused. See PORTFOLIO.md.

The three scale mechanisms

Partition pruning on site and year. Filters on site skip other buildings' directories. Crucially, a start/end time range is translated into bounds on the year and month partition fields as well as the ts data column (_build_filter), so a one-month query across a multi-year store opens only the relevant month partition(s) instead of scanning every year. Proven in tests/test_store_scale.py via dataset.get_fragments(filter=…).
Column projection. Reads pull only the columns they need from Parquet. points() (catalog enumeration) projects just site/equip/role and never reads the ts/value payload; read_role_frame projects ts/role/value for the one equipment requested.
Fast-path pivot. read_role_frame uses a plain pivot when each (ts, role) is unique and only falls back to the slower mean-aggregating pivot_table when the store actually holds duplicate observations.
Cached catalog. points() enumerates distinct (site, equip, role) keys from a small _catalog.json written alongside the dataset, so it needs no partition scan. Writes invalidate the catalog (an O(1) file removal — they never rewrite it, so bulk ingestion stays cheap); the first points() after a write burst rebuilds it once with a projected scan and caches the result. A points(facility_id=…) / equipment(facility_id=…) call on a stale catalog scans only that facility's partition (memoized in-process until the partition changes) and leaves the store-wide rebuild to the next unscoped call, so its cost does not grow with the rest of the store (0.111). (Arrow's dataset discovery ignores leading-_ paths, so the file is invisible to reads. rebuild_catalog() materializes it for a store that predates the cache.)

One writer at a time (0.104)

Every ParquetStore write method (write_long, write_role_frame, register_facility, write_rollup, prune, drop_partition, drop_facility, migrate_partitions(apply=True)) holds the store's single-writer lock while it writes: <store>/_lock for a plain store, the workspace's lock for a workspace store. Another process that writes the same store waits (30 s by default; --lock-timeout, $CAMBER_LOCK_TIMEOUT or ParquetStore(root, lock_timeout=…) change it), then gets StoreLocked naming the holder. Reads never take it.

Taking the lock afresh costs a file open, an flock and a short holder record (under a millisecond on a local disk). For a bulk load, take it once around the whole load. The writes inside re-enter it, which costs microseconds:

st = ParquetStore("lake")
with st.lock():  # once for the whole load
    for fid, equip, frame in frames:
        st.write_role_frame(frame, facility_id=fid, equip=equip)

camber datasets ingest, the lab, camber interop openfdd ingest and synth_portfolio() already do this: each takes the store lock once per command.

A read that rebuilds the cached catalog (points(), equipment(), rebuild_catalog()) still takes no lock. Instead, each writer stamps a fresh random generation token in _catalog.gen when it invalidates the catalog. A rebuild records the token before it scans and stamps it into the catalog it writes. It skips the write-back if the token has changed by then, and a catalog whose token is no longer current is never served. A rebuild that races a write can therefore never hide that write (#141).

Measuring it

A synthetic generator + benchmark ships in the package:

python -m camber.store.bench --facilities 50 --equips 10 --days 30 --freq 1h

It builds a portfolio and times the hot paths (points, single-equipment read, time-ranged read, rollup). benchmark() / synth_portfolio() are importable for custom runs.

Measured behaviour

On a developer laptop (numbers are illustrative, not a guarantee):

portfolio	rows	write	single-equip read	time-ranged read	`points()` cold → warm
150 sites × 10 equips × 4 roles × 30 d hourly	4.32 M	~8.8 s	~36 ms	~240 ms	~3.5 s → ~22 ms

The headline property: a single-equipment read stays roughly flat as the portfolio grows (tens of ms) — partition pruning + projection mean it opens one building's data regardless of how many other buildings exist. This is the access pattern the rules/resolve layer uses, so per-equipment analytics scale.

Catalog enumeration (points()) is served from the cached catalog in ~tens of ms once warm. The first call after a write burst rebuilds it once (a projected scan — ~3.5 s here); writes themselves stay cheap because they only invalidate the catalog, never rewrite it (an earlier rewrite-on-every-write design made a bulk load ~5× slower — the invalidate-on-write design keeps write throughput intact). Further mitigations if even the cold rebuild matters: scope it with points(facility_id=…), or roll up + prune (rollup / write_rollup / prune) so old partitions are smaller.