Skip to content

Data Refresh

Owner: Bryan · Trigger: the client drops a new dated export folder into team-data/.

How to refresh a client hub's numbers when new data arrives. Data comes as manual exports (the default: no credentials, no PII).

The loop

  1. Client drops a dated folder in team-data/ (e.g. team-data/June 23 - July 1/).
  2. Run the data builder from the client repo:
.\.venv\Scripts\python.exe tools\build_overview_data.py

It merges all period folders and dedupes by day (newest file wins on overlapping days, so no double-count), then writes docs/assets/data/overview.js. Commit that file; the dashboards light up.

  1. Re-run your analysis over the same merged data (totals, top channels + "Other", top products, weekly table newest-first, vitals latest + range).
  2. Update the static tables and the period label in docs/*.md.
  3. Build clean:
.\.venv\Scripts\python.exe -m mkdocs build --strict
  1. Commit and push. Cloudflare rebuilds in about 2 minutes.
  2. Verify the deploy and that Access is still gating both URLs.

Currency

Set the currency once in tools/build_overview_data.py ("$" → e.g. "A$", "US$"). The builder already filters glitched vitals rows (for example an absurd multi-minute LCP).

Export naming

Shopify exports come in two naming styles and the builder handles both:

  • "All Metrics / Conversions…" (TLC-style) or "Total sales by day…" (Sundae-style) for the KPI, funnel, and weekly widgets.
  • "Website Performance Metrics…" or "LCP…p75" for the vitals chart.

Data quirks (don't get burned)

  • Drop null-title product rows.
  • If the "Other" channel bucket shows an impossible CVR (orders exceed tracked sessions from attribution spread), blank its rate and add a caveat.
  • Call out traffic-quality anomaly weeks (session spikes with collapsed ATC/CVR) rather than letting them distort the story.
  • Frame all insights as hypotheses to validate. Mark partial weeks.

More in Gotchas.

TLC exception

The Lad Collective is multi-region (AU/US) and uses tools/generate_reports.py instead of build_overview_data.py because it is region-aware. Command from client-theladcollective/:

.\.venv\Scripts\python.exe tools\generate_reports.py au   # or: us

AU CRO test records are intentionally unpublished for now (PUBLISH_CRO_TESTS = {"au": False}). Keep that unless Bryan decides the source tracker is accurate enough to show again.