|
| 1 | +# Data Brief — *What's in a Barrel?* |
| 2 | + |
| 3 | +> A data-first reference. **Part A** explains every dataset we have, what's in it, |
| 4 | +> and the one takeaway it proves. **Part B** explores and plans how to make the |
| 5 | +> site lean harder on that data. This is a working/planning doc — not the process |
| 6 | +> book, not a changelog. All numbers verified against source on 2026-05-28 |
| 7 | +> (EIA refinery yield, EIA STEO, IEA OMR, OPEC MOMR via the IEF comparative analysis). |
| 8 | +
|
| 9 | +--- |
| 10 | + |
| 11 | +## Part A — What we have, and what each dataset proves |
| 12 | + |
| 13 | +We have **four datasets**: three public, one I wrote by hand. Two notebooks |
| 14 | +(`01_explore.py`, `02_steo.py`) turn the raw downloads into **seven processed |
| 15 | +files** the D3 scenes read at runtime. Below, each is described by *what it is → |
| 16 | +what's in it → the takeaway → how much the site currently uses*. |
| 17 | + |
| 18 | +### 1. The barrel's composition — EIA U.S. Refinery Yield |
| 19 | + |
| 20 | +- **What it is:** the % of a U.S. barrel that becomes each refined product, every |
| 21 | + year, from the same agency on the same basis. |
| 22 | +- **What's in it:** 1993–2025, 16 products → grouped into **12 boiling-point |
| 23 | + bands**. Files: `us_refinery_yield.csv` (raw 16), `yield_by_band_over_time.csv` |
| 24 | + (12 bands × 33 years), `tower.json` (the 2025 snapshot). |
| 25 | +- **Takeaway:** a 2025 barrel is **46% gasoline · 30% diesel · 11% jet**, and the |
| 26 | + yields **sum to ~106%** — refining cracks heavy molecules into lighter, larger- |
| 27 | + volume ones (the **+5.9% processing gain**). Over 32 years the one cut that |
| 28 | + *climbed* is **diesel: +8.1 pp** (21.9 → 30.0); gasoline held flat; refinery- |
| 29 | + side petrochem feedstock actually *fell*. |
| 30 | +- **Site usage:** ✅ well used. Refinery column (tower.json), Sankey sources, |
| 31 | + climax companion tower, and the trend stacked-area (the full 33-year series). |
| 32 | + **This is our most data-driven dataset.** |
| 33 | + |
| 34 | +### 2. The barrel's destinations — hand-authored mapping |
| 35 | + |
| 36 | +- **What it is:** the one file with no public dataset behind it. Maps each |
| 37 | + fraction → end-use sectors. **Editorial — a model, not a measurement.** |
| 38 | +- **What's in it:** `product_sector_mapping.json` — 12 fractions → 11 sectors, |
| 39 | + weights summing to 1.0 per fraction, **each row cited** to IEA *Oil 2024* / EIA |
| 40 | + *AEO 2025*. |
| 41 | +- **Takeaway:** there is *no* clean public fraction→end-use matrix at barrel |
| 42 | + granularity, so the Sankey is necessarily a cited estimate. Its honesty (every |
| 43 | + row sourced, copy hedged) is the point. |
| 44 | +- **Site usage:** ✅ used by the Sankey. It's the hardest claim to defend and the |
| 45 | + most transparent about being a model. |
| 46 | + |
| 47 | +### 3. The long view — Energy Institute / OWID consumption history |
| 48 | + |
| 49 | +- **What it is:** harmonized world + country oil consumption, narrow "oil" basis. |
| 50 | +- **What's in it:** `consumption_by_country.csv` — **6,405 rows, 1965–2024, 112 |
| 51 | + entities (79 real countries + 33 regional/income aggregates)**, columns |
| 52 | + `entity, code, year, oil_twh, mbd, is_country`. Plus `world_consumption.csv` |
| 53 | + (the World row alone). |
| 54 | +- **Takeaway:** world demand **tripled** since 1965 (29.0 → 89.1 Mb/d, ×3.07), |
| 55 | + with visible shocks (1973, the 2008–09 GFC, the 2020 COVID drop to 79.1). In |
| 56 | + 2024 the top consumers are **US 16.0 · China 14.4 · India 4.9 Mb/d** (narrow |
| 57 | + basis). *(No population / per-capita column — per-capita would need a new |
| 58 | + source.)* |
| 59 | +- **Site usage:** ⚠️ **barely scratched.** The counter shows **only the latest |
| 60 | + year's top-N country bars**. The 60-year time dimension, every country |
| 61 | + trajectory, and all 33 regional/income aggregates are **unused**. |
| 62 | + |
| 63 | +### 4. The right-now — EIA STEO (May 2026) |
| 64 | + |
| 65 | +- **What it is:** current + forecast demand on the **broad** "petroleum and other |
| 66 | + liquids" basis (the ~104 figure the press uses). |
| 67 | +- **What's in it:** `steo_monthly_consumption.csv` — **6 series** (World, OECD, |
| 68 | + Non-OECD, US, China, India), **monthly Jan 2022 → Dec 2027** (504 rows). The |
| 69 | + only forward-looking and only monthly data we have. |
| 70 | +- **Takeaway:** world demand 102.8 (2024) → 104.16 (2026) → **105.6 (2027)**. The |
| 71 | + May-2026 agency split is genuine and **validated against the IEF comparative |
| 72 | + analysis**: **IEA −0.4 · EIA +0.2 · OPEC +1.2 Mb/d**, a ~1.6 Mb/d spread. |
| 73 | +- **Site usage:** ⚠️ **partial.** The counter rate (1,205 bbl/s) derives from it. |
| 74 | + The crisis scene uses **only the World series**: its **EIA** forecast line is the |
| 75 | + **real deseasonalised STEO `patc_world` trend**, and the **IEA/OPEC** lines are |
| 76 | + labelled editorial offsets sized to each agency's published 2026 delta (see Part B |
| 77 | + and the `crisis.js` header — *this supersedes the earlier "hard-coded slopes" note*). |
| 78 | + The US/China/India/OECD/non-OECD series are **completely unused**. |
| 79 | + |
| 80 | +### The "two numbers" that govern everything |
| 81 | + |
| 82 | +| Basis | 2024 world | Used for | |
| 83 | +|---|---|---| |
| 84 | +| **Narrow** "oil" (OWID/EI) | 89.1 Mb/d | the long 1965→2024 history, country bars | |
| 85 | +| **Broad** "petroleum + other liquids" (STEO) | 102.8 Mb/d | the live counter, the crisis epilogue | |
| 86 | + |
| 87 | +A 13.7 Mb/d (13%) **definitional** gap — not an error. Any data-driven change must |
| 88 | +keep declaring which basis it's on. |
| 89 | + |
| 90 | +### What we're sitting on but not showing |
| 91 | + |
| 92 | +| Untapped data | Where it lives | What it could power | |
| 93 | +|---|---|---| |
| 94 | +| 60 years × 79 countries of consumption | `consumption_by_country.csv` | a "who burns it, and since when" view; country trajectories | |
| 95 | +| 33 regional / income-group aggregates | same | OECD vs non-OECD divergence; the demand-shift-east story | |
| 96 | +| World 1965→2024 growth (×3) | `world_consumption.csv` | a rising-demand context beat before the mix story | |
| 97 | +| STEO US/China/India/OECD forecasts | `steo_monthly_consumption.csv` | real per-region forecast lines in the crisis scene | |
| 98 | +| Per-capita (NOT in our data) | needs new OWID source | the single most reordering reveal (Saudi/US ≫ China/India) | |
| 99 | + |
| 100 | +--- |
| 101 | + |
| 102 | +## Part B — Making the site more data-driven |
| 103 | + |
| 104 | +### The principle |
| 105 | + |
| 106 | +"Data-driven" here means two distinct things, and we should be honest about which |
| 107 | +each change buys: |
| 108 | + |
| 109 | +1. **Integrity** — every on-screen number provably comes from the pipeline, not |
| 110 | + typed by hand. (Helps *Technical 15%* and the process-book data story.) |
| 111 | +2. **Revelation** — interaction/views expose data the reader can't see at rest, as |
| 112 | + the interaction lecture demands. (Helps *Visualization 35%*.) |
| 113 | + |
| 114 | +### The honest trade-off first |
| 115 | + |
| 116 | +Per `HANDOFF.md`, the site is content-complete and **scene polish is now |
| 117 | +low-yield** versus the **screencast (25%) and process book (25%), both at zero**. |
| 118 | +So this plan is ranked for **integrity-per-hour**, and the recommendation is to |
| 119 | +take only the cheap, high-credibility wins now and defer the big builds unless the |
| 120 | +two 25% deliverables are done. |
| 121 | + |
| 122 | +### Options, ranked by payoff-per-hour |
| 123 | + |
| 124 | +| # | Change | Data it unlocks | Effort | Payoff | Risk | |
| 125 | +|---|---|---|---|---|---| |
| 126 | +| 1 | **Generate `figures.json` from the notebooks** — every on-screen constant (yields, deltas, counter rate, spread) emitted by the pipeline; site imports it; `constants.js` stops hard-coding numbers | makes *all* scenes provably data-derived | S (1–1.5 h) | **High** integrity; clean process-book claim "no number is typed" | Low | |
| 127 | +| 2 | **Make the crisis EIA line real** — draw the actual STEO World forecast (it runs to Dec 2027) as the EIA trajectory; keep IEA/OPEC as annotated end-points clearly labelled "each agency's published 2026 delta" | the STEO forecast we already ship but don't plot | S–M (1–2 h) | **High** — kills the most "made-up-feeling" scene | Low–Med | |
| 128 | +| 3 | **Surface the ×3 growth** — a small sparkline/line of world demand 1965→2024 as a context beat before the mix story | `world_consumption.csv` | S (1 h) | Med — sets stakes; uses idle data | Low | |
| 129 | +| 4 | **Counter → add a per-region forecast toggle** — plot STEO US/China/India lines, not just World | unused STEO country series | M (2 h) | Med — shows demand shifting east | Med | |
| 130 | +| 5 | **Revive Scene 7 drill-down** — click a fraction → its yield-over-time + sector split + top consumers (joins 3 datasets) | yield_by_band + mapping + countries together | L (3–4 h) | Med–High viz, but expensive | Med–High | |
| 131 | +| 6 | **Per-capita reveal in the counter** — reorder bars by barrels-per-person | **needs a new source** (OWID per-capita oil, CC-BY) | M (2–3 h) | **High** viz "aha", but new-data risk | Med | |
| 132 | + |
| 133 | +### Decision taken (2026-05-28) — shipped #1 + #2 + #3, no new sources |
| 134 | + |
| 135 | +| # | Change | Status | What shipped | |
| 136 | +|---|---|---|---| |
| 137 | +| 1 | `figures.json` | ✅ done | `notebooks/03_figures.py` emits `data/processed/figures.json`; `constants.js` imports it at build time. Processing gain, counter rate, world demand, diesel delta, yields, growth, and EIA's 2026 delta are all derived. No site-wide number is hand-typed. IEA/OPEC deltas stay as cited editorial constants (not in our data). | |
| 138 | +| 2 | Real crisis line | ✅ done | `crisis.js` EIA line is now the **real STEO `patc_world` forecast**, deseasonalized to its 12-month trend and anchored at the real May-2026 fork. IEA/OPEC are honest offsets of it (their published 2026 delta vs EIA's). End-labels staggered to avoid collision. | |
| 139 | +| 3 | Growth context | ✅ done | `world_consumption.csv` now loaded; trend scene shows a 1965→2024 sparkline + "demand grew 3.1×" caption that bridges into the mix story. | |
| 140 | +| 4–6 | per-region toggle / drill-down / per-capita | ⬜ deferred | v2 — left untouched to protect screencast + process-book time. | |
| 141 | + |
| 142 | +**Decision on new data:** none pulled — stayed with the four validated datasets. |
| 143 | +Per-capita (#6), Brent price, and Hormuz trade-flow remain noted as future |
| 144 | +options if a v2 wants them. |
| 145 | + |
| 146 | +**Verified:** production build clean (159.57 KB / 52.44 KB gz, within budget), |
| 147 | +zero console/page errors, scenes screenshot-checked (refinery +5.9%, counter |
| 148 | +1,205/s, crisis fork, trend sparkline all render from data). |
| 149 | + |
| 150 | +--- |
| 151 | + |
| 152 | +## Appendix — file → scene map (current state) |
| 153 | + |
| 154 | +| Processed file | Loaded by | Depth of use | |
| 155 | +|---|---|---| |
| 156 | +| `tower.json` | refinery, sankey, climax, tower | full (2025 snapshot) | |
| 157 | +| `yield_by_band_over_time.csv` | trend | full (33-year series) | |
| 158 | +| `product_sector_mapping.json` | sankey | full | |
| 159 | +| `consumption_by_country.csv` | counter | latest-year top-N only (still room: #4) | |
| 160 | +| `steo_monthly_consumption.csv` | crisis (+ counter rate) | World series; **EIA forecast line now real (deseasonalized)**, IEA/OPEC offset | |
| 161 | +| `world_consumption.csv` | trend | **now used — the 1965→2024 growth sparkline** | |
| 162 | +| `us_refinery_yield.csv` (raw 16) | — | unused (bands version is used instead) | |
| 163 | +| `figures.json` *(new)* | `constants.js` (build-time import) | **all site-wide numbers derived here** | |
0 commit comments