Skip to content

dataset-sources.md

Latest commit

 

History

History
415 lines (306 loc) · 14.5 KB

dataset-sources.md

File metadata and controls

415 lines (306 loc) · 14.5 KB
Raw

Data overview

Data snapshot (tree ./data)

data
├── iso
│   └── countries.csv
├── nasdaq
│   ├── etf
│   │   ├── EWA.csv
│   │   ├── EWC.csv
│   │   ├── EWCO.csv
│   │   ├── EWD.csv
│   │   ├── EWG.csv
│   │   ├── EWGS.csv
│   │   ├── EWH.csv
│   │   ├── EWI.csv
│   │   ├── EWJ.csv
│   │   ├── EWJE.csv
│   │   ├── EWJV.csv
│   │   ├── EWK.csv
│   │   ├── EWL.csv
│   │   ├── EWM.csv
│   │   ├── EWMC.csv
│   │   ├── EWN.csv
│   │   ├── EWO.csv
│   │   ├── EWP.csv
│   │   ├── EWQ.csv
│   │   ├── EWRE.csv
│   │   ├── EWS.csv
│   │   ├── EWSC.csv
│   │   ├── EWT.csv
│   │   ├── EWU.csv
│   │   ├── EWUS.csv
│   │   ├── EWV.csv
│   │   ├── EWW.csv
│   │   ├── EWX.csv
│   │   ├── EWY.csv
│   │   ├── EWZ.csv
│   │   └── EWZS.csv
│   └── symbols-valid-meta.csv
└── worldbank
    ├── gdp-capita-current-usd-2026.csv
    ├── gdp-capita-ppp-international-usd-2021.csv
    ├── gdp-current-usd-2026.csv
    └── gdp-ppp-international-usd-2021.csv

Data sources

World Bank GDP Data

Main website: https://data.worldbank.org

GDP: https://data.worldbank.org/indicator/NY.GDP.MKTP.CD GDP, PPP: https://data.worldbank.org/indicator/NY.GDP.MKTP.PP.KD GDP per capita: https://data.worldbank.org/indicator/NY.GDP.PCAP.CD GDP per capita, PPP: https://data.worldbank.org/indicator/NY.GDP.PCAP.PP.KD

Current preprocessing uses the PPP-adjusted constant-dollar files for GDP and GDP per capita:

  • data/worldbank/gdp-ppp-international-usd-2021.csv
  • data/worldbank/gdp-capita-ppp-international-usd-2021.csv

Current-USD GDP is still loaded as a conversion factor for asset-market proxies.

ISO country data

Country codes, names, and region and continent classification

https://www.iso.org/iso-3166-country-codes.html

Actually acquired from: https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes

Stock Listing Data

Data sourced from: https://www.kaggle.com/datasets/qks1lver/amex-nyse-nasdaq-stock-histories

Due to the large amount of not immediately relevant data, a few ETFs are chosen for now.

Stock symbol listing data:

Stock Indices Data

https://www.kaggle.com/datasets/guillemservera/global-stock-indices-historical-data

Market Cap Data

World Bank market capitalisation of listed domestic companies (current USD): https://data.worldbank.org/indicator/CM.MKT.LCAP.CD

File: data/worldbank/market-cap-current-usd-2026.csv

Exclusions: Russia, China, India, and Sweden, due to spotty data.

ETF-proxy gap filling:
For several countries the World Bank series has gaps in recent years. Gaps are filled using the country's iShares MSCI ETF price as a proxy:

MC(year) ≈ MC(last_known_year) × (ETF_price(year) / ETF_price(last_known_year))

Market cap is converted from current USD to a constant-2021 PPP-equivalent valuation proxy:

MC_real_ppp = MC_current_usd * GDP_PPP_constant_2021 / GDP_current_USD

ETF Price Data (iShares MSCI country ETFs)

iShares MSCI country ETF daily price history. Files: data/nasdaq/etf/EW*.csv

Data after 2020 was supplemented using yfinance.

Countries for which reliable data from 2000 was not easily available are excluded for now.

ETF prices are source adjusted-close prices, not inflation-adjusted prices. The preprocessing converts them to a constant-2021 PPP-equivalent valuation proxy:

ETF_real_ppp = ETF_nominal_usd * GDP_PPP_constant_2021 / GDP_current_USD

This is an approximation: ETF prices are asset valuations, not GDP. The purpose is to keep normalized trend comparisons on one broad real/PPP basis.

Regional equity benchmarks

These are not yet shown in the UI, but the preprocessing setup now tracks them as documented regional market proxies for later comparison views.

Europe

Asia-Pacific

Alternative local file:

  • Fund: iShares Core MSCI Pacific ETF (IPAC)
  • Official source: https://www.ishares.com/us/products/264619/
  • Local history file: data/stock/history/IPAC.csv
  • Coverage in repo: 2014 to 2020
  • Caveat: too late for the full 2000-to-2008 story

Latin America

North America

Regional macro aggregates

World Bank regional aggregates exist for major regions, including:

  • Europe & Central Asia
  • North America
  • East Asia & Pacific

Official explorer examples:

These are useful later for region selector views, but they should be treated as separate benchmark series, not mixed blindly with country averages, because the World Bank regional definitions are broader than the current country subset used in the narrative.

ECB household balance-sheet add-on

This is a separate narrative module, not part of the core country/region preprocessing pipeline. It exists to test the "growth for whom?" question without coupling the main map/spider workflow to ECB-specific data.

Current local file:

  • public/ecb_household_assets.json

Refresh command:

python scripts/fetch_ecb_household_assets.py

Current series:

Display rule in UI:

  • both series are rebased to 2008-Q4 = 1
  • panel remains separate from the core post-2008 market/fundamentals analysis so it can be removed cleanly later if the final story changes

OECD living-cost and pay add-on

This is a separate narrative module for testing whether post-2008 growth showed up as wage gains, broad living-cost pressure, or asset-price pressure.

Current local file:

  • public/living_cost_pay_data.json

Refresh command:

python scripts/fetch_living_cost_pay.py

Current sources:

Current derived series:

  • Real average annual wages: OECD WG, USD_PPP, constant 2024 prices, indexed to 2008.
  • Housing CPI / wage: broad housing CPI index divided by nominal average-wage index, then indexed to 2008.
  • House price / wage: residential house-price index divided by nominal average-wage index, then indexed to 2008.
  • Headline lens check: World Bank GDP per capita in current USD versus GDP per capita in constant-PPP international dollars, both indexed to 2008.

Caveats:

  • Europe is a simple average of available countries from the project sample, not an official EU aggregate.
  • OECD broad housing CPI coverage is narrower than wage coverage for this comparison; the Europe housing CPI / wage series currently uses Germany, the United Kingdom, the Netherlands, Sweden, and Switzerland.
  • Actual-rent CPI has no comparable US annual series in the fetched OECD slice, so the visualization uses broad housing CPI and residential house-price indexes instead.

Inequality add-on

This is a second separate narrative module focused on the question: who captured the post-2008 rebound?

Current local file:

  • public/inequality_addon.json

Refresh command:

python scripts/fetch_inequality_addon.py

Series included:

Three-part 2008 story data

This module is generated by scripts/build_analysis_lab_data.py and stored in public/analysis_lab_data.json. It keeps the main story reversible: all new panels depend on this generated JSON, not on ad-hoc calculations inside the renderer.

Refresh command:

python scripts/build_analysis_lab_data.py

Pre-2008 growth uses the existing processed project series:

  • GDP
  • GDP per capita
  • ETF price proxy
  • listed market capitalization

The pre-2008 panel also stores a runup view:

  • 2000 baseline is 1
  • 2007 dot shows the pre-crisis run-up
  • 2008 dot shows the immediate break-year endpoint
  • this avoids hiding the 2007-to-2008 market-cap drop inside a single 2000-to-2008 multiple

Industrial-structure data is fetched directly from the World Bank API for USA and EUU, using the latest available observation up to 2008:

  • Manufacturing value added, % GDP: NV.IND.MANF.ZS
  • Industry value added, % GDP: NV.IND.TOTL.ZS
  • Services value added, % GDP: NV.SRV.TOTL.ZS
  • Gross capital formation, % GDP: NE.GDI.TOTL.ZS
  • R&D expenditure, % GDP: GB.XPD.RSDV.GD.ZS

These industrial indicators are not additive:

  • manufacturing is a subset of industry
  • industry and services are value-added shares
  • investment and R&D are capacity/intensity measures, not output sectors

Current interpretation rule:

  • the pre-2008 section is descriptive, not causal
  • industrial structure is shown as context only
  • any claim that sector composition caused the post-2008 divergence needs a separate sector-level analysis with output, market, employment, and export exposure data

Normalization explorer and break-fit fields:

  • PPP GDP: World Bank NY.GDP.MKTP.PP.KD, constant 2021 international dollars
  • current-USD GDP: World Bank NY.GDP.MKTP.CD, current US dollars
  • ETF and market cap: processed project valuation proxies
  • breakFinder: piecewise-linear break-year fit for US/Europe gaps, including PPP GDP, current-USD GDP, ETF, and market capitalization

Other generated story fields:

  • counterfactual: actual post-2008 path versus extrapolated 2000-2008 trend
  • scorecard: post-2008 market, macro, labour-share, life-expectancy, and happiness comparison
  • heatmap, timeline, archetypes, and gainMatrix: backup views parked in the UI under "Backup analysis"

Section-3 scorecard units:

  • GDP per capita uses World Bank NY.GDP.PCAP.PP.KD: PPP-adjusted GDP per person in constant 2021 international dollars
  • market rebound uses the PPP-equivalent ETF price proxy generated by preprocessing
  • capital gains / income is a household financial-asset valuation measure
  • labour share is percentage-point change in labour share of GDP
  • life expectancy is change in years
  • happiness is change in Cantril ladder score

Break-year methodology:

  • gap series is log(US / Europe) for each metric
  • candidate break years are 2005 through 2015
  • model is gap = intercept + time + post_break_time
  • each candidate model is scored with BIC
  • displayed score is inverted and min-max normalized across candidate years: 1 = lowest BIC / best candidate, 0 = highest BIC / worst candidate
  • this score is a relative fit ranking, not a probability or confidence interval

Counterfactual methodology:

  • current counterfactual panel uses GDP per capita, not total GDP
  • model fits log(GDP per capita) from 2000 through 2008
  • post-2008 line compares actual normalized GDP per capita against that projected log-linear pre-break trend

Normalization explorer adjustment:

  • all lines use the same preprocessed public/spider_data.json output
  • GDP/cap lines use World Bank NY.GDP.PCAP.PP.KD
  • ETF and market-cap lines use the PPP-equivalent valuation proxies generated in preprocessing
  • for regions, GDP and market cap are summed across countries in the current project sample; GDP/cap and ETF proxy are averaged across countries