Skip to content

REPORT.md

Latest commit

 

History

History
89 lines (73 loc) · 6.04 KB

REPORT.md

File metadata and controls

89 lines (73 loc) · 6.04 KB
Raw

Tabaqa enricher — accuracy report

Generated by eval/eval_enrich.py over eval/labeled_transactions.jsonl (a hand-curated, deliberately-messy Arabic set). Re-run to regenerate.

Headline

  • Income-class accuracy: 94.2% — does a transaction count as income (salary/gig/p2p) or not. This is the number that drives the reveal.
  • Exact txn_type accuracy: 112/138 = 81.2% across 7 classes (macro-F1 0.809).
  • Reconciliation precision 1.00 / recall 0.80 — matched bank↔wallet transfers are never double-counted as income or spend.
  • Merchant coverage 50/65 = 76.9% on rows with a real brand (the rest is the long-tail the production AraBERT/LLM layer absorbs).

Per-txn_type

type precision recall F1 support
salary 1.00 0.86 0.92 21
gig_income 0.95 0.91 0.93 22
p2p 0.92 0.80 0.86 15
loan_obligation 1.00 0.79 0.88 14
purchase 0.97 0.76 0.85 49
internal_movement 1.00 0.80 0.89 10
unknown 0.21 0.86 0.33 7

Income-class (the reveal metric)

  • accuracy 94.2%, precision 0.98, recall 0.88, F1 0.93
  • Positive class = salary ∪ gig_income ∪ p2p. A wrong sub-type (e.g. a salary worded with «دفعة» read as gig) still counts as income here — the reveal total is unaffected even when the label is imperfect.

Reconciliation (anti-double-count)

  • precision 1.00, recall 0.80, F1 0.89 (tp=8, fp=0, fn=2).
  • Precision is the one that matters: we never falsely net out a real P2P inflow as an internal transfer. Recall misses are transfers whose two legs fall outside the 3-day matching window — they stay visible, just untagged.

Merchant resolution

  • coverage 50/65 (76.9%), correct 50/65 (76.9%).
  • Long-tail merchants the dictionary misses (the honest gap — these are what the production embedding/LLM tier is for):
    • DELIVERY HERO PAYOUT → expected DeliveryHero
    • ToYou كابتن - دفعة → expected ToYou
    • موبايلي - باقة بيانات → expected Mobily
    • مدى - البيك الطائف → expected Albaik
    • مدى - كودو → expected Kudu
    • POS - DOMINOS PIZZA → expected Dominos
    • مدى - دانكن دونتس → expected Dunkin
    • مدى - صيدلية النهدي → expected Nahdi
    • نينجا - توصيل بقالة → expected Ninja
    • ماكدونالدز الطائف → expected McDonalds
    • مدى - بترومين بلس → expected Petromin
    • موبايلي fiber → expected Mobily
    • مدى - صيدلية الدواء → expected AlDawaa
    • POS - SUBWAY RIYADH → expected Subway
    • Apple Store اشتراك iCloud → expected Apple

Every disagreement (full honesty)

26 of 138 rows disagree with the human label. Each is a known rule limitation, annotated in the dataset:

raw_desc human label rule predicted why
رابت شهر مايو salary unknown EXPECTED MISS: misspelled راتب->رابت
دفعة راتب أبريل salary gig_income EXPECTED MISS: دفعة triggers gig rule before salary
مستحقات شهر يونيو salary unknown EXPECTED MISS: no salary keyword
مرسول مستحقات التوصيل gig_income unknown —
توصيل - مستحقات اسبوع gig_income unknown EXPECTED MISS: no gig keyword
حوالة من خالد p2p unknown EXPECTED MISS: حوالة synonym, not تحويل من
تحويل وارد من محمد p2p unknown EXPECTED MISS: تحويل وارد من, not contiguous تحويل من
tahweel min ahmad p2p unknown EXPECTED MISS: transliterated p2p
خصم مديونية بطاقة loan_obligation unknown EXPECTED MISS: مديونية not in obligation keywords
موبايلي - باقة بيانات purchase unknown —
مدى - البيك الطائف purchase unknown type ok via acquirer; merchant long-tail (coverage miss)
مدى - كودو purchase unknown coverage miss: long-tail merchant
مدى - دانكن دونتس purchase unknown coverage miss: long-tail merchant
مدى - صيدلية النهدي purchase unknown coverage miss: long-tail merchant
نينجا - توصيل بقالة purchase unknown EXPECTED MISS: no acquirer keyword + long-tail merchant
ماكدونالدز الطائف purchase unknown EXPECTED MISS: long-tail merchant, no acquirer keyword
هابي - مطعم purchase unknown EXPECTED MISS: no keyword/merchant
تحويل الى محفظة urpay internal_movement purchase EXPECTED MISS: pair legs 7 days apart (outside 3-day window)
urpay إيداع من البنك internal_movement unknown EXPECTED MISS: pair legs 7 days apart (outside 3-day window)
تحويل من شركة كاش باك unknown p2p EXPECTED MISS (false-positive risk): matches تحويل من but is a cashback, not real p2p income
مدى - بترومين بلس purchase unknown —
موبايلي fiber purchase unknown —
اقساط تابي loan_obligation unknown EXPECTED MISS: اقساط plural not in keyword list (قسط)
مدى - صيدلية الدواء purchase unknown coverage miss: long-tail merchant
Apple Store اشتراك iCloud purchase unknown EXPECTED MISS: long-tail merchant, no acquirer keyword
تحويل لتسديد بطاقة ائتمان loan_obligation unknown EXPECTED MISS: credit-card payment, no qist/tamweel keyword

Method note

This measures the deterministic rule tier only (enrich_all + reconcile, pure stdlib, zero network). Production routes the residual long-tail to AraBERT embeddings + Claude Opus for cleaning; this set is constructed so the gap between the two tiers is exactly the rows listed above — nothing is hidden.