Skip to content

Commit 44eb63e

Browse files
authored
fix: keep articles with missing fields (#14)
The generator dropped 10 articles over an empty or oddly written field. They are kept now, with the missing value as `NULL`. Mounts also record which currency their price is in, so Landsailer's 140 reads as Event Points, not Tibia Coins. changes: - spells without `type`, `subclass`, `effect`, level or a whole-number cooldown are kept: Broadcast, Force Strike, Poison Storm, Undead Legion, Sniper, Second Wind and Ultimate Explosion - spell mana keeps `0` for special conditions, now also when it is empty - charm costs split by commas are read: Low Blow - NPCs without a city are kept: A Blue Stone - achievements without a name or description are kept: Achievement 563 - added `mount.price_currency`, following Template:Infobox Mount: `pricecurrency`, else Tournament Coins for tournament mounts, else Tibia Coins Parsing all 18,378 articles from one snapshot with main and with this branch recovers exactly those 10 and changes no existing value.
2 parents 1f31703 + fdd4ea2 commit 44eb63e

26 files changed

Lines changed: 608 additions & 43 deletions

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,23 @@
11
<!-- Changed by tibia.sh in 2026. See "About this copy" in README.md. -->
22
# Changelog
33

4+
## Unreleased
5+
6+
- Store `spell.effect`, `spell.spell_type`, `spell.group_spell`, `spell.level` and `spell.cooldown` as `NULL` when a
7+
spell's article has no such field, and `spell.cooldown` as `NULL` when it is not a whole number of seconds, like
8+
`1-2s`, instead of dropping the spell. A spell with no `mana` field gets `0`, which means special conditions apply.
9+
Force Strike, Broadcast, Poison Storm, Undead Legion, Sniper, Second Wind and Ultimate Explosion are now kept.
10+
- Read a charm's three costs split by commas, like `800, 1200, 4000`, when its `cost` field has no slash, instead of
11+
dropping the charm. Low Blow is now kept.
12+
- Store `npc.city` as `NULL` when an NPC's article has no `city` field, instead of dropping the NPC. Such an NPC's
13+
travel legs start where a `From [[Place]]` note or a shuttle names, and otherwise have a `NULL` origin. A Blue Stone
14+
is now kept.
15+
- Store `achievement.name` and `achievement.description` as `NULL` when an achievement's article has no such field,
16+
instead of dropping the achievement. `achievement.description` is now nullable. Achievement 563 is now kept.
17+
- Add the `mount.price_currency` column. It holds the article's `pricecurrency`, otherwise `Tournament Coins` for a
18+
tournament mount, otherwise `Tibia Coins`, and `NULL` when the mount has no price. Landsailer's price of 140 is now
19+
in `Event Points`, not Tibia Coins.
20+
421
## 9.0.0+tibiash.4
522

623
- Store `version` as `NULL` in `game_update`, `house`, `outfit` and `mount` when an article has no `implemented`

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ The copy adds this data to the database:
1818
types) of outfits and mounts.
1919
* The `creature.race_id` column. It holds the client race ID of a creature, `NULL` when the wiki has none or only a
2020
comment.
21+
* The `mount.price_currency` column. It holds the currency of a mount's price, like `Tibia Coins` or `Event Points`,
22+
`NULL` when the mount has no price.
2123

2224
The [database schema](docs/schema.md) describes each of them.
2325

‎docs/schema.md‎

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -425,23 +425,24 @@ that doesn't form a range.
425425

426426
### mount
427427

428-
| Column | Type | Description |
429-
| ------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
430-
| article_id | `INTEGER` / `PRIMARY` | The id of the article containing this mount. |
431-
| title | `TEXT` | The title of the article containing the mount. |
432-
| name | `TEXT` | The name of the mount. |
433-
| speed | `INTEGER` | The speed given by the mount. |
434-
| taming_method | `TEXT` | A brief description on how the mount is obtained. |
435-
| is_buyable | `BOOLEAN` | Whether the mount can be bought from the store or not. |
436-
| price | `INTEGER` | The price in Tibia coins to buy the mount. |
437-
| achievement | `TEXT` | The achievement obtained for obtaining this mount. |
438-
| light_color | `INTEGER` | The color of the light emitted by this mount, if any. |
439-
| light_radius | `INTEGER` | The radius of the light emitted by this mount, if any. |
440-
| client_id | `INTEGER` | The client ID (look type) of the mount, the first integer in the value after wiki comments are removed, `NULL` when there is none. |
441-
| version | `TEXT` | The client version where this mount was introduced to the game. |
442-
| image | `BLOB` | The mount's image bytes. |
443-
| status | `TEXT` | The status of the mount in game. |
444-
| timestamp | `TIMESTAMP` | ISO 8601 timestamp of the article's last edit. |
428+
| Column | Type | Description |
429+
| -------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
430+
| article_id | `INTEGER` / `PRIMARY` | The id of the article containing this mount. |
431+
| title | `TEXT` | The title of the article containing the mount. |
432+
| name | `TEXT` | The name of the mount. |
433+
| speed | `INTEGER` | The speed given by the mount. |
434+
| taming_method | `TEXT` | A brief description on how the mount is obtained. |
435+
| is_buyable | `BOOLEAN` | Whether the mount can be bought from the store or not. |
436+
| price | `INTEGER` | The price to buy the mount, in `price_currency`. |
437+
| price_currency | `TEXT` | The currency of the price: the article's `pricecurrency`, otherwise Tournament Coins for a tournament mount, otherwise Tibia Coins. `NULL` when there is no price. |
438+
| achievement | `TEXT` | The achievement obtained for obtaining this mount. |
439+
| light_color | `INTEGER` | The color of the light emitted by this mount, if any. |
440+
| light_radius | `INTEGER` | The radius of the light emitted by this mount, if any. |
441+
| client_id | `INTEGER` | The client ID (look type) of the mount, the first integer in the value after wiki comments are removed, `NULL` when there is none. |
442+
| version | `TEXT` | The client version where this mount was introduced to the game. |
443+
| image | `BLOB` | The mount's image bytes. |
444+
| status | `TEXT` | The status of the mount in game. |
445+
| timestamp | `TIMESTAMP` | ISO 8601 timestamp of the article's last edit. |
445446

446447

447448

‎tests/models/test_mount.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ def setUp(self):
1212
self.conn.row_factory = sqlite3.Row
1313
schema.create_tables(self.conn)
1414

15-
def _mount(self, client_id: int | None) -> Mount:
15+
def _mount(self, client_id: int | None, price_currency: str | None = "Tibia Coins") -> Mount:
1616
return Mount(
1717
article_id=1,
1818
title="Doombringer",
@@ -21,6 +21,7 @@ def _mount(self, client_id: int | None) -> Mount:
2121
taming_method="Buying it on Tibia.com or via the Store.",
2222
is_buyable=True,
2323
price=780,
24+
price_currency=price_currency,
2425
achievement=None,
2526
light_color=None,
2627
light_radius=None,
@@ -43,3 +44,10 @@ def test_mount_client_id_round_trip_none(self):
4344
loaded = Mount.get_one_by_field(self.conn, "article_id", 1)
4445

4546
self.assertIsNone(loaded.client_id)
47+
48+
def test_mount_price_currency_round_trip(self):
49+
self._mount(644, "Event Points").insert(self.conn)
50+
51+
loaded = Mount.get_one_by_field(self.conn, "article_id", 1)
52+
53+
self.assertEqual("Event Points", loaded.price_currency)

‎tests/parsers/test_achievement.py‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
1+
# Changed by tibia.sh in 2026. See "About this copy" in README.md.
12
import datetime
3+
import sqlite3
24
import unittest
35

46
from tests import load_resource
57
from tibiawikisql.api import Article
68
from tibiawikisql.models import Achievement
79
from tibiawikisql.parsers import AchievementParser
10+
from tibiawikisql.schema import AchievementTable
811

912

1013
class TestAchievementParser(unittest.TestCase):
@@ -20,3 +23,63 @@ def test_achievement_parser_from_article(self):
2023
achievement = AchievementParser.from_article(article)
2124

2225
self.assertIsInstance(achievement, Achievement)
26+
27+
def test_unknown_achievement(self):
28+
article = Article(
29+
article_id=1,
30+
title="Achievement 563",
31+
timestamp=datetime.datetime.fromisoformat("2026-08-06T21:35:46+00:00"),
32+
content=load_resource("content_achievement_563.txt"),
33+
)
34+
35+
achievement = AchievementParser.from_article(article)
36+
37+
self.assertIsNone(achievement.name)
38+
self.assertIsNone(achievement.description)
39+
self.assertEqual(563, achievement.achievement_id)
40+
self.assertIs(True, achievement.is_secret)
41+
42+
conn = sqlite3.connect(":memory:")
43+
self.addCleanup(conn.close)
44+
conn.executescript(AchievementTable.get_create_table_statement())
45+
achievement.insert(conn)
46+
stored = conn.execute("SELECT name, description FROM achievement WHERE article_id = 1").fetchone()
47+
self.assertEqual((None, None), stored)
48+
49+
def _parse(self, content: str) -> Achievement:
50+
article = Article(
51+
article_id=1,
52+
title="Demonic Barkeeper",
53+
timestamp=datetime.datetime.fromisoformat("2018-08-20T04:33:15+00:00"),
54+
content=content,
55+
)
56+
57+
return AchievementParser.from_article(article)
58+
59+
def _edited(self, old: str, new: str) -> str:
60+
content = load_resource("content_achievement.txt")
61+
self.assertIn(old, content)
62+
return content.replace(old, new)
63+
64+
def test_achievement_name_from_actualname(self):
65+
achievement = self._parse(self._edited(
66+
"| name = Demonic Barkeeper\n",
67+
"| name = Demonic Barkeeper (Achievement)\n| actualname = Demonic Barkeeper\n",
68+
))
69+
70+
self.assertEqual("Demonic Barkeeper", achievement.name)
71+
72+
def test_achievement_name_without_actualname(self):
73+
achievement = self._parse(load_resource("content_achievement.txt"))
74+
75+
self.assertEqual("Demonic Barkeeper", achievement.name)
76+
77+
def test_achievement_description_cleaned(self):
78+
achievement = self._parse(self._edited("your demon blood", "your [[Demon|demon]] blood"))
79+
80+
self.assertEqual(
81+
"Thick, red - shaken, not stirred - and with a straw in it: that's the way you prefer your demon blood. "
82+
"Served with an onion ring, the subtle metallic aftertaste is almost not noticeable. Beneficial effects on "
83+
"health or mana are welcome.",
84+
achievement.description,
85+
)

‎tests/parsers/test_charm.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
import datetime
2+
import unittest
3+
4+
from tests import load_resource
5+
from tibiawikisql.api import Article
6+
from tibiawikisql.errors import ArticleParsingError
7+
from tibiawikisql.models import Charm
8+
from tibiawikisql.parsers import CharmParser
9+
10+
11+
class TestCharmParserCosts(unittest.TestCase):
12+
def _parse(self, title: str, content: str) -> Charm:
13+
article = Article(
14+
article_id=1,
15+
title=title,
16+
timestamp=datetime.datetime.fromisoformat("2026-08-06T21:35:46+00:00"),
17+
content=content,
18+
)
19+
20+
return CharmParser.from_article(article)
21+
22+
def _costs(self, charm: Charm) -> tuple[int, int, int]:
23+
return charm.cost_level_1, charm.cost_level_2, charm.cost_level_3
24+
25+
def test_low_blow_costs_split_by_commas(self):
26+
charm = self._parse("Low Blow", load_resource("content_charm_low_blow.txt"))
27+
28+
self.assertEqual((800, 1200, 4000), self._costs(charm))
29+
30+
def test_costs_with_thousands_separator(self):
31+
content = load_resource("content_charm.txt")
32+
self.assertIn("| cost = 360 / 540 / 1,800\n", content)
33+
34+
charm = self._parse("Curse (Charm)", content)
35+
36+
self.assertEqual((360, 540, 1800), self._costs(charm))
37+
38+
def test_costs_not_three(self):
39+
content = load_resource("content_charm.txt").replace("360 / 540 / 1,800", "100 / 150")
40+
41+
with self.assertRaisesRegex(ArticleParsingError, "Expected three charm costs"):
42+
self._parse("Curse (Charm)", content)

‎tests/parsers/test_mount.py‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,84 @@ def test_mount_with_taming_method_and_version(self):
7575

7676
self.assertEqual("Buying it on Tibia.com or via the Store.", mount.taming_method)
7777
self.assertEqual("10.56", mount.version)
78+
79+
80+
class TestMountPriceCurrency(unittest.TestCase):
81+
def _parse(self, title: str, content: str) -> Mount:
82+
article = Article(
83+
article_id=1,
84+
title=title,
85+
timestamp=datetime.datetime.fromisoformat("2026-08-06T21:35:46+00:00"),
86+
content=content,
87+
)
88+
89+
return MountParser.from_article(article)
90+
91+
def _edited(self, old: str, new: str) -> str:
92+
content = load_resource("content_mount.txt")
93+
self.assertIn(old, content)
94+
return content.replace(old, new)
95+
96+
def test_price_currency_from_pricecurrency(self):
97+
mount = self._parse("Landsailer", load_resource("content_mount_landsailer.txt"))
98+
99+
self.assertEqual(140, mount.price)
100+
self.assertEqual("Event Points", mount.price_currency)
101+
102+
def test_price_currency_defaults_to_tibia_coins(self):
103+
mount = self._parse("Doombringer", load_resource("content_mount.txt"))
104+
105+
self.assertEqual(780, mount.price)
106+
self.assertEqual("Tibia Coins", mount.price_currency)
107+
108+
def test_price_currency_tournament_coins(self):
109+
mount = self._parse("Doombringer", self._edited("| price = 780\n",
110+
"| price = 780\n| tournament = yes\n"))
111+
112+
self.assertEqual(780, mount.price)
113+
self.assertEqual("Tournament Coins", mount.price_currency)
114+
115+
def test_price_currency_none_without_price(self):
116+
mount = self._parse("Doombringer", self._edited("| price = 780\n",
117+
"| pricecurrency = Event Points\n"))
118+
119+
self.assertIsNone(mount.price)
120+
self.assertIsNone(mount.price_currency)
121+
122+
def test_pricecurrency_wins_over_tournament(self):
123+
mount = self._parse("Doombringer", self._edited(
124+
"| price = 780\n",
125+
"| price = 780\n| pricecurrency = Event Points\n| tournament = yes\n",
126+
))
127+
128+
self.assertEqual("Event Points", mount.price_currency)
129+
130+
def _priced_in(self, currency: str) -> Mount:
131+
return self._parse("Doombringer", self._edited("| price = 780\n",
132+
f"| price = 780\n| pricecurrency = {currency}\n"))
133+
134+
def test_price_currency_cleans_links(self):
135+
mount = self._priced_in("[[Silver Token]]s")
136+
137+
self.assertEqual("Silver Tokens", mount.price_currency)
138+
139+
def test_price_currency_tibia_coins_template(self):
140+
mount = self._priced_in("{{TC}}")
141+
142+
self.assertEqual("Tibia Coins", mount.price_currency)
143+
144+
def test_price_currency_tournament_coins_template(self):
145+
mount = self._priced_in("{{TC3}}")
146+
147+
self.assertEqual("Tournament Coins", mount.price_currency)
148+
149+
def test_price_currency_template_first_letter_any_case(self):
150+
mount = self._priced_in("{{tC}}")
151+
152+
self.assertEqual("Tibia Coins", mount.price_currency)
153+
154+
def test_price_currency_kept_for_zero_price(self):
155+
mount = self._parse("Doombringer", self._edited("| price = 780\n", "| price = 0\n"))
156+
157+
self.assertEqual(0, mount.price)
158+
self.assertEqual("Tibia Coins", mount.price_currency)

0 commit comments

Comments
 (0)