Validate README data source URLs against city data (#628) - #791
Validate README data source URLs against city data (#628)#791alexschnapper wants to merge 1 commit into
Conversation
| [fuerth-markets]: https://www.fuerthwiki.de/wiki/index.php/Kategorie:M%C3%A4rkte | ||
| [Gelsenkirchen-wikipedia]: https://en.wikipedia.org/wiki/Gelsenkirchen | ||
| [Gelsenkirchen-markets]: https://essener-wochenmaerkte.de | ||
| [Gelsenkirchen-markets]: https://www.gelsendienste.de/privathaushalte/maerkte/ |
There was a problem hiding this comment.
That was likely not wrong. Gelsenkirchen is near Essen - markets might have been maintained on that website.
There was a problem hiding this comment.
Thanks for pointing this out. You are right that the previous URL was not necessarily incorrect merely because it used the Essen markets domain.
The PR updates the README to match the current data_source value in cities/gelsenkirchen.json, which is the consistency this validation enforces. I have adjusted the PR description so it no longer describes the previous URL as a confirmed content error.
There was a problem hiding this comment.
Feel free to replace the former text. GitHub keeps a version history of the text anyway.
johnjohndoe
left a comment
johnjohndoe
left a comment
There was a problem hiding this comment.
Please squash your commits and only update the single commit.
| return dataSources; | ||
| } | ||
|
|
||
| function readCityDataSources() { |
There was a problem hiding this comment.
Can the helper function be private?
| }); | ||
| } | ||
|
|
||
| test('each configured city data source is documented in README', () => { |
There was a problem hiding this comment.
Can you move the test function to the top and the "not so important" helper functions to the bottom?
4b8cf80 to
8977203
Compare
2 participants
Summary
Saw the issue from @johnjohndoe which was there since a couple of years and gave it a look and thought I might help and created this PR. If not needed, just let me know 😁
This pull request adds an automated validation that ensures the data source URL documented for each city in
README.mdmatches the URL stored in the correspondingcities/*.jsonfile.It also synchronizes the currently divergent README links with the city data.
Closes #628.
Changes
Add
spec/data-source-url.test.js.Parse the supported-cities table and Markdown link definitions from
README.md.Match every README entry to its corresponding city in
cities/cities.json.Compare the README URL with
metadata.data_source.urlfrom the city JSON file.Fail the existing test workflow when:
Normalize city names to handle:
(Rheinland)or(Taunus); andcities.json.Treat Markdown reference labels case-insensitively, as Markdown itself does.
Synchronize 35 stale README data source URLs with their city JSON counterparts.
Treat
cities/cities.jsonand the corresponding city files as the source of truth.Check the relationship in both directions:
Parse the supported-cities table and Markdown reference definitions with small, documented parsing helpers.
Findings
Issue #628 was created in January 2024 and is still open without comments. A review of the current codebase, commit history, and related pull requests did not find an existing implementation of the requested README-to-JSON validation.
Several later pull requests updated individual data source URLs, but the README and city JSON files continued to be maintained independently. This caused the two sources to diverge over time.
The initial validation found 35 inconsistent README references. These included outdated URLs, HTTP-to-HTTPS changes, moved city pages, and different URL paths.
It also exposed an existing content error:
cities/gelsenkirchen.json.The existing Gelsenkirchen and Kaiserslautern Markdown reference labels use different capitalization from their references in the city table. The test therefore compares Markdown reference labels case-insensitively.
edit:
The initial validation found 35 mismatches, including outdated URLs, HTTP-to-HTTPS changes, moved pages, different URL paths, and the Gelsenkirchen entry.
The previous Gelsenkirchen README URL may have been valid historically because markets for nearby cities can be maintained by the same provider. This PR does not make a judgment about that historical URL; it updates the README to match the current source-of-truth value in
cities/gelsenkirchen.json.Why this approach
The validation compares URLs per city instead of comparing only the combined set of URLs.
A set-based comparison could incorrectly pass if two cities had exchanged URLs. Matching each README entry to its city JSON file ensures that the correct URL belongs to the correct city and produces a useful city-specific failure message.
The test uses only Node.js built-in modules and is included automatically in the existing
npm testcommand through the currentnode --test spec/**.test.jsconfiguration.Validation
node --test spec/**.test.jsgit diff --checkBranch comparison:
masterThe complete
npm testand build commands could not be completed in the Codex environment because npm repeatedly failed while extracting registry tarballs from its cache. This failure occurred during dependency installation and was unrelated to the implementation.The new validation itself and the existing dependency-free Node.js test suite pass successfully.