[goodreads] add review and search scrapers - #186
Open
HishamM1 wants to merge 1 commit into
Open
Conversation
Reviews come from the __NEXT_DATA__ Apollo records of a book page, search reuses the list row parser since both pages share the same table markup. Adds the missing scraper README and root README row, registers both tests in the CI matrix. Also fixes run.py writing results with the platform encoding, which cannot encode the non latin characters review text contains.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the existing Goodreads scraper with two capabilities it did not cover.
scrape_reviewsreads the review sample of a book page from its__NEXT_DATA__Apollo recordsscrape_searchreusesparse_list, search and list pages share the same table markuptest_review_scrapingandtest_search_scrapingin the CI matrixrun.pywriting results with the platform encoding, which crashes on review textpytest test.pypasses all 4 tests against the live site.