[autoscout24] fix swallowed page failures, pagination, dedupe and render cost - #183
Open
HishamM1 wants to merge 1 commit into
Open
[autoscout24] fix swallowed page failures, pagination, dedupe and render cost#183HishamM1 wants to merge 1 commit into
HishamM1 wants to merge 1 commit into
Conversation
…der cost concurrent_scrape yields errors instead of raising them and force-disables upstream raising, so a failed page arrived as a normal response carrying an empty __NEXT_DATA__ payload and was counted as a page of zero results. - guard on the success type instead of one error class, the error classes are siblings and an asp target most likely yields ScrapflyAspError - add change_page() so a url that already carries ?page= is not given a second one, keeping repeated query keys used by the multi select filters - cap pages at numberOfPages, dedupe listings by url, warn on empty payloads - retry a page that came back empty once, sequentially, since AutoScout24 intermittently bounces a page to the homepage under concurrency - drop render_js, the payload is server rendered and costs 1 credit not 6 - take run.py detail urls from the listings instead of expired offers - add an offline test for change_page, flaky markers with pytest-rerunfailures, required price and url in the schemas, and a page relative count assertion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #182
concurrent_scrapeyields errors instead of raising them, so a failed page used to land in the results as a page of zero listings. Guarding it positively is the core of this PR.What changed in
autoscout24.py:isinstance(response, ScrapeApiResponse)- not on an error class. The error classes are siblings, and anasp: Truetarget most likely yieldsScrapflyAspError, which aScrapflyScrapeErrorcheck misses and then crashes on.selector. Same idiom aszoominfo.py:74.change_page()replaces the string concat. It keeps repeated query keys, which AutoScout24 uses for multi-select filters (eq=1&eq=15) -dict(parse_qsl(...))would drop all but the last one.min(max_pages, numberOfPages). Listings are deduped by url. An empty payload logs a warning instead of passing as zero results.render_jsdropped. Identical__NEXT_DATA__payload, 1 credit instead of 6.Tests: offline
test_page_url, flaky markers on both live tests pluspytest-rerunfailuresin the dev group (the marker was a silent no-op without it),required: Trueonpriceandurl, and>= 30replaced by> 2 * len(first_page)with a unique-url assertion. The old count passed on 20 real listings plus 10 duplicates.run.pytakes detail urls from the listings it just scraped instead of three expired offers.The workflow gets the new offline test folded into an existing runner rather than a new one, and
-kis now quoted so a multi-name expression survives the shell.Live validation: suite
3 passed in 49.26s.run.py-> 57 listings, 57 unique urls, 3/3 car details. The homepage bounce hit page 3 and one offer page during separate runs and the retry recovered both.