Skip to content

Commit fdbfb37

Browse files
author
stlc-bot
committed
feat(scrape): add reddit-post prebuilt parser with full comment trees (#2217)
Stainless-Generated-From: d234654d8641e13b92b5d05272b22385441bbeb0
1 parent a36d82e commit fdbfb37

4 files changed

Lines changed: 204 additions & 27 deletions

File tree

‎src/context/dev/types/batch_get_results_response.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,10 @@ class DataOkResultParsed(BaseModel):
323323
"tiktok-followers",
324324
"tiktok-following",
325325
"tiktok-user-search",
326+
"reddit-post",
327+
"reddit-search",
328+
"reddit-subreddit",
329+
"reddit-user",
326330
"yelp-search",
327331
"yelp-reviews",
328332
"emails",
@@ -861,6 +865,10 @@ class DataErrorResultParsed(BaseModel):
861865
"tiktok-followers",
862866
"tiktok-following",
863867
"tiktok-user-search",
868+
"reddit-post",
869+
"reddit-search",
870+
"reddit-subreddit",
871+
"reddit-user",
864872
"yelp-search",
865873
"yelp-reviews",
866874
"emails",

‎src/context/dev/types/batch_submit_params.py‎

Lines changed: 128 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -894,6 +894,10 @@ class InputScrapeDataBatchUnifiedScrapeDataParseParamsPrebuiltParser(TypedDict,
894894
"tiktok-followers",
895895
"tiktok-following",
896896
"tiktok-user-search",
897+
"reddit-post",
898+
"reddit-search",
899+
"reddit-subreddit",
900+
"reddit-user",
897901
"yelp-search",
898902
"yelp-reviews",
899903
"emails",
@@ -960,13 +964,62 @@ class InputScrapeDataBatchUnifiedScrapeDataParseParamsPrebuiltParser(TypedDict,
960964
(/search/user?q={query}) as data.users: id, username, nickname, biography,
961965
avatar_url, sec_uid, follower_count, like_count, and is_verified. The TikTok
962966
parsers accept tiktok.com, www.tiktok.com, and m.tiktok.com URLs but not short
963-
links (vm.tiktok.com, vt.tiktok.com, tiktok.com/t/…). yelp-search accepts Yelp
964-
search URLs with find_desc and find_loc, reads start as the result offset, and
965-
returns up to 20 ranked organic businesses in data.businesses with total and
966-
pagination. yelp-reviews accepts a Yelp business URL (yelp.com/biz/{alias}),
967-
uses optional start for the offset, and always sorts newest first; unsupported
968-
sort and review filters are rejected, while tracking params are ignored. It
969-
returns up to 20 latest public reviews in data.reviews with business details and
967+
links (vm.tiktok.com, vt.tiktok.com, tiktok.com/t/…). reddit-post parses a post
968+
(reddit.com/r/{subreddit}/comments/{id}, reddit.com/comments/{id}, or
969+
redd.it/{id}) into data.post: title, body as Markdown, type (text, link, image,
970+
gallery, or video), link_url, subreddit, author, flairs, score, upvote_ratio,
971+
comment and crosspost counts, NSFW, spoiler, locked, stickied, and archived
972+
flags, images (every gallery image in order), and video (an MP4 url plus an
973+
hls_url that carries the audio). data.comments holds the comment tree, each
974+
comment with id, author, body as Markdown, score, posted_at, depth, parent_id,
975+
url, is_submitter, is_deleted, and its replies; limit caps the total number of
976+
comments (default 100), filled shallowest first so whole conversations are kept,
977+
and data.has_more_comments says whether the thread has more. The sort query of
978+
the URL (best, top, new, controversial, old, qa) orders the comments.
979+
reddit-search returns search results for a search URL
980+
(reddit.com/search?q={query}, or reddit.com/r/{subreddit}/search?q={query} to
981+
search one subreddit). With type=link or no type it returns posts in the
982+
reddit-post shape as data.posts, ordered by the sort (relevance, hot, top, new,
983+
comments) and limited to the t time range (hour, day, week, month, year, all) of
984+
the URL; type=sr returns communities as data.communities (name, title,
985+
description, sidebar, subscriber_count, created_at, type, NSFW and quarantine
986+
flags, language, category, icon and banner URLs, and allowed post types);
987+
type=user returns accounts as data.users (username, display_name, bio,
988+
created_at, post and comment karma, avatar and banner URLs, and verified,
989+
employee, moderator, premium, and NSFW flags). limit caps the number of results
990+
(default 100), in the order Reddit returns them, and data.has_more says whether
991+
Reddit has more. reddit-subreddit reads a subreddit URL
992+
(reddit.com/r/{subreddit}, optionally followed by /hot, /new, /top, /rising, or
993+
/controversial, with t setting the time range of top and controversial, default
994+
day) into data.subreddit in the community shape of reddit-search, data.rules
995+
(name, description, applies_to, violation_reason, created_at), and data.posts in
996+
the reddit-post shape, ordered like the tab (default hot) and with stickied
997+
posts flagged; r/all and r/popular return posts with a null subreddit and rules.
998+
limit caps the posts (default 100) and data.has_more says whether Reddit has
999+
more. Private and quarantined subreddits return WEBSITE_ACCESS_ERROR.
1000+
reddit-user reads a profile URL (reddit.com/user/{username} or
1001+
reddit.com/u/{username}, optionally followed by /overview, /submitted, or
1002+
/comments, with sort set to new, hot, top, or controversial and t setting the
1003+
time range of top and controversial, default all) into data.user (id, username,
1004+
display_name, bio, created_at, total, post, comment, awardee, and awarder karma,
1005+
avatar and banner URLs, and verified, suspended, employee, moderator, premium,
1006+
and NSFW flags), data.trophies (name, description, icon_url, granted_at, url),
1007+
data.moderated_communities (name, title, url, subscriber_count, icon_url,
1008+
is_nsfw), data.posts in the reddit-post shape, and data.comments, each with id,
1009+
author, body as Markdown, score, posted_at, edited_at, url, subreddit, parent_id
1010+
(null for a top-level comment), is_submitter, is_stickied, distinguished,
1011+
award_count, and the post it was left on (id, title, url, author, is_nsfw). The
1012+
overview tab returns both posts and comments; submitted returns null comments
1013+
and comments returns null posts. limit caps the posts and comments together
1014+
(default 100) and data.has_more says whether Reddit has more. Users who hide
1015+
their activity return empty lists, and suspended accounts return their profile
1016+
with null posts and comments. yelp-search accepts Yelp search URLs with
1017+
find_desc and find_loc, reads start as the result offset, and returns up to 20
1018+
ranked organic businesses in data.businesses with total and pagination.
1019+
yelp-reviews accepts a Yelp business URL (yelp.com/biz/{alias}), uses optional
1020+
start for the offset, and always sorts newest first; unsupported sort and review
1021+
filters are rejected, while tracking params are ignored. It returns up to 20
1022+
latest public reviews in data.reviews with business details and
9701023
pagination.next_offset. Each review has rating, full text, updated_at, and
9711024
author; owner replies and photos are not included. emails accepts any page and
9721025
returns its email addresses as data.emails ([{ email }]), read from mailto
@@ -985,9 +1038,11 @@ class InputScrapeDataBatchUnifiedScrapeDataParseParamsPrebuiltParser(TypedDict,
9851038
"""Use a prebuilt parser instead of custom rules."""
9861039

9871040
limit: int
988-
"""google-maps-reviews only.
1041+
"""google-maps-reviews and reddit parsers only.
9891042
990-
Maximum number of reviews to return. Defaults to 100; maximum 1000.
1043+
Maximum number of reviews (google-maps-reviews, default 100), comments
1044+
(reddit-post, default 100), or results (reddit-search, reddit-subreddit, and
1045+
reddit-user, default 100) to return; maximum 1000.
9911046
"""
9921047

9931048
sort: Literal["newest", "oldest", "relevant", "highest", "lowest"]
@@ -2544,6 +2599,10 @@ class InputCrawlDataBatchUnifiedCrawlDataParseParamsPrebuiltParser(TypedDict, to
25442599
"tiktok-followers",
25452600
"tiktok-following",
25462601
"tiktok-user-search",
2602+
"reddit-post",
2603+
"reddit-search",
2604+
"reddit-subreddit",
2605+
"reddit-user",
25472606
"yelp-search",
25482607
"yelp-reviews",
25492608
"emails",
@@ -2610,13 +2669,62 @@ class InputCrawlDataBatchUnifiedCrawlDataParseParamsPrebuiltParser(TypedDict, to
26102669
(/search/user?q={query}) as data.users: id, username, nickname, biography,
26112670
avatar_url, sec_uid, follower_count, like_count, and is_verified. The TikTok
26122671
parsers accept tiktok.com, www.tiktok.com, and m.tiktok.com URLs but not short
2613-
links (vm.tiktok.com, vt.tiktok.com, tiktok.com/t/…). yelp-search accepts Yelp
2614-
search URLs with find_desc and find_loc, reads start as the result offset, and
2615-
returns up to 20 ranked organic businesses in data.businesses with total and
2616-
pagination. yelp-reviews accepts a Yelp business URL (yelp.com/biz/{alias}),
2617-
uses optional start for the offset, and always sorts newest first; unsupported
2618-
sort and review filters are rejected, while tracking params are ignored. It
2619-
returns up to 20 latest public reviews in data.reviews with business details and
2672+
links (vm.tiktok.com, vt.tiktok.com, tiktok.com/t/…). reddit-post parses a post
2673+
(reddit.com/r/{subreddit}/comments/{id}, reddit.com/comments/{id}, or
2674+
redd.it/{id}) into data.post: title, body as Markdown, type (text, link, image,
2675+
gallery, or video), link_url, subreddit, author, flairs, score, upvote_ratio,
2676+
comment and crosspost counts, NSFW, spoiler, locked, stickied, and archived
2677+
flags, images (every gallery image in order), and video (an MP4 url plus an
2678+
hls_url that carries the audio). data.comments holds the comment tree, each
2679+
comment with id, author, body as Markdown, score, posted_at, depth, parent_id,
2680+
url, is_submitter, is_deleted, and its replies; limit caps the total number of
2681+
comments (default 100), filled shallowest first so whole conversations are kept,
2682+
and data.has_more_comments says whether the thread has more. The sort query of
2683+
the URL (best, top, new, controversial, old, qa) orders the comments.
2684+
reddit-search returns search results for a search URL
2685+
(reddit.com/search?q={query}, or reddit.com/r/{subreddit}/search?q={query} to
2686+
search one subreddit). With type=link or no type it returns posts in the
2687+
reddit-post shape as data.posts, ordered by the sort (relevance, hot, top, new,
2688+
comments) and limited to the t time range (hour, day, week, month, year, all) of
2689+
the URL; type=sr returns communities as data.communities (name, title,
2690+
description, sidebar, subscriber_count, created_at, type, NSFW and quarantine
2691+
flags, language, category, icon and banner URLs, and allowed post types);
2692+
type=user returns accounts as data.users (username, display_name, bio,
2693+
created_at, post and comment karma, avatar and banner URLs, and verified,
2694+
employee, moderator, premium, and NSFW flags). limit caps the number of results
2695+
(default 100), in the order Reddit returns them, and data.has_more says whether
2696+
Reddit has more. reddit-subreddit reads a subreddit URL
2697+
(reddit.com/r/{subreddit}, optionally followed by /hot, /new, /top, /rising, or
2698+
/controversial, with t setting the time range of top and controversial, default
2699+
day) into data.subreddit in the community shape of reddit-search, data.rules
2700+
(name, description, applies_to, violation_reason, created_at), and data.posts in
2701+
the reddit-post shape, ordered like the tab (default hot) and with stickied
2702+
posts flagged; r/all and r/popular return posts with a null subreddit and rules.
2703+
limit caps the posts (default 100) and data.has_more says whether Reddit has
2704+
more. Private and quarantined subreddits return WEBSITE_ACCESS_ERROR.
2705+
reddit-user reads a profile URL (reddit.com/user/{username} or
2706+
reddit.com/u/{username}, optionally followed by /overview, /submitted, or
2707+
/comments, with sort set to new, hot, top, or controversial and t setting the
2708+
time range of top and controversial, default all) into data.user (id, username,
2709+
display_name, bio, created_at, total, post, comment, awardee, and awarder karma,
2710+
avatar and banner URLs, and verified, suspended, employee, moderator, premium,
2711+
and NSFW flags), data.trophies (name, description, icon_url, granted_at, url),
2712+
data.moderated_communities (name, title, url, subscriber_count, icon_url,
2713+
is_nsfw), data.posts in the reddit-post shape, and data.comments, each with id,
2714+
author, body as Markdown, score, posted_at, edited_at, url, subreddit, parent_id
2715+
(null for a top-level comment), is_submitter, is_stickied, distinguished,
2716+
award_count, and the post it was left on (id, title, url, author, is_nsfw). The
2717+
overview tab returns both posts and comments; submitted returns null comments
2718+
and comments returns null posts. limit caps the posts and comments together
2719+
(default 100) and data.has_more says whether Reddit has more. Users who hide
2720+
their activity return empty lists, and suspended accounts return their profile
2721+
with null posts and comments. yelp-search accepts Yelp search URLs with
2722+
find_desc and find_loc, reads start as the result offset, and returns up to 20
2723+
ranked organic businesses in data.businesses with total and pagination.
2724+
yelp-reviews accepts a Yelp business URL (yelp.com/biz/{alias}), uses optional
2725+
start for the offset, and always sorts newest first; unsupported sort and review
2726+
filters are rejected, while tracking params are ignored. It returns up to 20
2727+
latest public reviews in data.reviews with business details and
26202728
pagination.next_offset. Each review has rating, full text, updated_at, and
26212729
author; owner replies and photos are not included. emails accepts any page and
26222730
returns its email addresses as data.emails ([{ email }]), read from mailto
@@ -2635,9 +2743,11 @@ class InputCrawlDataBatchUnifiedCrawlDataParseParamsPrebuiltParser(TypedDict, to
26352743
"""Use a prebuilt parser instead of custom rules."""
26362744

26372745
limit: int
2638-
"""google-maps-reviews only.
2746+
"""google-maps-reviews and reddit parsers only.
26392747
2640-
Maximum number of reviews to return. Defaults to 100; maximum 1000.
2748+
Maximum number of reviews (google-maps-reviews, default 100), comments
2749+
(reddit-post, default 100), or results (reddit-search, reddit-subreddit, and
2750+
reddit-user, default 100) to return; maximum 1000.
26412751
"""
26422752

26432753
sort: Literal["newest", "oldest", "relevant", "highest", "lowest"]

0 commit comments

Comments
 (0)