a shortened amount of data in the scraper

[FabrizioMazzucco]

When i scrape comments only get 20 comments for publication + replies for these comments, same with links videos for an user. Did anyone else experience this or manage to solve it?

[SunnyChopra7]

Something similar with me, when I am trying to scrape user for a particular keyword, I am getting only 10 results

[SanclerMoonshot]

It was happening to me as well, looking into the library code to retrieve comments I saw that the count value in the params variable is hardcoded to be always 20, see:

    async def comments(self, count=20, cursor=0, **kwargs) -> Iterator[Comment]:
        """
        Returns the comments of a TikTok Video.

        Parameters:
            count (int): The amount of comments you want returned.
            cursor (int): The the offset of comments from 0 you want to get.

        Returns:
            async iterator/generator: Yields TikTokApi.comment objects.

        Example Usage
        .. code-block:: python

            async for comment in api.video(id='7041997751718137094').comments():
                # do something
        ```
        """
        found = 0
        while found < count:
            params = {
                "aweme_id": self.id,
                "count": 20,
                "cursor": cursor,
            }

            resp = await self.parent.make_request(
                url="https://www.tiktok.com/api/comment/list/",
                params=params,
                headers=kwargs.get("headers"),
                session_index=kwargs.get("session_index"),
            )

            if resp is None:
                raise InvalidResponseException(
                    resp, "TikTok returned an invalid response."
                )

            for video in resp.get("comments", []):
                yield self.parent.comment(data=video)
                found += 1

            if not resp.get("has_more", False):
                return

            cursor = resp.get("cursor")

What i've done to collect all messages was set a cursor:

...
cursor = 0
        total_collected_comments = 0
        total_fetched_comments = 0
        previous_fetched_comments = deque(maxlen=10)  # Keep track of the last 10 fetch counts
        collected_comments = list()

        print("STARTING COLLECTION")
        while True:
            comments = video.comments(count=20, cursor=cursor)

            try:
                async for comment in comments:
                    try:
                        if not is_only_emoji(comment.text):
                            # print(emoji.replace_emoji(comment.text, replace='').strip())
                            comment_dict = comment.as_dict

                            response_dict = {
                                "platform": "TikTok",
                                "collected_at": datetime.now(),
                                "id": comment.id,
                                "video_id": video_id,
                                "video_url": video_url,
                                "text": emoji.replace_emoji(comment.text, replace='').strip(),
                                "author": comment.author.username,
                                "author_id": comment.author.user_id,
                            }
                            collected_comments.append(response_dict)

                            total_replies = comment_dict["reply_comment_total"]
                            if total_replies > 0:
                                async for reply in comment.replies(count=total_replies):
                                    print(reply.text)

                            total_collected_comments += 1
                        total_fetched_comments += 1
                    except KeyError as e:
                        print(f"Missing key in comment data: {e}")

                cursor += 20  # Increment cursor by 20
...

I also noted that the number of comments in a tiktok video doesn't look to be accurate.