Home

Twitter (X) Comment Scraper Wiki

Welcome to the Twitter (X) Comment Scraper documentation wiki. This guide provides comprehensive information on setting up, configuring, and integrating the scraper for various data extraction workflows.

📚 Table of Contents

Overview & Architecture
Getting Started
Input Parameters & Configuration
Output Data Schema & Field Dictionary
Integration Guides
Best Practices & Rate Limiting
Troubleshooting & FAQ

⚡ Overview & Architecture

The Twitter (X) Comment Scraper is a high-speed, cookieless Apify actor designed to pull public post replies and commenter metadata from X (Twitter) without needing account credentials or API tokens.

Key Capabilities

Cookieless Operation: No session cookies, user logins, or API keys needed.
High Throughput: Capable of processing 20+ replies in under 2 seconds per target thread.
Cost Efficient: Billed on compute usage at approximately $5.00 per 1,000 items.
Data Filtering: Filter out low-engagement content via minimum like thresholds (minLikes).
Sorting Modes: Scrape by relevance, recency, or likes.

🚀 Getting Started

Apify Console Execution

Navigate to the Actor page on Apify:
👉 https://apify.com/mikolabs/twitter-comment-scraper
Paste the target tweet URLs into the Tweet URLs input field.
Adjust optional parameters (maxComments, sortBy, minLikes).
Click Start to run the scraper.
Export data in JSON, CSV, Excel, XML, or HTML table format.

⚙️ Input Parameters & Configuration

Parameter	Type	Required	Default	Description
`tweetUrls`	Array of Strings	Yes	`[]`	List of target X (Twitter) post URLs (e.g., `https://x.com/username/status/123456789`).
`maxComments`	Integer	No	`100`	Maximum number of top-level comments/replies to retrieve per post.
`sortBy`	String	No	`"relevance"`	Sorting order for replies. Options: `relevance`, `recency`, `likes`.
`minLikes`	Integer	No	`0`	Exclude comments with fewer likes than this value.
`includeUserStats`	Boolean	No	`true`	When `true`, extracts author profile metadata (bio, followers, location, etc.).

📊 Output Data Schema & Field Dictionary

Each item exported into the Apify dataset represents a single comment/reply with the following schema:

{
  "id": "1815123456789012345",
  "text": "Great insights on AI agent architectures!",
  "createdAt": "Sun Jul 26 01:00:00 +0000 2026",
  "lang": "en",
  "likeCount": 85,
  "retweetCount": 4,
  "replyCount": 2,
  "quoteCount": 1,
  "bookmarkCount": 12,
  "viewCount": 3400,
  "conversationId": "1815000000000000000",
  "inReplyToStatusId": "1815000000000000000",
  "inReplyToUserId": "44196397",
  "author": {
    "id": "123456789",
    "username": "tech_dev",
    "name": "Jane Developer",
    "description": "Full-stack engineer & AI researcher.",
    "followersCount": 9400,
    "followingCount": 450,
    "tweetCount": 1200,
    "location": "San Francisco, CA",
    "isVerified": true,
    "profileImageUrl": "https://pbs.twimg.com/profile_images/..."
  },
  "media": [
    {
      "type": "photo",
      "mediaUrl": "https://pbs.twimg.com/media/..."
    }
  ]
}

Field Definitions

id: Unique status ID of the reply.
text: Full text content of the reply.
conversationId: Target tweet's root conversation ID.
inReplyToStatusId: Direct parent tweet/comment ID.
author: Full profile information of the commenter.
media: List of attached images, GIFs, or video URLs.

🔌 Integration Guides

Python Integration

Install the official Apify Python SDK:

pip install apify-client

Run the actor programmatically:

from apify_client import ApifyClient

# Initialize client
client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Set up input payload
run_input = {
    "tweetUrls": ["https://x.com/elonmusk/status/1815000000000000000"],
    "maxComments": 200,
    "sortBy": "likes",
    "minLikes": 10
}

# Run the actor
run = client.actor("mikolabs/twitter-comment-scraper").call(run_input=run_input)

# Print scraped comments
dataset = client.dataset(run["defaultDatasetId"])
for item in dataset.iterate_items():
    author = item.get("author", {}).get("username", "anonymous")
    text = item.get("text", "")
    likes = item.get("likeCount", 0)
    print(f"@{author} ({likes} likes): {text}")

Node.js Integration

Install the official Apify JavaScript SDK:

npm install apify-client

Run the actor programmatically:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: 'YOUR_APIFY_API_TOKEN',
});

const runInput = {
    tweetUrls: ['https://x.com/elonmusk/status/1815000000000000000'],
    maxComments: 100,
    sortBy: 'relevance'
};

(async () => {
    const run = await client.actor('mikolabs/twitter-comment-scraper').call(runInput);
    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    
    console.log(`Successfully scraped ${items.length} comments.`);
    items.forEach(comment => {
        console.log(`[${comment.author?.username}]: ${comment.text}`);
    });
})();

cURL / REST API Integration

You can trigger runs using standard HTTP requests:

curl -X POST "https://api.apify.com/v2/acts/mikolabs~twitter-comment-scraper/runs?token=YOUR_APIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tweetUrls": ["https://x.com/elonmusk/status/1815000000000000000"],
    "maxComments": 50
  }'

💡 Best Practices & Rate Limiting

Use minLikes for Filtering: If analyzing high-volume tweets (10,000+ replies), use minLikes: 5 or higher to discard spam and low-value bot comments.
Batch Multiple URLs: Submit multiple URLs in a single tweetUrls array to maximize compute efficiency.
Webhooks & Integrations: Set up Apify Webhooks to stream results directly to your database, Slack, Zapier, or custom Webhook endpoint when scraping completes.

❓ Troubleshooting & FAQ

Q: Do I need a Twitter developer account or API key?

No. The scraper operates entirely cookieless and requires no Twitter developer access or user login credentials.

Q: Can I scrape comments from private accounts?

No. The scraper only accesses publicly viewable tweets and replies on X (Twitter).

Q: How can I copy this page into GitHub Wiki?

Go to your GitHub repository and click on the Wiki tab.
Click Create the first page or New Page.
Set title to Home or Twitter-Comment-Scraper-Guide.
Paste the content of this file and click Save Page.

Maintained by MikoLabs • Apify Actor Page

Navigation Menu