Add TwexAPI Haystack components for tweet search and timelines.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: lanbytes

.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Python ${{ matrix.python-version }}
+    runs-on: ubuntu-24.04
+    timeout-minutes: 20
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version:
+          - "3.10"
+          - "3.11"
+          - "3.12"
+          - "3.13"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v6
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --dev
+      - run: uv run ruff format --check .
+      - run: uv run ruff check .
+      - run: uv run pytest

.github/workflows/publish.yml

@@ -0,0 +1,32 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types:
+      - published
+
+permissions: {}
+
+jobs:
+  publish:
+    if: github.event.release.prerelease == false
+    runs-on: ubuntu-24.04
+    environment: pypi
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.release.tag_name }}
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v6
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: uv sync --dev
+      - run: uv run pytest
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/

.gitignore

@@ -0,0 +1,11 @@
+.venv/
+__pycache__/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+.coverage
+htmlcov/
+.DS_Store
+.env

CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 0.1.0
+
+- First release with TwexAPI tweet search and user timeline Haystack components.

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Code of Conduct
+
+Be respectful. Do not harass maintainers, contributors, or users.
+
+Unacceptable behavior includes harassment, hate speech, and posting secrets or personal data.
+
+Report issues on GitHub. Maintainers may remove comments, close issues, or block accounts that violate this policy.
+
+This project is not affiliated with X Corp.

CONTRIBUTING.md

@@ -0,0 +1,6 @@
+# Contributing
+
+- Open an issue for bugs or missing components.
+- Keep PRs small. Do not add endpoints that are not in the TwexAPI OpenAPI.
+- Do not commit API keys or `.env` files.
+- Run `uv run ruff check .` and `uv run pytest` before opening a PR.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TwexAPI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,75 @@
+# Twitter search and user timeline components for Haystack
+
+TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.
+
+Search Twitter and fetch public user timelines in Haystack RAG pipelines. Each TwexAPI result becomes a Haystack `Document`.
+
+## Components
+
+| Task | Haystack component | TwexAPI route | Output |
+| --- | --- | --- | --- |
+| Search tweets for RAG | `TwexApiTweetSearch` | `POST /twitter/advanced_search/page` | Matching posts as `Document` objects |
+| Retrieve a user's timeline | `TwexApiUserTweetsFetcher` | `POST /twitter/{screen_name}/timeline/page` | Recent user posts as `Document` objects |
+
+```python
+from haystack_integrations.components.websearch.x_api_scraper import (
+    TwexApiTweetSearch,
+    TwexApiUserTweetsFetcher,
+)
+```
+
+Both components read `X_API_SCRAPER_KEY` by default and accept a Haystack `Secret`. They send `Authorization: Bearer`. Set `base_url` for another TwexAPI-compatible endpoint. Results include `documents`, `links`, `has_more`, and `next_cursor`.
+
+Search returns up to 20 tweets per page. Timeline `top_k` maps to the page `count` (1-100). Use REST or an SDK for follower pagination or approved posting.
+
+## Install
+
+```bash
+pip install x-api-scraper-haystack
+```
+
+## Build a Haystack RAG pipeline with Twitter search
+
+### Search posts
+
+```python
+from haystack import Pipeline
+from haystack.utils import Secret
+from haystack_integrations.components.websearch.x_api_scraper import TwexApiTweetSearch
+
+search = TwexApiTweetSearch(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"), top_k=10)
+
+pipeline = Pipeline()
+pipeline.add_component("x_search", search)
+
+result = pipeline.run({"x_search": {"query": "haystack ai"}})
+documents = result["x_search"]["documents"]
+```
+
+### Fetch user posts
+
+```python
+from haystack.utils import Secret
+from haystack_integrations.components.websearch.x_api_scraper import (
+    TwexApiUserTweetsFetcher,
+)
+
+fetcher = TwexApiUserTweetsFetcher(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"))
+
+result = fetcher.run(screen_name="elonmusk")
+documents = result["documents"]
+```
+
+## Document mapping
+
+Each tweet becomes a Haystack `Document` with this mapping.
+
+- `Document.content`: `full_text` or `text`, or an empty string when both are missing
+- `Document.meta["endpoint"]`: `search` or `timeline`
+- `Document.meta`: available `id`, `url`, and `created_at` source values
+- `Document.meta["author"]`: available author identity and verification data
+- `Document.meta`: available like, retweet, reply, quote, view, and bookmark counts
+
+## License
+
+MIT. TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

SECURITY.md

@@ -0,0 +1,5 @@
+# Security
+
+Do not file public issues that include API keys, cookies, `auth_token` values, or personal data.
+
+Report suspected vulnerabilities privately through GitHub Security Advisories on this repository.

examples/search_pipeline.py

@@ -0,0 +1,12 @@
+from haystack import Pipeline
+from haystack.utils import Secret
+
+from haystack_integrations.components.websearch.x_api_scraper import TwexApiTweetSearch
+
+search = TwexApiTweetSearch(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"), top_k=10)
+
+pipeline = Pipeline()
+pipeline.add_component("x_search", search)
+
+result = pipeline.run({"x_search": {"query": "haystack ai"}})
+print(result["x_search"]["documents"])