The system follows these principles:
- Compliance-first design: manual content input is the default; no bypassing access control.
- Modular pipeline: input, cleaning, extraction, clustering, risk analysis, and reporting are independent modules.
- LLM provider abstraction: business logic should not depend on a specific LLM vendor.
- Traceability: every extracted topic should preserve source URLs.
- Testability: core modules should be testable without external APIs.
- Extensibility: the system should support future WeCom integration, API service, and knowledge base.
+----------------------+ +----------------------+
| User JSON Input | | Future WeCom Input |
| Manual content | | Message / Command |
+----------+-----------+ +----------+-----------+
| |
v v
+-------------------------------------------------------+
| Input Layer |
| - JSON parser |
| - Input validator |
| - Source URL tracker |
+--------------------------+----------------------------+
|
v
+-------------------------------------------------------+
| Content Processing Layer |
| - Content loader |
| - Content cleaner |
| - Text normalizer |
+--------------------------+----------------------------+
|
v
+-------------------------------------------------------+
| Intelligence Layer |
| - LLM client abstraction |
| - Topic extractor |
| - Topic clusterer |
| - Risk analyzer |
+--------------------------+----------------------------+
|
v
+-------------------------------------------------------+
| Output Layer |
| - JSON result generator |
| - Markdown report generator |
| - Local file writer |
| - Future WeCom sender |
+-------------------------------------------------------+
xhs-topic-intelligence-bot/
│
├── app/
│ ├── main.py
│ ├── config.py
│ │
│ ├── api/
│ │ ├── routes_links.py
│ │ ├── routes_reports.py
│ │ └── routes_wecom.py
│ │
│ ├── services/
│ │ ├── content_loader.py
│ │ ├── content_cleaner.py
│ │ ├── topic_extractor.py
│ │ ├── topic_clusterer.py
│ │ ├── risk_analyzer.py
│ │ ├── report_generator.py
│ │ └── wecom_sender.py
│ │
│ ├── llm/
│ │ ├── client.py
│ │ ├── provider_base.py
│ │ └── prompts/
│ │ ├── summarize.md
│ │ ├── extract_topics.md
│ │ ├── cluster_topics.md
│ │ ├── analyze_risks.md
│ │ └── generate_report.md
│ │
│ ├── models/
│ │ ├── input.py
│ │ ├── topic.py
│ │ ├── risk.py
│ │ └── report.py
│ │
│ ├── storage/
│ │ ├── database.py
│ │ └── repository.py
│ │
│ └── utils/
│ ├── file_utils.py
│ ├── text_utils.py
│ └── validators.py
│
├── inputs/
│ └── sample_input.json
│
├── outputs/
│ └── .gitkeep
│
├── tests/
│ ├── test_content_cleaner.py
│ ├── test_topic_extractor.py
│ ├── test_topic_clusterer.py
│ ├── test_risk_analyzer.py
│ └── test_report_generator.py
│
├── docs/
│ ├── PRD.md
│ ├── ARCHITECTURE.md
│ ├── COMPLIANCE.md
│ └── PROJECT_BRIEF.md
│
├── .env.example
├── .gitignore
├── requirements.txt
├── README.md
└── CLAUDE.md
1. User prepares input JSON.
2. main.py loads JSON file.
3. Input validator validates required fields.
4. content_cleaner.py normalizes text.
5. topic_extractor.py extracts per-item topics with LLM.
6. topic_clusterer.py merges similar topics across items.
7. risk_analyzer.py flags sensitive and unsupported claims.
8. report_generator.py creates Markdown report.
9. JSON result and Markdown report are saved locally.
1. User calls POST /analyze.
2. FastAPI validates request body.
3. Pipeline runs asynchronously or synchronously.
4. Report result is stored.
5. API returns report_id and output paths.
6. User calls GET /reports/{report_id}.
7. User optionally calls POST /reports/{report_id}/send-wecom.
1. User sends a command or content to WeCom bot.
2. Backend receives message callback.
3. Backend parses links and manual content.
4. Analysis pipeline generates report.
5. WeCom sender pushes Markdown summary or report file.
Responsibilities:
- Load JSON input files.
- Convert raw JSON into Pydantic models.
- Preserve source URL, title, content, and comments.
- Reject invalid input.
MVP behavior:
- Manual content only.
- No automated Xiaohongshu scraping.
Responsibilities:
- Normalize spaces and line breaks.
- Remove noisy decorative symbols.
- Deduplicate hashtags.
- Preserve important factual information.
- Merge post content and comments with clear labels.
Output example:
{
"url": "...",
"title": "...",
"cleaned_text": "...",
"metadata": {
"has_comments": true,
"comment_count": 3
}
}Responsibilities:
- Provide a unified interface for LLM calls.
- Hide provider-specific implementation.
- Load prompts from
app/llm/prompts/. - Handle retries and basic errors.
Suggested interface:
class LLMClient:
def generate_json(self, prompt: str, schema_name: str) -> dict:
...
def generate_text(self, prompt: str) -> str:
...Provider abstraction should allow future support for:
- MiniMax.
- Claude.
- OpenAI-compatible APIs.
- Local models.
Responsibilities:
- Generate per-item summary.
- Extract topics, subtopics, keywords, pain points, questions, and evidence.
- Produce structured JSON.
- Validate LLM output.
Important rules:
- Do not quote large blocks from the original post.
- Evidence snippets should be short.
- Keep source URL.
Responsibilities:
- Merge similar topics across multiple items.
- Combine keywords and questions.
- Preserve all related source URLs.
- Detect possible duplicate topics.
- Prepare consolidated topic list for reporting.
Clustering options:
- MVP: LLM-based semantic clustering.
- Later: embedding-based clustering.
Responsibilities:
- Detect sensitive domains.
- Identify medical or infant-care claims.
- Identify advertising bias.
- Identify unsupported or absolute claims.
- Generate risk warnings.
Risk types:
medical
infant_care
advertising
unsupported_claim
privacy
safety
other
Severity levels:
low
medium
high
Responsibilities:
- Generate Markdown report.
- Generate structured JSON output.
- Ensure report format is stable.
- Include source links.
- Include disclaimers where necessary.
Markdown report sections:
# Topic Intelligence Report
## 1. Input Summary
## 2. Executive Summary
## 3. Core Topics
## 4. High-frequency Keywords
## 5. User Pain Points
## 6. Frequently Asked Questions
## 7. Conflicting Opinions
## 8. Risk Warnings
## 9. Actionable Recommendations
## 10. Suggested Follow-up Questions
## 11. Source Links
Responsibilities:
- Send Markdown report to WeCom.
- Support webhook mode first.
- Later support self-built application mode.
- Handle errors, timeout, and retry.
MVP status:
- Stub or optional implementation.
Future configuration:
WECOM_WEBHOOK_URL=
WECOM_CORP_ID=
WECOM_AGENT_ID=
WECOM_SECRET={
"project": "maternal-infant-research",
"language": "zh-CN",
"items": [
{
"url": "https://www.xiaohongshu.com/explore/example1",
"title": "待产包避坑清单",
"content": "这里是手动粘贴的小红书正文。",
"comments": [
"评论1",
"评论2"
]
}
]
}{
"report_id": "20260627_235900",
"input_summary": {
"total_items": 3,
"processed_items": 3,
"failed_items": 0
},
"topics": [
{
"topic": "待产包准备",
"summary": "多篇内容围绕入院准备、妈妈用品和宝宝用品展开。",
"confidence": 0.91,
"subtopics": ["妈妈用品", "宝宝用品", "医院清单"],
"keywords": ["待产包", "产褥垫", "尿不湿"],
"pain_points": ["不知道哪些必须买", "担心买多浪费"],
"questions": ["医院会提供哪些物品？"],
"source_urls": ["https://www.xiaohongshu.com/explore/example1"]
}
],
"risks": [
{
"risk_type": "medical",
"severity": "medium",
"description": "涉及产后恢复建议，需要结合医生意见。",
"recommendation": "仅作为信息整理，不作为医疗建议。"
}
]
}MVP uses local files:
outputs/
├── topic_result_20260627_235900.json
└── report_20260627_235900.md
Suggested tables:
reports
- id
- report_id
- title
- created_at
- language
- markdown_path
- json_path
source_items
- id
- report_id
- url
- title
- content_hash
- created_at
topics
- id
- report_id
- topic
- summary
- confidence
- source_urls_json
risks
- id
- report_id
- risk_type
- severity
- description
- recommendation
Examples:
- Missing URL.
- Missing title.
- Empty content.
- Invalid JSON.
Behavior:
- Return clear validation error.
- Do not run LLM pipeline.
Examples:
- Timeout.
- Invalid JSON output.
- Provider unavailable.
Behavior:
- Retry configurable times.
- Save partial results if possible.
- Return actionable error message.
Examples:
- Invalid webhook URL.
- Token expired.
- Message too long.
- API rate limit.
Behavior:
- Log sanitized error.
- Retry if safe.
- Fall back to local report file.
APP_ENV=development
APP_LANGUAGE=zh-CN
OUTPUT_DIR=outputs
LLM_PROVIDER=minimax
LLM_API_KEY=
LLM_BASE_URL=
LLM_MODEL=
WECOM_MODE=webhook
WECOM_WEBHOOK_URL=
WECOM_CORP_ID=
WECOM_AGENT_ID=
WECOM_SECRET=
LOG_LEVEL=INFORequired tests:
- Content cleaning.
- Input validation.
- Topic model validation.
- Report generation.
- Risk detection rules.
Use mocked LLM responses to test:
- Valid JSON parsing.
- Invalid JSON handling.
- Missing field recovery.
Later milestones:
- End-to-end JSON input to Markdown output.
- FastAPI endpoint test.
- WeCom sender dry-run.
- Do not log API keys.
- Do not commit
.env. - Mask URLs if they contain sensitive tokens.
- Do not store private user data unless explicitly needed.
- Do not store raw content longer than necessary if privacy mode is enabled.
- Provide a delete function for reports in future versions.
Best for MVP.
python -m app.main --input inputs/sample_input.json
Best for testing API mode.
uvicorn app.main:app --reload
Future deployment option.
Future deployment for WeCom callback and scheduled tasks.
- Embedding-based topic clustering.
- Domain-specific report templates.
- Maternal and infant checklist generation.
- Knowledge base retrieval.
- WeCom interactive commands.
- Scheduled topic digest.
- Multi-language report generation.
- Report comparison across time periods.