Skip to content

Commit bea3843

Browse files
tsrdatatechtsrdatatech
committed
feat: Add comprehensive LangChain integration for advanced content analysis
- Implement AdvancedContentAnalyzer with sophisticated prompt engineering - Add sentiment analysis, topic classification, and quality scoring - Integrate structured output parsing with Pydantic models - Create AIEnhancedNewsParser extending existing GenericNewsParser - Add SmartParserFactory for flexible parser instantiation - Implement comprehensive test suite with 12 test cases covering all AI features - Add demo script showcasing LangChain capabilities - Support both mock LLM for development and production LLM integration - Enhance NewsArticle schema with optional AI analysis results - Add fallback mechanisms for robust error handling - Update dependencies with LangChain framework components Technical improvements: - Async/await pattern for efficient AI processing - Factory pattern implementation for parser management - Comprehensive logging and monitoring - Multi-language content support - Configurable quality thresholds and analysis parameters
1 parent cda02fc commit bea3843

9 files changed

Lines changed: 812 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ This project demonstrates enterprise-level software design patterns:
3030

3131
## 🚀 Technical Stack
3232

33+
- **AI/ML Framework**: LangChain with prompt engineering and structured output parsing
3334
- **Browser Automation**: Crawlee + Playwright for sophisticated queue management
3435
- **Data Validation**: Pydantic v2 with advanced type checking and serialization
3536
- **Content Extraction**: Multi-method approach (Newspaper3k + Trafilatura + custom)
@@ -55,6 +56,14 @@ This project demonstrates enterprise-level software design patterns:
5556
- **Data Validation**: Runtime type checking with Pydantic schemas
5657
- **Error Handling**: Comprehensive exception management with graceful degradation
5758

59+
### AI/ML Engineering with LangChain
60+
- **Content Analysis**: AI-powered article summarization and sentiment analysis
61+
- **Topic Classification**: Automated topic extraction and entity recognition
62+
- **Quality Assessment**: Intelligent content quality scoring and readability analysis
63+
- **Prompt Engineering**: Sophisticated LangChain prompt templates and output parsing
64+
- **Fallback Systems**: Graceful degradation when AI services are unavailable
65+
- **Mock Integration**: Development-friendly mock LLM for testing and demonstration
66+
5867
### Advanced Kubernetes Orchestration
5968
- **Batch Processing**: Enterprise-grade job orchestration similar to AWS Step Functions + Batch
6069
- **Auto-scaling**: Dynamic resource allocation with horizontal and vertical scaling
@@ -82,6 +91,7 @@ This project demonstrates enterprise-level software design patterns:
8291

8392
## 🏛️ Core Features
8493

94+
- **AI-Enhanced Content Analysis**: LangChain-powered summarization, sentiment analysis, and topic classification
8595
- **Multi-Parser Architecture**: Automatic parser selection based on URL fingerprinting
8696
- **Advanced Kubernetes Orchestration**: Enterprise-grade batch processing with auto-scaling
8797
- **Production Pipelines**: Complete CI/CD with automated testing, building, and deployment
@@ -106,6 +116,33 @@ class BaseParser:
106116
"""Core extraction logic with type safety"""
107117
```
108118

119+
### AI-Powered Content Analysis
120+
```python
121+
# LangChain integration for intelligent content analysis
122+
from langchain_core.prompts import PromptTemplate
123+
from langchain_community.llms import FakeListLLM
124+
125+
class AdvancedContentAnalyzer:
126+
def __init__(self):
127+
self.summary_prompt = PromptTemplate(
128+
input_variables=["title", "content"],
129+
template="Analyze this article and provide a concise summary..."
130+
)
131+
132+
async def analyze_article(self, article: NewsArticle) -> ContentAnalysis:
133+
"""AI-powered analysis with sentiment, topics, and quality scoring"""
134+
analysis = await self.llm.ainvoke({
135+
"title": article.title,
136+
"content": article.content
137+
})
138+
return ContentAnalysis(
139+
summary=analysis.summary,
140+
sentiment=analysis.sentiment,
141+
topics=analysis.topics,
142+
quality_score=self.calculate_quality_score(article)
143+
)
144+
```
145+
109146
### Data Validation Pipeline
110147
```python
111148
# Pydantic v2 schema with advanced validation
@@ -115,6 +152,7 @@ class NewsArticle(BaseModel):
115152
url: HttpUrl
116153
published_date: Optional[datetime] = None
117154
author: Optional[str] = Field(None, max_length=200)
155+
ai_analysis: Optional[Dict[str, Any]] = None # AI insights
118156

119157
@field_validator('content')
120158
@classmethod
@@ -268,4 +306,4 @@ jobs:
268306
269307
## 🔧 Technical Keywords
270308
271-
`Python` • `Async/Await` • `Pydantic` • `Playwright` • `Docker` • `Kubernetes` • `GitHub Actions` • `Test-Driven Development` • `Clean Architecture` • `Design Patterns` • `Type Safety` • `CI/CD` • `Container Orchestration` • `Web Scraping` • `Parser Registry` • `Strategy Pattern`
309+
`Python` • `LangChain` • `AI/ML Engineering` • `Async/Await` • `Pydantic` • `Playwright` • `Docker` • `Kubernetes` • `GitHub Actions` • `Test-Driven Development` • `Clean Architecture` • `Design Patterns` • `Type Safety` • `CI/CD` • `Container Orchestration` • `Web Scraping` • `Parser Registry` • `Strategy Pattern` • `Prompt Engineering` • `Content Analysis`

‎demo_ai_features.py‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Demo script showcasing LangChain AI/ML integration capabilities.
4+
5+
This script demonstrates the AI-enhanced web scraping features including:
6+
- Content analysis and summarization
7+
- Sentiment analysis
8+
- Topic classification
9+
- Entity extraction
10+
- Quality assessment
11+
12+
Run with: python demo_ai_features.py
13+
"""
14+
15+
import asyncio
16+
import json
17+
from pathlib import Path
18+
19+
from src.ai.content_analyzer import create_content_analyzer
20+
from src.schemas.news import NewsArticle
21+
from src.parsers.ai_enhanced_news import AIEnhancedNewsParser, SmartParserFactory
22+
23+
24+
async def demo_content_analysis():
25+
"""Demonstrate AI-powered content analysis capabilities."""
26+
print("🤖 AI-Enhanced Web Scraper Demo")
27+
print("=" * 50)
28+
29+
# Create a sample article for demonstration
30+
# Only title and url are required - all other fields are Optional
31+
sample_article = NewsArticle( # type: ignore
32+
title="Revolutionary AI Breakthrough Achieves Human-Level Performance",
33+
content="""
34+
In a groundbreaking development, researchers at leading technology institutes
35+
have successfully created an artificial intelligence system that demonstrates
36+
human-level performance across multiple cognitive tasks. This breakthrough
37+
represents years of dedicated research in machine learning and neural networks.
38+
39+
The AI system, trained on diverse datasets, shows remarkable capabilities in
40+
natural language understanding, reasoning, and problem-solving. Early tests
41+
indicate that the technology could have profound implications for industries
42+
ranging from healthcare to autonomous vehicles.
43+
44+
Scientists involved in the project express cautious optimism about the future
45+
applications of this technology, while emphasizing the importance of responsible
46+
development and deployment of advanced AI systems.
47+
""",
48+
url="https://example.com/ai-breakthrough-2024"
49+
)
50+
51+
# Create content analyzer
52+
analyzer = create_content_analyzer(use_mock_llm=True)
53+
54+
print("\n📊 Analyzing the sample article...\n")
55+
56+
print(f"Article: {sample_article.title}")
57+
print("-" * 60)
58+
59+
# Run AI analysis
60+
analysis = await analyzer.analyze_article(sample_article)
61+
62+
# Display results
63+
print(f"📝 Summary: {analysis.summary}")
64+
print(f"😊 Sentiment: {analysis.sentiment} (confidence: {analysis.confidence:.2f})")
65+
print(f"⭐ Quality Score: {analysis.quality_score:.1f}/10")
66+
print(f"📖 Readability: {analysis.readability}")
67+
print(f"🏷️ Topics: {', '.join(analysis.topics)}")
68+
print(f"👥 Entities: {', '.join(analysis.entities[:5])}...")
69+
print(f"🔤 Language: {analysis.language}")
70+
print(f"📊 Word Count: {analysis.word_count}")
71+
print(f"✅ Completeness: {analysis.completeness:.1%}")
72+
print()
73+
74+
75+
async def demo_ai_enhanced_parser():
76+
"""Demonstrate the AI-enhanced parser factory."""
77+
print("🏭 AI-Enhanced Parser Factory Demo")
78+
print("=" * 40)
79+
80+
# Create different parser configurations
81+
configurations = [
82+
("Standard Parser", SmartParserFactory.create_standard_parser()),
83+
("AI-Enhanced Parser", SmartParserFactory.create_enhanced_parser(ai_enabled=True)),
84+
]
85+
86+
for name, parser in configurations:
87+
print(f"\n{name}:")
88+
print(f" • Parser ID: {parser.id}")
89+
print(f" • AI Analysis: {'✅ Enabled' if parser.enable_ai_analysis else '❌ Disabled'}")
90+
print(f" • Content Analyzer: {'✅ Available' if parser.content_analyzer else '❌ None'}")
91+
92+
93+
async def demo_langchain_integration():
94+
"""Demonstrate LangChain integration details."""
95+
print("🔗 LangChain Integration Details")
96+
print("=" * 35)
97+
98+
analyzer = create_content_analyzer()
99+
100+
print("📦 LangChain Components:")
101+
print(f" • Core Framework: {type(analyzer).__name__}")
102+
print(f" • Mock LLM: {'✅ Active' if analyzer.use_mock_llm else '❌ Disabled'}")
103+
print(f" • Prompt Templates: ✅ Configured")
104+
print(f" • Output Parsers: ✅ Available")
105+
106+
print("\n🧠 AI Capabilities:")
107+
capabilities = [
108+
"Content Summarization",
109+
"Sentiment Analysis",
110+
"Topic Classification",
111+
"Entity Extraction",
112+
"Quality Assessment",
113+
"Language Detection",
114+
"Readability Analysis"
115+
]
116+
117+
for capability in capabilities:
118+
print(f" • {capability}: ✅ Implemented")
119+
120+
print("\n🏗️ Architecture Benefits:")
121+
benefits = [
122+
"Pluggable AI components",
123+
"Fallback analysis without LLM",
124+
"Structured output with Pydantic",
125+
"Async processing support",
126+
"Mock LLM for development/testing",
127+
"Extensible prompt templates"
128+
]
129+
130+
for benefit in benefits:
131+
print(f" • {benefit}")
132+
133+
134+
async def main():
135+
"""Run the complete AI features demonstration."""
136+
try:
137+
await demo_content_analysis()
138+
await demo_ai_enhanced_parser()
139+
await demo_langchain_integration()
140+
141+
print("\n🎯 Demo Complete!")
142+
print("\nThis demonstrates:")
143+
print("• LangChain integration for AI-powered content analysis")
144+
print("• Sophisticated prompt engineering and output parsing")
145+
print("• Production-ready error handling and fallbacks")
146+
print("• Type-safe AI results with Pydantic schemas")
147+
print("• Scalable architecture for ML/AI enhancements")
148+
149+
except Exception as e:
150+
print(f"❌ Demo failed: {e}")
151+
import traceback
152+
traceback.print_exc()
153+
154+
155+
if __name__ == "__main__":
156+
print("🚀 Starting AI-Enhanced Web Scraper Demo...")
157+
asyncio.run(main())

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ httpx = "^0.25.0"
1919
aiofiles = "^23.0.0"
2020
kubernetes = "^29.0.0"
2121
structlog = "^23.2.0"
22+
langchain = "^0.3.27"
23+
langchain-community = "^0.3.29"
24+
langchain-huggingface = "^0.3.1"
2225

2326
[tool.poetry.group.dev.dependencies]
2427
pytest = "^7.4.0"

‎requirements.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ aiofiles>=23.0.0
1212
kubernetes>=29.0.0
1313
structlog>=23.2.0
1414

15+
# LangChain AI/ML dependencies
16+
langchain>=0.3.27
17+
langchain-community>=0.3.29
18+
langchain-huggingface>=0.3.1
19+
1520
# Development dependencies
1621
pytest>=7.4.0
1722
pytest-asyncio>=0.21.0

‎src/ai/__init__.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
"""
2+
AI module for advanced content analysis and processing.
3+
"""
4+
5+
from .content_analyzer import AdvancedContentAnalyzer, ContentAnalysis, create_content_analyzer
6+
7+
__all__ = [
8+
"AdvancedContentAnalyzer",
9+
"ContentAnalysis",
10+
"create_content_analyzer"
11+
]

0 commit comments

Comments
 (0)