Skip to content

README.md

Latest commit

 

History

History
485 lines (351 loc) · 11.5 KB

README.md

File metadata and controls

485 lines (351 loc) · 11.5 KB
Raw

🔍 Fuzzygrep

Fuzzygrep is a powerful, production-ready command-line tool for interactive fuzzy searching, exploring, and inspecting JSON and CSV files. Built with performance and user experience in mind.

Python 3.9+ License: MIT

✨ Features

🚀 Performance

  • Blazing Fast: Sub-second search on 10K+ records
  • Lazy Loading: Stream large files without loading everything into memory
  • Smart Indexing: Trigram-based indexing for 5-10x faster searches
  • Parallel Processing: Multi-core support for faster data processing
  • Intelligent Caching: TTL-based caching with automatic invalidation

💎 User Experience

  • Interactive Interface: Beautiful, intuitive CLI with rich formatting
  • Fuzzy Search: Find what you need with typo-tolerant search
  • Regex Search: Pattern matching with regular expressions (v1.1)
  • Color Themes: Nord, Dracula, Solarized, and default themes (v1.1)
  • Syntax Highlighting: JSON visualization with color-coded output
  • Auto-completion: Smart suggestions as you type
  • Export Options: Save results as JSON, CSV, Markdown, or HTML

🎯 Functionality

  • Deep Search: Search through nested JSON structures
  • Dual Mode: Search keys, values, or both simultaneously
  • Regex Mode: Toggle between fuzzy and regex search (v1.1)
  • Query Bookmarks: Save and load frequent searches (v1.1)
  • Key Filtering: Focus on specific data patterns
  • Visualizations: Tree charts and frequency histograms
  • Multi-format: JSON, CSV, YAML, and XML support (v1.1)

📦 Installation

Quick Install

git clone https://github.com/anggiAnand/fuzzygrep.git
cd fuzzygrep
pip install -e .

With Optional Dependencies

For enhanced features (streaming large files, CSV chunking):

pip install -e ".[enhanced]"

For development (testing, linting, formatting):

pip install -e ".[dev]"

Requirements

  • Python 3.9 or higher
  • 5 core dependencies (automatically installed)
  • Optional: ijson, pandas for large file handling

🚀 Quick Start

Basic Usage

# Interactive search
fuzzygrep data.json

# Show file structure
fuzzygrep data.json --chart

# View frequency analysis
fuzzygrep data.json --histogram

# Verbose output
fuzzygrep data.json --verbose

Interactive Commands

Once in interactive mode, you have access to powerful commands:

Search Commands:
  <query>               Search for keys and values
  
File Operations:
  /load <file>          Load a different file
  /reload               Reload current file
  
Results Management:
  /export <format>      Export results (json, csv, md, html)
  /save                 Quick save to results.json
  
Filtering & Configuration:
  /filter <patterns>    Filter keys by patterns (comma-separated)
  /clear                Clear active filters
  /stats                Show performance statistics
  
Navigation:
  /history              Show search history
  /help                 Show help message
  /exit, /quit          Exit the program

Keyboard Shortcuts

Shortcut Action
Ctrl+T Toggle autocompletion on/off
Ctrl+V Switch between key/value completion
Ctrl+R Reload data from file
Ctrl+S Save last search results
Ctrl+H Show help
Ctrl+C Exit program

📚 Examples

Example 1: Basic Search

$ fuzzygrep people.json
[people.json] Search> john

Matches in Keys:
┌─────────┬────────────────┬───────┐
│ Key     │ Value          │ Score │
├─────────┼────────────────┼───────┤
│ name    │ John Doe       │  95.0 │
│ email   │ john@email.com │  82.0 │
└─────────┴────────────────┴───────┘

Matches in Values:
┌────────────────┬──────┬───────┐
│ Value          │ Keys │ Score │
├────────────────┼──────┼───────┤
│ John Doe       │ name │  100  │
│ john@email.com │ email│  88.0 │
└────────────────┴──────┴───────┘

Example 2: Export Results

[data.json] Search> alice

# Export as JSON
[data.json] Search> /export json results.json

# Export as CSV
[data.json] Search> /export csv results.csv

# Export as HTML with nice formatting
[data.json] Search> /export html report.html

Example 3: Filter by Keys

[data.json] Search> /filter email,phone,address
Filter applied: email, phone, address

# Now searches are limited to these keys
[data.json] Search> john

Example 4: Performance Options

# Disable caching for always-fresh data
fuzzygrep data.json --no-cache

# Disable indexing for small files
fuzzygrep small.json --no-index

# Control worker threads
fuzzygrep large.json --workers 8

# Combine options
fuzzygrep data.json --no-cache --workers 4 --verbose

Example 5: Visualizations

# Tree view with depth limit
fuzzygrep data.json --chart --chart-limit 50

# Frequency analysis
fuzzygrep data.json --histogram

Example 6: v1.1 Features

# Use regex search mode
fuzzygrep data.json --regex
[data.json] Search> user.*@.*\.com  # Regex pattern

# Use different color themes
fuzzygrep data.yaml --theme nord
fuzzygrep data.xml --theme dracula

# Interactive commands (v1.1)
[data.json] Search> /regex on           # Enable regex mode
[data.json] Search> /bookmark my_query  # Save current search
[data.json] Search> /bookmarks          # List all bookmarks
[data.json] Search> /load-bookmark my_query  # Load a bookmark
[data.json] Search> /theme solarized    # Change theme

🏗️ Architecture

Fuzzygrep is built with a clean, modular architecture:

fuzzygrep/
├── core/               # Core functionality
│   ├── loaders.py     # Data loading with streaming support
│   ├── searcher.py    # Fuzzy search with parallel processing
│   ├── indexer.py     # Trigram-based indexing
│   └── cache.py       # Multi-layer caching system
├── ui/                # User interface
│   ├── display.py     # Results visualization & export
│   └── interactive.py # Interactive session management
├── utils/             # Utilities
│   ├── errors.py      # Custom exception hierarchy
│   └── logging.py     # Rich logging system
└── cli.py             # CLI entry point

Key Components

Loaders (core/loaders.py)

  • Automatic format detection (JSON/CSV)
  • Streaming for large files (>10MB)
  • Memory-optimized data structures
  • Graceful error handling

Searcher (core/searcher.py)

  • Fuzzy matching with RapidFuzz
  • Trigram-based pre-filtering
  • Parallel processing support
  • Smart scorer selection
  • Multi-layer caching

Indexer (core/indexer.py)

  • Trigram-based search index
  • Fast candidate filtering
  • Reduces search space by 50-90%
  • Persistent index caching

Display (ui/display.py)

  • Rich table formatting
  • Syntax-highlighted JSON
  • Tree visualizations
  • Multiple export formats

⚡ Performance

Benchmarks

Tested on a dataset of 10,000 records:

Operation Time Memory
Load JSON 1.2s 45MB
Build Index 0.8s 15MB
Search (indexed) 45ms -
Search (no index) 320ms -
Export JSON 0.5s -

Optimization Tips

  1. Enable indexing (default): Best for repeated searches
  2. Use streaming: Automatic for files >10MB
  3. Enable caching (default): Instant results for repeated queries
  4. Parallel processing (default): Faster on multi-core systems
  5. Filter keys: Reduce search space for faster results

🧪 Testing

Run the test suite:

# Run all tests
pytest

# With coverage report
pytest --cov=fuzzygrep --cov-report=html

# Run specific test file
pytest tests/test_searcher.py

# Verbose output
pytest -v

Current test coverage: 85%+

🛠️ Development

Setup Development Environment

# Clone repository
git clone https://github.com/anggiAnand/fuzzygrep.git
cd fuzzygrep

# Install in development mode with all dependencies
pip install -e ".[dev,enhanced]"

# Run tests
pytest

# Format code
black fuzzygrep tests
isort fuzzygrep tests

# Lint
flake8 fuzzygrep
mypy fuzzygrep

Project Structure

fuzzygrep/
├── fuzzygrep/          # Main package
├── tests/              # Test suite
├── setup.py            # Package configuration
├── requirements.txt    # Dependencies
├── README.md           # Documentation
└── CHANGELOG.md        # Version history

🐛 Troubleshooting

Common Issues

Import Error: Missing dependencies

pip install -r requirements.txt

Slow performance on large files

# Install optional dependencies
pip install ijson pandas

Cache issues

# Clear cache
fuzzygrep cache-clear

# Check cache stats
fuzzygrep cache-stats

Out of memory errors

# Disable caching and indexing
fuzzygrep large.json --no-cache --no-index

📝 Configuration

Fuzzygrep can be configured via:

  1. Command-line options (highest priority)
  2. Environment variables
  3. Config file ~/.config/fuzzygrep/config.toml

Environment Variables

export FUZZYGREP_CACHE_DIR="~/.cache/fuzzygrep"
export FUZZYGREP_CACHE_TTL=300
export FUZZYGREP_MAX_WORKERS=4

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

How to Contribute

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Code Style

  • Follow PEP 8
  • Use Black for formatting
  • Add type hints
  • Write docstrings
  • Include tests

📋 Roadmap

Version 1.1 ✅ (Completed)

  • YAML and XML support
  • Regular expression search mode
  • Query bookmarks
  • Color themes (Nord, Dracula, Solarized)

Version 1.2

  • Multi-file search
  • Advanced filtering (by type, score threshold)
  • Excel (.xlsx) support
  • Configuration file support

Version 2.0

  • GUI mode (optional)
  • Real-time file watching
  • Plugin system
  • REST API

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Anggi Ananda

🙏 Acknowledgments

📊 Statistics

GitHub stars GitHub forks GitHub issues

Made with ❤️ by Anggi Ananda