Skip to content

README.md

Latest commit

 

History

History
231 lines (168 loc) · 9.69 KB

README.md

File metadata and controls

231 lines (168 loc) · 9.69 KB
Raw

FastFileScrape 0.1.0 [ALPHA-2026-05-25] — Ultra‑Fast File Tree & Content Scraper for Java

Status License: MIT Java Platform JitPack

⚡ Scrape and process millions of files in milliseconds with zero latency.

FastFileScrape is the high‑speed file scraping module of the FastJava ecosystem.
It provides two core capabilities:

  • FastFileTree — build complete directory trees with include/exclude rules
  • FastFileScrapeContent — extract file contents with chunking for LLMs and agents

Watch Demo (YouTube) | Watch JMH Benchmark (Youtube)

FastFileScrape Showcase

Quick Start

import fastfilescrape.*;
import java.nio.file.Path;
import java.util.List;

public class Demo {
    public static void main(String[] args) throws Exception {
        // 1. FastFileTree — Build and print directory tree
        var tcfg = new FastFileTree.Config();
        tcfg.root = Path.of(".");
        var tree = FastFileTree.build(tcfg);
        FastFileTree.printTree(tree, System.out);

        // 2. FastFileScrapeContent — Extract chunked contents for LLMs & agents
        var ccfg = new FastFileScrapeContent.Config();
        ccfg.root = Path.of(".");
        ccfg.includeGlobs = List.of("**/*.java");

        FastFileScrapeContent.scrape(ccfg, (file, chunk, text) -> {
            System.out.println("=== " + file + " (chunk " + chunk + ") ===");
            System.out.println(text);
        });
    }
}

CLI Tool — fastfilescrape

# Show directory tree
fastfilescrape tree --root . --include "**/*.java"

# Extract file contents
fastfilescrape content --root . --include "**/*.java" --out repo.txt

# Tree + Content in JSONL
fastfilescrape all --root . --include "**/*.java" --format jsonl --out repo.jsonl

Table of Contents

Why FastFileScrape?

Java's standard Files.walk() and Files.readString() were never designed for scraping millions of files at AI agent speed:

  1. JVM Syscall Bottlenecks: Standard Files.walk() invokes individual JVM filesystem syscalls per directory entry, choking on deep hierarchies with hundreds of thousands of files.
  2. Regex Compilation Overhead: Java's PathMatcher compiles and evaluates regex patterns per match, wasting millions of CPU cycles during glob filtering.
  3. Sequential I/O & Memory Bloat: Reading files one-by-one via Files.readString() creates severe I/O bottlenecks and massive JVM heap churn from intermediate String allocations.
  4. Token-Unsafe LLM Chunking: Naive string slicing splits multi-byte UTF-8 codepoints or cuts mid-token, corrupting prompt context for AI and RAG ingestion pipelines.

FastFileScrape solves this by combining native Win32 FastGLOB directory scanning with parallel token-safe chunking:

  • Native Directory Traversal: Uses FastGLOB native Win32 batch traversal to discover files orders of magnitude faster than Files.walk().
  • Fast-Path Glob Matching: Performs instant substring checks with regex fallback, eliminating regex compile bottlenecks.
  • Parallel Chunked Ingestion: Streams and parses multiple files concurrently with boundary-safe UTF-8 chunking.
  • Built-In LLM Guards: Automatic binary detection, size limits (maxFileSizeBytes), and UTF-8 boundary protection.
Feature Standard Java (java.nio) Apache Commons IO FastFileScrape
Directory Traversal Files.walk() (Slow syscalls) FileUtils.listFiles() (Sequential) FastGLOB native Win32 batch scan
Glob Matching PathMatcher (Regex per match) WildcardFileFilter (Linear) FastGLOB native + string fast-path
File Ingestion Sequential single-file read Iterative FileUtils reads Parallel multi-file concurrent stream
LLM Token Chunking ❌ Manual (Risk of broken tokens) ❌ None Built-in UTF-8 boundary-safe 64 KB chunks
Binary & Size Guards ❌ Manual checking required ⚠️ Basic filter utils Auto binary detection + size limits
Heap Memory Overhead High String & Path churn High collection object churn Low-allocation streaming callbacks

When an AI agent needs to read an entire codebase into context — say, 2 000 .java files across 400 folders — standard Java spends most of its time in filesystem overhead and sequential I/O. FastFileScrape does the traversal natively, filters with a string fast-path, and reads all matching files in parallel. The Sink callback streams chunks directly to the agent pipeline without buffering the entire repo in memory.

Technical Demos & Benchmarks

Run standalone verification demos or execute JMH throughput benchmarks:

Type Target / Launcher Source File Description
Interactive Demo run-demo.bat Demo.java Live directory tree generation and chunked repository content streaming
Throughput Benchmark run-benchmark.bat Benchmark.java JMH benchmark evaluating tree building and file ingestion throughput

Installation

Option 1: Maven (Recommended)

Add the JitPack repository and the dependencies to your pom.xml:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileScrape</artifactId>
        <version>0.1.0</version>
    </dependency>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastGLOB</artifactId>
        <version>0.1.0</version>
    </dependency>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastCore</artifactId>
        <version>v1.0.0</version>
    </dependency>
</dependencies>

Option 2: Gradle (via JitPack)

repositories {
    maven { url 'https://jitpack.io' }
}
dependencies {
    implementation 'com.github.andrestubbe:FastFileScrape:0.1.0'
    implementation 'com.github.andrestubbe:FastGLOB:0.1.0'
    implementation 'com.github.andrestubbe:FastCore:v1.0.0'
}

Option 3: Direct Download (No Build Tool)

Download the pre-compiled JARs to add them to your classpath:

  1. 📦 FastFileScrape-0.1.0.jar (The Scraper Core Library)
  2. 📦 FastGlob-0.1.0.jar (The Native Glob Matching Library)
  3. ⚙️ fastcore-v1.0.0.jar (The Mandatory JNI Loader)

Important

Since FastFileScrape is natively accelerated, all three JARs must be present in your classpath for the JNI-accelerated directory walking to operate correctly on Windows.

API Reference

FastFileTree

Method Description
Node build(Config cfg) Builds the directory tree
printTree(Node, Appendable) Prints ASCII tree

FastFileScrapeContent

Method Description
scrape(Config cfg, Sink sink) Reads files and emits chunks

Documentation

  • COMPILE.md: Full compilation guide (MSVC C++17 build chain + JNI Setup).
  • REFERENCE.md: Full API descriptions, border configurations, and codepoint index.
  • PHILOSOPHY.md: The engineering rationale for zero-allocation performance.
  • ROADMAP.md: Future milestones and planned features.

Platform Support

Platform Status
Windows 10/11 ✅ Fully Supported
Linux 🚧 Planned
macOS 🚧 Planned

License

MIT License — See LICENSE file for details.

Related Projects

Part of the FastJava Ecosystem — Making the JVM faster. Small package. Maximum speed. Zero bloat. 🚀📋