common-crawl

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

Process Common Crawl data with Python and Spark

News crawling with StormCrawler - stores content as WARC

Statistics of Common Crawl monthly archives mined from URL index files

🕷️ The pipeline for the OSCAR corpus

Drill into WARC web archives

Tools to construct and process Common Crawl webgraphs

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

Various Jupyter notebooks about Common Crawl data

The largest open corpus of classified docx documents

A dataset for knowledge base population research using Common Crawl and DBpedia.

DuckDB extension to fetch pages from Wayback Machine & Common Crawl

[NeurIPS 2024] 🕸 GlotCC Dataset and Pipline

German small and large versions of GPT2.

The website of the Oscar Project

Common Crawl's processing tools

A fast, friendly command line for Common Crawl: URL index search, WARC fetch, Parquet columnar queries, and dataset building.

Lightweight Python utility for retrieving individual pages from the Common Crawl archives.

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

We explore data by using Big Data Analysis and Visualization skills. To obtain this, we perform 3 main operations. i.e. i)Data Aggregation through different sources. ii) Big Data Analysis using MapReduce and iii) Visualization through Tableau. Data Analysis is very critical in understanding the data, and what we can do with the data. For small d…