Periodic Table Scraper

A Scrapy spider that extracts all 118 elements from the PubChem Periodic Table using Playwright for JavaScript rendering.

Features

• Scrapes all 118 chemical elements from PubChem
• Uses Playwright to handle JavaScript-rendered content
• Extracts: symbol, name, atomic number, atomic mass, and chemical group
• Clean data processing with ItemLoaders

Prerequisites

• Python 3.10+
• Playwright browsers

Installation

1. Clone the repository:

   git clone <repository-url>
   cd scrapingproject/elements

2. Install dependencies:

   pip install scrapy scrapy-playwright itemloaders w3lib

3. Install Playwright browsers:

   playwright install chromium

Usage

Run the spider from the elements directory:

# Output to console
scrapy crawl periodic_elements

# Export to JSON
scrapy crawl periodic_elements -o elements.json

# Export to CSV
scrapy crawl periodic_elements -o elements.csv

Output Schema

Each element includes:

Field	Type	Description
atomic_number	int	Element's atomic number (1-118)
symbol	str	Chemical symbol (e.g., "H", "He")
name	str	Element name (e.g., "Hydrogen")
atomic_mass	float	Atomic mass in unified atomic mass units
chemical_group	str	Classification (e.g., "Alkali metal", "Noble gas")

Project Structure

elements/
├── scrapy.cfg
└── elements/
    ├── __init__.py
    ├── items.py          # Item definitions with processors
    ├── middlewares.py    # Spider/downloader middlewares
    ├── pipelines.py      # Item pipelines
    ├── settings.py       # Scrapy + Playwright configuration
    └── spiders/
        ├── __init__.py
        └── periodic_elements.py  # Main spider

Configuration

Key settings in settings.py:

• Playwright handlers enabled for HTTP/HTTPS
• ROBOTSTXT_OBEY: True (respects robots.txt)
• DOWNLOAD_DELAY: 1 second between requests
• CONCURRENT_REQUESTS_PER_DOMAIN: 1

License

MIT