Skip to content

Commit 843f5fb

Browse files
devilscrapesdevilscrapes
committed
Open Library scraper for JavaScript and Node.js
0 parents  commit 843f5fb

11 files changed

Lines changed: 1395 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
node-version: [20, 24]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
19+
with:
20+
node-version: ${{ matrix.node-version }}
21+
cache: npm
22+
- run: npm ci
23+
- run: node --check index.js
24+
- run: npm test

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.env
2+
results.json
3+
results.csv
4+
node_modules/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/openlibrary-books-scraper"><img src="assets/banner.svg" alt="Open Library Scraper — Book Metadata in Bulk — JavaScript example" width="100%"></a></p>
2+
3+
# Open Library Scraper for JavaScript and Node.js: book lookup by title, author or ISBN
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/openlibrary-books-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="JavaScript" src="https://img.shields.io/badge/Node.js-20%2B-339933?logo=nodedotjs&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/openlibrary-scraper-javascript/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/openlibrary-scraper-javascript/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
This repository calls the **[Open Library Books Scraper](https://apify.com/DevilScrapes/openlibrary-books-scraper)** from Node.js using one ES module and the official `apify-client` package. Send a search string, optionally tied to the title, author, subject or ISBN field, and get back plain objects describing works from Open Library: names, years, edition counts, ISBN arrays, publishers, a large cover URL and the e-book status. The work happens on Apify, so a route handler, a queue worker or a scheduled function only has to start a run and collect rows. Reading apps, bookshop backends and catalogue tools all get the same shape every time, which is the whole bargain; no deals with the devil required.
13+
14+
> **Actor:** [Open Library Scraper — Book Metadata in Bulk](https://apify.com/DevilScrapes/openlibrary-books-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in Python:** [openlibrary-scraper-python](https://github.com/devilscrapes/openlibrary-scraper-python)
15+
16+
## 🔥 What you get
17+
18+
- **Works, not editions**: each row is one Open Library work, with `edition_count` telling you how many printings and translations sit behind it.
19+
- **Point the query at one field**: `searchField` sends your text to title, author, subject or ISBN matching, or leaves it free-form with `all`.
20+
- **A language filter**: a three-letter code such as `eng` or `spa` in `language` is passed to Open Library's search as a language filter.
21+
- **Identifiers you can join on**: the work key, a canonical `work_url`, every ISBN-10 and ISBN-13 Open Library lists, and the numeric `cover_id`.
22+
- **A ready cover link**: `cover_url_l` is built from the cover ID and points at the large JPEG on Open Library's cover service.
23+
- **Reader signals where they exist**: average rating, rating count and an `ebook_access` value that says whether a scan is public, borrowable or absent.
24+
- **Bounded subject lists**: subjects are cut at 30 per work, so one heavily tagged classic cannot bloat a CSV column.
25+
26+
## 🎯 Use cases
27+
28+
### Power an ISBN lookup endpoint
29+
30+
Expose `GET /books/:isbn` in Express or Fastify, start a run with `searchField: 'isbn'` and `maxResults: 1`, cache the row by ISBN and answer from the cache next time. Your users get title, author and cover from one scan of a barcode.
31+
32+
### Fill in cover art for a reading app
33+
34+
When a user adds a book by title, look it up and store `cover_url_l` next to it. The link points straight at Open Library's cover service, so your front end can render it in an `<img>` tag without hosting images yourself.
35+
36+
### Flag books that can be read online
37+
38+
Filter rows on `ebook_access` and show a 'read now' badge for `public` or `borrowable` works, linking to `work_url`. One `Set` lookup on a single field is all the logic it takes.
39+
40+
### Seed a recommendations service
41+
42+
Pull works for a few subjects on a schedule and store `subjects`, `authors` and `ratings_average` in your database. A recommender can then suggest books that share tags with the ones a reader already finished.
43+
44+
## ⚡ Quick start
45+
46+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
47+
48+
```bash
49+
git clone https://github.com/devilscrapes/openlibrary-scraper-javascript.git
50+
cd openlibrary-scraper-javascript
51+
npm install
52+
export APIFY_TOKEN=your_token_here
53+
node index.js # uses input.json
54+
node index.js my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [Open Library Scraper — Book Metadata in Bulk](https://apify.com/DevilScrapes/openlibrary-books-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
The integration your service needs is just the following. `call()` resolves after the run has finished, and `listItems()` hands back that run's dataset as one array of objects.
62+
63+
```javascript
64+
import { ApifyClient } from 'apify-client';
65+
66+
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
67+
const run = await client.actor('DevilScrapes/openlibrary-books-scraper').call({"searchQuery": "isaac asimov foundation", "searchField": "all", "maxResults": 10, "language": "", "proxyConfiguration": {"useApifyProxy": false}});
68+
const { items } = await client.dataset(run.defaultDatasetId).listItems();
69+
console.log(items);
70+
```
71+
72+
## 📥 Input
73+
74+
`index.js` reads `input.json` and sends it unchanged. This sample matches the Actor's shipped defaults: Asimov's Foundation as free text, at most 10 works, every language, direct connection, for about 21.5 cents. Only `searchQuery` is mandatory. From a request handler, set `searchField` to the kind of value the user typed and keep `maxResults` small for interactive lookups; bulk jobs can go up to 1,000.
75+
76+
```json
77+
{
78+
"searchQuery": "isaac asimov foundation",
79+
"searchField": "all",
80+
"maxResults": 10,
81+
"language": "",
82+
"proxyConfiguration": {
83+
"useApifyProxy": false
84+
}
85+
}
86+
```
87+
88+
| Field | Type | Required | Default | Allowed values |
89+
|---|---|---|---|---|
90+
| `searchQuery` | string | yes | `"isaac asimov foundation"` | — |
91+
| `searchField` | string | no | `"all"` | `all`, `title`, `author`, `subject`, `isbn` |
92+
| `maxResults` | integer | no | `30` | — |
93+
| `language` | string | no | `""` | — |
94+
| `proxyConfiguration` | object | no | `{"useApifyProxy": false}` | — |
95+
96+
## 📤 Output
97+
98+
You get an array of works. Treat `openlibrary_key`, `title` and `work_url` as guaranteed and type the rest as optional. Array fields come back as empty arrays rather than null when Open Library has nothing, so `.join()` and `.length` are safe on them.
99+
100+
| Field | Type | Always present |
101+
|---|---|---|
102+
| `openlibrary_key` | string | yes |
103+
| `title` | string | yes |
104+
| `subtitle` | string or null | no |
105+
| `authors` | array | no |
106+
| `first_publish_year` | integer or null | no |
107+
| `edition_count` | integer | no |
108+
| `languages` | array | no |
109+
| `subjects` | array | no |
110+
| `isbns` | array | no |
111+
| `publishers` | array | no |
112+
| `cover_id` | integer or null | no |
113+
| `cover_url_l` | string or null | no |
114+
| `ratings_average` | number or null | no |
115+
| `ratings_count` | integer or null | no |
116+
| `ebook_access` | string or null | no |
117+
| `work_url` | string | yes |
118+
| `scraped_at` | string | no |
119+
120+
## 💰 Pricing
121+
122+
| Event | Price (USD) |
123+
|---|---|
124+
| Actor start | $0.2 |
125+
| Result emitted | $0.0015 |
126+
127+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/openlibrary-books-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
128+
129+
## 🛠️ Working with the data in JavaScript
130+
131+
This module turns the saved rows into a small 'shelf' for a front end: only works with a public or borrowable e-book, with the first ISBN-13 picked out, sorted oldest first and written to `shelf.json`. Node.js built-ins only.
132+
133+
```javascript
134+
import { readFile, writeFile } from 'node:fs/promises';
135+
136+
const books = JSON.parse(await readFile('results.json', 'utf8'));
137+
const readable = new Set(['public', 'borrowable']);
138+
139+
const shelf = books
140+
.filter((book) => readable.has(book.ebook_access))
141+
.map((book) => ({
142+
id: book.openlibrary_key.replace('/works/', ''),
143+
title: book.subtitle ? `${book.title}: ${book.subtitle}` : book.title,
144+
by: (book.authors ?? []).join(', '),
145+
year: book.first_publish_year,
146+
cover: book.cover_url_l,
147+
isbn13: (book.isbns ?? []).find((code) => /^97[89]\d{10}$/.test(code)) ?? null,
148+
link: book.work_url,
149+
}))
150+
.sort((a, b) => (a.year ?? 9999) - (b.year ?? 9999));
151+
152+
await writeFile('shelf.json', JSON.stringify(shelf, null, 2));
153+
console.table(shelf.map(({ title, by, year, isbn13 }) => ({ title, by, year, isbn13 })));
154+
```
155+
156+
## ❓ FAQ
157+
158+
### Should I call Open Library's search endpoint directly instead?
159+
160+
For a handful of lookups you can. The Actor earns its keep when you want paging, a fixed row shape, scheduling and run history handled for you, and when Apify's dataset is where the results should live anyway.
161+
162+
### Does it work in a serverless function?
163+
164+
Small lookups finish quickly, so awaiting `call()` usually fits in a function's time limit. For bulk pulls, start the run with `.start()` and pick up the dataset from a webhook when Apify reports it finished.
165+
166+
### How do I type the rows in TypeScript?
167+
168+
`apify-client` brings its own declarations for runs and datasets. Add a `Work` interface from the output table above, with `openlibrary_key`, `title` and `work_url` as required strings, and cast `items` to `Work[]`.
169+
170+
### Why did a search return fewer works than `maxResults`?
171+
172+
Either the query genuinely matched fewer works, or Open Library answered one page with an error, which ends paging without a retry. A language filter also narrows results more than people expect.
173+
174+
## 🧯 Troubleshooting
175+
176+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
177+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
178+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
179+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/openlibrary-books-scraper) so the maintainers see it.
180+
181+
## 🔗 More from Devil Scrapes
182+
183+
- [Open Library Scraper — Book Metadata in Bulk on Apify Store](https://apify.com/DevilScrapes/openlibrary-books-scraper)
184+
- [Python version of this example](https://github.com/devilscrapes/openlibrary-scraper-python)
185+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
186+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
187+
188+
Related Actors:
189+
190+
- [Goodreads Book & Reviews Scraper](https://apify.com/DevilScrapes/goodreads-book-reviews-scraper) — reader reviews to set next to Open Library's catalogue data.
191+
- [Crossref Works Scraper — DOIs, Authors & Citations](https://apify.com/DevilScrapes/crossref-works-scraper) — DOI metadata for journal articles and scholarly books.
192+
- [arXiv Scraper — Search & Export Paper Metadata](https://apify.com/DevilScrapes/arxiv-papers-scraper) — preprint records when the reading list turns academic.
193+
194+
---
195+
196+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/openlibrary-books-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with Open Library or the Internet Archive.

‎assets/banner.svg‎

Lines changed: 49 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)