Skip to content

Commit 28153fc

Browse files
docs(storage): document GCS migration and recovery workflow
Document bucket and authentication setup, provider configuration, verified transfer and URL cutover, validation results, rollback, and delayed source cleanup.
1 parent 12d7dfc commit 28153fc

6 files changed

Lines changed: 382 additions & 5 deletions

File tree

‎docs/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010

1111
- [Getting Started](getting-started.md) — Prerequisites, environment setup, database, running the app
1212
- [Deployment](deployment.md) — VPS, Docker, Vercel+Railway deployment options
13+
- [Paper PDF Storage Migration](paper-storage-gcs-migration-plan.md) — GCS configuration, transfer, URL cutover, rollback, and delayed source cleanup
14+
- [PDF Migration Record](paper-storage-gcs-migration-report.md) — Completed cutover, validation evidence, and recovery-window cleanup
1315

1416
## Architecture
1517

‎docs/api/papers.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@ Query parameters:
2727

2828
Returns an array of Paper objects.
2929

30+
Paper responses include the read-only `hasStoredPdf` boolean. It identifies a URL
31+
in the configured Supabase PDF bucket or GCS PDF bucket; it does not perform a
32+
live file-existence check. External publisher URLs return `false`. Clients should
33+
use this field for stored-PDF filters and controls rather than inspecting URL
34+
substrings. No database column or paper-ID migration is required.
35+
3036
### POST /api/papers
3137

3238
Optional query parameter: `?check_duplicates=true`
@@ -120,7 +126,7 @@ Response on duplicate (`200`):
120126
}
121127
```
122128

123-
After creation, if the paper has a `pdf_url`, a background task auto-downloads the PDF to Supabase Storage. If the library has `auto_note_enabled`, AI notes are generated in the background.
129+
After creation, if the paper has a `pdf_url`, a background task auto-downloads the PDF to the configured PDF storage provider. If the library has `auto_note_enabled`, AI notes are generated in the background.
124130

125131
---
126132

@@ -204,19 +210,19 @@ Upload a PDF file for a paper.
204210

205211
Request: `multipart/form-data` with field `file` (must be `application/pdf` or `application/octet-stream`).
206212

207-
Stores the PDF in Supabase Storage at `{paper_id}.pdf`, updates `papers.pdf_url` to the public storage URL, then queues AI note generation as a background task.
213+
Stores the PDF in the configured storage provider at `{paper_id}.pdf`, updates `papers.pdf_url` to the public storage URL, invalidates cached extracted text, then queues AI note generation as a background task.
208214

209215
Returns the updated Paper object.
210216

211217
### DELETE /api/papers/{id}/pdf
212218

213-
Remove the paper's PDF from Supabase Storage, clear `pdf_url`, and delete any cached extracted text.
219+
Remove the paper's PDF from its recorded managed storage location, clear `pdf_url`, and delete any cached extracted text. Storage failures preserve the database reference for retry.
214220

215221
Returns `204 No Content`.
216222

217223
### POST /api/papers/{id}/pdf/fetch
218224

219-
Download the paper's existing external `pdf_url` into Supabase Storage.
225+
Download the paper's existing external `pdf_url` into the configured storage provider. PDFs already in either configured managed bucket are rejected as already stored.
220226

221227
Use this for papers imported via DOI/arXiv where the PDF URL was set but not yet downloaded. Returns `422` if the paper has no `pdf_url` or the PDF is already in storage.
222228

‎docs/database/schema.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ Academic papers and preprints. The primary library item type.
7979
| `agent_reasoning` | TEXT | YES | — | Agent's explanation of relevance |
8080
| `rejected` | BOOLEAN | NO | `false` | Whether rejected during agent review |
8181
| `collections` | JSONB | NO | `[]` | Array of collection IDs this paper belongs to |
82-
| `pdf_url` | TEXT | YES | — | Public URL to PDF (Supabase storage or external) |
82+
| `pdf_url` | TEXT | YES | — | Public URL to PDF (configured GCS/Supabase storage or external) |
8383
| `github_url` | TEXT | YES | — | Related GitHub repository URL |
8484
| `website_url` | TEXT | YES | — | Related project website URL |
8585
| `links` | JSONB | NO | `[]` | Array of `{name, url}` named links |

‎docs/deployment.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,20 @@ The backend reads from `backend/.env`:
3737
| `SUPABASE_URL` | Yes | Your Supabase project URL (e.g. `https://abc.supabase.co`) |
3838
| `SUPABASE_KEY` | Yes | Supabase publishable (anon) key |
3939
| `OPENAI_API_KEY` | No | OpenAI API key. AI features disabled without it. |
40+
| `PAPER_STORAGE_PROVIDER` | No | `supabase` (default) or `gcs`; controls new PDF uploads only |
41+
| `GCS_PROJECT_ID` | For GCS | Google Cloud project used by the storage client |
42+
| `GCS_PDF_BUCKET` | For GCS | Dedicated public-read PDF bucket; also used to recognize managed GCS URLs |
43+
44+
For local GCS access, run `gcloud auth application-default login`. CLI login alone
45+
does not authenticate the Python client. On a hosted backend, supply an attached
46+
service account or workload identity with bucket-level Storage Object User access.
47+
Never expose credentials in frontend configuration. Supabase credentials remain
48+
required for the database. Follow the [PDF migration runbook](paper-storage-gcs-migration-plan.md)
49+
for bucket setup, CORS, copying, verified URL updates, rollback, and delayed cleanup.
50+
51+
For a brief storage cutover, create `backend/data/migration/maintenance.flag` to
52+
return HTTP 503 for API mutations while allowing reads; remove it when finished.
53+
The marker is local to this checkout, so coordinate every instance before using it.
4054

4155
## Supabase Setup
4256

0 commit comments

Comments
 (0)