FlowCraft is an intelligent, visual web automation and scraping platform. It enables engineers, analysts, and teams to build resilient browser automation workflows using an intuitive drag-and-drop node graph canvas—paired with Google Gemini AI to dynamically parse, understand, and extract unstructured web data without brittle CSS/XPath selectors.
- Visual DAG Workflow Builder: Design complex scraping logic with drag-and-drop nodes powered by
@xyflow/react. - AI-Enhanced Extraction: Harness Google Gemini 1.5 LLM reasoning to extract structured JSON from raw HTML, immune to website layout changes.
- Headless Browser Execution: Automated page navigation, typing, clicking, waiting, and scrolling via Puppeteer.
- Scheduled Cron Triggers: Publish workflows and automate recurring execution with cron-parser scheduling and telemetry logs.
- Enterprise-Grade Security: Encrypt sensitive third-party API keys and credentials with AES-256-CBC at rest.
- Modular Data Delivery: Stream extracted data directly to custom webhook endpoints or export as structured JSON.
| Layer | Technologies |
|---|---|
| Frontend & UI | Next.js 15 (App Router), React 18, Tailwind CSS, Radix UI, Lucide Icons, Framer Motion |
| Workflow Canvas | React Flow (@xyflow/react), TanStack React Query |
| Automation Engine | Puppeteer (Headless Chrome), Cheerio |
| AI Integration | Google Generative AI SDK (@google/generative-ai / Gemini 1.5 Flash) |
| Database & ORM | SQLite (Default for zero-config local run) / PostgreSQL / MySQL via Prisma ORM |
| Authentication | Custom stateless JWT with signed HTTP-only cookies (jose, bcryptjs) |
| Scheduling | cron-parser, cronstrue |
Follow these steps to set up and run FlowCraft on your local machine for development.
- Node.js:
v18.xor later (tested on Node v20+) - npm:
v9.xor later - Git: Version control system
-
Clone the repository:
git clone https://github.com/rishab2211/AI-Powered-No-Code-Web-Scraping-Automation-tool.git cd AI-Powered-No-Code-Web-Scraping-Automation-tool -
Install dependencies:
npm install
-
Configure environment variables: Copy the example environment template:
cp .env.example .env
Generate secure secrets for JWT and encryption:
NEXT_PUBLIC_APP_URL="http://localhost:3000" DATABASE_URL="file:./dev.db" JWT_SECRET="your_custom_jwt_secret_key_minimum_32_characters" ENCRYPTION_SECRET_KEY="your_32_byte_aes_encryption_key_here" API_SECRET="your_internal_api_secret_key_here" GEMINI_API_KEY="your_google_gemini_api_key_here"
-
Initialize database schema: Push the Prisma schema to create the local SQLite database:
npx prisma db push
-
Start local development server:
npm run dev
Open http://localhost:3000 in your browser.
Choreograph execution graphs with specialized node types:
- Browser Nodes:
Launch Browser,Navigate to URL,Scroll Page. - Interaction Nodes:
Click Element,Fill Input,Wait for Element. - Extraction Nodes:
Page to HTML,Extract Text via Selector,Extract Data with AI (Gemini). - Delivery Nodes:
Deliver via Webhook,JSON Storage.
- All sensitive credentials (such as external Gemini API keys and basic auth passwords) are encrypted using AES-256-CBC with unique initialization vectors before writing to the database.
- Route handlers enforce strict JWT validation and ownership checks so users can only view, edit, and trigger their own automation pipelines.
For exhaustive architecture specifications, database schemas, API references, operational runbooks, and design decisions (ADR), refer to DOCUMENTATION.md.
This project is licensed under the MIT License.