Create Doc behind login

[JC1738]

Is there a way to crawl a knowledge base that is behind a login?

Thank you

[Shubham-Khichi]

No not yet but it's a feature requested by our enterprise users. I have added it to the roadmap and I will implement it soon. Thanks for bringing it up.

[traycerai]

Plan

Observations

The current DevDocs architecture is well-structured with clear separation between frontend (Next.js), backend (FastAPI), and crawling service (Crawl4AI). However, the Crawl4AI service limitation is a significant constraint that requires a pragmatic implementation approach.

The codebase shows good security practices with no credential leakage in logs, proper error handling, and clean separation of concerns. The frontend uses modern React patterns with Radix UI components that can be extended for authentication forms.

The status management system is robust and can be extended to track authentication states. The Docker environment is properly configured and can accommodate the new authentication features with minimal changes.

Approach

Based on my analysis of the DevDocs codebase, I've identified that the current Crawl4AI service only supports Bearer token authentication and lacks advanced authentication features like storage state injection, persistent browser profiles, or scripted login hooks.

To implement authentication-based crawling for enterprise users, I'm proposing a hybrid approach that starts with Basic HTTP Authentication support (which can work with the current service) and provides a foundation for future advanced authentication features.

This approach balances immediate value delivery with long-term extensibility, allowing enterprise users to access Basic Auth-protected knowledge bases while establishing the infrastructure for more sophisticated authentication methods.

Reasoning

I explored the codebase systematically to understand the current architecture and authentication capabilities. I examined the backend FastAPI structure, the Crawl4AI integration patterns, the frontend React components, and the Docker environment setup. I discovered that the current Crawl4AI service (unclecode/crawl4ai:all) only supports Bearer token authentication for API access and lacks advanced browser automation features needed for form-based login or session management. I analyzed the frontend architecture and found it uses local state management with Radix UI components, making it suitable for adding authentication dialogs. I also reviewed the security practices and found good foundations but identified the need for credential handling safeguards.

Proposed File Changes

📄 lib/types.ts ^(MODIFY) 🔗

Add comprehensive authentication type definitions to support the new auth features:

1. Add AuthType union type for supported authentication methods: 'none' | 'basic' | 'form' | 'storage_state'

2. Add AuthStatus union type for tracking authentication state: 'none' | 'pending' | 'authenticated' | 'failed'

3. Add AuthProfile interface with fields:

   • auth_type: AuthType
   • login_url?: string (for form-based auth)
   • username?: string
   • password?: string
   • cookies?: string (JSON string for storage state)
   • headers?: Record<string, string> (custom headers)
   • description?: string (user-friendly label)

4. Add AuthProfileRequest interface for API payload to create auth profiles

5. Add AuthProfileResponse interface with auth_id and expires_at fields

6. Extend existing interfaces:

   • Add optional auth_id field to DiscoverOptions
   • Add optional auth_id field to CrawlRequest
   • Add 'auth_pending' and 'auth_failed' to OverallStatus union
   • Add auth_status and auth_error fields to CrawlJobStatus

7. Add component prop interfaces:

   • AuthConfigDialogProps for the authentication configuration dialog
   • CredentialFormProps for credential input forms

📄 backend/app/models.py ^(NEW) 🔗

Create a new models file to centralize authentication-related Pydantic models:

1. AuthProfile class with validation:

   • auth_type field with enum validation
   • conditional field validation based on auth_type
   • password field with SecretStr for security
   • custom validators for URL format and required fields

2. AuthProfileRequest class for API input validation:

   • inherits from AuthProfile
   • additional validation for required fields per auth type
   • sanitization of input data

3. AuthProfileResponse class for API responses:

   • auth_id field
   • expires_at timestamp
   • success status

4. AuthProfileStore class for in-memory management:

   • TTL-based expiration
   • secure cleanup methods
   • thread-safe access patterns

Implement proper validation to ensure Basic Auth profiles have username/password, form-based profiles have login_url, and storage state profiles have cookies.

📄 backend/app/auth_manager.py ^(NEW) 🔗

References

• 📄 backend/app/status_manager.py ^(MODIFY)

Create a dedicated authentication manager to handle auth profile lifecycle and security:

1. AuthProfileManager class with methods:

   • create_auth_profile(): generate unique auth_id, store profile with TTL
   • get_auth_profile(): retrieve profile by auth_id with expiration check
   • cleanup_expired_profiles(): background cleanup of expired profiles
   • delete_auth_profile(): explicit cleanup for job completion

2. Security features:

   • In-memory storage only (no disk persistence)
   • 30-minute TTL for all auth profiles
   • Automatic cleanup on retrieval
   • Secure deletion of sensitive data

3. Thread safety:

   • Use threading.Lock for concurrent access
   • Atomic operations for profile management
   • Safe cleanup during background tasks

4. Integration with status manager:

   • Cleanup auth profiles when jobs complete/cancel
   • Track auth status per job
   • Handle auth failures and retries

Implement comprehensive logging for auth operations while ensuring no credentials are logged.

📄 backend/app/main.py ^(MODIFY) 🔗

References

• 📄 backend/app/status_manager.py ^(MODIFY)

Extend the FastAPI application to support authentication profile management:

1. Import new modules:

   • Import AuthProfile, AuthProfileRequest, AuthProfileResponse from models
   • Import AuthProfileManager from auth_manager
   • Initialize global auth_manager instance

2. Add new API endpoint POST /api/auth-profile:

   • Accept AuthProfileRequest payload
   • Validate authentication credentials
   • Create auth profile using auth_manager
   • Return AuthProfileResponse with auth_id and expiration
   • Handle validation errors and return appropriate HTTP status codes

3. Modify existing endpoints:

   • Update DiscoverRequest model to include optional auth_id field
   • Update CrawlRequest model to include optional auth_id field
   • Modify /api/discover endpoint to extract auth_id and retrieve auth_profile
   • Modify /api/crawl endpoint to extract auth_id and retrieve auth_profile
   • Pass auth_profile to background tasks (discover_pages, crawl_pages)

4. Error handling:

   • Add specific error responses for invalid/expired auth_id
   • Handle auth profile validation failures
   • Return 401 for authentication failures, 404 for missing profiles

5. Security enhancements:

   • Add request logging filter to redact sensitive auth data
   • Implement cleanup of auth profiles on job completion
   • Add rate limiting for auth profile creation if needed

Ensure all auth-related operations are logged appropriately without exposing credentials.

📄 backend/app/status_manager.py ^(MODIFY) 🔗

Extend the status management system to track authentication state and integrate with auth profile cleanup:

1. Update CrawlJobStatus model:

   • Add auth_status field with AuthStatus enum
   • Add auth_error field for authentication-specific error messages
   • Update serialization to handle new fields

2. Add new status management functions:

   • update_auth_status(): update job's authentication state
   • cleanup_auth_data(): remove auth profiles when jobs complete
   • get_jobs_with_auth(): helper to find jobs using specific auth profiles

3. Modify existing functions:

   • update_overall_status(): trigger auth cleanup on final states ('completed', 'cancelled', 'error')
   • request_cancellation(): ensure auth profiles are cleaned up during cancellation
   • initialize_job(): set initial auth_status to 'none'

4. Integration with auth_manager:

   • Import and use AuthProfileManager for cleanup operations
   • Track which jobs are using which auth profiles
   • Coordinate cleanup between job completion and auth expiration

5. Enhanced error handling:

   • Distinguish between crawl errors and auth errors
   • Provide specific error messages for auth failures
   • Track auth retry attempts if needed

Ensure the status manager properly coordinates with the auth manager for secure cleanup of sensitive data.

📄 backend/app/crawler.py ^(MODIFY) 🔗

References

• 📄 backend/app/status_manager.py ^(MODIFY)

Modify the crawler to support authentication by extending HTTP requests with auth headers:

1. Update function signatures:

   • Add optional auth_profile parameter to discover_pages() function
   • Add optional auth_profile parameter to crawl_pages() function
   • Update all callers to pass auth_profile when available

2. Implement Basic Authentication support:

   • Create _build_auth_headers() helper function
   • Generate Authorization header for Basic Auth when auth_profile.auth_type is 'basic'
   • Encode username:password in base64 for Basic Auth header
   • Add custom headers from auth_profile.headers if provided

3. Modify HTTP requests to Crawl4AI:

   • Update all requests.post() calls to include auth headers
   • Merge auth headers with existing API token headers
   • Ensure auth headers are passed to both discovery and crawl requests

4. Add authentication status tracking:

   • Update job status to 'auth_pending' before first authenticated request
   • Update to 'authenticated' on successful response
   • Update to 'auth_failed' on 401/403 responses
   • Use update_auth_status() from status_manager

5. Error handling for auth failures:

   • Detect authentication failures from HTTP status codes
   • Provide specific error messages for auth issues
   • Distinguish between network errors and auth errors
   • Implement proper cleanup on auth failures

6. Future extensibility:

   • Structure code to easily add form-based auth and storage state support
   • Add placeholder comments for advanced auth features
   • Ensure session_id handling is compatible with future auth methods

Note: This implementation focuses on Basic Auth which can work with the current Crawl4AI service, while laying groundwork for future advanced authentication methods.

📄 lib/crawl-service.ts ^(MODIFY) 🔗

References

• 📄 lib/types.ts ^(MODIFY)

Update the crawl service to support authentication profiles and auth_id flow:

1. Add new service functions:

   • createAuthProfile(): POST to /api/auth-profile with AuthProfileRequest
   • validateAuthProfile(): client-side validation for auth profile data
   • deleteAuthProfile(): cleanup auth profile when no longer needed

2. Modify existing functions:

   • Update discoverSubdomains() to accept optional auth_id parameter
   • Update crawlPages() to accept optional auth_id parameter
   • Include auth_id in request payloads when provided
   • Update return types to handle auth-related errors

3. Enhanced error handling:

   • Add specific error handling for 401 (Unauthorized) responses
   • Add handling for 404 (auth profile not found) responses
   • Add handling for auth profile validation errors
   • Provide user-friendly error messages for auth failures

4. Type integration:

   • Import AuthProfile, AuthProfileRequest, AuthProfileResponse from types
   • Update function signatures to use proper auth types
   • Ensure type safety for auth_id parameters

5. Request/response handling:

   • Add proper JSON serialization for auth profile data
   • Handle auth profile expiration scenarios
   • Add retry logic for expired auth profiles
   • Implement proper cleanup of auth data

6. Security considerations:

   • Ensure auth profiles are not logged or cached inappropriately
   • Handle sensitive data properly in error scenarios
   • Implement proper cleanup when components unmount

The service layer will provide a clean interface for components to manage authentication without exposing implementation details.

📄 components/AuthConfigDialog.tsx ^(NEW) 🔗

References

• 📄 components/MCPConfigDialog.tsx
• 📄 components/ui/dialog.tsx
• 📄 components/ui/input.tsx
• 📄 components/ui/button.tsx

Create a comprehensive authentication configuration dialog component using the existing Radix UI patterns:

1. Component structure:

   • Use Radix Dialog components following the pattern from MCPConfigDialog.tsx
   • Accept props: open, onOpenChange, onSave, initialProfile, errors
   • Manage local state for form data and validation

2. Authentication type selection:

   • Radio button group for auth types: 'none', 'basic', 'form', 'storage_state'
   • Dynamic form fields based on selected auth type
   • Clear visual indication of selected auth method

3. Form fields by auth type:

   • Basic Auth: username and password fields with proper input types
   • Form Auth: login URL, username, password fields with URL validation
   • Storage State: textarea for cookies/localStorage JSON with validation
   • Custom Headers: key-value pairs for additional headers

4. Form validation:

   • Real-time validation using existing error display patterns
   • Required field validation based on auth type
   • URL format validation for login URLs
   • JSON validation for storage state data
   • Password strength indicators for security

5. UI/UX features:

   • Password visibility toggle for password fields
   • Test connection button to validate credentials
   • Save and cancel buttons with proper loading states
   • Error display using existing toast patterns
   • Responsive design for mobile compatibility

6. Security features:

   • Clear form data on cancel or close
   • Mask sensitive data in form previews
   • Proper cleanup of sensitive state on unmount
   • Warning messages about credential storage

Implement following the existing component patterns in components/ui/ and components/MCPConfigDialog.tsx for consistency.

📄 components/UrlInput.tsx ^(MODIFY) 🔗

References

• 📄 lib/crawl-service.ts ^(MODIFY)
• 📄 lib/types.ts ^(MODIFY)

Extend the URL input component to support authentication configuration:

1. Add authentication state management:

   • Add requiresAuth boolean state for toggling auth requirement
   • Add authProfile state to store current auth configuration
   • Add showAuthDialog state for dialog visibility
   • Add authId state to track created auth profiles

2. Update component UI:

   • Add "Requires Login" toggle/checkbox after the depth selector
   • Add "Configure Auth" button that appears when requiresAuth is true
   • Add auth status indicator showing current auth state (configured/not configured)
   • Use existing icon patterns and styling for consistency

3. Integrate AuthConfigDialog:

   • Import and render AuthConfigDialog component
   • Handle dialog open/close state
   • Pass current authProfile as initialProfile to dialog
   • Handle auth profile save/update from dialog

4. Update form submission logic:

   • Modify handleSubmit to create auth profile if configured
   • Call createAuthProfile() from crawl-service when authProfile exists
   • Store returned auth_id for use in discovery/crawl requests
   • Pass auth_id to onSubmit callback along with URL and depth
   • Handle auth profile creation errors with appropriate user feedback

5. Update props interface:

   • Modify UrlInputProps.onSubmit to accept optional auth_id parameter
   • Add optional onAuthConfigured callback for parent components
   • Maintain backward compatibility with existing usage

6. Error handling and validation:

   • Show auth-specific errors using existing error display patterns
   • Validate auth configuration before form submission
   • Provide clear feedback for auth profile creation success/failure
   • Handle auth profile expiration scenarios

7. Accessibility and UX:

   • Proper ARIA labels for auth controls
   • Keyboard navigation support
   • Clear visual hierarchy for auth options
   • Helpful tooltips explaining auth requirements

Ensure the component maintains its existing functionality while seamlessly integrating authentication features.

📄 components/CrawlUrls.tsx ^(MODIFY) 🔗

References

• 📄 lib/types.ts ^(MODIFY)

Update the CrawlUrls component to display authentication status and handle auth-related errors:

1. Add auth status display:

   • Show authentication status in the component header or status area
   • Display auth method being used (Basic Auth, Form Auth, etc.)
   • Add auth status badge/indicator similar to existing status badges

2. Update props interface:

   • Add authStatus prop to CrawlUrlsProps for displaying current auth state
   • Add authError prop for showing auth-specific error messages
   • Maintain compatibility with existing prop structure

3. Enhanced error handling:

   • Display auth-specific errors separately from crawl errors
   • Show helpful messages for common auth failures (401, 403, expired profiles)
   • Add retry options for auth failures
   • Use existing error display patterns and styling

4. Status integration:

   • Update status polling to include auth status from CrawlJobStatus
   • Handle auth_pending, authenticated, and auth_failed states
   • Show appropriate loading indicators during auth operations

5. User feedback:

   • Add tooltips explaining auth status and any issues
   • Provide clear actions for resolving auth problems
   • Use existing toast notifications for auth events
   • Maintain consistent styling with existing status displays

6. Accessibility:

   • Proper ARIA labels for auth status elements
   • Screen reader friendly auth status announcements
   • Color and text indicators for auth states (not just color)

The component should provide clear visibility into authentication status while maintaining its existing crawl management functionality.

📄 app/page.tsx ^(MODIFY) 🔗

References

• 📄 components/UrlInput.tsx ^(MODIFY)
• 📄 components/CrawlUrls.tsx ^(MODIFY)

Update the main page component to handle authentication flow and pass auth_id through the component hierarchy:

1. Add auth state management:

   • Add state to track current auth_id from URL input
   • Add state to track auth status for display in CrawlUrls
   • Handle auth_id lifecycle (creation, expiration, cleanup)

2. Update component integration:

   • Modify UrlInput onSubmit handler to receive and store auth_id
   • Pass auth_id to discover and crawl service calls
   • Pass authStatus and authError props to CrawlUrls component
   • Handle auth profile cleanup when jobs complete

3. Add auth status tracking:

   • Monitor job status for auth-related state changes
   • Update UI based on authentication success/failure
   • Handle auth profile expiration during long-running jobs

4. Error handling:

   • Catch and display auth-related errors from service calls
   • Provide user-friendly error messages for auth failures
   • Handle auth profile expiration gracefully
   • Use existing toast system for auth notifications

5. Cleanup and lifecycle:

   • Clean up auth profiles when component unmounts
   • Handle browser refresh scenarios with active auth profiles
   • Ensure proper cleanup on navigation away from page

6. State coordination:

   • Coordinate auth state between UrlInput and CrawlUrls components
   • Handle auth state updates from job status polling
   • Maintain consistency between auth configuration and job execution

The page component should orchestrate the authentication flow while maintaining the existing user experience for non-authenticated crawling.

📄 backend/requirements.txt ^(MODIFY) 🔗

Add any additional Python dependencies needed for authentication support:

1. Security libraries:

   • Add cryptography for secure password handling if not already present
   • Add python-jose for JWT token handling if needed for future auth methods

2. HTTP authentication:

   • Ensure requests supports all needed auth methods (should be included)
   • Add httpx if async HTTP client improvements are needed

3. Validation and parsing:

   • Ensure pydantic version supports SecretStr for password fields
   • Add email-validator if email validation is needed for usernames

4. Background tasks:

   • Ensure asyncio and threading support for auth profile cleanup
   • Add apscheduler if scheduled cleanup tasks are needed

Review existing dependencies to ensure compatibility and avoid conflicts. Most authentication features should work with existing dependencies, but add specific libraries as needed for security and validation.

📄 docs/features/authentication_implementation.md ^(NEW) 🔗

Create comprehensive documentation for the authentication feature implementation:

1. Feature Overview:

   • Explain the authentication-based crawling capability
   • Document supported authentication methods (Basic Auth initially)
   • Describe the enterprise use case and benefits

2. Architecture Documentation:

   • Document the auth profile management system
   • Explain the security model and TTL-based cleanup
   • Describe the integration between frontend and backend
   • Document the API endpoints and payload formats

3. Usage Guide:

   • Step-by-step instructions for configuring authentication
   • Examples of different auth types and when to use them
   • Troubleshooting guide for common auth issues
   • Security best practices for credential management

4. API Reference:

   • Document the /api/auth-profile endpoint
   • Explain the auth_id parameter in discover/crawl endpoints
   • Document error codes and responses
   • Provide example requests and responses

5. Security Considerations:

   • Document credential handling and storage policies
   • Explain TTL-based expiration and cleanup
   • Security recommendations for enterprise deployment
   • Logging and audit considerations

6. Future Roadmap:

   • Document planned advanced authentication features
   • Explain the extensibility of the current implementation
   • Roadmap for form-based auth and storage state support

This documentation will help enterprise users understand and implement authentication-based crawling while providing guidance for future development.

Diagram

sequenceDiagram
    participant User
    participant Frontend
    participant Backend
    participant AuthManager
    participant Crawler
    participant Crawl4AI

    User->>Frontend: Configure authentication
    Frontend->>Frontend: Open AuthConfigDialog
    User->>Frontend: Enter credentials (Basic Auth)
    Frontend->>Backend: POST /api/auth-profile
    Backend->>AuthManager: Create auth profile
    AuthManager->>AuthManager: Store with TTL (30min)
    AuthManager-->>Backend: Return auth_id
    Backend-->>Frontend: Return auth_id + expires_at

    User->>Frontend: Submit URL for discovery
    Frontend->>Backend: POST /api/discover (with auth_id)
    Backend->>AuthManager: Get auth profile
    AuthManager-->>Backend: Return auth profile
    Backend->>Crawler: discover_pages(url, auth_profile)
    Crawler->>Crawler: Build auth headers (Basic Auth)
    Crawler->>Crawl4AI: POST /crawl (with auth headers)
    Crawl4AI-->>Crawler: Return task_id
    Crawler->>Crawl4AI: Poll /task/{task_id}
    Crawl4AI-->>Crawler: Return discovered pages
    Crawler-->>Backend: Update job status
    Backend-->>Frontend: Return job_id

    User->>Frontend: Select pages to crawl
    Frontend->>Backend: POST /api/crawl (with auth_id)
    Backend->>AuthManager: Get auth profile
    Backend->>Crawler: crawl_pages(pages, auth_profile)
    Crawler->>Crawl4AI: POST /crawl (with auth headers)
    Crawl4AI-->>Crawler: Return crawl results
    Crawler->>Crawler: Save consolidated markdown
    Crawler-->>Backend: Update job status to completed
    Backend->>AuthManager: Cleanup auth profile
    AuthManager->>AuthManager: Delete expired profiles

Loading

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end. Make sure you fix all the linting, compilation or validation issues after successful implementation of the plan.

### Observations

The current DevDocs architecture is well-structured with clear separation between frontend (Next.js), backend (FastAPI), and crawling service (Crawl4AI). However, the Crawl4AI service limitation is a significant constraint that requires a pragmatic implementation approach.

The codebase shows good security practices with no credential leakage in logs, proper error handling, and clean separation of concerns. The frontend uses modern React patterns with Radix UI components that can be extended for authentication forms.

The status management system is robust and can be extended to track authentication states. The Docker environment is properly configured and can accommodate the new authentication features with minimal changes.

### Approach

Based on my analysis of the DevDocs codebase, I've identified that the current Crawl4AI service only supports Bearer token authentication and lacks advanced authentication features like storage state injection, persistent browser profiles, or scripted login hooks. 

To implement authentication-based crawling for enterprise users, I'm proposing a **hybrid approach** that starts with Basic HTTP Authentication support (which can work with the current service) and provides a foundation for future advanced authentication features.

This approach balances immediate value delivery with long-term extensibility, allowing enterprise users to access Basic Auth-protected knowledge bases while establishing the infrastructure for more sophisticated authentication methods.

### Reasoning

I explored the codebase systematically to understand the current architecture and authentication capabilities. I examined the backend FastAPI structure, the Crawl4AI integration patterns, the frontend React components, and the Docker environment setup. I discovered that the current Crawl4AI service (unclecode/crawl4ai:all) only supports Bearer token authentication for API access and lacks advanced browser automation features needed for form-based login or session management. I analyzed the frontend architecture and found it uses local state management with Radix UI components, making it suitable for adding authentication dialogs. I also reviewed the security practices and found good foundations but identified the need for credential handling safeguards.

## Mermaid Diagram

sequenceDiagram
    participant User
    participant Frontend
    participant Backend
    participant AuthManager
    participant Crawler
    participant Crawl4AI

    User->>Frontend: Configure authentication
    Frontend->>Frontend: Open AuthConfigDialog
    User->>Frontend: Enter credentials (Basic Auth)
    Frontend->>Backend: POST /api/auth-profile
    Backend->>AuthManager: Create auth profile
    AuthManager->>AuthManager: Store with TTL (30min)
    AuthManager-->>Backend: Return auth_id
    Backend-->>Frontend: Return auth_id + expires_at

    User->>Frontend: Submit URL for discovery
    Frontend->>Backend: POST /api/discover (with auth_id)
    Backend->>AuthManager: Get auth profile
    AuthManager-->>Backend: Return auth profile
    Backend->>Crawler: discover_pages(url, auth_profile)
    Crawler->>Crawler: Build auth headers (Basic Auth)
    Crawler->>Crawl4AI: POST /crawl (with auth headers)
    Crawl4AI-->>Crawler: Return task_id
    Crawler->>Crawl4AI: Poll /task/{task_id}
    Crawl4AI-->>Crawler: Return discovered pages
    Crawler-->>Backend: Update job status
    Backend-->>Frontend: Return job_id

    User->>Frontend: Select pages to crawl
    Frontend->>Backend: POST /api/crawl (with auth_id)
    Backend->>AuthManager: Get auth profile
    Backend->>Crawler: crawl_pages(pages, auth_profile)
    Crawler->>Crawl4AI: POST /crawl (with auth headers)
    Crawl4AI-->>Crawler: Return crawl results
    Crawler->>Crawler: Save consolidated markdown
    Crawler-->>Backend: Update job status to completed
    Backend->>AuthManager: Cleanup auth profile
    AuthManager->>AuthManager: Delete expired profiles

## Proposed File Changes

### lib/types.ts(MODIFY)

Add comprehensive authentication type definitions to support the new auth features:

1. **Add AuthType union type** for supported authentication methods: 'none' | 'basic' | 'form' | 'storage_state'

2. **Add AuthStatus union type** for tracking authentication state: 'none' | 'pending' | 'authenticated' | 'failed'

3. **Add AuthProfile interface** with fields:
   - auth_type: AuthType
   - login_url?: string (for form-based auth)
   - username?: string
   - password?: string
   - cookies?: string (JSON string for storage state)
   - headers?: Record<string, string> (custom headers)
   - description?: string (user-friendly label)

4. **Add AuthProfileRequest interface** for API payload to create auth profiles

5. **Add AuthProfileResponse interface** with auth_id and expires_at fields

6. **Extend existing interfaces**:
   - Add optional auth_id field to DiscoverOptions
   - Add optional auth_id field to CrawlRequest
   - Add 'auth_pending' and 'auth_failed' to OverallStatus union
   - Add auth_status and auth_error fields to CrawlJobStatus

7. **Add component prop interfaces**:
   - AuthConfigDialogProps for the authentication configuration dialog
   - CredentialFormProps for credential input forms

### backend/app/models.py(NEW)

Create a new models file to centralize authentication-related Pydantic models:

1. **AuthProfile class** with validation:
   - auth_type field with enum validation
   - conditional field validation based on auth_type
   - password field with SecretStr for security
   - custom validators for URL format and required fields

2. **AuthProfileRequest class** for API input validation:
   - inherits from AuthProfile
   - additional validation for required fields per auth type
   - sanitization of input data

3. **AuthProfileResponse class** for API responses:
   - auth_id field
   - expires_at timestamp
   - success status

4. **AuthProfileStore class** for in-memory management:
   - TTL-based expiration
   - secure cleanup methods
   - thread-safe access patterns

Implement proper validation to ensure Basic Auth profiles have username/password, form-based profiles have login_url, and storage state profiles have cookies.

### backend/app/auth_manager.py(NEW)

References: 

- backend/app/status_manager.py(MODIFY)

Create a dedicated authentication manager to handle auth profile lifecycle and security:

1. **AuthProfileManager class** with methods:
   - create_auth_profile(): generate unique auth_id, store profile with TTL
   - get_auth_profile(): retrieve profile by auth_id with expiration check
   - cleanup_expired_profiles(): background cleanup of expired profiles
   - delete_auth_profile(): explicit cleanup for job completion

2. **Security features**:
   - In-memory storage only (no disk persistence)
   - 30-minute TTL for all auth profiles
   - Automatic cleanup on retrieval
   - Secure deletion of sensitive data

3. **Thread safety**:
   - Use threading.Lock for concurrent access
   - Atomic operations for profile management
   - Safe cleanup during background tasks

4. **Integration with status manager**:
   - Cleanup auth profiles when jobs complete/cancel
   - Track auth status per job
   - Handle auth failures and retries

Implement comprehensive logging for auth operations while ensuring no credentials are logged.

### backend/app/main.py(MODIFY)

References: 

- backend/app/status_manager.py(MODIFY)

Extend the FastAPI application to support authentication profile management:

1. **Import new modules**:
   - Import AuthProfile, AuthProfileRequest, AuthProfileResponse from models
   - Import AuthProfileManager from auth_manager
   - Initialize global auth_manager instance

2. **Add new API endpoint POST /api/auth-profile**:
   - Accept AuthProfileRequest payload
   - Validate authentication credentials
   - Create auth profile using auth_manager
   - Return AuthProfileResponse with auth_id and expiration
   - Handle validation errors and return appropriate HTTP status codes

3. **Modify existing endpoints**:
   - Update DiscoverRequest model to include optional auth_id field
   - Update CrawlRequest model to include optional auth_id field
   - Modify /api/discover endpoint to extract auth_id and retrieve auth_profile
   - Modify /api/crawl endpoint to extract auth_id and retrieve auth_profile
   - Pass auth_profile to background tasks (discover_pages, crawl_pages)

4. **Error handling**:
   - Add specific error responses for invalid/expired auth_id
   - Handle auth profile validation failures
   - Return 401 for authentication failures, 404 for missing profiles

5. **Security enhancements**:
   - Add request logging filter to redact sensitive auth data
   - Implement cleanup of auth profiles on job completion
   - Add rate limiting for auth profile creation if needed

Ensure all auth-related operations are logged appropriately without exposing credentials.

### backend/app/status_manager.py(MODIFY)

Extend the status management system to track authentication state and integrate with auth profile cleanup:

1. **Update CrawlJobStatus model**:
   - Add auth_status field with AuthStatus enum
   - Add auth_error field for authentication-specific error messages
   - Update serialization to handle new fields

2. **Add new status management functions**:
   - update_auth_status(): update job's authentication state
   - cleanup_auth_data(): remove auth profiles when jobs complete
   - get_jobs_with_auth(): helper to find jobs using specific auth profiles

3. **Modify existing functions**:
   - update_overall_status(): trigger auth cleanup on final states ('completed', 'cancelled', 'error')
   - request_cancellation(): ensure auth profiles are cleaned up during cancellation
   - initialize_job(): set initial auth_status to 'none'

4. **Integration with auth_manager**:
   - Import and use AuthProfileManager for cleanup operations
   - Track which jobs are using which auth profiles
   - Coordinate cleanup between job completion and auth expiration

5. **Enhanced error handling**:
   - Distinguish between crawl errors and auth errors
   - Provide specific error messages for auth failures
   - Track auth retry attempts if needed

Ensure the status manager properly coordinates with the auth manager for secure cleanup of sensitive data.

### backend/app/crawler.py(MODIFY)

References: 

- backend/app/status_manager.py(MODIFY)

Modify the crawler to support authentication by extending HTTP requests with auth headers:

1. **Update function signatures**:
   - Add optional auth_profile parameter to discover_pages() function
   - Add optional auth_profile parameter to crawl_pages() function
   - Update all callers to pass auth_profile when available

2. **Implement Basic Authentication support**:
   - Create _build_auth_headers() helper function
   - Generate Authorization header for Basic Auth when auth_profile.auth_type is 'basic'
   - Encode username:password in base64 for Basic Auth header
   - Add custom headers from auth_profile.headers if provided

3. **Modify HTTP requests to Crawl4AI**:
   - Update all requests.post() calls to include auth headers
   - Merge auth headers with existing API token headers
   - Ensure auth headers are passed to both discovery and crawl requests

4. **Add authentication status tracking**:
   - Update job status to 'auth_pending' before first authenticated request
   - Update to 'authenticated' on successful response
   - Update to 'auth_failed' on 401/403 responses
   - Use update_auth_status() from status_manager

5. **Error handling for auth failures**:
   - Detect authentication failures from HTTP status codes
   - Provide specific error messages for auth issues
   - Distinguish between network errors and auth errors
   - Implement proper cleanup on auth failures

6. **Future extensibility**:
   - Structure code to easily add form-based auth and storage state support
   - Add placeholder comments for advanced auth features
   - Ensure session_id handling is compatible with future auth methods

Note: This implementation focuses on Basic Auth which can work with the current Crawl4AI service, while laying groundwork for future advanced authentication methods.

### lib/crawl-service.ts(MODIFY)

References: 

- lib/types.ts(MODIFY)

Update the crawl service to support authentication profiles and auth_id flow:

1. **Add new service functions**:
   - createAuthProfile(): POST to /api/auth-profile with AuthProfileRequest
   - validateAuthProfile(): client-side validation for auth profile data
   - deleteAuthProfile(): cleanup auth profile when no longer needed

2. **Modify existing functions**:
   - Update discoverSubdomains() to accept optional auth_id parameter
   - Update crawlPages() to accept optional auth_id parameter
   - Include auth_id in request payloads when provided
   - Update return types to handle auth-related errors

3. **Enhanced error handling**:
   - Add specific error handling for 401 (Unauthorized) responses
   - Add handling for 404 (auth profile not found) responses
   - Add handling for auth profile validation errors
   - Provide user-friendly error messages for auth failures

4. **Type integration**:
   - Import AuthProfile, AuthProfileRequest, AuthProfileResponse from types
   - Update function signatures to use proper auth types
   - Ensure type safety for auth_id parameters

5. **Request/response handling**:
   - Add proper JSON serialization for auth profile data
   - Handle auth profile expiration scenarios
   - Add retry logic for expired auth profiles
   - Implement proper cleanup of auth data

6. **Security considerations**:
   - Ensure auth profiles are not logged or cached inappropriately
   - Handle sensitive data properly in error scenarios
   - Implement proper cleanup when components unmount

The service layer will provide a clean interface for components to manage authentication without exposing implementation details.

### components/AuthConfigDialog.tsx(NEW)

References: 

- components/MCPConfigDialog.tsx
- components/ui/dialog.tsx
- components/ui/input.tsx
- components/ui/button.tsx

Create a comprehensive authentication configuration dialog component using the existing Radix UI patterns:

1. **Component structure**:
   - Use Radix Dialog components following the pattern from `MCPConfigDialog.tsx`
   - Accept props: open, onOpenChange, onSave, initialProfile, errors
   - Manage local state for form data and validation

2. **Authentication type selection**:
   - Radio button group for auth types: 'none', 'basic', 'form', 'storage_state'
   - Dynamic form fields based on selected auth type
   - Clear visual indication of selected auth method

3. **Form fields by auth type**:
   - **Basic Auth**: username and password fields with proper input types
   - **Form Auth**: login URL, username, password fields with URL validation
   - **Storage State**: textarea for cookies/localStorage JSON with validation
   - **Custom Headers**: key-value pairs for additional headers

4. **Form validation**:
   - Real-time validation using existing error display patterns
   - Required field validation based on auth type
   - URL format validation for login URLs
   - JSON validation for storage state data
   - Password strength indicators for security

5. **UI/UX features**:
   - Password visibility toggle for password fields
   - Test connection button to validate credentials
   - Save and cancel buttons with proper loading states
   - Error display using existing toast patterns
   - Responsive design for mobile compatibility

6. **Security features**:
   - Clear form data on cancel or close
   - Mask sensitive data in form previews
   - Proper cleanup of sensitive state on unmount
   - Warning messages about credential storage

Implement following the existing component patterns in `components/ui/` and `components/MCPConfigDialog.tsx` for consistency.

### components/UrlInput.tsx(MODIFY)

References: 

- lib/crawl-service.ts(MODIFY)
- lib/types.ts(MODIFY)

Extend the URL input component to support authentication configuration:

1. **Add authentication state management**:
   - Add requiresAuth boolean state for toggling auth requirement
   - Add authProfile state to store current auth configuration
   - Add showAuthDialog state for dialog visibility
   - Add authId state to track created auth profiles

2. **Update component UI**:
   - Add "Requires Login" toggle/checkbox after the depth selector
   - Add "Configure Auth" button that appears when requiresAuth is true
   - Add auth status indicator showing current auth state (configured/not configured)
   - Use existing icon patterns and styling for consistency

3. **Integrate AuthConfigDialog**:
   - Import and render AuthConfigDialog component
   - Handle dialog open/close state
   - Pass current authProfile as initialProfile to dialog
   - Handle auth profile save/update from dialog

4. **Update form submission logic**:
   - Modify handleSubmit to create auth profile if configured
   - Call createAuthProfile() from crawl-service when authProfile exists
   - Store returned auth_id for use in discovery/crawl requests
   - Pass auth_id to onSubmit callback along with URL and depth
   - Handle auth profile creation errors with appropriate user feedback

5. **Update props interface**:
   - Modify UrlInputProps.onSubmit to accept optional auth_id parameter
   - Add optional onAuthConfigured callback for parent components
   - Maintain backward compatibility with existing usage

6. **Error handling and validation**:
   - Show auth-specific errors using existing error display patterns
   - Validate auth configuration before form submission
   - Provide clear feedback for auth profile creation success/failure
   - Handle auth profile expiration scenarios

7. **Accessibility and UX**:
   - Proper ARIA labels for auth controls
   - Keyboard navigation support
   - Clear visual hierarchy for auth options
   - Helpful tooltips explaining auth requirements

Ensure the component maintains its existing functionality while seamlessly integrating authentication features.

### components/CrawlUrls.tsx(MODIFY)

References: 

- lib/types.ts(MODIFY)

Update the CrawlUrls component to display authentication status and handle auth-related errors:

1. **Add auth status display**:
   - Show authentication status in the component header or status area
   - Display auth method being used (Basic Auth, Form Auth, etc.)
   - Add auth status badge/indicator similar to existing status badges

2. **Update props interface**:
   - Add authStatus prop to CrawlUrlsProps for displaying current auth state
   - Add authError prop for showing auth-specific error messages
   - Maintain compatibility with existing prop structure

3. **Enhanced error handling**:
   - Display auth-specific errors separately from crawl errors
   - Show helpful messages for common auth failures (401, 403, expired profiles)
   - Add retry options for auth failures
   - Use existing error display patterns and styling

4. **Status integration**:
   - Update status polling to include auth status from CrawlJobStatus
   - Handle auth_pending, authenticated, and auth_failed states
   - Show appropriate loading indicators during auth operations

5. **User feedback**:
   - Add tooltips explaining auth status and any issues
   - Provide clear actions for resolving auth problems
   - Use existing toast notifications for auth events
   - Maintain consistent styling with existing status displays

6. **Accessibility**:
   - Proper ARIA labels for auth status elements
   - Screen reader friendly auth status announcements
   - Color and text indicators for auth states (not just color)

The component should provide clear visibility into authentication status while maintaining its existing crawl management functionality.

### app/page.tsx(MODIFY)

References: 

- components/UrlInput.tsx(MODIFY)
- components/CrawlUrls.tsx(MODIFY)

Update the main page component to handle authentication flow and pass auth_id through the component hierarchy:

1. **Add auth state management**:
   - Add state to track current auth_id from URL input
   - Add state to track auth status for display in CrawlUrls
   - Handle auth_id lifecycle (creation, expiration, cleanup)

2. **Update component integration**:
   - Modify UrlInput onSubmit handler to receive and store auth_id
   - Pass auth_id to discover and crawl service calls
   - Pass authStatus and authError props to CrawlUrls component
   - Handle auth profile cleanup when jobs complete

3. **Add auth status tracking**:
   - Monitor job status for auth-related state changes
   - Update UI based on authentication success/failure
   - Handle auth profile expiration during long-running jobs

4. **Error handling**:
   - Catch and display auth-related errors from service calls
   - Provide user-friendly error messages for auth failures
   - Handle auth profile expiration gracefully
   - Use existing toast system for auth notifications

5. **Cleanup and lifecycle**:
   - Clean up auth profiles when component unmounts
   - Handle browser refresh scenarios with active auth profiles
   - Ensure proper cleanup on navigation away from page

6. **State coordination**:
   - Coordinate auth state between UrlInput and CrawlUrls components
   - Handle auth state updates from job status polling
   - Maintain consistency between auth configuration and job execution

The page component should orchestrate the authentication flow while maintaining the existing user experience for non-authenticated crawling.

### backend/requirements.txt(MODIFY)

Add any additional Python dependencies needed for authentication support:

1. **Security libraries**:
   - Add `cryptography` for secure password handling if not already present
   - Add `python-jose` for JWT token handling if needed for future auth methods

2. **HTTP authentication**:
   - Ensure `requests` supports all needed auth methods (should be included)
   - Add `httpx` if async HTTP client improvements are needed

3. **Validation and parsing**:
   - Ensure `pydantic` version supports SecretStr for password fields
   - Add `email-validator` if email validation is needed for usernames

4. **Background tasks**:
   - Ensure `asyncio` and threading support for auth profile cleanup
   - Add `apscheduler` if scheduled cleanup tasks are needed

Review existing dependencies to ensure compatibility and avoid conflicts. Most authentication features should work with existing dependencies, but add specific libraries as needed for security and validation.

### docs/features/authentication_implementation.md(NEW)

Create comprehensive documentation for the authentication feature implementation:

1. **Feature Overview**:
   - Explain the authentication-based crawling capability
   - Document supported authentication methods (Basic Auth initially)
   - Describe the enterprise use case and benefits

2. **Architecture Documentation**:
   - Document the auth profile management system
   - Explain the security model and TTL-based cleanup
   - Describe the integration between frontend and backend
   - Document the API endpoints and payload formats

3. **Usage Guide**:
   - Step-by-step instructions for configuring authentication
   - Examples of different auth types and when to use them
   - Troubleshooting guide for common auth issues
   - Security best practices for credential management

4. **API Reference**:
   - Document the /api/auth-profile endpoint
   - Explain the auth_id parameter in discover/crawl endpoints
   - Document error codes and responses
   - Provide example requests and responses

5. **Security Considerations**:
   - Document credential handling and storage policies
   - Explain TTL-based expiration and cleanup
   - Security recommendations for enterprise deployment
   - Logging and audit considerations

6. **Future Roadmap**:
   - Document planned advanced authentication features
   - Explain the extensibility of the current implementation
   - Roadmap for form-based auth and storage state support

This documentation will help enterprise users understand and implement authentication-based crawling while providing guidance for future development.

---

Developer Humor

🔐 Why did the crawler need a password?
Because it was tired of being locked out of the cool docs!
Now it can sneak past the login guards,
And feast on knowledge like a digital bard.
🕷️ No more Access Denied blues,
Just authenticated crawling cruise! 🚀

---

Execution Information

Branch: feature-main
Commit: 9845479

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.