Skip to content

aws_lambda_playwright_crawler.mdx

Latest commit

 

History

History
107 lines (67 loc) · 5.04 KB

aws_lambda_playwright_crawler.mdx

File metadata and controls

107 lines (67 loc) · 5.04 KB
Raw
id aws-lambda-playwright
title PlaywrightCrawler on AWS Lambda
description Prepare your PlaywrightCrawler to run in Lambda functions on Amazon Web Services.

import ApiLink from '@site/src/components/ApiLink';

import CodeBlock from '@theme/CodeBlock';

import PlaywrightCrawlerLambda from '!!raw-loader!./code_examples/aws/playwright_crawler_lambda.py'; import PlaywrightCrawlerDockerfile from '!!raw-loader!./code_examples/aws/playwright_dockerfile';

AWS Lambda is a serverless compute service that runs code without provisioning or managing servers. For crawlers that require browser rendering, you need to deploy using Docker container images because Playwright and browser binaries exceed Lambda's ZIP deployment size limits.

Updating the code

For the project foundation, use BeautifulSoupCrawler as described in this example. We will update it to work with PlaywrightCrawler.

When instantiating a crawler, use MemoryStorageClient. By default, Crawlee uses file-based storage, but the Lambda filesystem is read-only (except for /tmp). Using MemoryStorageClient tells Crawlee to use in-memory storage instead. Replace BeautifulSoupCrawler with PlaywrightCrawler and configure browser_launch_options with flags optimized for serverless environments. These flags disable sandboxing and GPU features that aren't available in Lambda's containerized runtime.

Wrap the crawler logic in a lambda_handler function. This is the entry point that AWS will execute.

:::important

Make sure to always instantiate a new crawler for every Lambda invocation. AWS keeps the environment running for some time after the first execution (to reduce cold-start times), so subsequent calls may access an already-used crawler instance.

TLDR: Keep your Lambda stateless.

:::

Finally, return the scraped data from the Lambda when the crawler run ends.

{PlaywrightCrawlerLambda}

Deploying the project

Installing and configuring AWS CLI

Install AWS CLI following the official documentation according to your operating system.

Authenticate by running:

aws login

Preparing the project

Initialize the project by running uvx 'crawlee[cli]' create.

Or use a single command if you don't need interactive mode:

uvx 'crawlee[cli]' create aws_playwright --crawler-type playwright --http-client impit --package-manager uv --no-apify --start-url 'https://crawlee.dev' --install

Add additional dependencies:

uv add awslambdaric aws-lambda-powertools boto3

The project is created with a Dockerfile that needs to be modified for AWS Lambda by adding ENTRYPOINT and updating CMD:

{PlaywrightCrawlerDockerfile}

Building and pushing the Docker image

Create a repository lambda/aws-playwright in Amazon Elastic Container Registry in the same region where your Lambda functions will run. To learn more, refer to the official documentation.

Navigate to the created repository and click the "View push commands" button. This will open a window with console commands for uploading the Docker image to your repository. Execute them.

Example:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin {user-specific-data}
docker build --platform linux/amd64 --provenance=false -t lambda/aws-playwright .
docker tag lambda/aws-playwright:latest {user-specific-data}/lambda/aws-playwright:latest
docker push {user-specific-data}/lambda/aws-playwright:latest

Creating the Lambda function

  1. In the AWS Lambda Console, click "Create function"
  2. Select "Container image"
  3. Browse and select your ECR image
  4. Configure the function settings

:::tip Configuration

Playwright crawlers require more resources than HTTP-based crawlers:

  • Memory: Minimum 1024 MB recommended. Browser operations are memory-intensive, so 2048 MB or more may be needed for complex pages.
  • Timeout: Set according to crawl size. Browser startup adds overhead, so allow at least 5 minutes even for simple crawls.
  • Ephemeral storage: Default 512 MB is usually sufficient unless downloading large files.

See the official documentation to learn how performance and cost scale with memory.

:::

Testing the function

After the Lambda deploys, click the "Test" button to invoke it. The event contents don't matter for a basic test, but you can parameterize your crawler by parsing the event object that AWS passes as the first argument to the handler.