nutch-grafana-resources

Grafana Dashboards, Alerts, Collector resources and Infrastructure as Code for monitoring Apache Nutch.

Overview

This repository provides ready-to-use Grafana resources for observing Apache Nutch crawl jobs, including:

• Docker Compose stack for quick local deployment of the complete monitoring stack
• Grafana Alloy configuration for collecting logs and extracting metrics
• Grafana dashboards for visualizing crawler performance and activity
• OpenTofu configuration for infrastructure-as-code deployment of dashboards and alert rules

Quick Start with Docker

The fastest way to get started is using Docker Compose, which deploys Grafana, Alloy, Loki, and Prometheus in a single command.

Prerequisites

• Docker and Docker Compose
• Apache Nutch with logs directory

Local Deployment

1. Configure environment:

   cp .env.example .env
   # Edit .env and set NUTCH_LOGS_PATH to your Nutch logs directory
   # Example: NUTCH_LOGS_PATH=/opt/nutch/runtime/local/logs

2. Start the stack:

   docker compose --profile local up -d

3. Access services:

   • Grafana: http://localhost:3000 (admin/admin)
   • Alloy UI: http://localhost:12345
   • Prometheus: http://localhost:9090
   • Loki: http://localhost:3100

4. Provision dashboards (via OpenTofu):

   cd tofu
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars: grafana_url = "http://localhost:3000", grafana_auth = "admin:admin"
   tofu init && tofu apply

Grafana Cloud Deployment

To send data to Grafana Cloud instead of local services:

1. Configure environment:

   cp .env.example .env
   # Edit .env with your Grafana Cloud credentials:
   # DEPLOYMENT_MODE=cloud
   # LOKI_URL=https://logs-prod-xxx.grafana.net
   # LOKI_USERNAME=<your-loki-username>
   # PROMETHEUS_URL=https://prometheus-prod-xxx.grafana.net
   # PROMETHEUS_USERNAME=<your-prometheus-username>
   # GRAFANA_CLOUD_API_KEY=glc_xxx

2. Start the stack (without local profile):

   docker compose up -d

For more details, see the Docker README.

---

Manual Setup

If you prefer to run components individually or integrate with existing infrastructure, follow the manual setup instructions below.

Requirements

• Apache Nutch (with metrics logging enabled)
• Grafana Alloy for log/metrics collection
• Grafana with Loki and Prometheus datasources
• OpenTofu >= 1.6.0 (optional, for IaC deployment)
• Docker (optional, for containerized deployment)

Required Grafana Permissions

Create a service account token with the following permissions:

• folders:write
• dashboards:write
• alert.rules:write

Resources

alloy/config.alloy

Grafana Alloy configuration that:

• Collects Nutch logs from the local runtime directory
• Parses Log4j2 format with multiline support for stack traces
• Extracts Prometheus metrics from Hadoop counter output in logs
• Forwards logs to Loki (Grafana Cloud or local)
• Forwards metrics to Prometheus (Grafana Cloud or local)

Metrics Extracted

The configuration extracts metrics from the following Nutch components (based on org.apache.nutch.metrics.NutchMetrics):

Component	Metrics
Fetcher	Active threads, spin waiting, queue sizes, bytes downloaded, robots denied, redirects, timeouts, latency (p50/p95/p99)
Generator	URL filter rejections, schedule rejections, score filtering, malformed URLs
Indexer	Documents indexed, deleted (robots/gone/redirects/duplicates), skipped, errors, latency
CrawlDB	URLs filtered, gone/orphan records removed, status counts (fetched/unfetched/gone/redirects)
Injector	URLs injected, unique URLs, merged URLs, purged URLs
HostDB	Host counts (new/existing/purged), filtered records
Parser	Parse success count, latency metrics
Deduplication	Documents marked as duplicate
WebGraph	Links added/removed
Sitemap	Seeds extracted, sitemaps discovered, failed fetches
WARC Exporter	Records generated, missing content/metadata, invalid URIs
Domain Stats	Fetched/not fetched URLs per domain

Setup

1. Create credential files for Grafana Cloud (or configure local endpoints):

   mkdir -p ~/.config/alloy
   echo -n "your_loki_username" > ~/.config/alloy/loki_username
   echo -n "your_prometheus_username" > ~/.config/alloy/prometheus_username
   echo -n "your_api_key" > ~/.config/alloy/grafana_cloud_api_key
   chmod 600 ~/.config/alloy/grafana_cloud_api_key

2. Update paths in config.alloy:

   Credential file paths — Update the local.file blocks with your paths:

   local.file "loki_username" {
     filename  = "/home/youruser/.config/alloy/loki_username"
     is_secret = false
   }

   Nutch logs path — Update the declare "nutch_log_targets" block to point to your Nutch logs directory:

   declare "nutch_log_targets" {
     export "targets" {
       value = [
         {
           "__path__" = "/home/youruser/nutch/runtime/local/logs/*.log",
           "job"      = "nutch",
           "hostname" = constants.hostname,
         },
       ]
     }
   }

   This target configuration is shared by both loki.source.file components, which use the built-in file_match block for automatic file discovery.

   Grafana Cloud URLs — Update loki.write and prometheus.remote_write with your instance details (found in your Grafana Cloud portal):

   loki.write "grafanacloud" {
     endpoint {
       url = "https://logs-prod-<INSTANCE>.grafana.net/loki/api/v1/push"
       ...
     }
   }
   
   prometheus.remote_write "grafanacloud" {
     endpoint {
       url = "https://prometheus-prod-<INSTANCE>-<REGION>.grafana.net/api/prom/push"
       ...
     }
   }

3. Run Alloy with the configuration:

   alloy run config.alloy

dashboards/

Pre-built Grafana dashboards for monitoring Nutch crawls.

nutch_crawler_monitoring.json

Log-based monitoring dashboard using Loki as the datasource:

• Log rate by level (INFO, WARN, ERROR)
• Real-time log viewer with filtering
• Crawler activity timeline

nutch_metrics_dashboard.json

Comprehensive metrics dashboard using Prometheus as the datasource:

• Fetcher Metrics: Active threads, queue sizes, bytes downloaded, latency percentiles
• Parser Statistics: Parse success rates and latency
• CrawlDB State: URL status distribution (fetched, unfetched, gone, redirects)
• Indexer Performance: Documents indexed, deleted, and error rates
• Generator Stats: URL filtering and rejection reasons

Importing Dashboards

See Import dashboards in the Grafana documentation.

docker/

Docker Compose configuration for deploying the complete monitoring stack:

docker/
├── grafana/
│   └── provisioning/
│       ├── dashboards/
│       │   └── dashboards.yaml      # Dashboard provisioning config
│       └── datasources/
│           └── datasources.yaml     # Loki & Prometheus datasources
├── loki/
│   └── loki-config.yaml             # Loki configuration
├── prometheus/
│   └── prometheus.yml               # Prometheus configuration
└── README.md                        # Detailed Docker documentation

The stack supports two deployment modes:

Mode	Command	Services
Local	docker compose --profile local up -d	Grafana, Alloy, Loki, Prometheus
Cloud	docker compose up -d	Grafana, Alloy (sends to Grafana Cloud)

See docker/README.md for complete documentation.

Troubleshooting

macOS file watching: On macOS, Docker volume mounts may not immediately detect file changes. If logs aren't appearing, restart Alloy:

docker compose --profile local restart alloy

tofu/

OpenTofu configuration for infrastructure-as-code deployment using the Grafana Terraform Provider.

Features

• Declarative provisioning of dashboards and alert rules
• Version-controlled infrastructure changes
• CI/CD integration via GitHub Actions
• Support for Grafana Cloud and self-hosted instances

Alert Rules

The OpenTofu configuration deploys the following Grafana Alerting rules:

Loki (Log-based) Alerts:

• Critical: Crawler stopped, high error rate
• Warning: Error spike, no activity, high warn rate, OOM detection
• Info: Job started/finished, indexing completed

Mimir (Metric-based) Alerts:

• Critical: Hung threads, zero throughput, high exception rate
• Warning: Robots denied, timeouts, high latency, queue backlog
• Info: High duplicate rate, generator rejections

Setup

1. Install OpenTofu:

   # macOS
   brew install opentofu
   
   # Linux
   curl -fsSL https://get.opentofu.org/install-opentofu.sh | sh

2. Configure credentials:

   cd tofu
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your Grafana URL and API token

3. Deploy resources:

   tofu init
   tofu plan
   tofu apply

Destroying Resources

To remove all Grafana resources (folder, dashboards, and alert rules):

cd tofu
tofu destroy

Additional destroy commands:

# Preview what will be destroyed (dry run)
tofu plan -destroy

# Destroy without confirmation prompt
tofu destroy -auto-approve

# Destroy specific resources only
tofu destroy -target=grafana_dashboard.nutch_metrics
tofu destroy -target=grafana_rule_group.nutch_loki_critical

CI/CD Integration

The repository includes a GitHub Actions workflow (.github/workflows/tofu.yml) that:

• Validates configuration on all pull requests
• Plans changes and comments on PRs
• Applies changes when merged to main

Required GitHub secrets:

• GRAFANA_URL — Your Grafana instance URL
• GRAFANA_AUTH — Service account token

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.