diff --git a/deploy/README.md b/deploy/README.md index 7b95baa..3ccff13 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -50,6 +50,15 @@ vm.max_map_count=262144 Then run `sudo sysctl -p` to reload the kernel runtime parameters configuration. +### Clone HealthyCode + +``` +git clone https://github.com/aboutcode-org/healthycode.git /opt/healthycode/ && cd /opt/healthycode/ +``` + +Open `/opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml` in a text editor, +and set the credentials as suggested in following steps. + ### Ports and networks The main GrimoireLab services needed from outside its docker compose network @@ -73,7 +82,7 @@ are minimally documented here. This various values for credentials are configured using environment variables and defaults in docker compose file at: -docker-compose/docker-compose-grimoirelab2.yml +`/opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml` You can update that file as needed. @@ -108,93 +117,90 @@ NOTE that this is not a good practice for production services, yet only containers inside this compose project can reach them. The good practice is to set passwords using the following variables for a -production enviroment: +production environment: - - For mariadb, set enviroment variables ending in: + - For mariadb, set environment variables ending in: - `*_DB_USER` - `*_DB_PASSWORD` - - For Valkey, set enviroment variables ending in: + - For Valkey, set environment variables ending in: - `*_REDIS_PASSWORD` ## GrimoireLab Startup -Once the initial configutaion above is completed, the next step if launch the +Once the initial configuration above is completed, the next step if launch the containers with these commands, from a clone of this repository: + ``` -cd docker-compose/ +cd /opt/healthycode/deploy/docker-compose/ docker compose -f docker-compose-grimoirelab2.yml up -d ``` -## Installation of with PurlDB and ScanCode.io - -### Base installation +## Installation of PurlDB and ScanCode.io The first step is to install PurlDB and ScanCode.io using their standard documentation. -- Setup PurlDB using code and documentation at https://github.com/aboutcode-org/purldb - -Once completed, make sure that PurlDB is up and running. - - -- Setup ScanCode.io as a PurlDB scan worker using code and documentation at -https://github.com/aboutcode-org/scancode.io - -### Add shared GrimoireLab docker network to ScanCode.io - -There is a temaplte for this docker network configuration in: -etc/scancodeio/network.yml. We copy that to the host: +### Install and setup PurlDB ``` -cd healthycode-deploy/ -sudo cp ./etc/scancodeio/network.yml /etc/scancodeio/network.yml +git clone https://github.com/aboutcode-org/purldb.git/ /opt/purldb/ && cd /opt/purldb/ +make envfile ``` -### Edit ScanCode.io systemctl config to run with healthycode profile +Open `/opt/purldb/.env` in a text editor and populate with this config: ``` -sudo vi /etc/systemd/system/scancodeio.service +ALLOWED_HOSTS= +SITE_URL=https:/// +PURLDB_PUBLIC_SETUP=True +REST_API_ENDPOINTS=scan_queue,health,packages ``` -Update ExecStart as below: +Build and run PurlDB ``` -ExecStart=/usr/bin/docker compose \ - -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml \ - -f /etc/scancodeio/networks.yml \ - --profile healthycode \ - up --scale worker=${NUM_WORKERS} +sudo docker compose -f /opt/purldb/docker-compose.yml build +sudo docker compose -f /opt/purldb/docker-compose.yml up -d ``` +Once completed, make sure that PurlDB is up and running. + + +### Setup ScanCode.io as a PurlDB scan worker + ``` -sudo systemctl daemon-reload +git clone https://github.com/aboutcode-org/scancode.io.git /opt/scancodeio/ && cd /opt/scancodeio/ +make envfile ``` -### Add PurlDB base URL and API key to ScanCode.io +#### Add PurlDB base URL and API key to ScanCode.io Get or create a PurlDB API key for the scanworker: ``` -# on purldb -docker compose -f /opt/purldb/docker-compose.production.yml exec web python \ - manage_purldb.py create-scan-queue-worker-user +sudo docker compose -f /opt/purldb/docker-compose.yml exec web purldb \ + create-scan-queue-worker-user scancode-worker --generate-api-key ``` +Copy the scancode-worker API Key, and open `/opt/scancodeio/.env` in a text editor. + ``` -sudo vi /etc/scancodeio/.env +sudo vi /opt/scancodeio/.env ``` -Populate with this config, using localhost assuming eveything is co-located on +Populate with this config, using localhost assuming everything is co-located on one server: + ``` -PURLDB_URL= -PURLDB_API_KEY="scanworker-api-key" +PURLDB_URL= +PURLDB_API_KEY="scancode-worker-api-key" ``` -### Configure GrimoireLab environment in ScanCode.io + +#### Configure GrimoireLab environment in ScanCode.io ``` -sudo vi /etc/scancodeio/.env +sudo vi /opt/scancodeio/.env ``` Populate with this config: @@ -203,20 +209,27 @@ GRIMOIRELAB_URL="http://grimoirelab-nginx:8000/" GRIMOIRELAB_OPENSEARCH_URL="https://grimoirelab-opensearch:9200/" GRIMOIRELAB_ECOSYSTEM="npmlargerecosystem" GRIMOIRELAB_PROJECT="NPMMostUsed" +GRIMOIRELAB_OPENSEARCH_INDEX="events" ``` Populate these fields based on the same secret that may have been set in -`./docker-compose/docker-compose-grimoirelab2.yml` +`/opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml` ``` +# Populate this using value from GRIMOIRELAB_SUPERUSER_USERNAME +# and GRIMOIRELAB_SUPERUSER_PASSWORD set earlier in +# /opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml GRIMOIRELAB_USERNAME="" GRIMOIRELAB_PASSWORD="" -GRIMOIRELAB_OPENSEARCH_INDEX="" + +# Populate this using value from GRIMOIRELAB_ARCHIVIST_STORAGE_USERNAME +# and GRIMOIRELAB_ARCHIVIST_STORAGE_PASSWORD set earlier in +# /opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml GRIMOIRELAB_OPENSEARCH_USERNAME="" GRIMOIRELAB_OPENSEARCH_PASSWORD="" ``` -### Expose ScanCode.io on port 8443 +#### Expose ScanCode.io on port 8443 This is optional, and enables access to the ScanCode.io scan worker. If using this, make sure you configure authentication in ScanCode.io first. @@ -227,19 +240,125 @@ sudo vi /opt/scancodeio/.env Add the below env variables: ``` +ALLOWED_HOSTS= +CSRF_TRUSTED_ORIGINS=https://:8443 + NGINX_PUBLISHED_HTTP_PORT=8081 NGINX_PUBLISHED_HTTPS_PORT=8443 ``` -### Restart ScanCode.io +If this is enabled ScanCode.io will be accessible at: `http://:8443` + + +#### Start ScanCode.io + +Remove any old healthycode volume. + +``` +docker volume rm scancodeio_healthycode_venv +``` + +Pull prebuilt ScanCode docker images + ``` -sudo systemctl restart scancodeio +sudo docker compose -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml pull ``` +Start ScanCode with HealthyCode profile + +``` +sudo docker compose \ + -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml \ + -f /opt/healthycode/deploy/etc/scancodeio/network.yml \ + --profile healthycode \ + up --scale worker=2 -d +``` + +Note: we attach shared GrimoireLab network to ScanCode service using `network.yml` + ## Verify everything is working end to end -Visit the PurlDB health API at: https:///api/health/ +Visit the PurlDB health API at: `https:///api/health/` + +Make a test request with `https:///api/health/?purl=pkg:npm/lodash&format=json` + + +## Some useful commands + +### Start and Stop ScanCode.io service + +Start ScanCode.io + +``` +sudo docker compose \ + -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml \ + -f /opt/healthycode/deploy/etc/scancodeio/network.yml \ + --profile healthycode \ + up --scale worker=2 -d +``` + +Stop ScanCode.io: + +``` +sudo docker compose -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml down +``` + +### Start and Stop PurlDB service + +Start PurlDB: + +``` +sudo docker compose -f /opt/purldb/docker-compose.yml up -d +``` + +Stop PurlDB: + +``` +sudo docker compose -f /opt/purldb/docker-compose.yml down +``` + +### Start and Stop GrimoireLab service + +Start GrimoireLab: + +``` +docker compose -f /opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml up -d +``` + +Stop GrimoireLab: + +``` +docker compose -f /opt/healthycode/deploy/docker-compose/docker-compose-grimoirelab2.yml down +``` + + +## Enable authentication for ScanCode.io (Optional): + +Add this to `/opt/scancodeio/.env` + +``` +SCANCODEIO_REQUIRE_AUTHENTICATION=True +``` + +Run this command to create username and password required to login to ScanCode: + +``` +docker compose -f /opt/scancodeio/docker-compose.yml exec web ./manage.py scanpipe create-user +``` + +Restart ScanCode.io service. + +## SSL/TLS Configuration + +Add `fullchain.pem` and `privkey.pem` for your host inside: + +- /opt/purldb/etc/nginx/conf.d/ +- /opt/scancodeio/etc/nginx/conf.d/ + +Restart PurlDB and ScanCode.io service. -Make a test request with https:///api/health/?purl=pkg:npm/lodash&format=json +## Reference documentation +- https://scancodeio.readthedocs.io/en/latest/installation.html +- https://purldb.readthedocs.io/en/latest/getting-started/install.html