Skip to content

Commit c697cc1

Browse files
keshav-spacekeshav-space
committed
Merge healthycode deployment docs
2 parents 8a61f29 + 6b6fd15 commit c697cc1

5 files changed

Lines changed: 476 additions & 0 deletions

File tree

‎deploy/README.md‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
# HealthyCode Deployment documentation
2+
3+
This is the installation documentation for HealthyCode.
4+
5+
It explains how to deploy HealthyCode four components:
6+
- PurlDB frontend
7+
- ScanCode.io worker with its health pipeline and HealthyCode
8+
- GrimoireLab backend
9+
10+
The sequence is to install GrimoireLab, then ScanCode.io and then PurlDB.
11+
The installation assume that we have a Linux installation (Ubuntu or Debian)
12+
with these packages:
13+
- git
14+
- docker and docker compose
15+
- build essentials
16+
17+
The machine should have at least 32GB RAM (more is better), and about 500GB
18+
for the base installation. For on-going production, more disk space is required
19+
in order to keep git clones of all the tracked open source projects.
20+
21+
22+
## GrimoireLab installation
23+
24+
This is deployed using the docker compose at:
25+
docker-compose/docker-compose-grimoirelab2.yml
26+
27+
28+
### Set the max_map_count on the server to support OpenSearch needs
29+
30+
GrimoireLab runs a combo of MariaDB, OpenSearch and Valkey(Redis). OpenSearch
31+
needs some specific memory config on the server that runs these containers.
32+
33+
Hence, before we start the OpenSearch container, there is a one time operation
34+
to make sure the Linux server host setting `vm.max_map_count` is set to at least
35+
262144 (instead of the default 65530). Set this value on the host machine as
36+
explained below:
37+
38+
- Check the current value, with this command:
39+
40+
```
41+
cat /proc/sys/vm/max_map_count
42+
```
43+
44+
- To increase the value, add (or edit) the following line to /etc/sysctl.conf
45+
(using sudo):
46+
47+
```
48+
vm.max_map_count=262144
49+
```
50+
51+
Then run `sudo sysctl -p` to reload the kernel runtime parameters configuration.
52+
53+
### Ports and networks
54+
55+
The main GrimoireLab services needed from outside its docker compose network
56+
are `grimoirelab_core`, served by `nginx` on port 8000` by default and `opensearch`
57+
served on port `9200` by default. Both of the two services are included in a
58+
docker network named `grimoirelab_shared`, and they will be reachable using
59+
these endpoints from inside the docker network:
60+
61+
- Web server for the GrimoireLab API: http://grimoirelab-nginx:8000/
62+
- OpenSearch endpoint: https://grimoirelab-opensearch:9200/
63+
64+
### GrimoireLab credentials and authentication
65+
66+
By default healthycode installation does not expose the bulk of GrimoireLab
67+
services outside of the docker network, and therefore we need limited password
68+
on each instance including OpenSearch. ScanCode.io access these through the
69+
private docker network.
70+
71+
There are provisions to set passwords on the various services if needed but
72+
are minimally documented here.
73+
74+
This various values for credentials are configured using environment variables
75+
and defaults in docker compose file at:
76+
docker-compose/docker-compose-grimoirelab2.yml
77+
78+
You can update that file as needed.
79+
80+
#### OpenSearch (aka. archivist)
81+
82+
The GrimoireLab events are stored in OpenSearch cluster.
83+
84+
The credentials for the `admin` user are set with the password variable
85+
`OPENSEARCH_INITIAL_ADMIN_PASSWORD`.
86+
87+
Note that OpenSearch demands a string passwords and will not start up unless
88+
the password is strong. These OpenSearch credentials are the ones needed by the
89+
`grimoirelab_core_archivists` component. The user is `admin` and the password
90+
is the same you set for OpenSearch:
91+
92+
- `GRIMOIRELAB_ARCHIVIST_STORAGE_USERNAME`
93+
- `GRIMOIRELAB_ARCHIVIST_STORAGE_PASSWORD`
94+
95+
#### Scheduler
96+
97+
Set the credentials for the GrimoireLab scheduler and its API. This component is
98+
used by healthycode to schedule the data collection. Set the following
99+
variables for the scheduler:
100+
101+
- `GRIMOIRELAB_SUPERUSER_USERNAME`
102+
- `GRIMOIRELAB_SUPERUSER_PASSWORD`
103+
104+
This uses two containers (`valkey` and `maria_db`) that are accessed with
105+
empty credentials through the docker network only.
106+
107+
NOTE that this is not a good practice for production services, yet only
108+
containers inside this compose project can reach them.
109+
110+
The good practice is to set passwords using the following variables for a
111+
production enviroment:
112+
113+
- For mariadb, set enviroment variables ending in:
114+
- `*_DB_USER`
115+
- `*_DB_PASSWORD`
116+
117+
- For Valkey, set enviroment variables ending in:
118+
- `*_REDIS_PASSWORD`
119+
120+
## GrimoireLab Startup
121+
122+
Once the initial configutaion above is completed, the next step if launch the
123+
containers with these commands, from a clone of this repository:
124+
125+
```
126+
cd docker-compose/
127+
docker compose -f docker-compose-grimoirelab2.yml up -d
128+
```
129+
130+
## Installation of with PurlDB and ScanCode.io
131+
132+
### Base installation
133+
134+
The first step is to install PurlDB and ScanCode.io using their standard
135+
documentation.
136+
137+
- Setup PurlDB using code and documentation at https://github.com/aboutcode-org/purldb
138+
139+
Once completed, make sure that PurlDB is up and running.
140+
141+
142+
- Setup ScanCode.io as a PurlDB scan worker using code and documentation at
143+
https://github.com/aboutcode-org/scancode.io
144+
145+
### Add shared GrimoireLab docker network to ScanCode.io
146+
147+
There is a temaplte for this docker network configuration in:
148+
etc/scancodeio/network.yml. We copy that to the host:
149+
150+
```
151+
cd healthycode-deploy/
152+
sudo cp ./etc/scancodeio/network.yml /etc/scancodeio/network.yml
153+
```
154+
155+
### Edit ScanCode.io systemctl config to run with healthycode profile
156+
157+
```
158+
sudo vi /etc/systemd/system/scancodeio.service
159+
```
160+
161+
Update ExecStart as below:
162+
```
163+
ExecStart=/usr/bin/docker compose \
164+
-f /opt/scancodeio/docker-compose.purldb-scan-worker.yml \
165+
-f /etc/scancodeio/networks.yml \
166+
--profile healthycode \
167+
up --scale worker=${NUM_WORKERS}
168+
```
169+
170+
```
171+
sudo systemctl daemon-reload
172+
```
173+
174+
### Add PurlDB base URL and API key to ScanCode.io
175+
176+
Get or create a PurlDB API key for the scanworker:
177+
```
178+
# on purldb
179+
docker compose -f /opt/purldb/docker-compose.production.yml exec web python \
180+
manage_purldb.py create-scan-queue-worker-user <username>
181+
```
182+
183+
```
184+
sudo vi /etc/scancodeio/.env
185+
```
186+
187+
Populate with this config, using localhost assuming eveything is co-located on
188+
one server:
189+
```
190+
PURLDB_URL=
191+
PURLDB_API_KEY="scanworker-api-key"
192+
```
193+
194+
### Configure GrimoireLab environment in ScanCode.io
195+
196+
```
197+
sudo vi /etc/scancodeio/.env
198+
```
199+
200+
Populate with this config:
201+
```
202+
GRIMOIRELAB_URL="http://grimoirelab-nginx:8000/"
203+
GRIMOIRELAB_OPENSEARCH_URL="https://grimoirelab-opensearch:9200/"
204+
GRIMOIRELAB_ECOSYSTEM="npmlargerecosystem"
205+
GRIMOIRELAB_PROJECT="NPMMostUsed"
206+
```
207+
208+
Populate these fields based on the same secret that may have been set in
209+
`./docker-compose/docker-compose-grimoirelab2.yml`
210+
211+
```
212+
GRIMOIRELAB_USERNAME=""
213+
GRIMOIRELAB_PASSWORD=""
214+
GRIMOIRELAB_OPENSEARCH_INDEX=""
215+
GRIMOIRELAB_OPENSEARCH_USERNAME=""
216+
GRIMOIRELAB_OPENSEARCH_PASSWORD=""
217+
```
218+
219+
### Expose ScanCode.io on port 8443
220+
221+
This is optional, and enables access to the ScanCode.io scan worker.
222+
If using this, make sure you configure authentication in ScanCode.io first.
223+
224+
```
225+
sudo vi /opt/scancodeio/.env
226+
```
227+
228+
Add the below env variables:
229+
```
230+
NGINX_PUBLISHED_HTTP_PORT=8081
231+
NGINX_PUBLISHED_HTTPS_PORT=8443
232+
```
233+
234+
### Restart ScanCode.io
235+
```
236+
sudo systemctl restart scancodeio
237+
```
238+
239+
## Verify everything is working end to end
240+
241+
242+
Visit the PurlDB health API at: https://<your host>/api/health/
243+
244+
Make a test request with https://<your host>/api/health/?purl=pkg:npm/lodash&format=json
245+

0 commit comments

Comments
 (0)