Skip to content

Commit e35157d

Browse files
skarreaskarrea
committed
Merge branch 'pr-3-edit'
2 parents d3628c3 + f1900cd commit e35157d

5 files changed

Lines changed: 298 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# HUNT Cloud internal wiki
22
This is internal documentation and guiding principles for how the CIMORe group uses the HUNT Cloud platform. It is meant as a reference document to ensure common practices.
33

4-
This documentation is markdown-based. To suggest a change or addition, edit or add a new markdown file to the [`docs`](docs) folder. The repository is set up with github actions to automatically rebuild after a new commit to the main branch.
4+
This documentation is markdown-based. To suggest a change or addition, edit or add a new markdown file to the [`docs`](docs) folder. The repository is set up with github actions to automatically rebuild after a new commit to the main branch. If a new markdown file is added, make sure to add it to the `nav` in the [mkdocs.yml](mkdocs.yml) file.
55

66
## Advanced edits
77
This documentation is build using the [Material theme](https://squidfunk.github.io/mkdocs-material/) for [MkDocs](https://www.mkdocs.org/). Please refer to the documentation for the [Material theme](https://squidfunk.github.io/mkdocs-material/getting-started/) if you want to make more advanced edits to the documentation. In this case, make sure to host the page locally to make sure that the modifications works as intented before pushing to github.

‎docs/data-treatment.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Data Treatment
2+
3+
## Overview
4+
5+
**Purpose:** Describe how you should to treat and deal with the data on the Hunt Cloud environment. <br>
6+
**Scope:** This guide covers guidelines for data treatment and handling.<br>
7+
**Audience:** All users.
8+
9+
## Core Principles
10+
11+
### 1. Data Protection and Compliance
12+
13+
All data handling must follow relevant legal and institutional frameworks, including:
14+
15+
- **REC** approval.
16+
17+
- **GDPR** for processing personal and sensitive personal data.
18+
19+
- **Helseforskningsloven (Norwegian Health Research Act)**.
20+
21+
- Institutional data handling agreements and ethical approvals.
22+
23+
**Only users with authorized access may interact with datasets.**
24+
25+
### 2. Data Minimization
26+
27+
- Access and process only the data strictly necessary for approved project tasks.
28+
29+
- Direct identifiers should be removed or pseudonymized.
30+
31+
### 3. Secure Environment Usage
32+
33+
- All data processing must take place **within the HUNT Cloud environment**.
34+
35+
- Data must **not** be downloaded, copied, or transferred outside the cloud unless **explicitly approved by the lab leader or space leader**.
36+
37+
### 4. Structured Organization
38+
39+
Data should follow a clear and consistent directory structure, typically separating:
40+
41+
- **Raw data** – read-only, original files.
42+
43+
> [NOTE] Raw data should be saved in the dedicated location under `/mnt/archive`.
44+
45+
- **Processed data** – intermediate outputs.
46+
47+
- **Results** – final outputs (e.g., models, statistics, figures).
48+
49+
### 5. Documentation and Traceability
50+
51+
All projects should maintain documentation (e.g., README file) covering:
52+
53+
- Data sources.
54+
55+
- Processing workflows and scripts.
56+
57+
- Software versions.
58+
59+
- Storage locations.
60+
61+
- Access permissions.
62+
63+
This ensures transparency and reproducibility of all analyses.

‎docs/good-practice.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# Good Practice
2+
3+
## Overview
4+
5+
**Purpose:** Describe the good practices users preferred to follow on the Hunt Cloud environment.<br>
6+
**Scope:** This guide covers good practice recommendations for programming, structuring, data handling.<br>
7+
**Audience:** All users.
8+
9+
## Data Usage
10+
11+
- Sensitive data **MUST** be uploaded and kept only on the cloud.
12+
13+
- Sensitive data **MUST NOT** be downloaded from the cloud.
14+
15+
> [NOTE] If deemed necessary to download sensitive data, you MUST have a written permission to do so from the Lab or Space leader.
16+
17+
- Avoid copying non-sensitive data to personal machines, USB drives, or external cloud services. _Except if deemed necessary (e.g., for publication)_.
18+
19+
- If data was downloaded, delete it as soon as you don't need it locally anymore.
20+
21+
- Raw data should be located under its dedicated folder under `/mnt/archive`.
22+
23+
- Don't save processed data under `/mnt/archive`, but save it under your project folder.
24+
25+
- Don't make multiple copies of raw data.
26+
27+
- Have code to process the data automatically depending on the raw data, if possible.
28+
29+
- Don't keep multiple copies of the processed data.
30+
31+
## Collaborative Projects
32+
33+
- If you are collaborating with others on the same project, consider locating the project folder under `/mnt/work/projects`.
34+
35+
## Data Structuring and Project Organization
36+
37+
- Create a `README` file, describing the name and objectives of the project, the name of those who work on it, and the timeline (start date, estimated end date).
38+
39+
- Use a clear and predictable structure with clear folder names, such as:
40+
41+
42+
```text
43+
project-name/
44+
├─ docs/ # Documentation, protocols, README files
45+
├─ config/ # Config files (YAML/JSON) and environment files
46+
├─ notebooks/ # Jupyter/R notebooks
47+
├─ src/ # Source code (modules, scripts)
48+
├─ results/ # Figures, tables, model outputs
49+
└─ logs/ # Logs and run metadata
50+
51+
project-data/
52+
├─ raw/ # Original, read-only input data
53+
├─ interim/ # Intermediate processed data
54+
└─ processed/ # Final analysis-ready datasets
55+
```
56+
57+
- We recommend storing the data separately from the code to reduces chances of:
58+
- Accidental pushes to GitHub or similar version control services.
59+
- Accidentally having data be in scope of AI agents (see [section on LLMs and agents](#using-large-language-models-llms-and-chatbots) below)
60+
61+
## Workbench Usage
62+
63+
- Prefer using the **provided workbench solutions** (e.g., JupyterLab, RStudio, VS Code, or similar managed environments) instead of local tools.
64+
65+
## Programming
66+
67+
### Processing
68+
69+
- Keep all code execution, data access, and processing **inside the cloud environment** to maintain security and compliance.
70+
71+
- If Nodes were used to process data, transfer output.
72+
73+
- Make a conda environment for each project and delete it when you are done with it.
74+
75+
- When you run tasks that include multi-processing, make sure to not use all the computing cores. **Preserve at least 30% of the cores** for others.
76+
77+
### Version Control and Reproducibility
78+
79+
- Use **Git** for version control of scripts and notebooks.
80+
- Create meaningful branches (e.g., `feature/qc-pipeline`, `bugfix/mri-loader`).
81+
- Write clear commit messages describing _what_ and _why_.
82+
83+
- Store **configuration files** (e.g., `environment.yml`, `requirements.txt`, or `renv.lock`) to document software dependencies.
84+
85+
- Prefer **notebooks for exploration** and **scripts/modules for pipelines**:
86+
- Use notebooks for initial analysis and visualization.
87+
- Move stable code into reusable functions and scripts.
88+
89+
### Coding Style and Documentation
90+
91+
- Follow a consistent coding style (e.g., **PEP 8** for Python).
92+
93+
- Add short, clear **docstrings** and comments, especially for:
94+
- Data loading functions.
95+
- Pre-processing pipelines.
96+
- Model training and evaluation scripts.
97+
98+
- Log key steps and parameters (e.g., using simple logging libraries or structured text files).
99+
100+
## Using GitHub
101+
102+
GitHub provides a secure and structured way to collaborate on code, notebooks, documentation, and workflows. It ensures version control, transparency, and reproducibility across all team members working in HUNT Cloud.
103+
104+
### Use CIMORe Group Lab
105+
106+
- If the code or repo will be public or cited in a publication, transfer ownership of the code to the [CIMORe GitHub account](https://github.com/ntnu-mr-cancer).
107+
108+
> [NOTE] if you don't have access to it, ask the CIMORe Group leader to point you towards who can add you to it.
109+
110+
- If the code or repo will be private and only meant for private or small group collaboration, it can be located on your private account or the CIMORe lab.
111+
112+
### Use Git for Version Control
113+
114+
- Track all scripts, notebooks, and configuration files through Git.
115+
116+
- Commit frequently with clear and descriptive messages.
117+
118+
- Avoid committing any sensitive data files (use `.gitignore` for this).
119+
120+
- Avoid committing large data files (use `.gitignore` for this).
121+
122+
### Branch-Based Workflow
123+
124+
- Create a new branch for each feature, analysis, or task:
125+
- Example: `feature/qc-pipeline`, `analysis/mri-cleaning`, `fix/dicom-loader`
126+
127+
- In case of collaboration, open a Pull Request to merge changes and request review from team members.
128+
129+
### Documentation in the Repository
130+
131+
- Include a `README.md` explaining:
132+
- How to run the code
133+
- Required dependencies
134+
- Folder structure
135+
136+
- Maintain additional documentation in `/docs` if needed.
137+
138+
### Avoid Storing Data in GitHub
139+
140+
- Do **not** upload raw or sensitive data.
141+
142+
- Use symbolic references or documented file paths to shared cloud folders.
143+
144+
- Store only:
145+
- Code.
146+
- Metadata templates.
147+
- Example dummy datasets (if allowed).
148+
149+
- Be very careful with notebooks as cell output is contained in the saved file.
150+
151+
### Repository Structure
152+
153+
Have an organized, clear structure, such as:
154+
155+
```text
156+
repo-name/
157+
├─ src/ # Scripts and modules
158+
├─ config/ # Environment and configuration files
159+
├─ docs/ # Documentation, SOPs, wiki files
160+
├─ tests/ # Unit tests if applicable
161+
├─ .gitignore # Files/folders to exclude
162+
└─ README.md
163+
```
164+
165+
## Using Large Language Models (LLMs) and Chatbots
166+
167+
### Keep All Sensitive Information Inside the Cloud
168+
169+
- Do **not** paste patient data, filenames containing identifiers, project paths, or other sensitive information into external chatbots (e.g., ChatGPT).
170+
171+
- Only share **generic** code fragments or simplified examples when seeking help.
172+
173+
- For tasks involving real data or sensitive logic, work entirely **within the cloud workbench**.
174+
175+
### Use LLMs as Assistants, Not Authorities
176+
177+
- Validate all model-generated code before use.
178+
179+
- Review logic, imports, file paths, and assumptions.
180+
181+
- Treat LLM responses as suggestions that require scientific and technical interpretation.
182+
183+
### Using Copilot in Secure Environments
184+
185+
- Do **NOT** use Copilot through tools (VS Code, JetBrains, etc.) **inside the cloud**.

‎docs/info.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Info
2+
3+
## Overview
4+
5+
**Purpose:** Provide basic information on the Hunt Cloud environment. <br>
6+
**Scope:** This guide covers labs, names, roles, contact details. <br>
7+
**Intended audience:** All users.
8+
9+
## Data Space
10+
11+
### Overview
12+
13+
| Name | Tag | Subscription Type |
14+
| --------- | --------- | ----------------- |
15+
| MR CANCER | MR-CANCER | BLUE |
16+
17+
### Data Space Administrator Team
18+
19+
Below is a table with the data space roles details.
20+
21+
The roles description can be found in the [relevant article of the HUNT cloud documentation.](https://docs.hdc.ntnu.no/administer-science/roles#data-space-roles)
22+
23+
| Role | Name | Email |
24+
| -------------------- | -------------------- | --------------------------- |
25+
| **Leader** | Tone F. Bathen | tone.f.bathen@ntnu.no |
26+
| **Compliance Admin** | Tone F. Bathen | tone.f.bathen@ntnu.no |
27+
| **Data Admin** | Tone F. Bathen | tone.f.bathen@ntnu.no |
28+
| **Training Admin** | Bendik S. Abrahamsen | bendik.s.abrahamsen@ntnu.no |
29+
| **Technical Admin** | Bendik S. Abrahamsen | bendik.s.abrahamsen@ntnu.no |
30+
| **Finance Admin** | Tone F. Bathen | tone.f.bathen@ntnu.no |
31+
32+
## Labs
33+
34+
Below is a table with the data space labs and their details.
35+
36+
The roles description can be found in the [relevant article of the HUNT cloud documentation.](https://docs.hdc.ntnu.no/administer-science/roles#lab-roles)
37+
38+
| Lab Name | Lab Tag | Lab Leader Name | Lab Leader Name Email | Lab Coordinator Name | Lab Coordinator Email |
39+
| -------------------- | -------------- | ---------------- | ------------------------ | -------------------- | --------------------------- |
40+
| **Digital Life Lab** | digitallifelab | Tone F. Bathen | tone.f.bathen@ntnu.no | Bendik S. Abrahamsen | bendik.s.abrahamsen@ntnu.no |
41+
| **Prostomics** | prostomics | May-Britt Tessem | may-britt.tessem@ntnu.no | Maximilian Wess | maximilian.wess@ntnu.no |
42+
| **Molecular breast** | molbreastlab | Tone F. Bathen | tone.f.bathen@ntnu.no | Guro F. Giskeødegård | guro.giskeodegard@ntnu.no |
43+
| **FLIP.AI** | flipai | Mattijs Elschot | mattijs.elschot@ntnu.no | N/A | N/A |
44+
| **180N-prostate** | 180n-prostate | Tone F. Bathen | tone.f.bathen@ntnu.no | Ingerid S. Knudtsen | ingerid.s.knudtsen@ntnu.no |
45+
| **Breast MRI** | breastmri | Tone F. Bathen | tone.f.bathen@ntnu.no | Tone F. Bathen | tone.f.bathen@ntnu.no |

‎mkdocs.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,5 +23,8 @@ theme:
2323
name: Switch to system preference
2424
nav:
2525
- Home: index.md
26+
- Info: info.md
2627
- On-boarding: on-boarding.md
27-
- Off-boarding: off-boarding.md
28+
- Off-boarding: off-boarding.md
29+
- Good practices: good-practice.md
30+
- Data treatment: data-treatment.md

0 commit comments

Comments
 (0)