feat: updating for smooth runs and good docs

Author: vancauwe

2025-mini-hackathon.md

@@ -4,19 +4,18 @@
 
 1. Choose your team captain !
 2. Each team member makes an account on [RenkuLab](https://renkulab.io/)
-3. The team captain copies the [project quickstart](https://renkulab.io/p/open-pulse-hackathon/quickstart) into their renku user profile
-4. Captain adds all his team members (renku users) to that project
-5. Team sets up their project: add your own code repository and add your own data (Notes: do any code/data connections before launching a work session.)
-6. If you want to launch the quickstart session, you will be prompted to enter the connection secrets (see documents).
-7. If you want to launch a session with all the code from all connected code repositories, we recommend launching the standard VSCode interface of Renku and working there.
+3. Join all together your project in the open-pulse-hackathon (same name as on discord).
+4. Team sets up their project: add your own code repository and add your own data (Notes: do any code/data connections before launching a work session.)
+5. If you want to launch the quickstart session, you will be prompted to enter the connection secrets (see documents).
+6. If you want to launch a session with all the code from all connected code repositories, we recommend launching the standard VSCode interface of Renku and working there.
 
 ![alt text](docs/images/Renku.png)
 
 Here are the [Open Pulse Hackathon Renku Project](https://renkulab.io/g/open-pulse-hackathon) and [the template project](https://renkulab.io/p/open-pulse-hackathon/quickstart)from which you can kick off.
 
 For further Jupyter Hub Documentation you can read [here](https://jupyterhub.readthedocs.io/en/stable/);
 
-You can also clone the repository and work locally via the `uv` setup if you prefer. 
+You can also clone the repository and work locally via the `uv` setup if you prefer (see README in each of the quickstarts for neoj and tentris). 
 
 ## How to submit your project?
 
@@ -34,6 +33,22 @@ You’re also welcome to include supplemental material such as:
 
 During the final session, all projects will be presented and participants will vote using stickers to recognize the
 
+## Data available
+
+Communities that have been crawled through (hence in the graph and metadata available):
+
+    - ENAC
+    - SDSC
+    - DeepLabCut
+    - SCALA Graph
+    - Materials Cloud
+    - EPFL Center for Imaging
+    - Imaging Plaza
+    - 50 top “EPFL” users 
+    - 50 top users among “ETHZ”,“UZH”, “UNINE”, “UNIBE”, “UNIGE” and “UNIL”
+
+Note: Total of ~1M nodes and growing.
+
 ## Credits
 
 TBD.

README.md

@@ -32,13 +32,20 @@ To be extended...
 
 [Link to example](https://dashboard.documentfoundation.org/app/kibana#/dashboard/Overview?_g=()&_a=(description:'Overview%20Panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:'Empty%20Commits',disabled:!f,index:git,key:files,negate:!t,params:(query:'0',type:phrase),type:phrase,value:'0'),query:(match:(files:(query:'0',type:phrase)))),('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:git,key:author_bot,negate:!t,params:(query:!t,type:phrase),type:phrase,value:true),query:(match:(author_bot:(query:!t,type:phrase))))),fullScreenMode:!f,options:(darkTheme:!f,useMargins:!t),panels:!((gridData:(h:2,i:'1',w:4,x:0,y:5),id:git_main_numbers,panelIndex:'1',title:Git,type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'2',w:4,x:4,y:5),id:git_evolution_commits,panelIndex:'2',title:'Git%20Commits',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'3',w:4,x:8,y:5),id:git_evolution_authors,panelIndex:'3',title:'Git%20Authors',type:visualization,version:'6.1.0-3'),(gridData:(h:5,i:'5',w:4,x:4,y:0),id:git_commits_organizations,panelIndex:'5',title:Organizations,type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'6',w:4,x:0,y:7),id:gerrit_main_numbers,panelIndex:'6',title:Gerrit,type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'7',w:4,x:4,y:7),id:gerrit_evolution_changesets,panelIndex:'7',title:'Gerrit%20Changesets',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'8',w:4,x:8,y:7),id:gerrit_evolution_submitters,panelIndex:'8',title:'Gerrit%20Changeset%20Submitters',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'96',w:4,x:4,y:11),id:mediawiki_editions,panelIndex:'96',title:'Mediawiki%20Edits',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'97',w:4,x:8,y:11),id:mediawiki_editors,panelIndex:'97',title:'Mediawiki%20Editors',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'98',w:4,x:0,y:11),id:mediawiki_main_numbers,panelIndex:'98',title:Mediawiki,type:visualization,version:'6.1.0-3'),(gridData:(h:5,i:'111',w:4,x:0,y:0),id:git_overview_top_authors,panelIndex:'111',title:'Git%20Top%20Authors',type:visualization,version:'6.1.0-3'),(gridData:(h:5,i:'112',w:4,x:8,y:0),id:git_overview_top_projects,panelIndex:'112',title:'Git%20Top%20Projects',type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'113',w:4,x:0,y:9),id:issues_main_metrics,panelIndex:'113',title:'Bugzilla%20Issues',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'114',w:4,x:4,y:9),id:issues_evolutionary,panelIndex:'114',title:'Bugzilla%20Issues',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'115',w:4,x:8,y:9),id:issues_evolutionary_submitters,panelIndex:'115',title:'Bugzilla%20Issue%20Submitters',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'116',w:4,x:0,y:15),id:bd562660-099c-11eb-b2a9-cfa8a0efe987,panelIndex:'116',title:Weblate,type:visualization,version:'6.1.4'),(gridData:(h:2,i:'119',w:4,x:4,y:15),id:'1fe9b9b0-4b9a-11eb-ba3f-e561effddee9',panelIndex:'119',title:'Weblate%20actions',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'120',w:4,x:8,y:15),id:cc367d00-4b97-11eb-ba3f-e561effddee9,panelIndex:'120',title:'Weblate%20contributors',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'121',w:4,x:0,y:13),id:discourse_main_metrics,panelIndex:'121',title:Discourse,type:visualization,version:'6.1.4'),(gridData:(h:2,i:'123',w:4,x:8,y:13),id:discourse_qa_evolutionary_submitters,panelIndex:'123',title:'People%20sending%20Q%26A',type:visualization,version:'6.1.4'),(gridData:(h:2,i:'124',w:4,x:4,y:13),id:discourse_qa_evolutionary,panelIndex:'124',title:'Questions%20and%20Answers',type:visualization,version:'6.1.4')),query:(language:lucene,query:(query_string:(analyze_wildcard:!t,default_field:'*',query:'*'))),timeRestore:!f,title:Overview,uiState:(P-1:(title:Git),P-10:(title:Emails,vis:(legendOpen:!f)),P-100:(title:Meetings),P-101:(title:Representatives%2FCountries),P-102:(title:'-'),P-104:(title:Telegram),P-105:(title:Messages,vis:(legendOpen:!f)),P-106:(title:Participants,vis:(legendOpen:!f)),P-107:(title:'Git%20Pair%20Programming'),P-108:(title:'-'),P-109:(title:Commits,vis:(legendOpen:!f)),P-11:(title:Participants,vis:(legendOpen:!f)),P-110:(title:Authors,vis:(legendOpen:!f)),P-111:(title:'Git%20Top%20Authors',vis:(params:(config:(searchKeyword:''),sort:(columnIndex:!n,direction:!n)))),P-112:(title:'Git%20Top%20Projects',vis:(params:(config:(searchKeyword:''),sort:(columnIndex:!n,direction:!n)))),P-113:(title:'-'),P-114:(title:'-',vis:(legendOpen:!f)),P-115:(title:'Reps%20Activities',vis:(legendOpen:!f)),P-116:(title:'Reps%20Events'),P-117:(title:Reports,vis:(legendOpen:!f)),P-118:(title:Events,vis:(legendOpen:!f)),P-119:(title:Representatives%2FCountries,vis:(legendOpen:!f)),P-12:(title:Issues),P-120:(title:Authors,vis:(legendOpen:!f)),P-121:(title:'-'),P-122:(title:Stars,vis:(legendOpen:!f)),P-123:(title:Pulls,vis:(legendOpen:!f)),P-124:(title:DockerHub),P-13:(title:Issues),P-132:(vis:(params:(config:(searchKeyword:''),sort:(columnIndex:!n,direction:!n)))),P-14:(title:'Issue%20Submitters',vis:(legendOpen:!f)),P-15:(title:Twitter),P-16:(title:Tweets,vis:(legendOpen:!f)),P-17:(title:Users,vis:(legendOpen:!f)),P-18:(title:Stackoverflow),P-19:(title:'Questions%20and%20Answers'),P-2:(title:Commits,vis:(legendOpen:!f)),P-20:(title:Participants),P-22:(title:IRC),P-23:(title:Messages,vis:(legendOpen:!f)),P-24:(title:Participants,vis:(legendOpen:!f)),P-28:(title:Confluence),P-29:(title:Participants,vis:(legendOpen:!f)),P-3:(title:Authors,vis:(legendOpen:!f)),P-30:(title:Editions),P-31:(title:'Github%20Issues'),P-32:(title:'GitHub%20Issues'),P-33:(title:'GitHub%20Issues%20Submitters'),P-34:(title:'GitHub%20Pull%20Requests'),P-35:(title:'Pull%20Requests'),P-36:(title:'Pull%20Request%20Submitters',vis:(legendOpen:!f)),P-37:(title:Jira),P-38:(title:Issues),P-39:(title:Submitters,vis:(legendOpen:!f)),P-40:(title:Issues,vis:(legendOpen:!f)),P-41:(title:Maniphest),P-42:(title:Submitters,vis:(legendOpen:!f)),P-43:(title:Redmine),P-44:(title:Issues),P-45:(title:Submitters,vis:(legendOpen:!f)),P-46:(title:RSS),P-47:(title:Posts),P-48:(title:Authors),P-49:(title:Meetup),P-5:(title:Organizations),P-50:(title:Events),P-51:(title:Participants,vis:(legendOpen:!f)),P-52:(title:Askbot),P-53:(title:'Questions,%20Answers%20and%20Comments',vis:(legendOpen:!f)),P-54:(title:Participants,vis:(legendOpen:!f)),P-55:(title:Discourse),P-56:(title:'Questions%20and%20Answers'),P-57:(title:Participants),P-58:(title:Jenkins),P-59:(title:'Jenkins%20Builds',vis:(legendOpen:!f)),P-6:(title:Gerrit),P-60:(title:'Jenkins%20Active%20Nodes',vis:(legendOpen:!f)),P-62:(title:Hits),P-65:(title:Slack),P-66:(title:'Slack%20Messages'),P-67:(title:'Slack%20Participants'),P-68:(title:'-'),P-69:(title:'-'),P-7:(title:Changesets,vis:(legendOpen:!f)),P-70:(title:'-'),P-71:(title:'-'),P-72:(title:'-'),P-73:(title:'-'),P-74:(title:'-'),P-75:(title:'-'),P-76:(title:'-'),P-77:(title:'-'),P-78:(title:'-'),P-79:(title:'-'),P-8:(title:'Changeset%20Submitters',vis:(legendOpen:!f)),P-80:(title:'-'),P-81:(title:'-'),P-82:(title:'-'),P-83:(title:'-'),P-84:(title:'-'),P-85:(title:'-'),P-86:(title:'-'),P-87:(title:'-'),P-88:(title:'Google%20Hits'),P-89:(title:Hits),P-9:(title:'Mailing%20Lists'),P-90:(title:'Hits%20by%20Keyword'),P-91:(title:'-'),P-92:(spy:(mode:(fill:!f,name:!n)),title:'Apache%20Logs'),P-93:(spy:(mode:(fill:!f,name:!n)),title:'Apache%20Visits'),P-94:(title:'Apache%20Packages'),P-95:(title:'-'),P-96:(title:Editions,vis:(legendOpen:!f)),P-97:(title:Editors,vis:(legendOpen:!f)),P-98:(title:Mediawiki),P-99:(title:'Mozilla%20Club')),viewMode:view))
 
-Link to Grimoire Open Pulse Instance: Waiting for final deployment
 Documentation: https://chaoss.community/
 CHAOSS Metrics: https://chaoss.community/kbtopic/metrics-and-metrics-models-by-topic/
 
+For entities of interest, you can get more information about them here: 
+
 ### Community Network
 
-To be extended....
+Relationships are structure in a property-graph and accessed via Cypher Queries.
+
+You can explore the graph via the Neo4J User Interface of via the `neo4j-quickstart` notebook or script.
+
+- Code there allows you to run CYPHER queries, remember you can explore [graph patterns (such as shortest path between 2 nodes)](https://neo4j.com/docs/cypher-manual/current/patterns/).
+- You can download into a pandas dataframe part of the graph. The Graph being huge, we do not recommend reading the entire thing into Pandas.
+- Visualization code is also made available for the graphs. 
 
 ![alt text](docs/images/Neo4J.png)
 
@@ -48,24 +55,18 @@ Link to Neo4j Open Pulse Instance:
 Documentation: https://neo4j.com/docs/
 Cypher Tutorial: https://neo4j.com/docs/cypher-manual/current/appendix/tutorials/
 
-#### CSV Example dataset
 
-As an alternative to the interaction with NEO4j we compiled a subset of the graph in csv format which can be found in  `data/community-network`. 
-
-Code for loading this csv is available in the `neo4j-quickstart` notebook.
 
 ### Repositories Metadata
 
-To be extended....
+Metadata and semantic properties are stored in a RDF-compatible database based on the Imaging Plaza Ontology (ORD Project). We can query its results with SPARQL queries.
 
-Mention Federated queries with Wikipedia
+You can query the repositories' metadata via the quickstart notebook of the `tentris-quickstart`
+You can also combine the queries to tentris about the extracted data with queries to WikiData for more complex information extraction. (see examples in notebook)
 
 ![alt text](docs/images/Tentris.png)
 
 Link to Tentris Open Pulse: http://128.178.219.51:7502/ui
 Documentation: https://docs.tentris.io/
 SPARQL Tutorial: https://sparql.dev/
 
-#### Document based Metadata
-
-As an alternative to SPARQL we compiled the metadata in a much more familiar formar and included a subset in `data/entities-metadata`

src/neo4j-quickstart/README.md

@@ -1,26 +1,71 @@
 # QuickStart Neo4J
 
-For open pulse project, you will manipulate graph data with Neo4J. The example notebook will guide you on how to download and visualize the graph. 
+For open pulse project, you will manipulate graph data with Neo4J. The example notebook will guide you on how to download and visualize the graph. Should you prefer scripts, one is also available to you.
 
 ## Working on Renku with Jupyter Notebook 
 
 This set-up provides jupyter notebook support if this is your preferred working method.
 
-Please see instructions in `2025-hackathon.md` at the root of the repository for more instructions.
+Please see instructions in `2025-mini-hackathon.md` at the root of the repository for more instructions.
 
 ## Working locally
 
-We are using uv ([installation instructions here](https://docs.astral.sh/uv/getting-started/installation/)).
+### 🚀 Part 1: Create the Environment (In Your Terminal)
 
-1. Create virtual environment: `uv venv`
-2. Activate it: `source .venv/bin/activate`
-3. Get all predefined dependencies from the `uv.lock` file by running the command: `uv sync`
-4. Run the code with `python quickstart.py`
+First, install `uv`. Then, run **all** of the following commands in your system terminal (like Terminal, PowerShell, or bash) from the root folder of the `neo4j-quickstart` project.
 
-Note: uv and Jupyter Notebook integration is not covered in this set-up so you will need to work with python scripts only.
+**Do not** run these in a notebook cell.
 
-## Build docker
+**1. Install `uv` (if not already done):**
+Run **one** of these commands in your terminal.
 
-Locally: `docker build -f tools/images/Dockerfile -t neo4j-quickstart .`
+* *On macOS / Linux:*
+    ```bash
+    curl -LsSf [https://astral.sh/uv/install.sh](https://astral.sh/uv/install.sh) | sh
+    ```
+* *On Windows (PowerShell):*
+    ```bash
+    pip install uv
+    ```
 
-Else there is an integrated github CI in the github workflows of this repository.
+> **Note:** After installing, you may need to **restart your terminal or VS Code** for the `uv` command to be recognized.
+
+**2. Create the Environment and Install Packages:**
+These commands will create a local `.venv` folder, install all project dependencies (including `ipykernel`), and register it as a named kernel for Jupyter/VS Code.
+
+You can go to `src/neo4j-quickstart` and run `uv sync`.
+
+You can then choose to use a python script or a jupyter notebook.
+
+**3. (if you will use Jupyter Notebooks) Register the Kernel:**
+Now, run the command for your operating system to make the new environment visible to Jupyter/VS Code.
+
+* *On macOS / Linux:*
+    ```bash
+    .venv/bin/python -m ipykernel install --user --name="neo4j-env" --display-name="Python (Neo4j)"
+    ```
+* *On Windows (PowerShell):*
+    ```bash
+    .venv\Scripts\python.exe -m ipykernel install --user --name="neo4j-env" --display-name="Python (Neo4j)"
+    ```
+
+---
+
+### 💻 Part 2: Run the Code (In VS Code: Script or Notebook)
+
+SCRIPT :
+
+Run the code with `python quickstart.py`
+
+NOTEBOOK :
+
+Now that your environment is built and registered, you can start the notebook.
+
+1.  **Open VS Code:** Open the *entire* `neo4j-quickstart` folder in VS Code (File > Open Folder...).
+2.  **Open this Notebook:** Open the `.ipynb` notebook file.
+3.  **Select the Kernel:**
+    * Click the "Select Kernel" button in the top-right corner of the notebook.
+    * From the dropdown list, choose **"Python (Neo4j)"**. This is the name you just registered.
+    * If VS Code prompts you, it may also auto-detect the `.venv` folder. Selecting that also works, but the named kernel is more explicit.
+
+You are now ready to run the cells in the notebook.

src/neo4j-quickstart/quickstart.ipynb

@@ -80,9 +80,11 @@
    "id": "d3707369",
    "metadata": {},
    "source": [
-    "## Neo4J Querrying: Examples of CYPHER queries\n",
+    "## Neo4J Querrying: Examples of CYPHER queries for Graph Exploring and Graph Patterns\n",
     "\n",
-    "Here is an example of using the Neo4J Downloader to run some custom queries on the graph."
+    "Here is an example of using the Neo4J Downloader to run some custom queries on the graph.\n",
+    "\n",
+    "You can explore [graph patterns (such as shortest path between 2 nodes)](https://neo4j.com/docs/cypher-manual/current/patterns/)."
    ]
   },
   {

src/neo4j-quickstart/quickstart.py

@@ -52,7 +52,10 @@ def connect_neo4j():
     return downloader
 
 # ------------------------------------------------------------------
-# EXTRACT CUSTOM SUBSET OF DATA FROM NEO4J WITH CYPHER QUERRIES
+# EXAMPLE OF CYPHER QUERIES FOR GRAPH EXPLORING AND GRAPH PATTERNS
+
+# Here is an example of using the Neo4J Downloader to run some custom queries on the graph.
+# You can explore graph patterns (such as shortest path between 2 nodes) : (https://neo4j.com/docs/cypher-manual/current/patterns/).
 # ------------------------------------------------------------------
 
 downloader = connect_neo4j()

src/tentris-quickstart/.env.dist

@@ -0,0 +1,2 @@
+TENTRIS_USER="YOUR_TENTRIS_USER"
+TENTRIS_PASSWORD="YOUR_TENTRIS_PASSWORD"

src/tentris-quickstart/README.md

@@ -1,3 +1,15 @@
+# QuickStart Tentris
+
+For open pulse project, you will manipulate graph data with Tentris. The example notebook will guide you on how to download and visualize the graph. Should you prefer scripts, one is also available to you.
+
+## Working on Renku with Jupyter Notebook 
+
+This set-up provides jupyter notebook support if this is your preferred working method.
+
+Please see instructions in `2025-mini-hackathon.md` at the root of the repository for more instructions.
+
+## Working locally
+
 ### 🚀 Part 1: Create the Environment (In Your Terminal)
 
 First, install `uv`. Then, run **all** of the following commands in your system terminal (like Terminal, PowerShell, or bash) from the root folder of the `tentris-quickstart` project.
@@ -21,6 +33,12 @@ Run **one** of these commands in your terminal.
 **2. Create the Environment and Install Packages:**
 These commands will create a local `.venv` folder, install all project dependencies (including `ipykernel`), and register it as a named kernel for Jupyter/VS Code.
 
+Option A: 
+
+You can also go to `src/tentris-quickstart` and run `uv sync`.
+
+Option B:
+
 ```bash
 # 1. Clean up any old environment (optional, but safe)
 rm -rf .venv