Skip to content

Commit e4d15b8

Browse files
author
Solveit
committed
add ability to find a particular evaluation by title or url
1 parent 150f212 commit e4d15b8

3 files changed

Lines changed: 116 additions & 207 deletions

File tree

README.md

Lines changed: 68 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -3,52 +3,50 @@
33

44
<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
55

6-
Why my `nbs/sidebar.yml` file is erased when running `nbdev_prepare`?
7-
8-
Here is my settings.ini:
9-
10-
![pasted_image_139aef0e-db47-4fc7-92b5-0d761129545c.png](index_files/figure-commonmark/37de56d3-1-139aef0e-db47-4fc7-92b5-0d761129545c.png)
11-
12-
##### 🤖Reply🤖<!-- SOLVEIT_SEPARATOR_7f3a9b2c -->
13-
14-
Looking at your `settings.ini`, I can see the issue. You have:
15-
16-
custom_sidebar = True
17-
18-
When `custom_sidebar = True`, nbdev expects you to manually maintain
19-
your `sidebar.yml` and won’t overwrite it. However, if the file doesn’t
20-
exist or gets erased, nbdev might be regenerating it.
6+
[![PyPI](https://img.shields.io/pypi/v/evaluatr.png)](https://pypi.org/project/evaluatr/)
7+
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
8+
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue.png)](https://franckalbinet.github.io/evaluatr/)
219

22-
The problem is likely that you also have:
10+
## What is Evaluatr?
2311

24-
doc_host = https://francklabinet.github.io
25-
doc_baseurl = /evaluatr/
12+
`Evaluatr` is an AI-powered system that automates mapping evaluation
13+
reports against structured frameworks while maintaining interpretability
14+
and human oversight. Initially developed for [IOM (International
15+
Organization for Migration)](https://www.iom.int) evaluation reports and
16+
the [Strategic Results Framework (SRF)](https://srf.iom.int), it
17+
transforms a traditionally manual, time-intensive process into an
18+
efficient, transparent workflow.
2619

27-
But your repository structure might not match what nbdev expects. Try
28-
this:
20+
The system maps evaluation reports against hierarchical frameworks like
21+
the SRF (objectives, enablers, cross-cutting priorities, outcomes,
22+
outputs, indicators) and connects to broader frameworks like the
23+
[Sustainable Development Goals (SDGs)](https://sdgs.un.org) for
24+
interoperability.
2925

30-
1. Set `custom_sidebar = False` temporarily to let nbdev generate the
31-
sidebar
32-
2. Run `nbdev_prepare`
33-
3. Once generated, modify the sidebar as needed
34-
4. Then set `custom_sidebar = True` again
26+
Beyond automation, `Evaluatr` prioritizes **interpretability and
27+
human-AI collaboration**—enabling evaluators to understand the mapping
28+
process, audit AI decisions, perform error analysis, and build training
29+
datasets over time, ensuring the system aligns with organizational needs
30+
through actionable, transparent, auditable methodology.
3531

36-
Alternatively, if you want to keep `custom_sidebar = True`, make sure
37-
your `nbs/sidebar.yml` file exists and has valid YAML content before
38-
running `nbdev_prepare`. Even a minimal valid structure like:
32+
## The Challenge We Solve
3933

40-
``` yaml
41-
website:
42-
sidebar:
43-
contents:
44-
- index.ipynb
45-
```
34+
IOM evaluators possess deep expertise in mapping evaluation reports
35+
against frameworks like the Strategic Results Framework (SRF), but face
36+
significant operational challenges when processing reports that often
37+
exceed 150 pages of diverse content across multiple projects and
38+
contexts.
4639

47-
Would you like me to help you create a proper `sidebar.yml` structure?
40+
The core challenges are:
4841

49-
[![PyPI](https://img.shields.io/pypi/v/evaluatr.png)](https://pypi.org/project/evaluatr/)
50-
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
51-
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue.png)](https://franckalbinet.github.io/evaluatr/)
42+
- **Time-intensive process**: Hundreds of staff-hours required per
43+
comprehensive mapping exercise
44+
- **Individual consistency**: Even expert evaluators may categorize the
45+
same content differently across sessions
46+
- **Cross-evaluator consistency**: Different evaluators may interpret
47+
and map identical content to different framework outputs
48+
- **Scale vs. thoroughness**: Growing volume of evaluation reports
49+
creates pressure to choose between speed and comprehensive analysis
5250

5351
## Understanding Evaluation Mapping in the UN Context
5452

@@ -80,47 +78,6 @@ relevant reports) and recall (avoiding irrelevant ones).
8078
> Throughout this documentation, we use “mapping” and “tagging”
8179
> interchangeably.
8280
83-
## The Challenge We Solve
84-
85-
IOM evaluators possess deep expertise in mapping evaluation reports
86-
against frameworks like the Strategic Results Framework (SRF), but face
87-
significant operational challenges when processing reports that often
88-
exceed 150 pages of diverse content across multiple projects and
89-
contexts.
90-
91-
The core challenges are:
92-
93-
- **Time-intensive process**: Hundreds of staff-hours required per
94-
comprehensive mapping exercise
95-
- **Individual consistency**: Even expert evaluators may categorize the
96-
same content differently across sessions
97-
- **Cross-evaluator consistency**: Different evaluators may interpret
98-
and map identical content to different framework outputs
99-
- **Scale vs. thoroughness**: Growing volume of evaluation reports
100-
creates pressure to choose between speed and comprehensive analysis
101-
102-
## What is Evaluatr?
103-
104-
`Evaluatr` is an AI-powered system that automates mapping evaluation
105-
reports against structured frameworks while maintaining interpretability
106-
and human oversight. Initially developed for [IOM (International
107-
Organization for Migration)](https://www.iom.int) evaluation reports and
108-
the [Strategic Results Framework (SRF)](https://srf.iom.int), it
109-
transforms a traditionally manual, time-intensive process into an
110-
efficient, transparent workflow.
111-
112-
The system maps evaluation reports against hierarchical frameworks like
113-
the SRF (objectives, enablers, cross-cutting priorities, outcomes,
114-
outputs, indicators) and connects to broader frameworks like the
115-
[Sustainable Development Goals (SDGs)](https://sdgs.un.org) for
116-
interoperability.
117-
118-
Beyond automation, `Evaluatr` prioritizes **interpretability and
119-
human-AI collaboration**—enabling evaluators to understand the mapping
120-
process, audit AI decisions, perform error analysis, and build training
121-
datasets over time, ensuring the system aligns with organizational needs
122-
through actionable, transparent, auditable methodology.
123-
12481
## Key Features
12582

12683
### 1. Document Preparation Pipeline ✅ **Available**
@@ -210,7 +167,7 @@ model provider beyond the examples above.
210167

211168
## Quick Start
212169

213-
### IOM Workflow (Programmatic)
170+
### Reading & standardizing evaluations repository
214171

215172
For IOM evaluators working with the official evaluation repository,
216173
download the most recent evaluations from
@@ -231,9 +188,37 @@ evals[0]
231188
**Year:** 2025 \| **Organization:** IOM \| **Countries:** Worldwide
232189

233190
**Documents:** 2 available
234-
**ID:** `99923109...`
191+
**ID:** `9992310969aa2f428bc8aba29f865cf3`
192+
193+
To find a particular evaluation by `title` or `url`:
194+
195+
``` python
196+
from evaluatr.readers import find_eval
197+
198+
title = 'Evaluation of IOM Accountability to Affected Populations'
199+
find_eval(evals, title, by='title')
200+
```
201+
202+
### Evaluation of IOM Accountability to Affected Populations
203+
204+
**Year:** 2025 \| **Organization:** IOM \| **Countries:** Worldwide
205+
206+
**Documents:** 4 available
207+
**ID:** `6c3c2cf3fa479112967612b0baddab72`
208+
209+
``` python
210+
url = "https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/AAP%20Evaluation%20Report_final_.pdf"
211+
find_eval(evals, url, by='url')
212+
```
213+
214+
### Evaluation of IOM Accountability to Affected Populations
215+
216+
**Year:** 2025 \| **Organization:** IOM \| **Countries:** Worldwide
217+
218+
**Documents:** 4 available
219+
**ID:** `6c3c2cf3fa479112967612b0baddab72`
235220

236-
**Downloading Evaluation Documents**
221+
### Downloading Evaluation Documents
237222

238223
``` python
239224
from evaluatr.downloaders import download_docs
@@ -244,7 +229,7 @@ base_dir = Path("files/test/pdf_library")
244229
download_docs(fname, base_dir=base_dir, n_workers=0, overwrite=True)
245230
```
246231

247-
### Universal CLI Workflow
232+
### Command Line Interface (CLI) Workflow
248233

249234
Process any evaluation report from PDF to tagged outputs using three
250235
streamlined commands.

0 commit comments

Comments
 (0)