Skip to content

Commit 380b123

Browse files
NEW: add CaptureHolder developer documentation (#162)
* Add CaptureHolder docs * Update API docs on CaptureHolder * Document the new view annotations on Pipelines Co-authored-by: Greg Caporaso <jgcap@fastmail.com>
1 parent 529aab8 commit 380b123

4 files changed

Lines changed: 133 additions & 19 deletions

File tree

‎book/_toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ parts:
4444
- file: plugins/how-to-guides/usage-examples
4545
- file: plugins/how-to-guides/format-validation-levels
4646
- file: plugins/how-to-guides/handle-exceptions-in-parallel-pipelines
47+
- file: plugins/how-to-guides/track-the-value-of-auto-params-in-provenance
4748
- file: plugins/how-to-guides/include-R-in-plugins
4849
- file: plugins/how-to-guides/raise-visible-warning
4950
- file: plugins/explanations/intro

‎book/back-matter/glossary.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ Artifact class
2323
Artifact API
2424
See {term}`Python 3 API`.
2525
26+
CaptureHolder
27+
A class used as the Python type annotation for {term}`Parameters <Parameter>` whose default value is determined algorithmically (e.g., random number generator seeds).
28+
Using this class ensures that the algorithmically set value is tracked in {term}`Provenance`.
29+
2630
Collection
2731
An ordered list of `key: value` pairs. Think of an ordered [Python dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries).
2832
These can be used as {term}`Input`, {term}`Parameter`, and {term}`Output` {term}`Types <Type>`.

‎book/plugins/how-to-guides/create-register-pipeline.md‎

Lines changed: 62 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -7,32 +7,58 @@ This is accomplished by stitching together one or more `Methods` and/or `Visuali
77

88
Defining a function that can be registered as a `Pipeline` is very similar to defining one that can be registered as a `Method` with a few distinctions.
99

10-
First, `Pipelines` do not use function annotations and instead receive `Artifact` objects as input and return `Artifact` and/or `Visualization` objects as output.
10+
First, `Pipelines` are not required to use function annotations unless you are using the {term}`CaptureHolder` API documented [here](howto-track-the-value-of-auto-params-in-provenance). Instead they implicitly receive `Artifact` objects as input and return `Artifact` and/or `Visualization` objects as output.
11+
12+
If you choose to use function annotations on a `Pipeline` you must annotate all inputs, parameters, outputs, and the special `ctx` argument (described below). The parameters follow the same [mypy](http://mypy-lang.org/) syntax as `Methods` and `Visualizers`; however, the inputs and outputs are annotated simply as `Artifact` or `Visualization` in the case of singles or `list[Artifact]`, `dict[str, Artifact]`, `list[Visualization]`, or `dict[str, Visualization]` in the case of `Collections`. `ctx` must use `IContext` as its annotation.
1113

1214
Second, `Pipelines` must have `ctx` as their first parameter, which provides the following API:
1315
- `ctx.get_action(plugin: str, action: str)`: returns a *sub-action* that can be called like a normal Artifact API call.
1416
- `ctx.make_artifact(type, view, view_type=None)`: this has the same behavior as `Artifact.import_data`. It is wrapped by `ctx` for pipeline book-keeping.
1517

16-
Let's take a look at [`q2_diversity.core_metrics`](https://github.com/qiime2/q2-diversity/blob/99a0ccaaec14838b95845dbfe57f874d092b65c7/q2_diversity/_core_metrics.py#L10) for an example of a function that we can register as a `Pipeline`:
18+
Let's take a look at [`q2_diversity.core_metrics`](https://github.com/qiime2/q2-diversity/blob/3fe491062b8a72939111ff66b2f4aeab8c12b16d/q2_diversity/_core_metrics.py#L14) for an example of a function that we can register as a `Pipeline`:
1719

1820
```python
19-
def core_metrics(ctx, table, sampling_depth, metadata, n_jobs=1):
21+
def core_metrics(ctx: IContext,
22+
table: Artifact,
23+
sampling_depth: int,
24+
metadata: Metadata,
25+
with_replacement: bool = False,
26+
n_jobs: int = 1,
27+
ignore_missing_samples: bool = False,
28+
random_seed: CaptureHolder[int] = None) -> \
29+
tuple[
30+
Artifact, Artifact, Artifact, Artifact, Artifact, Artifact,
31+
Artifact, Artifact, Visualization, Visualization
32+
]:
33+
random_int = CaptureHolder.get_or_set(random_seed, get_np_random_seed)
34+
biom_table = table.view(biom.Table)
35+
if biom_table.length() < 2:
36+
raise ValueError(
37+
'Table must have at least two samples as beta diversity will be'
38+
' applied later.'
39+
)
40+
2041
rarefy = ctx.get_action('feature_table', 'rarefy')
21-
alpha = ctx.get_action('diversity', 'alpha')
22-
beta = ctx.get_action('diversity', 'beta')
42+
observed_features = ctx.get_action('diversity_lib', 'observed_features')
43+
pielou_e = ctx.get_action('diversity_lib', 'pielou_evenness')
44+
shannon = ctx.get_action('diversity_lib', 'shannon_entropy')
45+
braycurtis = ctx.get_action('diversity_lib', 'bray_curtis')
46+
jaccard = ctx.get_action('diversity_lib', 'jaccard')
2347
pcoa = ctx.get_action('diversity', 'pcoa')
2448
emperor_plot = ctx.get_action('emperor', 'plot')
2549

2650
results = []
27-
rarefied_table, = rarefy(table=table, sampling_depth=sampling_depth)
51+
rarefied_table, = rarefy(table=table, sampling_depth=sampling_depth,
52+
with_replacement=with_replacement,
53+
random_seed=random_int)
2854
results.append(rarefied_table)
2955

30-
for metric in 'observed_otus', 'shannon', 'pielou_e':
31-
results += alpha(table=rarefied_table, metric=metric)
56+
for metric in (observed_features, shannon, pielou_e):
57+
results += metric(table=rarefied_table)
3258

3359
dms = []
34-
for metric in 'jaccard', 'braycurtis':
35-
beta_results = beta(table=rarefied_table, metric=metric, n_jobs=n_jobs)
60+
for metric in (jaccard, braycurtis):
61+
beta_results = metric(table=rarefied_table, n_jobs=n_jobs)
3662
results += beta_results
3763
dms += beta_results
3864

@@ -43,7 +69,8 @@ def core_metrics(ctx, table, sampling_depth, metadata, n_jobs=1):
4369
pcoas += pcoa_results
4470

4571
for pcoa in pcoas:
46-
results += emperor_plot(pcoa=pcoa, metadata=metadata)
72+
results += emperor_plot(pcoa=pcoa, metadata=metadata,
73+
ignore_missing_samples=ignore_missing_samples)
4774

4875
return tuple(results)
4976
```
@@ -61,7 +88,7 @@ A description of this output should be included in `output_descriptions`
6188
Citations do not need to be added for the pipeline unless unique citations are required for the pipeline that are not appropriate for the underlying `Methods` and `Visualizers` that it calls.
6289
Citations for these underlying actions are automatically logged in citation provenance for this pipeline.
6390

64-
As an example for registering a `Pipeline`, we can look at `q2_diversity.core_metrics` (find the original source [here](https://github.com/qiime2/q2-diversity/blob/99a0ccaaec14838b95845dbfe57f874d092b65c7/q2_diversity/plugin_setup.py#L494)):
91+
As an example for registering a `Pipeline`, we can look at `q2_diversity.core_metrics` (find the original source [here](https://github.com/qiime2/q2-diversity/blob/3fe491062b8a72939111ff66b2f4aeab8c12b16d/q2_diversity/plugin_setup.py#L496-L565)):
6592

6693
```python
6794
plugin.pipelines.register_function(
@@ -72,11 +99,14 @@ plugin.pipelines.register_function(
7299
parameters={
73100
'sampling_depth': Int % Range(1, None),
74101
'metadata': Metadata,
75-
'n_jobs': Int % Range(0, None),
102+
'with_replacement': Bool,
103+
'n_jobs': Threads,
104+
'ignore_missing_samples': Bool,
105+
'random_seed': Int
76106
},
77107
outputs=[
78108
('rarefied_table', FeatureTable[Frequency]),
79-
('observed_otus_vector', SampleData[AlphaDiversity]),
109+
('observed_features_vector', SampleData[AlphaDiversity]),
80110
('shannon_vector', SampleData[AlphaDiversity]),
81111
('evenness_vector', SampleData[AlphaDiversity]),
82112
('jaccard_distance_matrix', DistanceMatrix),
@@ -88,17 +118,30 @@ plugin.pipelines.register_function(
88118
],
89119
input_descriptions={
90120
'table': 'The feature table containing the samples over which '
91-
'diversity metrics should be computed.',
121+
'diversity metrics should be computed.',
92122
},
93123
parameter_descriptions={
94124
'sampling_depth': 'The total frequency that each sample should be '
95-
'rarefied to prior to computing diversity metrics.',
125+
'rarefied to prior to computing diversity metrics.',
96126
'metadata': 'The sample metadata to use in the emperor plots.',
97-
'n_jobs': '[beta methods only] - %s' % sklearn_n_jobs_description
127+
'with_replacement': with_replacement_description,
128+
'n_jobs': '[beta methods only] - %s' % n_jobs_description,
129+
'ignore_missing_samples': 'If set to `True` samples and features '
130+
'without metadata are included by '
131+
'setting all metadata values to: '
132+
'"This element has no metadata". By '
133+
'default an exception will be raised if '
134+
'missing elements are encountered. Note, '
135+
'this flag only takes effect if there is at '
136+
'least one overlapping element.',
137+
'random_seed': 'Seed for the random number generation used to rarefy '
138+
'your feature table.'
139+
98140
},
99141
output_descriptions={
100142
'rarefied_table': 'The resulting rarefied feature table.',
101-
'observed_otus_vector': 'Vector of Observed OTUs values by sample.',
143+
'observed_features_vector': 'Vector of Observed Features values by '
144+
'sample.',
102145
'shannon_vector': 'Vector of Shannon diversity values by sample.',
103146
'evenness_vector': 'Vector of Pielou\'s evenness values by sample.',
104147
'jaccard_distance_matrix':
@@ -116,7 +159,7 @@ plugin.pipelines.register_function(
116159
},
117160
name='Core diversity metrics (non-phylogenetic)',
118161
description=("Applies a collection of diversity metrics "
119-
"(non-phylogenetic) to a feature table.")
162+
"(non-phylogenetic) to a feature table.")
120163
)
121164
```
122165

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
(howto-track-the-value-of-auto-params-in-provenance)=
2+
# Track the value of random seeds and other automated parameter settings in data provenance
3+
4+
There are cases where you might want an {term}`Action` to take a {term}`Parameter` that can be set to an explicit value by the user, or be algorithmically determined by the {term}`Action` itself.
5+
The most common use case for this is a random seed, where you may allow the user to pass a set seed (usually because they are trying to reproduce a previous result) or have the {term}`Action` set a random seed for general use (usually the default approach, achieved by setting the default value to `None`).
6+
This creates a challenge for reproducibility: if `None` is passed into the {term}`Action`, then `None` will be recorded in the output's {term}`Provenance`.
7+
The value that the seed was set to internally is lost, making it impossible to exactly reproduce a prior {term}`Action` execution because the random seed is not known.
8+
9+
This problem is solved by using the {term}`CaptureHolder` object.
10+
11+
```{note}
12+
The {term}`CaptureHolder` only works if the value of the algorithmically set parameter is actually accessible in the Python implementation of the function.
13+
If you are passing a sentinel value into an underlying tool (e.g., R code that is used under the hood by your {term}`Action`) which sets its value, that value will be inaccessible in the {term}`Provenance`.
14+
```
15+
16+
The {term}`Action` registration is unchanged:
17+
18+
```python
19+
my_plugin.methods.register_function(
20+
function=method_with_random_seed,
21+
inputs={},
22+
parameters={
23+
'random_seed': Int
24+
},
25+
outputs=[('seed', SingleInt)],
26+
name='Takes a random seed',
27+
description='Takes an integer as a random seed and returns that same'
28+
' integer. If no integer is provided, it generates one at'
29+
' random and captures that randomly generated integer in'
30+
' provenance.'
31+
)
32+
```
33+
34+
What changes is the implementation of the underlying Python function:
35+
36+
```python
37+
from qiime2.plugin.type import CaptureHolder
38+
39+
def random_seed_method(random_seed: CaptureHolder[int] = None) -> int:
40+
# Resolve the seed: if the user passed None, generate a random value and
41+
# record it in provenance; otherwise use the user-supplied value as-is.
42+
random_int = CaptureHolder.get_or_set(
43+
random_seed, lambda: random.randrange(sys.maxsize)
44+
)
45+
46+
# Use the resolved integer value (guaranteed to never be None here)
47+
my_value = my_function(random_int)
48+
49+
return my_value
50+
```
51+
52+
The following rules must be followed to use the {term}`CaptureHolder` object:
53+
54+
1. The type annotation on the {term}`CaptureHolder` {term}`Parameter` must be `CaptureHolder[T]`, where `T` is the Python view type that corresponds to the QIIME 2 {term}`Semantic Type` used for the {term}`Parameter` at registration (e.g., `CaptureHolder[int]` for a parameter registered as `Int`).
55+
2. The default value of the {term}`CaptureHolder` {term}`Parameter` must be `None`.
56+
3. `CaptureHolder.get_or_set(<instance>, <callable>)` must be called exactly once per {term}`CaptureHolder` {term}`Parameter`, before the parameter is used. The return value is the resolved value that should be used in place of the {term}`CaptureHolder` going forward.
57+
58+
`CaptureHolder.get_or_set` takes two arguments: the {term}`CaptureHolder` {term}`Parameter` instance, and a zero-argument callable that generates a value when one is needed.
59+
If the user passed `None`, the callable is invoked and its return value is used.
60+
If the user passed an explicit value, that value is returned directly.
61+
In both cases the resolved value is written back into the {term}`Action`'s {term}`Provenance` as though it had been passed in by the user originally.
62+
63+
```{note}
64+
When calling the underlying function directly during testing (rather than through QIIME 2), `CaptureHolder.get_or_set` behaves correctly whether the parameter is a `CaptureHolder` instance, a plain value, or `None`.
65+
This means you can write unit tests that call the function directly without any special handling.
66+
```

0 commit comments

Comments
 (0)