Skip to content

Commit 382d3fa

Browse files
deploy: f1dc78a
0 parents  commit 382d3fa

173 files changed

Lines changed: 80600 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.nojekyll

Whitespace-only changes.

CNAME

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
nbdev.fast.ai

api/clean.html

Lines changed: 1098 additions & 0 deletions
Large diffs are not rendered by default.

api/clean.html.md

Lines changed: 307 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,307 @@
1+
# clean
2+
3+
4+
<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
5+
6+
To avoid pointless conflicts while working with jupyter notebooks (with
7+
different execution counts or cell metadata), it is recommended to clean
8+
the notebooks before committing anything (done automatically if you
9+
install the git hooks with `nbdev-install-hooks`). The following
10+
functions are used to do that. Cleaning also adds cell `id`s if missing
11+
(required by nbformat 4.5+).
12+
13+
## Trust
14+
15+
------------------------------------------------------------------------
16+
17+
<a
18+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L28"
19+
target="_blank" style="float:right; font-size:smaller">source</a>
20+
21+
### nbdev_trust
22+
23+
``` python
24+
def nbdev_trust(
25+
fname:str=None, # A notebook name or glob to trust
26+
force_all:bool=False, # Also trust notebooks that haven't changed
27+
):
28+
```
29+
30+
*Trust notebooks matching `fname`.*
31+
32+
## Clean
33+
34+
------------------------------------------------------------------------
35+
36+
<a
37+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L90"
38+
target="_blank" style="float:right; font-size:smaller">source</a>
39+
40+
### clean_nb
41+
42+
``` python
43+
def clean_nb(
44+
nb, # The notebook to clean
45+
clear_all:bool=False, # Remove all cell metadata and cell outputs?
46+
allowed_metadata_keys:list=None, # Preserve the list of keys in the main notebook metadata
47+
allowed_cell_metadata_keys:list=None, # Preserve the list of keys in cell level metadata
48+
clean_ids:bool=True, # Remove ids from plaintext reprs?
49+
allowed_out_metadata_keys:list=None, # Preserve the list of keys in output metadata
50+
repair:bool=True, # Fix structural problems first (see `repair_nb`)?
51+
):
52+
```
53+
54+
*Clean `nb` from superfluous metadata*
55+
56+
Jupyter adds a trailing <code></code> to images in cell outputs.
57+
Vscode-jupyter does not.\
58+
Notebooks should be brought to a common style to avoid unnecessary
59+
diffs:
60+
61+
``` python
62+
test_nb = read_nb('../../tests/image.ipynb')
63+
assert test_nb.cells[0].outputs[0].data['image/png'][-1] == "\n" # Make sure it was not converted by acccident
64+
clean_nb(test_nb)
65+
assert test_nb.cells[0].outputs[0].data['image/png'][-1] != "\n"
66+
```
67+
68+
The test notebook has metadata in both the main metadata section and
69+
contains cell level metadata in the second cell:
70+
71+
``` python
72+
test_nb = read_nb('../../tests/metadata.ipynb')
73+
74+
assert {'meta', 'jekyll', 'nbdev', 'my_extra_key', 'my_removed_key'} <= test_nb.metadata.keys()
75+
assert {'meta', 'hide_input', 'my_extra_cell_key', 'nbdev', 'my_removed_cell_key'} == test_nb.cells[1].metadata.keys()
76+
```
77+
78+
After cleaning the notebook, all extra metadata is removed, only some
79+
keys are allowed by default:
80+
81+
``` python
82+
clean_nb(test_nb)
83+
84+
assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys()
85+
assert {'hide_input', 'nbdev'} == test_nb.cells[1].metadata.keys()
86+
```
87+
88+
[`clean_nb`](https://nbdev.fast.ai/api/clean.html#clean_nb) also repairs
89+
structural problems by default (via `repair_nb`), so notebooks that
90+
Jupyter would reject, such as a markdown cell carrying an `outputs`
91+
attr, are fixed on every clean:
92+
93+
``` python
94+
_nb = dict2nb(dict(cells=[dict(cell_type='markdown', source='hi', outputs=[], execution_count=1, id='m1', metadata={})],
95+
metadata=dict(kernelspec=dict(name='python3', display_name='Python 3')), nbformat=4, nbformat_minor=5))
96+
clean_nb(_nb)
97+
assert 'outputs' not in _nb.cells[0] and 'execution_count' not in _nb.cells[0]
98+
validate_nb(_nb)
99+
```
100+
101+
We can preserve some additional keys at the notebook or cell levels:
102+
103+
``` python
104+
test_nb = read_nb('../../tests/metadata.ipynb')
105+
clean_nb(test_nb, allowed_metadata_keys={'my_extra_key'}, allowed_cell_metadata_keys={'my_extra_cell_key'})
106+
107+
assert {'jekyll', 'kernelspec', 'nbdev', 'my_extra_key'} == test_nb.metadata.keys()
108+
assert {'hide_input', 'nbdev', 'my_extra_cell_key'} == test_nb.cells[1].metadata.keys()
109+
```
110+
111+
Passing `clear_all=True` removes everything from the cell metadata:
112+
113+
``` python
114+
test_nb = read_nb('../../tests/metadata.ipynb')
115+
clean_nb(test_nb, clear_all=True)
116+
117+
assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys()
118+
test_eq(test_nb.cells[1].metadata, {})
119+
```
120+
121+
Passing `clean_ids=True` removes `id`s from plaintext repr outputs, to
122+
avoid notebooks whose contents change on each run since they often lead
123+
to git merge conflicts. For example:
124+
125+
<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x7FB4F8979690>
126+
127+
becomes:
128+
129+
<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28>
130+
131+
*Cell* IDs, on the other hand, are always added if missing
132+
133+
``` python
134+
test_cell = {'source': 'x=1', 'cell_type': 'code', 'metadata': {}}
135+
_clean_cell(test_cell, False, set(), True, set())
136+
test_cell['id']
137+
```
138+
139+
'64e20756'
140+
141+
------------------------------------------------------------------------
142+
143+
<a
144+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L120"
145+
target="_blank" style="float:right; font-size:smaller">source</a>
146+
147+
### process_write
148+
149+
``` python
150+
def process_write(
151+
warn_msg, proc_nb, f_in, f_out:NoneType=None, disp:bool=False
152+
):
153+
```
154+
155+
*Call self as a function.*
156+
157+
## Directive migrations
158+
159+
Deliberate, opt-in rewrites for moving to the directive conventions
160+
introduced in v3.3: directives can live in cell metadata (an `nbdev`
161+
dict key), notebook-scope directives like `default_exp` can live in
162+
notebook metadata, and comment spellings have one canonical form. None
163+
of these run by default; they’re flags on `nbdev-clean` for one-off
164+
migration runs, so the git hook never triggers them.
165+
166+
[`_to_meta`](https://nbdev.fast.ai/api/clean.html#_to_meta) and
167+
[`_to_comments`](https://nbdev.fast.ai/api/clean.html#_to_comments) move
168+
named directives between a cell’s comments and its `nbdev` metadata key,
169+
preserving values through the same mapping the parser uses (bare
170+
directives become `true`, and so on):
171+
172+
``` python
173+
c = mk_cell('#| hide\n#| eval: false\n#| export: utils\n1+1')
174+
_to_meta(c, ['hide','eval'])
175+
test_eq(c.metadata['nbdev'], dict(hide='true', eval='false'))
176+
test_eq(c.source, '#| export: utils\n1+1')
177+
178+
_to_comments(c, ['hide','eval'])
179+
assert 'nbdev' not in c.metadata
180+
test_eq(c.directives, {'export':'utils', 'hide':'', 'eval':'false'})
181+
```
182+
183+
[`_canon_dirs`](https://nbdev.fast.ai/api/clean.html#_canon_dirs)
184+
respells each directive line canonically without touching anything else,
185+
and
186+
[`_hoist_nb_meta`](https://nbdev.fast.ai/api/clean.html#_hoist_nb_meta)
187+
moves `default_exp` to notebook metadata, deleting a first cell that
188+
held nothing but directives (a bare `#| hide` on an otherwise-empty cell
189+
hides nothing).
190+
[`_dir_moves`](https://nbdev.fast.ai/api/clean.html#_dir_moves) bundles
191+
the migrations for the CLI, converting the raw loaded dict to `NbCell`s
192+
first:
193+
194+
``` python
195+
c = mk_cell('#| default_exp core\n#| eval:false\n1+1')
196+
_canon_dirs(c)
197+
test_eq(c.source, '#| default_exp: core\n#| eval: false\n1+1')
198+
199+
_nb = dict2nb(dict(cells=[mk_cell('#| hide\n#| default_exp: core'), mk_cell('#| export\n1+1')],
200+
metadata={}, nbformat=4, nbformat_minor=5))
201+
_hoist_nb_meta(_nb)
202+
test_eq(_nb.metadata['nbdev'], dict(default_exp='core'))
203+
test_eq(len(_nb.cells), 1)
204+
test_eq(_nb.cells[0].source, '#| export\n1+1')
205+
```
206+
207+
------------------------------------------------------------------------
208+
209+
<a
210+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L203"
211+
target="_blank" style="float:right; font-size:smaller">source</a>
212+
213+
### nbdev_clean
214+
215+
``` python
216+
def nbdev_clean(
217+
fname:str=None, # A notebook name or glob to clean
218+
clear_all:bool=False, # Remove all cell metadata and cell outputs?
219+
disp:bool=False, # Print the cleaned outputs
220+
stdin:bool=False, # Read notebook from input stream
221+
repair:bool_arg=True, # Fix structural problems, e.g. stray outputs on non-code cells (see `repair_nb`)?
222+
dirs:bool=False, # Rewrite comment directives in canonical form?
223+
to_meta:str=None, # Space-separated directive names to move from comments to cell metadata
224+
to_comments:str=None, # Space-separated directive names to move from cell metadata to comments
225+
nb_meta:bool=False, # Move `default_exp` into notebook metadata?
226+
):
227+
```
228+
229+
*Clean all notebooks in `fname` to avoid merge conflicts*
230+
231+
By default (`fname` left to `None`), all the notebooks in
232+
`config.nbs_path` are cleaned. You can opt in to fully clean the
233+
notebook by removing every bit of metadata and the cell outputs by
234+
passing `clear_all=True`.
235+
236+
If you want to keep some keys in the main notebook metadata you can set
237+
`allowed_metadata_keys` in `[tool.nbdev]` in `pyproject.toml`. Similarly
238+
for cell level metadata use `allowed_cell_metadata_keys`, and for output
239+
metadata use `allowed_out_metadata_keys`. For example, to preserve both
240+
`k1` and `k2` at both the notebook and cell level add the following to
241+
`pyproject.toml`:
242+
243+
``` toml
244+
[tool.nbdev]
245+
allowed_metadata_keys = ["k1", "k2"]
246+
allowed_cell_metadata_keys = ["k1", "k2"]
247+
allowed_out_metadata_keys = ["k1", "k2"]
248+
```
249+
250+
------------------------------------------------------------------------
251+
252+
<a
253+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L223"
254+
target="_blank" style="float:right; font-size:smaller">source</a>
255+
256+
### clean_jupyter
257+
258+
``` python
259+
def clean_jupyter(
260+
path, model, **kwargs
261+
):
262+
```
263+
264+
*Clean Jupyter `model` pre save to `path`*
265+
266+
This cleans notebooks on-save to avoid unnecessary merge conflicts. The
267+
easiest way to install it for both Jupyter Notebook and Lab is by
268+
running `nbdev-install-hooks`. It works by implementing a
269+
`pre_save_hook` from Jupyter’s [file save hook
270+
API](https://jupyter-server.readthedocs.io/en/latest/developers/savehooks.html).
271+
272+
## Hooks
273+
274+
------------------------------------------------------------------------
275+
276+
<a
277+
href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L281"
278+
target="_blank" style="float:right; font-size:smaller">source</a>
279+
280+
### nbdev_install_hooks
281+
282+
``` python
283+
def nbdev_install_hooks(
284+
merge:bool_arg=True, # Install the notebook merge driver?
285+
diff:bool_arg=True, # Install the notebook diff driver?
286+
globally:bool_arg=False, # Define the drivers in `~/.gitconfig` and the global attributes file, instead of repo files?
287+
):
288+
```
289+
290+
*Install Jupyter and git hooks to automatically clean, trust, and fix
291+
merge conflicts in notebooks*
292+
293+
See
294+
[`clean_jupyter`](https://nbdev.fast.ai/api/clean.html#clean_jupyter)
295+
and `nbdev-merge` for more about how each hook works.
296+
297+
Both git drivers are registered under the name `jupyternotebook`, the
298+
same name nbdime uses. Repo installs define them in a repo-local
299+
`.gitconfig` (wired in via `include.path`) and activate them in the
300+
committed `.gitattributes`; `--globally` instead defines them in
301+
`~/.gitconfig` and activates them in the global attributes file
302+
(`core.attributesFile`, defaulting to `~/.config/git/attributes`). Since
303+
the attribute lines just name a driver, a committed `.gitattributes`
304+
means “use your preferred notebook driver”: whichever tool defined
305+
`jupyternotebook` last in a config git consults wins, so switching
306+
between nbdev and nbdime is just re-running either tool’s enable
307+
command. Global installs skip the repo-only `post-merge` trust hook.

0 commit comments

Comments
 (0)