|
| 1 | +# clean |
| 2 | + |
| 3 | + |
| 4 | +<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! --> |
| 5 | + |
| 6 | +To avoid pointless conflicts while working with jupyter notebooks (with |
| 7 | +different execution counts or cell metadata), it is recommended to clean |
| 8 | +the notebooks before committing anything (done automatically if you |
| 9 | +install the git hooks with `nbdev-install-hooks`). The following |
| 10 | +functions are used to do that. Cleaning also adds cell `id`s if missing |
| 11 | +(required by nbformat 4.5+). |
| 12 | + |
| 13 | +## Trust |
| 14 | + |
| 15 | +------------------------------------------------------------------------ |
| 16 | + |
| 17 | +<a |
| 18 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L28" |
| 19 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 20 | + |
| 21 | +### nbdev_trust |
| 22 | + |
| 23 | +``` python |
| 24 | +def nbdev_trust( |
| 25 | + fname:str=None, # A notebook name or glob to trust |
| 26 | + force_all:bool=False, # Also trust notebooks that haven't changed |
| 27 | +): |
| 28 | +``` |
| 29 | + |
| 30 | +*Trust notebooks matching `fname`.* |
| 31 | + |
| 32 | +## Clean |
| 33 | + |
| 34 | +------------------------------------------------------------------------ |
| 35 | + |
| 36 | +<a |
| 37 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L90" |
| 38 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 39 | + |
| 40 | +### clean_nb |
| 41 | + |
| 42 | +``` python |
| 43 | +def clean_nb( |
| 44 | + nb, # The notebook to clean |
| 45 | + clear_all:bool=False, # Remove all cell metadata and cell outputs? |
| 46 | + allowed_metadata_keys:list=None, # Preserve the list of keys in the main notebook metadata |
| 47 | + allowed_cell_metadata_keys:list=None, # Preserve the list of keys in cell level metadata |
| 48 | + clean_ids:bool=True, # Remove ids from plaintext reprs? |
| 49 | + allowed_out_metadata_keys:list=None, # Preserve the list of keys in output metadata |
| 50 | + repair:bool=True, # Fix structural problems first (see `repair_nb`)? |
| 51 | +): |
| 52 | +``` |
| 53 | + |
| 54 | +*Clean `nb` from superfluous metadata* |
| 55 | + |
| 56 | +Jupyter adds a trailing <code></code> to images in cell outputs. |
| 57 | +Vscode-jupyter does not.\ |
| 58 | +Notebooks should be brought to a common style to avoid unnecessary |
| 59 | +diffs: |
| 60 | + |
| 61 | +``` python |
| 62 | +test_nb = read_nb('../../tests/image.ipynb') |
| 63 | +assert test_nb.cells[0].outputs[0].data['image/png'][-1] == "\n" # Make sure it was not converted by acccident |
| 64 | +clean_nb(test_nb) |
| 65 | +assert test_nb.cells[0].outputs[0].data['image/png'][-1] != "\n" |
| 66 | +``` |
| 67 | + |
| 68 | +The test notebook has metadata in both the main metadata section and |
| 69 | +contains cell level metadata in the second cell: |
| 70 | + |
| 71 | +``` python |
| 72 | +test_nb = read_nb('../../tests/metadata.ipynb') |
| 73 | + |
| 74 | +assert {'meta', 'jekyll', 'nbdev', 'my_extra_key', 'my_removed_key'} <= test_nb.metadata.keys() |
| 75 | +assert {'meta', 'hide_input', 'my_extra_cell_key', 'nbdev', 'my_removed_cell_key'} == test_nb.cells[1].metadata.keys() |
| 76 | +``` |
| 77 | + |
| 78 | +After cleaning the notebook, all extra metadata is removed, only some |
| 79 | +keys are allowed by default: |
| 80 | + |
| 81 | +``` python |
| 82 | +clean_nb(test_nb) |
| 83 | + |
| 84 | +assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys() |
| 85 | +assert {'hide_input', 'nbdev'} == test_nb.cells[1].metadata.keys() |
| 86 | +``` |
| 87 | + |
| 88 | +[`clean_nb`](https://nbdev.fast.ai/api/clean.html#clean_nb) also repairs |
| 89 | +structural problems by default (via `repair_nb`), so notebooks that |
| 90 | +Jupyter would reject, such as a markdown cell carrying an `outputs` |
| 91 | +attr, are fixed on every clean: |
| 92 | + |
| 93 | +``` python |
| 94 | +_nb = dict2nb(dict(cells=[dict(cell_type='markdown', source='hi', outputs=[], execution_count=1, id='m1', metadata={})], |
| 95 | + metadata=dict(kernelspec=dict(name='python3', display_name='Python 3')), nbformat=4, nbformat_minor=5)) |
| 96 | +clean_nb(_nb) |
| 97 | +assert 'outputs' not in _nb.cells[0] and 'execution_count' not in _nb.cells[0] |
| 98 | +validate_nb(_nb) |
| 99 | +``` |
| 100 | + |
| 101 | +We can preserve some additional keys at the notebook or cell levels: |
| 102 | + |
| 103 | +``` python |
| 104 | +test_nb = read_nb('../../tests/metadata.ipynb') |
| 105 | +clean_nb(test_nb, allowed_metadata_keys={'my_extra_key'}, allowed_cell_metadata_keys={'my_extra_cell_key'}) |
| 106 | + |
| 107 | +assert {'jekyll', 'kernelspec', 'nbdev', 'my_extra_key'} == test_nb.metadata.keys() |
| 108 | +assert {'hide_input', 'nbdev', 'my_extra_cell_key'} == test_nb.cells[1].metadata.keys() |
| 109 | +``` |
| 110 | + |
| 111 | +Passing `clear_all=True` removes everything from the cell metadata: |
| 112 | + |
| 113 | +``` python |
| 114 | +test_nb = read_nb('../../tests/metadata.ipynb') |
| 115 | +clean_nb(test_nb, clear_all=True) |
| 116 | + |
| 117 | +assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys() |
| 118 | +test_eq(test_nb.cells[1].metadata, {}) |
| 119 | +``` |
| 120 | + |
| 121 | +Passing `clean_ids=True` removes `id`s from plaintext repr outputs, to |
| 122 | +avoid notebooks whose contents change on each run since they often lead |
| 123 | +to git merge conflicts. For example: |
| 124 | + |
| 125 | + <PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x7FB4F8979690> |
| 126 | + |
| 127 | +becomes: |
| 128 | + |
| 129 | + <PIL.PngImagePlugin.PngImageFile image mode=L size=28x28> |
| 130 | + |
| 131 | +*Cell* IDs, on the other hand, are always added if missing |
| 132 | + |
| 133 | +``` python |
| 134 | +test_cell = {'source': 'x=1', 'cell_type': 'code', 'metadata': {}} |
| 135 | +_clean_cell(test_cell, False, set(), True, set()) |
| 136 | +test_cell['id'] |
| 137 | +``` |
| 138 | + |
| 139 | + '64e20756' |
| 140 | + |
| 141 | +------------------------------------------------------------------------ |
| 142 | + |
| 143 | +<a |
| 144 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L120" |
| 145 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 146 | + |
| 147 | +### process_write |
| 148 | + |
| 149 | +``` python |
| 150 | +def process_write( |
| 151 | + warn_msg, proc_nb, f_in, f_out:NoneType=None, disp:bool=False |
| 152 | +): |
| 153 | +``` |
| 154 | + |
| 155 | +*Call self as a function.* |
| 156 | + |
| 157 | +## Directive migrations |
| 158 | + |
| 159 | +Deliberate, opt-in rewrites for moving to the directive conventions |
| 160 | +introduced in v3.3: directives can live in cell metadata (an `nbdev` |
| 161 | +dict key), notebook-scope directives like `default_exp` can live in |
| 162 | +notebook metadata, and comment spellings have one canonical form. None |
| 163 | +of these run by default; they’re flags on `nbdev-clean` for one-off |
| 164 | +migration runs, so the git hook never triggers them. |
| 165 | + |
| 166 | +[`_to_meta`](https://nbdev.fast.ai/api/clean.html#_to_meta) and |
| 167 | +[`_to_comments`](https://nbdev.fast.ai/api/clean.html#_to_comments) move |
| 168 | +named directives between a cell’s comments and its `nbdev` metadata key, |
| 169 | +preserving values through the same mapping the parser uses (bare |
| 170 | +directives become `true`, and so on): |
| 171 | + |
| 172 | +``` python |
| 173 | +c = mk_cell('#| hide\n#| eval: false\n#| export: utils\n1+1') |
| 174 | +_to_meta(c, ['hide','eval']) |
| 175 | +test_eq(c.metadata['nbdev'], dict(hide='true', eval='false')) |
| 176 | +test_eq(c.source, '#| export: utils\n1+1') |
| 177 | + |
| 178 | +_to_comments(c, ['hide','eval']) |
| 179 | +assert 'nbdev' not in c.metadata |
| 180 | +test_eq(c.directives, {'export':'utils', 'hide':'', 'eval':'false'}) |
| 181 | +``` |
| 182 | + |
| 183 | +[`_canon_dirs`](https://nbdev.fast.ai/api/clean.html#_canon_dirs) |
| 184 | +respells each directive line canonically without touching anything else, |
| 185 | +and |
| 186 | +[`_hoist_nb_meta`](https://nbdev.fast.ai/api/clean.html#_hoist_nb_meta) |
| 187 | +moves `default_exp` to notebook metadata, deleting a first cell that |
| 188 | +held nothing but directives (a bare `#| hide` on an otherwise-empty cell |
| 189 | +hides nothing). |
| 190 | +[`_dir_moves`](https://nbdev.fast.ai/api/clean.html#_dir_moves) bundles |
| 191 | +the migrations for the CLI, converting the raw loaded dict to `NbCell`s |
| 192 | +first: |
| 193 | + |
| 194 | +``` python |
| 195 | +c = mk_cell('#| default_exp core\n#| eval:false\n1+1') |
| 196 | +_canon_dirs(c) |
| 197 | +test_eq(c.source, '#| default_exp: core\n#| eval: false\n1+1') |
| 198 | + |
| 199 | +_nb = dict2nb(dict(cells=[mk_cell('#| hide\n#| default_exp: core'), mk_cell('#| export\n1+1')], |
| 200 | + metadata={}, nbformat=4, nbformat_minor=5)) |
| 201 | +_hoist_nb_meta(_nb) |
| 202 | +test_eq(_nb.metadata['nbdev'], dict(default_exp='core')) |
| 203 | +test_eq(len(_nb.cells), 1) |
| 204 | +test_eq(_nb.cells[0].source, '#| export\n1+1') |
| 205 | +``` |
| 206 | + |
| 207 | +------------------------------------------------------------------------ |
| 208 | + |
| 209 | +<a |
| 210 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L203" |
| 211 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 212 | + |
| 213 | +### nbdev_clean |
| 214 | + |
| 215 | +``` python |
| 216 | +def nbdev_clean( |
| 217 | + fname:str=None, # A notebook name or glob to clean |
| 218 | + clear_all:bool=False, # Remove all cell metadata and cell outputs? |
| 219 | + disp:bool=False, # Print the cleaned outputs |
| 220 | + stdin:bool=False, # Read notebook from input stream |
| 221 | + repair:bool_arg=True, # Fix structural problems, e.g. stray outputs on non-code cells (see `repair_nb`)? |
| 222 | + dirs:bool=False, # Rewrite comment directives in canonical form? |
| 223 | + to_meta:str=None, # Space-separated directive names to move from comments to cell metadata |
| 224 | + to_comments:str=None, # Space-separated directive names to move from cell metadata to comments |
| 225 | + nb_meta:bool=False, # Move `default_exp` into notebook metadata? |
| 226 | +): |
| 227 | +``` |
| 228 | + |
| 229 | +*Clean all notebooks in `fname` to avoid merge conflicts* |
| 230 | + |
| 231 | +By default (`fname` left to `None`), all the notebooks in |
| 232 | +`config.nbs_path` are cleaned. You can opt in to fully clean the |
| 233 | +notebook by removing every bit of metadata and the cell outputs by |
| 234 | +passing `clear_all=True`. |
| 235 | + |
| 236 | +If you want to keep some keys in the main notebook metadata you can set |
| 237 | +`allowed_metadata_keys` in `[tool.nbdev]` in `pyproject.toml`. Similarly |
| 238 | +for cell level metadata use `allowed_cell_metadata_keys`, and for output |
| 239 | +metadata use `allowed_out_metadata_keys`. For example, to preserve both |
| 240 | +`k1` and `k2` at both the notebook and cell level add the following to |
| 241 | +`pyproject.toml`: |
| 242 | + |
| 243 | +``` toml |
| 244 | +[tool.nbdev] |
| 245 | +allowed_metadata_keys = ["k1", "k2"] |
| 246 | +allowed_cell_metadata_keys = ["k1", "k2"] |
| 247 | +allowed_out_metadata_keys = ["k1", "k2"] |
| 248 | +``` |
| 249 | + |
| 250 | +------------------------------------------------------------------------ |
| 251 | + |
| 252 | +<a |
| 253 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L223" |
| 254 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 255 | + |
| 256 | +### clean_jupyter |
| 257 | + |
| 258 | +``` python |
| 259 | +def clean_jupyter( |
| 260 | + path, model, **kwargs |
| 261 | +): |
| 262 | +``` |
| 263 | + |
| 264 | +*Clean Jupyter `model` pre save to `path`* |
| 265 | + |
| 266 | +This cleans notebooks on-save to avoid unnecessary merge conflicts. The |
| 267 | +easiest way to install it for both Jupyter Notebook and Lab is by |
| 268 | +running `nbdev-install-hooks`. It works by implementing a |
| 269 | +`pre_save_hook` from Jupyter’s [file save hook |
| 270 | +API](https://jupyter-server.readthedocs.io/en/latest/developers/savehooks.html). |
| 271 | + |
| 272 | +## Hooks |
| 273 | + |
| 274 | +------------------------------------------------------------------------ |
| 275 | + |
| 276 | +<a |
| 277 | +href="https://github.com/AnswerDotAI/nbdev/blob/main/nbdev/clean.py#L281" |
| 278 | +target="_blank" style="float:right; font-size:smaller">source</a> |
| 279 | + |
| 280 | +### nbdev_install_hooks |
| 281 | + |
| 282 | +``` python |
| 283 | +def nbdev_install_hooks( |
| 284 | + merge:bool_arg=True, # Install the notebook merge driver? |
| 285 | + diff:bool_arg=True, # Install the notebook diff driver? |
| 286 | + globally:bool_arg=False, # Define the drivers in `~/.gitconfig` and the global attributes file, instead of repo files? |
| 287 | +): |
| 288 | +``` |
| 289 | + |
| 290 | +*Install Jupyter and git hooks to automatically clean, trust, and fix |
| 291 | +merge conflicts in notebooks* |
| 292 | + |
| 293 | +See |
| 294 | +[`clean_jupyter`](https://nbdev.fast.ai/api/clean.html#clean_jupyter) |
| 295 | +and `nbdev-merge` for more about how each hook works. |
| 296 | + |
| 297 | +Both git drivers are registered under the name `jupyternotebook`, the |
| 298 | +same name nbdime uses. Repo installs define them in a repo-local |
| 299 | +`.gitconfig` (wired in via `include.path`) and activate them in the |
| 300 | +committed `.gitattributes`; `--globally` instead defines them in |
| 301 | +`~/.gitconfig` and activates them in the global attributes file |
| 302 | +(`core.attributesFile`, defaulting to `~/.config/git/attributes`). Since |
| 303 | +the attribute lines just name a driver, a committed `.gitattributes` |
| 304 | +means “use your preferred notebook driver”: whichever tool defined |
| 305 | +`jupyternotebook` last in a config git consults wins, so switching |
| 306 | +between nbdev and nbdime is just re-running either tool’s enable |
| 307 | +command. Global installs skip the repo-only `post-merge` trust hook. |
0 commit comments