Skip to content

Commit 02066c8

Browse files
authored
Docs (#523)
Improve clarity on the `pickle` format during `load_simrun_general.init`. See #522
2 parents fc76e61 + 642b946 commit 02066c8

1 file changed

Lines changed: 16 additions & 12 deletions

File tree

  • data_base/db_initializers/load_simrun_general

data_base/db_initializers/load_simrun_general/__init__.py

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,23 @@
4444
- Dask dataframe containing the spike times of the postsynaptic cell for all trials.
4545
4646
47-
If you intialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :py:mod:`~data_base.isf_data_base.db_initializers.load_simrun_general.config`).
47+
If you initialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :py:mod:`~data_base.db_initializers.load_simrun_general.config`).
4848
If ``rewrite_in_optimized_format=False`` instead, these keys are pickled dask dataframes, containing the instructions to build the dataframe, not the data itself.
49-
This is useful for fast intermediate analysis, but strongly discouraged for long term storage, since these instructions contain absolute paths to the original data files, which invalidates once they are moved or deleted.
50-
Individual keys can afterwards be set to permanent, self-contained and efficient dask dataframes by calling
51-
:py:meth:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database
52-
keys.
49+
50+
.. warning::
51+
52+
Using ``pickle`` as a data format is useful for fast intermediate analysis, but **strongly discouraged** for long-term storage.
53+
These pickle files are instructions to build the data, not the data itself.
54+
The instructions contain absolute paths to the original data files, which invalidate once they are moved or deleted.
55+
They have a hard dependency on the loader functions, which invalidate if those are renamed or moved in the source code.
56+
57+
58+
.. hint::
59+
60+
Individual data keys can afterwards be re-optimized to other data formats by calling
61+
:py:meth:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database keys.
62+
This is useful to convert intermediate pickle data to long-term stsorage formats.
63+
5364
5465
Example::
5566
@@ -93,13 +104,6 @@
93104
>>> db['spike_times']
94105
<spike times dataframe>
95106
96-
If you intialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :mod:`~data_base.isf_data_base.db_initializers.load_simrun_general.config`).
97-
If ``rewrite_in_optimized_format=False`` instead, these keys are pickled dask dataframes, containing the instructions to build the dataframe, not the data itself.
98-
This is useful for fast intermediate analysis, but strongly discouraged for long term storage, since these instructions contain absolute paths to the original data files, which invalidates once they are moved or deleted.
99-
Individual keys can afterwards be set to permanent, self-contained and efficient dask dataframes by calling
100-
:func:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database
101-
keys.
102-
103107
See also:
104108
:ref:`simresult_dir_format` for more information on the raw output format of :mod:`simrun`.
105109

0 commit comments

Comments
 (0)