|
44 | 44 | - Dask dataframe containing the spike times of the postsynaptic cell for all trials. |
45 | 45 |
|
46 | 46 | |
47 | | -If you intialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :py:mod:`~data_base.isf_data_base.db_initializers.load_simrun_general.config`). |
| 47 | +If you initialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :py:mod:`~data_base.db_initializers.load_simrun_general.config`). |
48 | 48 | If ``rewrite_in_optimized_format=False`` instead, these keys are pickled dask dataframes, containing the instructions to build the dataframe, not the data itself. |
49 | | -This is useful for fast intermediate analysis, but strongly discouraged for long term storage, since these instructions contain absolute paths to the original data files, which invalidates once they are moved or deleted. |
50 | | -Individual keys can afterwards be set to permanent, self-contained and efficient dask dataframes by calling |
51 | | -:py:meth:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database |
52 | | -keys. |
| 49 | +
|
| 50 | +.. warning:: |
| 51 | +
|
| 52 | + Using ``pickle`` as a data format is useful for fast intermediate analysis, but **strongly discouraged** for long-term storage. |
| 53 | + These pickle files are instructions to build the data, not the data itself. |
| 54 | + The instructions contain absolute paths to the original data files, which invalidate once they are moved or deleted. |
| 55 | + They have a hard dependency on the loader functions, which invalidate if those are renamed or moved in the source code. |
| 56 | + |
| 57 | +
|
| 58 | +.. hint:: |
| 59 | +
|
| 60 | + Individual data keys can afterwards be re-optimized to other data formats by calling |
| 61 | + :py:meth:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database keys. |
| 62 | + This is useful to convert intermediate pickle data to long-term stsorage formats. |
| 63 | +
|
53 | 64 |
|
54 | 65 | Example:: |
55 | 66 |
|
|
93 | 104 | >>> db['spike_times'] |
94 | 105 | <spike times dataframe> |
95 | 106 | |
96 | | -If you intialize the database with ``rewrite_in_optimized_format=True`` (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see :mod:`~data_base.isf_data_base.db_initializers.load_simrun_general.config`). |
97 | | -If ``rewrite_in_optimized_format=False`` instead, these keys are pickled dask dataframes, containing the instructions to build the dataframe, not the data itself. |
98 | | -This is useful for fast intermediate analysis, but strongly discouraged for long term storage, since these instructions contain absolute paths to the original data files, which invalidates once they are moved or deleted. |
99 | | -Individual keys can afterwards be set to permanent, self-contained and efficient dask dataframes by calling |
100 | | -:func:`~data_base.db_initializers.load_simrun_general.load_simrun_general.optimize` on specific database |
101 | | -keys. |
102 | | -
|
103 | 107 | See also: |
104 | 108 | :ref:`simresult_dir_format` for more information on the raw output format of :mod:`simrun`. |
105 | 109 |
|
|
0 commit comments