Skip to content

Time series feature extractor error with spectral can not reindex on an axis with duplicate labels #183

Description

@MillerQuintero2001

Hi, I was applying feature extraction in a time series dataframe, but I found this issue:

import tsfel
cfg = tsfel.get_features_by_domain(["statistical", "temporal", "spectral"])

X_train = tsfel.time_series_features_extractor(cfg, x_train_array, fs=1/86400)
X_test = tsfel.time_series_features_extractor(cfg, x_test_array, fs=1/86400)

But I got this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[10], line 4
      1 import tsfel
      2 cfg = tsfel.get_features_by_domain(["statistical", "temporal", "spectral"])
----> 4 X_train = tsfel.time_series_features_extractor(cfg, x_train_array, fs=1/86400)
      5 X_test = tsfel.time_series_features_extractor(cfg, x_test_array, fs=1/86400)

File ~/Documentos/Dev/Startup/poc-data-china-energy/.venv/lib/python3.13/site-packages/tsfel/feature_extraction/calc_features.py:390, in time_series_features_extractor(config, timeseries, fs, window_size, overlap, verbose, **kwargs)
    379     features_final = calc_window_features(
    380         config,
    381         timeseries,
   (...)    386         single_window=True,
    387     )
    389 # Assuring the same feature extraction order
--> 390 features_final = features_final.reindex(sorted(features_final.columns), axis=1)
    391 return features_final.reset_index(drop=True)

File ~/Documentos/Dev/Startup/poc-data-china-energy/.venv/lib/python3.13/site-packages/pandas/core/frame.py:6078, in DataFrame.reindex(self, labels, index, columns, axis, method, copy, level, fill_value, limit, tolerance)
   5843 def reindex(
   5844     self,
   5845     labels=None,
   (...)   5855     tolerance=None,
   5856 ) -> DataFrame:
   5857     """
   5858     Conform DataFrame to new index with optional filling logic.
   5859 
   (...)   6076     See the :ref:`user guide <basics.reindexing>` for more.
   6077     """
-> 6078     return super().reindex(
   6079         labels=labels,
   6080         index=index,
   6081         columns=columns,
   6082         axis=axis,
   6083         method=method,
   6084         level=level,
   6085         fill_value=fill_value,
   6086         limit=limit,
   6087         tolerance=tolerance,
   6088         copy=copy,
   6089     )

File ~/Documentos/Dev/Startup/poc-data-china-energy/.venv/lib/python3.13/site-packages/pandas/core/generic.py:5476, in NDFrame.reindex(self, labels, index, columns, axis, method, copy, level, fill_value, limit, tolerance)
   5473     return self._reindex_multi(axes, fill_value)
   5475 # perform the reindex on the axes
-> 5476 return self._reindex_axes(
   5477     axes, level, limit, tolerance, method, fill_value
   5478 ).__finalize__(self, method="reindex")

File ~/Documentos/Dev/Startup/poc-data-china-energy/.venv/lib/python3.13/site-packages/pandas/core/generic.py:5498, in NDFrame._reindex_axes(self, axes, level, limit, tolerance, method, fill_value)
   5495     continue
   5497 ax = self._get_axis(a)
-> 5498 new_index, indexer = ax.reindex(
   5499     labels, level=level, limit=limit, tolerance=tolerance, method=method
   5500 )
   5502 axis = self._get_axis_number(a)
   5503 obj = obj._reindex_with_indexers(
   5504     {axis: [new_index, indexer]},
   5505     fill_value=fill_value,
   5506     allow_dups=False,
   5507 )

File ~/Documentos/Dev/Startup/poc-data-china-energy/.venv/lib/python3.13/site-packages/pandas/core/indexes/base.py:4253, in Index.reindex(self, target, method, level, limit, tolerance)
   4250     raise ValueError("cannot handle a non-unique multi-index!")
   4251 elif not self.is_unique:
   4252     # GH#42568
-> 4253     raise ValueError("cannot reindex on an axis with duplicate labels")
   4254 else:
   4255     indexer, _ = self.get_indexer_non_unique(target)

ValueError: cannot reindex on an axis with duplicate labels

It seems like internally is creating a columns dataframe that already exists with that name, so, at the moment to apply reindex the error raises.

When I don't specify a fs (use the default that until I know is 100 Hz), this error doesn't raise.

Maybe it can't work with non integer sample frequency.

Steps to recreate

I was using a public dataset, so I don't have problem to give the exact data that I was using.

  1. Download .parquet file
  2. Read the file in a dataframe (pandas or polars)
  3. Use the following code:
# Series to list of numpy arrays
x_train_array = []
for array in x_train[:, 2:].to_numpy():
    x_train_array.append(array)
import tsfel
cfg = tsfel.get_features_by_domain(["statistical", "temporal", "spectral"])

X_train = tsfel.time_series_features_extractor(cfg, x_train_array, fs=1/86400)

I know that the problem is with spectral feature, cause a run an experiment individually with each feature:

Image

Data: https://drive.google.com/file/d/1sTXC_SCIWi-oKoYOC5QxBN9ZlFl_ha7W/view?usp=sharing
Version: tsfel == 0.2.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions