Skip to content

_prepare_returns cache key collision changes Series/DataFrame type and column metadata #543

Description

@bf2363

_prepare_returns cache key collision changes Series/DataFrame type and column metadata

Description

utils._prepare_returns() can return a different pandas container type from the input because its cache key does not distinguish a Series from an equivalent single-column DataFrame.

The result depends on call order:

  • If the DataFrame is prepared first, preparing the equivalent Series returns a DataFrame.
  • If the Series is prepared first, preparing the equivalent DataFrame returns a Series.

DataFrame column labels are also missing from the key, so two DataFrames with identical values but different column names share a cache entry and the second result receives the first DataFrame's columns.

This appears to be a possible root cause of the downstream HTML report error reported in #383:

Index(...) must be called with a collection of some kind, 'Strategy' was passed

Minimal reproduction

import pandas as pd
import quantstats as qs

index = pd.date_range("2024-01-01", periods=3)
series = pd.Series([0.01, -0.02, 0.03], index=index, name="Strategy")
frame = series.to_frame()

qs.utils._PREPARE_RETURNS_CACHE.clear()

prepared_frame = qs.utils._prepare_returns(frame)
prepared_series = qs.utils._prepare_returns(series)

print(type(prepared_frame).__name__)
print(type(prepared_series).__name__)
print(
    qs.utils._generate_cache_key(frame, 0.0, None)
    == qs.utils._generate_cache_key(series, 0.0, None)
)

Actual output

DataFrame
DataFrame
True

Reversing the two _prepare_returns() calls produces:

Series
Series

Column metadata is affected as well:

frame_a = pd.DataFrame({"A": [0.01, -0.02, 0.03]}, index=index)
frame_b = pd.DataFrame({"B": [0.01, -0.02, 0.03]}, index=index)

qs.utils._PREPARE_RETURNS_CACHE.clear()
qs.utils._prepare_returns(frame_a)

print(qs.utils._prepare_returns(frame_b).columns.tolist())

Actual output:

['A']

Expected output:

['B']

Expected behavior

_prepare_returns() should preserve the input container type and metadata independently of earlier calls:

  • Series input returns a Series.
  • DataFrame input returns a DataFrame.
  • DataFrame column labels are preserved.

Root cause

_generate_cache_key() uses the same expression for Series and DataFrame inputs:

data_hash = pd.util.hash_pandas_object(data).sum()

For a Series and an equivalent single-column DataFrame, this produces the same value. The generated key contains only data_hash, rf, and nperiods; it omits the pandas container type and column/name metadata.

Suggested fix

Include the following in the cache key:

  • Container type.
  • Series name for Series inputs.
  • Column labels for DataFrame inputs.
  • Potentially dtype metadata if it affects the prepared result.

Regression tests should call _prepare_returns() in both orders and verify that type and column/name metadata are preserved.

Environment

  • QuantStats: 0.0.81
  • pandas: 2.3.3
  • Python: 3.12.12

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions