Skip to content

BUG: Convert the array_like inputs of the IV, system and asset pricin… - #776

Merged
bashtage merged 15 commits into
mainfrom
lm71-bug2-input-types
Oct 12, 2026
Merged

bashtage merged 15 commits into
mainfrom
lm71-bug2-input-types

Conversation

@bashtage

Copy link
Copy Markdown
Owner

…g models

These parameters are documented as array_like, but a list or a tuple failed
with "AttributeError: 'list' object has no attribute 'ndim'", because IVData
and the code that uses it need ndim and shape. This affected dependent, exog,
endog, instruments and weights of IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and
OLS, and the exog and endog of predict; dependent, exog and weights of
AbsorbingLS, and the cat and cont of Interaction; the variables in the
equations of SUR, IV3SLS and IVSystemGMM, including weights, multivariate_ls
and multivariate_iv; and the portfolios and factors of TradedFactorModel,
LinearFactorModel and LinearFactorModelGMM.

Everything that is not a NumPy array, a pandas Series or DataFrame, an xarray
DataArray or an IVData is now converted with np.asarray when it enters IVData,
which is where all of these inputs are received. The raw input is still kept as
IVData.original, so that a list has no labels and is paired by position, like
an array.

Input that is not array_like now fails with an error that names the input:

  • Complex values raise a TypeError. They were converted to float64 with a
    ComplexWarning, which silently discards the imaginary part.
  • Text that cannot be converted to float64 raises a TypeError. NumPy raised a
    ValueError that did not say which input was the problem.
  • A scalar, a string or a dict raises a TypeError, not an AttributeError. For
    a string, which is what weights="colname" passes to from_formula, the message
    says to pass the values of the column.
  • A column of a pandas object that is not numeric, text or categorical raises
    a ValueError that names the input and the column.
  • The number of observations of an IVData that is passed to a model is now
    checked, as it is for other inputs. It was accepted and failed later.

The weights of the from_formula of the system models have to be a dictionary-like
object with a key for each equation. Other containers were ignored with a
warning that the weights of the equations were not found, and now raise a
TypeError.

The tests compare the numbers that are used in estimation, and the estimates,
for each container (list, tuple, 1-d and 2-d arrays, named and unnamed Series,
DataFrames, DataArrays and IVData) and dtype with a pandas float64 input.

Comment thread linearmodels/iv/data.py
import pandas as pd
from pandas.api.types import is_numeric_dtype, is_string_dtype

from linearmodels.shared.inputs import as_array_like, as_float64
Comment thread linearmodels/iv/model.py Fixed
)
from pandas.api.types import is_datetime64_any_dtype, is_numeric_dtype, is_string_dtype

from linearmodels.shared.inputs import as_float64, reject_complex
ValueError
If a nested sequence is ragged.
"""
from linearmodels.iv.data import IVData # noqa: PLC0415
If a nested sequence is ragged.
"""
from linearmodels.iv.data import IVData # noqa: PLC0415
from linearmodels.panel.data import PanelData # noqa: PLC0415
TypeError
If value is not array_like, or if it contains complex values.
"""
from linearmodels.iv.data import IVData # noqa: PLC0415
If value is not array_like, or if it contains complex values.
"""
from linearmodels.iv.data import IVData # noqa: PLC0415
from linearmodels.panel.data import PanelData # noqa: PLC0415
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.65714% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.61%. Comparing base (6983567) to head (99cbdb7).
⚠️ Report is 52 commits behind head on main.

Files with missing lines Patch % Lines
linearmodels/panel/data.py 80.00% 2 Missing ⚠️
linearmodels/shared/inputs.py 96.55% 1 Missing and 1 partial ⚠️
linearmodels/panel/model.py 98.82% 0 Missing and 1 partial ⚠️
linearmodels/tests/test_input_types.py 99.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #776      +/-   ##
==========================================
+ Coverage   99.60%   99.61%   +0.01%     
==========================================
  Files         131      139       +8     
  Lines       24664    27922    +3258     
  Branches     1823     1966     +143     
==========================================
+ Hits        24566    27815    +3249     
- Misses         39       41       +2     
- Partials       59       66       +7     
Flag Coverage Δ
adder 99.60% <99.65%> (+0.01%) ⬆️
subtractor 99.60% <99.65%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…g models

These parameters are documented as array_like, but a list or a tuple failed
with "AttributeError: 'list' object has no attribute 'ndim'", because IVData
and the code that uses it need ndim and shape. This affected dependent, exog,
endog, instruments and weights of IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and
OLS, and the exog and endog of predict; dependent, exog and weights of
AbsorbingLS, and the cat and cont of Interaction; the variables in the
equations of SUR, IV3SLS and IVSystemGMM, including weights, multivariate_ls
and multivariate_iv; and the portfolios and factors of TradedFactorModel,
LinearFactorModel and LinearFactorModelGMM.

Everything that is not a NumPy array, a pandas Series or DataFrame, an xarray
DataArray or an IVData is now converted with np.asarray when it enters IVData,
which is where all of these inputs are received. The raw input is still kept as
IVData.original, so that a list has no labels and is paired by position, like
an array.

Input that is not array_like now fails with an error that names the input:

- Complex values raise a TypeError. They were converted to float64 with a
  ComplexWarning, which silently discards the imaginary part.
- Text that cannot be converted to float64 raises a TypeError. NumPy raised a
  ValueError that did not say which input was the problem.
- A scalar, a string or a dict raises a TypeError, not an AttributeError. For
  a string, which is what weights="colname" passes to from_formula, the message
  says to pass the values of the column.
- A column of a pandas object that is not numeric, text or categorical raises
  a ValueError that names the input and the column.
- The number of observations of an IVData that is passed to a model is now
  checked, as it is for other inputs. It was accepted and failed later.

The weights of the from_formula of the system models have to be a dictionary-like
object with a key for each equation. Other containers were ignored with a
warning that the weights of the equations were not found, and now raise a
TypeError.

The tests compare the numbers that are used in estimation, and the estimates,
for each container (list, tuple, 1-d and 2-d arrays, named and unnamed Series,
DataFrames, DataArrays and IVData) and dtype with a pandas float64 input.
…olumns

IV2SLS, IVLIML and AbsorbingLS convert clusters with np.asarray (in
ClusteredCovariance), but the one-way clustered weight matrix of IVGMM, IVGMMCUE
and IVJIVE did not. It failed with "'list' object has no attribute 'shape'" for
a list or a tuple, also when the clusters are given in weight_config. All of the
estimators now convert clusters, and so a list, a tuple, a pandas object (with
numbers, text or categories), an xarray DataArray and an IVData give the same
results. The clusters in the cov_config of IVGMM and IVGMMCUE results are now an
ndarray, as they are in the other estimators.

The number of columns of clusters was not checked either (#259):

- IV2SLS, IVLIML and AbsorbingLS used the first two columns of clusters that
  have 3 or more columns, and ignored the others without a message.
- IVGMM and IVGMMCUE failed with an error from NumPy about the dimensions of
  the arrays to concatenate when clusters had two columns, since the weight
  matrix only supports one-way clustering.

A ValueError that names clusters and the supported number of columns is raised
instead: 1 or 2 columns for IV2SLS, IVLIML and AbsorbingLS, and a single column
for IVGMM, IVGMMCUE and IVJIVE. Complex clusters raise a TypeError.

The clusters of the system models use the same conversion, and so complex
values are rejected there too. The ValueError for clusters with an unsupported
shape was raised as ValueError(message, ValueError), which printed a tuple.

The tests fit the estimators with each container, with one-way and two-way
clusters, and check the errors.
…g values

IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and OLS scaled the weights to have mean 1
before the observations with missing values were dropped. The weights of the
observations that remain then have a mean other than 1 whenever the weights of
the dropped observations differ from the others. The estimates are the same
since they do not depend on the scale of the weights, but the weighted
residuals (wresids), resid_ss, s2 and total_ss were multiplied by the ratio of
the mean of the weights of the observations that are used to the mean of all of
the weights, which is not what the model that is estimated using only the
complete observations gives. In a simulation with 300 observations of which 3
had a missing value the ratio was 1.0023.

The weights are now normalized after the observations are dropped, so that the
results are those of a model that is fit to the complete observations.
AbsorbingLS and the system models already did this, and so the estimators agree.
Parameters, covariances, R-squared and the test statistics are unchanged. The
numbers change only if weights and missing values are used together.
The IV estimators and the panel models raise a ValueError if a weight is zero or
negative. AbsorbingLS did not check its weights: a zero weight was accepted
without a message, and a negative weight failed with "LinAlgError: SVD did not
converge", which does not say that the weights are the problem. AbsorbingLS now
raises ValueError("weights must be strictly positive.") like the other
estimators. A missing weight still marks the observation as missing.
…orical

AbsorbingLS treats the columns of the DataFrame absorb that do not have a
categorical dtype as continuous variables to absorb. A column of strings is one
of those. Text failed with "ValueError: Cannot cast str dtype to float64",
which does not say what to do. Strings that look like numbers were converted
and silently absorbed as a continuous variable, not as an effect.

Interaction.from_frame, and the absorb and interactions of AbsorbingLS, which
use it, now check the columns that are not categorical. A column that is not
numeric raises a TypeError that names the input and the column and says to
convert the column to categorical, with the astype("category") call that does
it. Interaction.from_frame raises a TypeError that names frame if it is not a
DataFrame. The type lines of absorb, interactions and weights in the AbsorbingLS
docstring follow the docstring conventions and say what is accepted.
…supported inputs

The weights, clusters and other_effects of the panel models are documented as
array_like, but a list or a tuple failed: weights with "AttributeError: 'list'
object has no attribute 'ndim'", clusters with "TypeError: clusters found in the
dictionary but it is not array-like", and other_effects with "TypeError: Only
ndarrays, DataFrames or DataArrays are supported". They are now converted with
np.asarray when they are received. All of the shapes of weights (a weight for
each observation, time period, entity, or time period and entity) work with a
list or a tuple.

The documentation says that clusters contain 1 or 2 variables, but a 1-d NumPy
array was rejected with "2 or 3-d array required for numpy input", since a 2-d
array is read as time by entity and a 3-d array as variable by time by entity.
A 1-d array, list or tuple of clusters, or of other_effects, now has a value
for each row of dependent, in the order of the rows of dependent, and a
ValueError that names the input is raised if the number of values is wrong.

dependent and exog need labels or a layout, which a list does not have. They
accept the containers that PanelData accepts, and they are documented as that
list: a PanelData, a DataFrame or Series with a 2-level MultiIndex (entity, then
time), a DataFrame with the time periods as the index and the entities as the
columns, and 2-d and 3-d NumPy arrays and DataArrays. Anything else, such as a
list, raises a TypeError as before, but the message now starts with the name of
the input. The same is true of the weights, clusters and other_effects, of the
exog of predict, and of the data of from_formula, for all of the errors that
PanelData raises.

Other input that cannot be used also fails with a message that names the input:

- Complex values raise a TypeError. They were converted to float64 with a
  ComplexWarning, which discards the imaginary part.
- Text that cannot be converted to float64 raises a TypeError.
- A column name as the weights of from_formula raises a TypeError that says to
  pass the values of the column. It was an AttributeError.

dummy_matrix accepts a list or a tuple of category codes. It raises a ValueError
if the codes do not have 2 dimensions, and a TypeError if a DataFrame has a
column that is not categorical.
The clusters of the panel models can be given as a NumPy array or a list, which
has no labels and is paired with the rows of dependent by position. The
PanelData that was made from it had generated labels instead (entity.0, ...),
and the code that follows uses these labels:

- Combining clusters from an array with cluster_entity=True or
  cluster_time=True gave wrong standard errors without an error. The entity or
  time clusters are added to the clusters by label, using the index of
  dependent, so they did not match the generated labels and were all missing
  (NaN).
- FirstDifferenceOLS failed with a KeyError that lists the index of dependent,
  since it looks up the first differences of the clusters by label.

The clusters that are not labelled now take the labels of the rows of dependent.
Labelled clusters are still compared with the index of dependent and have to
match it.
…renceOLS

The documentation of the clusters of the panel models says that they should be
integers, but that other types are coerced to integers by treating them as
categories. PooledOLS, PanelOLS and RandomEffects did this, but BetweenOLS and
FirstDifferenceOLS read the clusters as numbers, and failed with "ValueError:
could not convert string to float" for text or categorical clusters.

The clusters that are not numeric are now replaced by the codes of their
categories when they are reformatted, which every estimator does, so that the
estimators accept the same clusters.
RandomEffects subtracts the entity means of dependent and exog, multiplied by
theta, from each row, by looking up the entity of each row of dependent in the
index of the means of exog. If dependent and exog do not have the same labels the
lookup fails with a KeyError that lists the labels of dependent. This happens
when one of them is a NumPy array, which gets generated labels (entity.0, ...)
that do not match the labels of a pandas object. The rows of the inputs are
paired by position when they are not labelled, so this combination must work.

The means are now selected by position, in the order in which the entities
appear in dependent, which is the order of the means, and the estimates are
unchanged when the labels match.
…has none

The estimators pair the rows of their inputs by position, and reject an input
that carries labels when its index is not the index of dependent. If dependent
had no labels (a NumPy array, or a pandas object with a default index) nothing
was compared, so labelled inputs in a different order from each other were still
silently attached to the wrong observations:

    IV2SLS(y.to_numpy(), w, x, z.iloc[::-1])  # accepted, results are wrong

When dependent has no labels, the first input that has labels is now the
reference, and every input with labels after it must have the same index, with
the same ValueError as for dependent. The message names the input that sets the
labels ("The row index of instruments does not match exog."). Inputs without
labels are paired by position and only need the right shape, as before, and
nothing changes when dependent has labels.

- IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and OLS use the order exog, endog,
  instruments, weights. clusters, in fit and in the weight configuration of the
  GMM estimators, use the same reference, and are compared with the labels of
  the observations that are kept after the observations with missing values are
  dropped.
- AbsorbingLS uses the order exog, absorb, interactions, weights, then clusters.
- The systems keep the check that every dependent variable with labels uses the
  same index. An equation whose dependent variable has no labels is compared
  with the labels of the other equations, and if no dependent variable has
  labels, the first input with labels in the order of the equations (exog,
  endog, instruments and weights of each) is the reference, also for clusters.
- The panel models use the order dependent, exog, weights, other_effects, then
  clusters in fit. A labelled input of every supported shape of weights is
  compared, including the weights of each time period or entity, which are
  compared with the time periods or entities of the reference. Only inputs that
  have a label for each row can set the reference.
- The asset pricing models only have two inputs with rows (portfolios and
  factors), so there is nothing else to compare.

Labelled clusters are paired with the rows of dependent by position and so they
are now given the labels of these rows when dependent has other labels. This
fixes two problems when dependent is an array and clusters is labelled:
cluster_entity and cluster_time set the identifiers of the entities or time
periods to missing values, since these were aligned with the labels of the
clusters, which gave incorrect standard errors, and FirstDifferenceOLS raised a
KeyError.

The tests that encoded the previous behavior for the IV and system models
(inputs in a different order from each other were accepted if dependent had no
labels) are changed to check that inputs that agree with each other are accepted
and that inputs that disagree are rejected. There are tests for each family, for
the MultiIndex, datetime, string and integer index styles, missing values, IVData
and xarray inputs, and the unit tests of the helper functions.
…ssage

The ValueError for clusters with an unsupported shape was raised as
ValueError(message, ValueError), which printed a tuple that starts with the
message. It is now raised with the message only, since the change that
converts clusters with the other inputs. The test checks the three shapes that
raise (too many columns, too many dimensions and the wrong number of rows) for
SUR and IV3SLS, and that the error has a single argument. It fails without that
change.
…ovariance

The one-way clustered weight matrix of the GMM estimators converted clusters to
an array and checked its columns in two statements that needed two imports in
iv/gmm.py. The conversion and the check are now one function next to
check_cluster_columns in iv/covariance.py, which the weight matrix already
imports from. The behavior does not change.
…etup_clusters

The clusters that are passed to fit were converted with as_array_like and then
replaced by the result of get_panel_data_like in the same variable, which has a
different type and made mypy report an incompatible assignment. The converted
clusters now have their own name. The behavior does not change.
The python310_minimums job of #776 failed in
test_ragged_list_raises_and_names_the_input. NumPy 1.24 and later raise a
ValueError when np.asarray gets a ragged nested sequence, and as_array_like
turns it into "weights could not be converted to an array". Older NumPy
returns an object array of the rows with a VisibleDeprecationWarning, so the
error came from the conversion to float64 instead, as a TypeError that does
not say what is wrong.

as_array_like now treats an object array whose elements are sequences as
ragged, raises the same ValueError on every NumPy version, and silences the
warning of the conversion. test_ragged_list_as_object_array_raises imitates
the old NumPy behavior so that it runs everywhere; it fails without this
change.
@bashtage
bashtage force-pushed the lm71-bug2-input-types branch from 2d314c6 to c04cf1b Compare October 12, 2026 19:02
@bashtage
bashtage force-pushed the lm71-bug2-input-types branch from c04cf1b to 99cbdb7 Compare October 12, 2026 19:06
@bashtage
bashtage merged commit 4e81e9c into main Oct 12, 2026
23 of 24 checks passed
@bashtage
bashtage deleted the lm71-bug2-input-types branch October 12, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants