Repository navigation
BUG: Convert the array_like inputs of the IV, system and asset pricin… - #776
Merged
Merged
Conversation
| import pandas as pd | ||
| from pandas.api.types import is_numeric_dtype, is_string_dtype | ||
|
|
||
| from linearmodels.shared.inputs import as_array_like, as_float64 |
| ) | ||
| from pandas.api.types import is_datetime64_any_dtype, is_numeric_dtype, is_string_dtype | ||
|
|
||
| from linearmodels.shared.inputs import as_float64, reject_complex |
| ValueError | ||
| If a nested sequence is ragged. | ||
| """ | ||
| from linearmodels.iv.data import IVData # noqa: PLC0415 |
| If a nested sequence is ragged. | ||
| """ | ||
| from linearmodels.iv.data import IVData # noqa: PLC0415 | ||
| from linearmodels.panel.data import PanelData # noqa: PLC0415 |
| TypeError | ||
| If value is not array_like, or if it contains complex values. | ||
| """ | ||
| from linearmodels.iv.data import IVData # noqa: PLC0415 |
| If value is not array_like, or if it contains complex values. | ||
| """ | ||
| from linearmodels.iv.data import IVData # noqa: PLC0415 | ||
| from linearmodels.panel.data import PanelData # noqa: PLC0415 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #776 +/- ##
==========================================
+ Coverage 99.60% 99.61% +0.01%
==========================================
Files 131 139 +8
Lines 24664 27922 +3258
Branches 1823 1966 +143
==========================================
+ Hits 24566 27815 +3249
- Misses 39 41 +2
- Partials 59 66 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…g models These parameters are documented as array_like, but a list or a tuple failed with "AttributeError: 'list' object has no attribute 'ndim'", because IVData and the code that uses it need ndim and shape. This affected dependent, exog, endog, instruments and weights of IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and OLS, and the exog and endog of predict; dependent, exog and weights of AbsorbingLS, and the cat and cont of Interaction; the variables in the equations of SUR, IV3SLS and IVSystemGMM, including weights, multivariate_ls and multivariate_iv; and the portfolios and factors of TradedFactorModel, LinearFactorModel and LinearFactorModelGMM. Everything that is not a NumPy array, a pandas Series or DataFrame, an xarray DataArray or an IVData is now converted with np.asarray when it enters IVData, which is where all of these inputs are received. The raw input is still kept as IVData.original, so that a list has no labels and is paired by position, like an array. Input that is not array_like now fails with an error that names the input: - Complex values raise a TypeError. They were converted to float64 with a ComplexWarning, which silently discards the imaginary part. - Text that cannot be converted to float64 raises a TypeError. NumPy raised a ValueError that did not say which input was the problem. - A scalar, a string or a dict raises a TypeError, not an AttributeError. For a string, which is what weights="colname" passes to from_formula, the message says to pass the values of the column. - A column of a pandas object that is not numeric, text or categorical raises a ValueError that names the input and the column. - The number of observations of an IVData that is passed to a model is now checked, as it is for other inputs. It was accepted and failed later. The weights of the from_formula of the system models have to be a dictionary-like object with a key for each equation. Other containers were ignored with a warning that the weights of the equations were not found, and now raise a TypeError. The tests compare the numbers that are used in estimation, and the estimates, for each container (list, tuple, 1-d and 2-d arrays, named and unnamed Series, DataFrames, DataArrays and IVData) and dtype with a pandas float64 input.
…olumns IV2SLS, IVLIML and AbsorbingLS convert clusters with np.asarray (in ClusteredCovariance), but the one-way clustered weight matrix of IVGMM, IVGMMCUE and IVJIVE did not. It failed with "'list' object has no attribute 'shape'" for a list or a tuple, also when the clusters are given in weight_config. All of the estimators now convert clusters, and so a list, a tuple, a pandas object (with numbers, text or categories), an xarray DataArray and an IVData give the same results. The clusters in the cov_config of IVGMM and IVGMMCUE results are now an ndarray, as they are in the other estimators. The number of columns of clusters was not checked either (#259): - IV2SLS, IVLIML and AbsorbingLS used the first two columns of clusters that have 3 or more columns, and ignored the others without a message. - IVGMM and IVGMMCUE failed with an error from NumPy about the dimensions of the arrays to concatenate when clusters had two columns, since the weight matrix only supports one-way clustering. A ValueError that names clusters and the supported number of columns is raised instead: 1 or 2 columns for IV2SLS, IVLIML and AbsorbingLS, and a single column for IVGMM, IVGMMCUE and IVJIVE. Complex clusters raise a TypeError. The clusters of the system models use the same conversion, and so complex values are rejected there too. The ValueError for clusters with an unsupported shape was raised as ValueError(message, ValueError), which printed a tuple. The tests fit the estimators with each container, with one-way and two-way clusters, and check the errors.
…g values IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and OLS scaled the weights to have mean 1 before the observations with missing values were dropped. The weights of the observations that remain then have a mean other than 1 whenever the weights of the dropped observations differ from the others. The estimates are the same since they do not depend on the scale of the weights, but the weighted residuals (wresids), resid_ss, s2 and total_ss were multiplied by the ratio of the mean of the weights of the observations that are used to the mean of all of the weights, which is not what the model that is estimated using only the complete observations gives. In a simulation with 300 observations of which 3 had a missing value the ratio was 1.0023. The weights are now normalized after the observations are dropped, so that the results are those of a model that is fit to the complete observations. AbsorbingLS and the system models already did this, and so the estimators agree. Parameters, covariances, R-squared and the test statistics are unchanged. The numbers change only if weights and missing values are used together.
The IV estimators and the panel models raise a ValueError if a weight is zero or
negative. AbsorbingLS did not check its weights: a zero weight was accepted
without a message, and a negative weight failed with "LinAlgError: SVD did not
converge", which does not say that the weights are the problem. AbsorbingLS now
raises ValueError("weights must be strictly positive.") like the other
estimators. A missing weight still marks the observation as missing.
…orical
AbsorbingLS treats the columns of the DataFrame absorb that do not have a
categorical dtype as continuous variables to absorb. A column of strings is one
of those. Text failed with "ValueError: Cannot cast str dtype to float64",
which does not say what to do. Strings that look like numbers were converted
and silently absorbed as a continuous variable, not as an effect.
Interaction.from_frame, and the absorb and interactions of AbsorbingLS, which
use it, now check the columns that are not categorical. A column that is not
numeric raises a TypeError that names the input and the column and says to
convert the column to categorical, with the astype("category") call that does
it. Interaction.from_frame raises a TypeError that names frame if it is not a
DataFrame. The type lines of absorb, interactions and weights in the AbsorbingLS
docstring follow the docstring conventions and say what is accepted.
…supported inputs The weights, clusters and other_effects of the panel models are documented as array_like, but a list or a tuple failed: weights with "AttributeError: 'list' object has no attribute 'ndim'", clusters with "TypeError: clusters found in the dictionary but it is not array-like", and other_effects with "TypeError: Only ndarrays, DataFrames or DataArrays are supported". They are now converted with np.asarray when they are received. All of the shapes of weights (a weight for each observation, time period, entity, or time period and entity) work with a list or a tuple. The documentation says that clusters contain 1 or 2 variables, but a 1-d NumPy array was rejected with "2 or 3-d array required for numpy input", since a 2-d array is read as time by entity and a 3-d array as variable by time by entity. A 1-d array, list or tuple of clusters, or of other_effects, now has a value for each row of dependent, in the order of the rows of dependent, and a ValueError that names the input is raised if the number of values is wrong. dependent and exog need labels or a layout, which a list does not have. They accept the containers that PanelData accepts, and they are documented as that list: a PanelData, a DataFrame or Series with a 2-level MultiIndex (entity, then time), a DataFrame with the time periods as the index and the entities as the columns, and 2-d and 3-d NumPy arrays and DataArrays. Anything else, such as a list, raises a TypeError as before, but the message now starts with the name of the input. The same is true of the weights, clusters and other_effects, of the exog of predict, and of the data of from_formula, for all of the errors that PanelData raises. Other input that cannot be used also fails with a message that names the input: - Complex values raise a TypeError. They were converted to float64 with a ComplexWarning, which discards the imaginary part. - Text that cannot be converted to float64 raises a TypeError. - A column name as the weights of from_formula raises a TypeError that says to pass the values of the column. It was an AttributeError. dummy_matrix accepts a list or a tuple of category codes. It raises a ValueError if the codes do not have 2 dimensions, and a TypeError if a DataFrame has a column that is not categorical.
The clusters of the panel models can be given as a NumPy array or a list, which has no labels and is paired with the rows of dependent by position. The PanelData that was made from it had generated labels instead (entity.0, ...), and the code that follows uses these labels: - Combining clusters from an array with cluster_entity=True or cluster_time=True gave wrong standard errors without an error. The entity or time clusters are added to the clusters by label, using the index of dependent, so they did not match the generated labels and were all missing (NaN). - FirstDifferenceOLS failed with a KeyError that lists the index of dependent, since it looks up the first differences of the clusters by label. The clusters that are not labelled now take the labels of the rows of dependent. Labelled clusters are still compared with the index of dependent and have to match it.
…renceOLS The documentation of the clusters of the panel models says that they should be integers, but that other types are coerced to integers by treating them as categories. PooledOLS, PanelOLS and RandomEffects did this, but BetweenOLS and FirstDifferenceOLS read the clusters as numbers, and failed with "ValueError: could not convert string to float" for text or categorical clusters. The clusters that are not numeric are now replaced by the codes of their categories when they are reformatted, which every estimator does, so that the estimators accept the same clusters.
RandomEffects subtracts the entity means of dependent and exog, multiplied by theta, from each row, by looking up the entity of each row of dependent in the index of the means of exog. If dependent and exog do not have the same labels the lookup fails with a KeyError that lists the labels of dependent. This happens when one of them is a NumPy array, which gets generated labels (entity.0, ...) that do not match the labels of a pandas object. The rows of the inputs are paired by position when they are not labelled, so this combination must work. The means are now selected by position, in the order in which the entities appear in dependent, which is the order of the means, and the estimates are unchanged when the labels match.
…has none
The estimators pair the rows of their inputs by position, and reject an input
that carries labels when its index is not the index of dependent. If dependent
had no labels (a NumPy array, or a pandas object with a default index) nothing
was compared, so labelled inputs in a different order from each other were still
silently attached to the wrong observations:
IV2SLS(y.to_numpy(), w, x, z.iloc[::-1]) # accepted, results are wrong
When dependent has no labels, the first input that has labels is now the
reference, and every input with labels after it must have the same index, with
the same ValueError as for dependent. The message names the input that sets the
labels ("The row index of instruments does not match exog."). Inputs without
labels are paired by position and only need the right shape, as before, and
nothing changes when dependent has labels.
- IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and OLS use the order exog, endog,
instruments, weights. clusters, in fit and in the weight configuration of the
GMM estimators, use the same reference, and are compared with the labels of
the observations that are kept after the observations with missing values are
dropped.
- AbsorbingLS uses the order exog, absorb, interactions, weights, then clusters.
- The systems keep the check that every dependent variable with labels uses the
same index. An equation whose dependent variable has no labels is compared
with the labels of the other equations, and if no dependent variable has
labels, the first input with labels in the order of the equations (exog,
endog, instruments and weights of each) is the reference, also for clusters.
- The panel models use the order dependent, exog, weights, other_effects, then
clusters in fit. A labelled input of every supported shape of weights is
compared, including the weights of each time period or entity, which are
compared with the time periods or entities of the reference. Only inputs that
have a label for each row can set the reference.
- The asset pricing models only have two inputs with rows (portfolios and
factors), so there is nothing else to compare.
Labelled clusters are paired with the rows of dependent by position and so they
are now given the labels of these rows when dependent has other labels. This
fixes two problems when dependent is an array and clusters is labelled:
cluster_entity and cluster_time set the identifiers of the entities or time
periods to missing values, since these were aligned with the labels of the
clusters, which gave incorrect standard errors, and FirstDifferenceOLS raised a
KeyError.
The tests that encoded the previous behavior for the IV and system models
(inputs in a different order from each other were accepted if dependent had no
labels) are changed to check that inputs that agree with each other are accepted
and that inputs that disagree are rejected. There are tests for each family, for
the MultiIndex, datetime, string and integer index styles, missing values, IVData
and xarray inputs, and the unit tests of the helper functions.
…ssage The ValueError for clusters with an unsupported shape was raised as ValueError(message, ValueError), which printed a tuple that starts with the message. It is now raised with the message only, since the change that converts clusters with the other inputs. The test checks the three shapes that raise (too many columns, too many dimensions and the wrong number of rows) for SUR and IV3SLS, and that the error has a single argument. It fails without that change.
…ovariance The one-way clustered weight matrix of the GMM estimators converted clusters to an array and checked its columns in two statements that needed two imports in iv/gmm.py. The conversion and the check are now one function next to check_cluster_columns in iv/covariance.py, which the weight matrix already imports from. The behavior does not change.
…etup_clusters The clusters that are passed to fit were converted with as_array_like and then replaced by the result of get_panel_data_like in the same variable, which has a different type and made mypy report an incompatible assignment. The converted clusters now have their own name. The behavior does not change.
The python310_minimums job of #776 failed in test_ragged_list_raises_and_names_the_input. NumPy 1.24 and later raise a ValueError when np.asarray gets a ragged nested sequence, and as_array_like turns it into "weights could not be converted to an array". Older NumPy returns an object array of the rows with a VisibleDeprecationWarning, so the error came from the conversion to float64 instead, as a TypeError that does not say what is wrong. as_array_like now treats an object array whose elements are sequences as ragged, raises the same ValueError on every NumPy version, and silences the warning of the conversion. test_ragged_list_as_object_array_raises imitates the old NumPy behavior so that it runs everywhere; it fails without this change.
bashtage
force-pushed
the
lm71-bug2-input-types
branch
from
October 12, 2026 19:02
2d314c6 to
c04cf1b
Compare
bashtage
force-pushed
the
lm71-bug2-input-types
branch
from
October 12, 2026 19:06
c04cf1b to
99cbdb7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…g models
These parameters are documented as array_like, but a list or a tuple failed
with "AttributeError: 'list' object has no attribute 'ndim'", because IVData
and the code that uses it need ndim and shape. This affected dependent, exog,
endog, instruments and weights of IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE and
OLS, and the exog and endog of predict; dependent, exog and weights of
AbsorbingLS, and the cat and cont of Interaction; the variables in the
equations of SUR, IV3SLS and IVSystemGMM, including weights, multivariate_ls
and multivariate_iv; and the portfolios and factors of TradedFactorModel,
LinearFactorModel and LinearFactorModelGMM.
Everything that is not a NumPy array, a pandas Series or DataFrame, an xarray
DataArray or an IVData is now converted with np.asarray when it enters IVData,
which is where all of these inputs are received. The raw input is still kept as
IVData.original, so that a list has no labels and is paired by position, like
an array.
Input that is not array_like now fails with an error that names the input:
ComplexWarning, which silently discards the imaginary part.
ValueError that did not say which input was the problem.
a string, which is what weights="colname" passes to from_formula, the message
says to pass the values of the column.
a ValueError that names the input and the column.
checked, as it is for other inputs. It was accepted and failed later.
The weights of the from_formula of the system models have to be a dictionary-like
object with a key for each equation. Other containers were ignored with a
warning that the weights of the equations were not found, and now raise a
TypeError.
The tests compare the numbers that are used in estimation, and the estimates,
for each container (list, tuple, 1-d and 2-d arrays, named and unnamed Series,
DataFrames, DataArrays and IVData) and dtype with a pandas float64 input.