Type endpoint parameters and bind them into project DB SQL

corehq/apps/case_search/CASE_SEARCH_ENDPOINTS.rst

@@ -18,7 +18,8 @@ Backend
 - ``endpoint_capability.py`` — domain capability metadata (case types, fields,
   operators, input schemas); drives both UI and query validation
 - ``endpoint_query_spec.py`` — query AST (``GroupNode``, ``ComponentNode``),
-  parameter spec (``Parameter``, ``ParameterInput``), and validation logic
+  parameter spec (``Parameter``, ``ParameterInput``), validation logic, and
+  the SQL parameter binding (``sql_placeholders``, ``bind_values``)
 - ``endpoint_views.py`` — Django views wired to the models
 - ``utils.py`` — ``CaseSearchEndpointQueryBuilder``: compiles the validated
   AST and parameter values into an ES query
@@ -57,17 +58,33 @@ This feature is gated behind the ``CASE_SEARCH_ENDPOINTS`` static toggle
 Parameters
 ----------
 
-Endpoints can declare named, typed parameters (``text``, ``number``, ``date``,
-``geopoint``). Parameters are stored as a JSON array on the
-``CaseSearchEndpointVersion`` and validated against ``FIELD_TYPES`` from
-``endpoint_capability``.
+Endpoints of both kinds declare named, typed parameters, stored as a JSON
+array on the ``CaseSearchEndpointVersion`` and validated against
+``PARAMETER_TYPES`` from ``endpoint_capability``. That is the field types
+(``text``, ``number``, ``date``, ``select``, ``geopoint``) plus ``daterange``,
+which is parameter-only: no case property has that type, so it has no
+operations and cannot be referenced from an Elasticsearch query spec.
 
 In the query spec, condition inputs can reference a parameter by name via a
 ``ParameterInput`` node (``{"type": "parameter", "value": "param_name"}``).
 At query execution time, ``CaseSearchEndpointQueryBuilder`` resolves each
 ``ParameterInput`` against the supplied criteria values before building the ES
 filter.
 
+Project DB endpoints bind parameters into their SQL instead.
+``sql_placeholders`` gives the placeholder names a spec implies — a
+``daterange`` named ``dob`` becomes ``:dob_from`` and ``:dob_to``, every other
+type keeps its own name — and ``bind_values`` maps a request's search criteria
+onto the values those placeholders take:
+
+- An absent or blank criterion binds as ``None``. NULL coerces to any column
+  type, so endpoint SQL guards each parameter with ``(:p IS NULL OR ...)``;
+  an empty string would fail against a numeric or date column.
+- A ``select`` parameter always binds as a list, however many values were
+  searched for, so that the SQL comparing it against an array column works
+  whatever the searcher chose.
+- Multiple values for a scalar parameter are a ``CaseSearchUserError``.
+
 Query Builder
 -------------
 
@@ -77,6 +94,29 @@ condition row triggers an HTMX fetch to ``condition_row.html``, which renders
 the appropriate operator/input controls for the selected field type. Condition
 inputs can be set to a literal value or bound to a declared parameter.
 
+Project DB Endpoints
+--------------------
+
+An endpoint's ``target_type`` selects its backend. A ``project_db`` endpoint
+stores SQL in ``dangerous_sql`` instead of a query spec, and runs it through
+``corehq.apps.project_db.user_sql``, which translates a restricted subset of
+SQL into SQLAlchemy Core. Rows are mapped back to ``CommCareCase`` objects by
+``_rows_to_cases``, so both kinds of endpoint return the same thing.
+
+The SQL is validated when the endpoint is saved (``sql_parameter_errors``),
+which reports at save time what would otherwise fail at run time:
+
+- a placeholder the parameter spec does not declare, or a declared parameter
+  the SQL never uses — ``UserSQL.run`` rejects a mismatched value set
+- a parameter used with ``IN``, whose unsupplied form renders nothing at all
+  and raises out of psycopg2
+- a ``select`` parameter not compared against a ``select_prop__`` array
+  column, or a scalar parameter that is
+
+List comparisons therefore go through the ``select_prop__`` columns, using
+``&&`` (any of) or ``@>`` (all of). Multi-value search over a plain text
+property is deliberately not expressible.
+
 Query Tester
 ------------
 

corehq/apps/case_search/endpoint_capability.py

@@ -20,6 +20,10 @@
 FIELD_TYPE_DATETIME = 'datetime'
 FIELD_TYPE_SELECT = 'select'
 FIELD_TYPE_GEOPOINT = 'geopoint'
+# Parameter-only: never the type of a case property, so it has no operations
+# and cannot be used in an Elasticsearch query builder spec. A project DB
+# endpoint binds it as two SQL placeholders -- see ``sql_placeholders``.
+FIELD_TYPE_DATERANGE = 'daterange'
 
 # DataType -> field type mapping
 _DATA_TYPE_MAP = {
@@ -85,6 +89,10 @@
 
 FIELD_TYPES = _OPERATOR_BY_TYPE.keys()
 
+# Types a parameter may declare. A superset of the field types, since a
+# parameter need not correspond to a case property.
+PARAMETER_TYPES = (*FIELD_TYPES, FIELD_TYPE_DATERANGE)
+
 # Sentinel input-slot type: the slot has no fixed type of its own and instead
 # takes the type of the field the condition is applied to. Used by operators
 # shared across field types (e.g. lt/gt work on both numbers and dates), where

corehq/apps/case_search/endpoint_query_spec.py

@@ -4,6 +4,11 @@
 attrs nodes. :func:`parse_filter_spec` validates a spec against capability
 metadata and, when valid, returns the typed tree a query builder can consume.
 
+Endpoint parameters are shared by both kinds of endpoint. Project DB
+endpoints additionally bind them into SQL: :func:`sql_placeholders` gives the
+placeholder names a spec implies, and :func:`bind_values` maps a request's
+search criteria onto the values those placeholders take.
+
 The nodes follow the ``type``/``to_json``/``from_json`` convention used by
 :mod:`corehq.apps.app_execution.data_model`, so they round-trip to and from
 the stored JSON. (If a node tree is ever persisted via a model field, add
@@ -13,14 +18,19 @@
 
 from typing import ClassVar
 
+from django.utils.translation import gettext as _
+
 from attr import Factory, define, field as attr_field, validators
 
 from corehq.apps.case_search.endpoint_capability import (
-    FIELD_TYPES,
+    FIELD_TYPE_DATERANGE,
+    FIELD_TYPE_SELECT,
     INPUT_TYPE_CHOICE,
     INPUT_TYPE_MATCH_FIELD,
     OPERATORS,
+    PARAMETER_TYPES,
 )
+from corehq.apps.case_search.exceptions import CaseSearchUserError
 
 # Group node types: all = AND, any = OR, none = NOR (no child matches).
 GROUP_TYPES = ('all', 'any', 'none')
@@ -32,10 +42,13 @@
 # Maximum total nodes across the entire query tree.
 MAX_TOTAL_NODES = 200
 
+# How a date range criterion arrives: __range__YYYY-MM-DD__YYYY-MM-DD
+DATE_RANGE_PREFIX = '__range__'
+
 @define
 class Parameter:
     name: str = attr_field(converter=str.strip, validator=validators.min_len(1))
-    type: str = attr_field(validator=validators.in_(FIELD_TYPES))
+    type: str = attr_field(validator=validators.in_(PARAMETER_TYPES))
 
 def parse_parameter_spec(spec):
     """Validate a parameter list spec and parse it.
@@ -64,21 +77,116 @@ def parse_parameter_spec(spec):
             seen_names.add(name)
 
         param_type = item.get('type', '')
-        if param_type not in FIELD_TYPES:
+        if param_type not in PARAMETER_TYPES:
             item_errors.append(
                 f"Parameter '{name or i}': invalid type '{param_type}'."
-                f" Must be one of: {', '.join(FIELD_TYPES)}"
+                f" Must be one of: {', '.join(PARAMETER_TYPES)}"
             )
 
         if item_errors:
             errors.extend(item_errors)
         else:
             parameters.append(Parameter(name=name, type=param_type))
 
+    errors.extend(_duplicate_placeholder_errors(parameters))
     if errors:
         return None, errors
     return parameters, []
 
+
+def _duplicate_placeholder_errors(parameters):
+    """A daterange derives two placeholder names, which may collide with
+    another parameter's (``dob`` as a daterange and a ``dob_from`` text
+    parameter both want ``:dob_from``)."""
+    seen = set()
+    for name in sql_placeholders(parameters):
+        if name in seen:
+            yield f"Duplicate SQL parameter name: '{name}'"
+        seen.add(name)
+
+
+def sql_placeholders(parameters):
+    """The SQL placeholder names a parameter spec implies.
+
+    A ``daterange`` parameter named ``dob`` is bound as ``:dob_from`` and
+    ``:dob_to``; every other type is bound under its own name.
+    """
+    return [name for param in parameters for name in placeholders_for(param)]
+
+
+def placeholders_for(param):
+    """The SQL placeholder names a single parameter is bound to."""
+    if param.type == FIELD_TYPE_DATERANGE:
+        return [f'{param.name}_from', f'{param.name}_to']
+    return [param.name]
+
+
+def bind_values(parameters, criteria):
+    """Map search criteria onto the values ``UserSQL.run`` expects.
+
+    A criterion that is absent or blank binds as ``None``: NULL coerces to any
+    column type, so endpoint SQL can guard every parameter with
+    ``(:p IS NULL OR ...)``.
+
+    :raises CaseSearchUserError: when a criterion's shape does not match the
+        type its parameter declares.
+    """
+    by_key = {c.key: c for c in criteria}
+    values = {}
+    for param in parameters:
+        values.update(_bind_parameter(param, by_key.get(param.name)))
+    return values
+
+
+def _bind_parameter(param, criterion):
+    value = _value_without_blanks(criterion)
+    if param.type == FIELD_TYPE_DATERANGE:
+        return dict(zip(placeholders_for(param), _as_date_range(param, value)))
+    if param.type == FIELD_TYPE_SELECT:
+        return {param.name: _as_list(value)}
+    return {param.name: _as_scalar(param, value)}
+
+
+def _value_without_blanks(criterion):
+    """The criterion's value, with blank terms dropped and blank read as unset"""
+    if criterion is None:
+        return None
+    if criterion.has_multiple_terms:
+        # A single remaining term is flattened back to a scalar
+        return criterion.clone_without_blanks().value or None
+    return criterion.value or None
+
+
+def _as_scalar(param, value):
+    if isinstance(value, list):
+        raise CaseSearchUserError(
+            _("Only one value may be given for '{}'").format(param.name)
+        )
+    return value
+
+
+def _as_list(value):
+    """Bound as a list whatever the number of values, so that endpoint SQL
+    comparing against an array column works no matter what was searched for."""
+    if value is None:
+        return None
+    return value if isinstance(value, list) else [value]
+
+
+def _as_date_range(param, value):
+    if value is None:
+        return None, None
+    if isinstance(value, list) or not str(value).startswith(DATE_RANGE_PREFIX):
+        raise CaseSearchUserError(
+            _("'{}' must be given as a date range").format(param.name)
+        )
+    start, _sep, end = str(value).removeprefix(DATE_RANGE_PREFIX).partition('__')
+    if not start or not end:
+        raise CaseSearchUserError(
+            _("Invalid date range for '{}'").format(param.name)
+        )
+    return start, end
+
 @define
 class ConstantInput:
     """A literal input value supplied directly in the spec."""