merge: bring CLARIN branch (main) into dtq

origin/main and origin/dtq had diverged since 5724696 (2024-10-24) with no
ancestry either way. main added +300/-0 lines and dtq +263/-71, both confined
to dspace_rest_client/client.py and dspace_rest_client/models.py.

main contributes the CLARIN/UFAL surface: License and Label models,
clarinlruallowances get/create, submit-group creation, eperson-to-group
membership, get_item_by_handle, get_items_from_collection, get_bundle_by_name,
get_user_by_email, api_put_uri and the dict-based resource-policy helpers.

dtq keeps its hardening: reauth on 401, request timeouts, proxy support,
verify_response, _last_err, the ResourcePolicy model and 404-safe bundles.

Conflict resolutions (3 hunks, 2 in client.py and 1 in models.py):

- get_items: kept dtq's paginated get_items(self, page=0, size=20) and dropped
  main's no-arg variants. main still carried two duplicate `def get_items`
  definitions; the shadowing second one tested for 'collections' in _embedded
  while iterating _embedded['items']. The paginated signature is required by
  DSpace-ISstag-integration src/ingest/_dspace.py, which calls
  get_items(page=page, size=page_size). main's additive get_item_by_handle is
  kept alongside it.

- remove_metadata: unified the two signatures into
  remove_metadata(self, dso, field, place=None). place=None removes every value
  of the field (dtq behaviour, PATCH /metadata/{field}); an explicit place
  removes a single value (main behaviour, PATCH /metadata/{field}/{place}).
  This keeps dtq's 2-arg call sites and dspace-import-clarin's
  remove_metadata(item, key, 0) working. main's module-level logging.error was
  replaced by _logger.error to match the rest of the file.

- models.py: kept all three new classes - License (with to_dict), Label and
  ResourcePolicy (with as_dict and __repr__). The conflict was purely
  positional, both sides having appended at EOF.

api_put_uri was added on main against the pre-hardening base, so it was
harmonised with its sibling api_put: reset self._last_err on entry, pass
proxies=self.proxies, and use _logger instead of module-level logging. Its
public signature is unchanged - dspace-rest-test depends on it.

dtq-dev is deliberately NOT merged. Its only unique commit, 145635a
(2023-06-12, "copied code from dtq main"), is a snapshot of a different
project: it relocates dspace_rest_client/{client,models}.py into
support/dspace_interface/, deletes setup.py, example.py, example_gets.py,
solr_example.py, publish.sh, CHANGELOG.md and MAINTAINING.md, and adds ~8.5k
lines of import tooling, test fixtures, license icons and a localization CSV.
Merging it would destroy the installable package.

Both resource-policy APIs are retained side by side, since main's dict-based
set (get_resource_policy, create_resource_policy, update_resource_policy_group)
and dtq's model-based set (get_resourcepolicy, create_resourcepolicy) each have
live consumers. Unification is left to a separate deprecation change.

Verified: compileall clean, package imports, no duplicate method definitions,
and the merged API surface is exactly the union of both branches (nothing from
either side lost, main's duplicate get_items aside). The
DSpace-ISstag-integration suite gives an identical result with and without this
merge - 579 passed, 40 failed - where the 40 failures are pre-existing and
caused by flask being absent from that environment, not by the library.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: jr-rk

dspace_rest_client/client.py

@@ -325,6 +325,38 @@ def api_put(self, url, params, json, retry=False):
 
         return r
 
+    def api_put_uri(self, url, params, uri_list, retry=False):
+        """
+        Perform a PUT request. Refresh XSRF token if necessary.
+        PUTs are typically used to update objects.
+        @param url:     DSpace REST API URL
+        @param params:  Any parameters to include (eg ?parent=abbc-....)
+        @param uri_list: One or more URIs referencing objects
+        @param retry:   Has this method already been retried? Used if we need to refresh XSRF.
+        @return:        Response from API
+        """
+        self._last_err = None
+        r = self.session.put(url, params=params, data=uri_list, headers=self.list_request_headers,
+                             proxies=self.proxies)
+        self.update_token(r)
+
+        if r.status_code == 403:
+            # 403 Forbidden
+            # If we had a CSRF failure, retry the request with the updated token
+            # After speaking in #dev it seems that these do need occasional refreshes but I suspect
+            # it's happening too often for me, so check for accidentally triggering it
+            _logger.debug(r.text)
+            # Parse response
+            r_json = parse_json(r)
+            if 'message' in r_json and 'CSRF token' in r_json['message']:
+                if retry:
+                    _logger.warning(f'Too many retries updating token: {r.status_code}: {r.text}')
+                else:
+                    _logger.debug("Retrying request with updated CSRF token")
+                    return self.api_put_uri(url, params=params, uri_list=uri_list, retry=True)
+
+        return r
+
     def api_delete(self, url, params, retry=False):
         """
         Perform a DELETE request. Refresh XSRF token if necessary.
@@ -978,6 +1010,29 @@ def get_item(self, uuid):
             _logger.error(f'Invalid item UUID: {uuid}')
             return None
 
+    def get_item_by_handle(self, handle):
+        """
+        Get item based on handle.
+        """
+        if handle is None:
+            return None
+        params = {
+            "handle": handle
+        }
+        url = f'{self.API_ENDPOINT}/core/items/search/byHandle'
+        try:
+            r = self.api_get(url, params, None)
+            r_json = parse_json(r)
+            if '_embedded' in r_json:
+                if 'items' in r_json['_embedded']:
+                    items = r_json['_embedded']['items']
+                    if len(items) > 0:
+                        return Item(items[0])
+            return None
+        except (TypeError, ValueError):
+            _logger.error(f'Invalid item handle: {handle}')
+            return None
+
     def get_items(self, page=0, size=20):
         """
         Get all archived items for a logged-in administrator. Admin only! Usually you will want to
@@ -1079,24 +1134,27 @@ def add_metadata(self, dso, field, value, language=None, authority=None, confide
 
         return dso_type(api_resource=parse_json(r))
 
-    def remove_metadata(self, dso, field):
+    def remove_metadata(self, dso, field, place=None):
         """
-        Remove metadata
+        Remove metadata from dso based on metadata field.
+        @param dso:   DSpace object to patch
+        @param field: metadata field, e.g. dc.title
+        @param place: if None, every value of the field is removed. Otherwise only
+                      the value at this place - a 0+ integer, or a hyphen meaning "last".
+        @return:      DSpace object constructed from the API response
         """
         if dso is None or field is None or not isinstance(dso, DSpaceObject):
             _logger.error('Invalid or missing DSpace object, field or value string')
             return self
 
         dso_type = type(dso)
 
-        # Place can be 0+ integer, or a hyphen - meaning "last"
-        path = f'/metadata/{field}'
+        path = f'/metadata/{field}' if place is None else f'/metadata/{field}/{place}'
         url = dso.links['self']['href']
 
         r = self.api_patch(url=url, operation=self.PatchOperation.REMOVE, path=path, value=None)
         return dso_type(api_resource=parse_json(r))
 
-
     def create_user(self, user, token=None):
         """
         Create a user
@@ -1154,6 +1212,45 @@ def create_group(self, group):
             # that you see for other DSO types - still figuring out the best way
         return Group(api_resource=parse_json(self.create_dso(url, params=None, data=data)))
 
+    def create_submit_group(self, collection):
+        """
+        Creates a submitter group for the given collection.
+        """
+        url = f'{self.API_ENDPOINT}/core/collections/{collection.uuid}/submittersGroup'
+        r = self.api_post(url, json={}, params=None)
+        if r.status_code == 201:
+            return Group(parse_json(r))
+        return None
+
+    def add_member(self, group, eperson):
+        """
+        Adds a user (EPerson) as a member of the specified group.
+
+        Args:
+            group (Group): The group to which the user will be added.
+            eperson (User): The EPerson to be added as a member of the group.
+
+        Returns:
+            bool: True if the user was successfully added (HTTP 204), False otherwise.
+        """
+        if not isinstance(group, Group):
+            _logger.error("Provided 'group' is not an instance of Group.")
+            return False
+
+        if not isinstance(eperson, User):
+            _logger.error("Provided 'eperson' is not an instance of User.")
+            return False
+
+        url = f'{self.API_ENDPOINT}/eperson/groups/{group.uuid}/epersons'
+        eperson_uri = f'{self.API_ENDPOINT}/epersons/{eperson.uuid}'
+        r = self.api_post_uri(url, params=None, uri_list=eperson_uri)
+        if r.status_code == 204:
+            return True
+        _logger.error(f"Failed to add user {eperson.uuid} to group {group.uuid}. "
+                        f"Status code: {r.status_code}")
+        return False
+
+
     def start_workflow(self, workspace_item):
         url = f'{self.API_ENDPOINT}/workflow/workflowitems'
         res = parse_json(self.api_post_uri(url, params=None, uri_list=workspace_item))
@@ -1204,3 +1301,139 @@ def solr_query(self, query, filters=None, fields=None, start=0, rows=999999999):
         return self.solr.search(query, fq=filters, start=start, rows=rows, **{
             'fl': ','.join(fields)
         })
+
+    def get_items_from_collection(self, collection_id, page=0, size=1000):
+        """
+        Get all items
+        @return:        list of Item objects
+        """
+        url = f'{self.API_ENDPOINT}/discover/search/objects?sort=dc.date.accessioned,DESC&page={page}&size={size}&scope={collection_id}&dsoType=ITEM&embed=thumbnail'
+
+        items = list()
+        r = self.api_get(url)
+        r_json = parse_json(r)
+        if '_embedded' in r_json:
+            if 'searchResult' in r_json['_embedded']:
+                if '_embedded' in r_json['_embedded']['searchResult']:
+                    for item_resource in r_json['_embedded']['searchResult']['_embedded']['objects']:
+                        items.append(Item(item_resource['_embedded']['indexableObject']))
+
+        return items
+
+    def get_bundle_by_name(self, name, item_uuid):
+        """
+        Get a bundle by name for a specific item
+        @param name:    Name of the bundle
+        @param item_uuid: UUID of the item
+        @return:        Bundle object
+        """
+        url = f'{self.API_ENDPOINT}/core/items/{item_uuid}/bundles'
+        r_json = self.fetch_resource(url, params=None)
+        if '_embedded' in r_json:
+            if 'bundles' in r_json['_embedded']:
+                for bundle in r_json['_embedded']['bundles']:
+                    if bundle['name'] == name:
+                        return Bundle(bundle)
+        return None
+
+    def get_resource_policy(self, bundle_uuid):
+        """
+        Get a resource policy for a specific bundle
+        """
+        url = f'{self.API_ENDPOINT}/authz/resourcepolicies/search/resource?uuid={bundle_uuid}&embed=eperson&embed=group'
+        r = self.api_get(url)
+        r_json = parse_json(r)
+        if '_embedded' in r_json:
+            if 'resourcepolicies' in r_json['_embedded']:
+                return r_json['_embedded']['resourcepolicies'][0]
+
+    def create_resource_policy(self, resource_uuid, data, group_uuid=None, eperson_uuid=None):
+        """
+        Creates a resource policy by sending a POST request to the API endpoint.
+        """
+        url = f'{self.API_ENDPOINT}/authz/resourcepolicies'
+        params = {"resource": resource_uuid}
+        if group_uuid:
+            params["group"] = group_uuid
+        if eperson_uuid:
+            params["eperson"] = eperson_uuid
+
+        r = self.api_post(url, params=params, json=data)
+        if r.status_code == 201:
+            return True
+        return False
+
+
+    def update_resource_policy_group(self, policy_id, group_uuid):
+        """
+        Update a resource policy with a new group
+        """
+        url = f'{self.API_ENDPOINT}/authz/resourcepolicies/{policy_id}/group'
+        body = f'{self.API_ENDPOINT}/eperson/groups/{group_uuid}'
+        r = self.api_put_uri(url, None, body, False)
+        return r
+
+    def get_clarinlruallowances(self):
+        """
+        Fetch all clarinlruallowances.
+        """
+        url = f'{self.API_ENDPOINT}/core/clarinlruallowances'
+        try:
+            response = self.api_get(url)
+            data = parse_json(response)
+            allowances = data.get('_embedded', {}).get('clarinlruallowances')
+            if allowances:
+                return allowances
+        except Exception as e:
+            _logger.error(f"Error fetching CLARIN LRU allowances [{url}]: {e}")
+        return None
+
+    def get_clarinlruallowances_by_bitstream_and_user(self, bitstream_uuid, user_uuid):
+        """
+        Fetch user allowances for a specific bitstream and user.
+        """
+        url = f'{self.API_ENDPOINT}/core/clarinlruallowances/search/byBitstreamAndUser'
+        params = {'bitstreamUUID': bitstream_uuid, 'userUUID': user_uuid}
+        try:
+            response = self.api_get(url, params=params)
+            data = parse_json(response)
+            allowances = data.get('_embedded', {}).get('clarinlruallowances')
+            if allowances:
+                return allowances
+        except Exception as e:
+            _logger.error(f"Error fetching user allowances: {e}")
+        return None
+
+
+    def create_clarinlruallowances(self, bitstream_uuid):
+        """
+        Create clarinlruallowances for a bitstream for logged user
+        by managing user metadata of bitstream.
+        """
+        url = f'{self.API_ENDPOINT}/core/clarinusermetadata/manage'
+        params = {'bitstreamUUID': bitstream_uuid}
+        metadata_payload = [
+            {"metadataKey": "NAME", "metadataValue": "Test"}
+        ]
+        try:
+            response = self.api_post(url, json=metadata_payload, params=params)
+            if response.status_code == 200:
+                return True
+        except Exception as e:
+            _logger.error(f"Error managing user metadata: {e}")
+        return False
+
+
+    def get_user_by_email(self, email):
+        """
+        Retrieve user details using their email address.
+        """
+        url = f'{self.API_ENDPOINT}/eperson/epersons/search/byEmail'
+        params = {'email': email}
+        try:
+            response = self.api_get(url, params=params)
+            user_data = parse_json(response)
+            return User(user_data)
+        except Exception as e:
+            _logger.error(f"Error retrieving user by email {email}: {e}")
+            return None

dspace_rest_client/models.py

@@ -521,6 +521,61 @@ class RelationshipType(AddressableHALResource):
     def __init__(self, api_resource):
         super(RelationshipType, self).__init__(api_resource)
 
+class License(AddressableHALResource):
+    """
+    Specific attributes and functions for licenses
+    """
+    def __init__(self, api_resource=None):
+        super(License, self).__init__(api_resource)
+        api_resource = api_resource or {}
+        self.type = 'clarinlicense'
+        self.name = api_resource.get('name')
+        self.definition = api_resource.get('definition')
+        self.confirmation = api_resource.get('confirmation', 0)
+        self.requiredInfo = api_resource.get('requiredInfo')
+        license_label_value = api_resource.get('clarinLicenseLabel')
+        self.licenseLabel = Label(license_label_value) if license_label_value else None
+        self.extendedLicenseLabel = [Label(label) for label in
+                                     api_resource.get('extendedClarinLicenseLabels', [])]
+        self.bitstream = api_resource.get('bitstreams')
+
+    def to_dict(self):
+        return {
+            'name': self.name,
+            'license_id': self.id,
+            'definition': self.definition,
+            'confirmation': self.confirmation,
+            'required_info': self.requiredInfo,
+            'label_id': self.licenseLabel.id if self.licenseLabel else None,
+        }
+
+
+class Label(AddressableHALResource):
+    """
+    Specific attributes and functions for licenses
+    """
+    def __init__(self, api_resource=None):
+        """
+        Default constructor. Call DSpaceObject init then set label-specific attributes
+        @param api_resource: API result object to use as initial data
+        """
+        super(Label, self).__init__(api_resource)
+        api_resource = api_resource or {}
+        self.type = 'clarinlicenselabel'
+        self.label = api_resource.get('label')
+        self.title = api_resource.get('title')
+        self.icon = api_resource.get('icon')
+        self.extended = api_resource.get('extended', False)
+
+    def to_dict(self):
+        return {
+            'label_id': self.id,
+            'label': self.label,
+            'title': self.title,
+            'icon': self.icon,
+            'is_extended': self.extended
+        }
+
 
 class ResourcePolicy(AddressableHALResource):
     """
@@ -544,6 +599,7 @@ def __init__(self, api_resource: dict):
             if 'group' in api_resource['_embedded']:
                 self.groupName = api_resource['_embedded']['group'].get('name')
                 self.groupUUID = api_resource['_embedded']['group'].get('uuid')
+
     def as_dict(self):
         return {
             'id': self.id,
@@ -559,4 +615,4 @@ def as_dict(self):
         }
 
     def __repr__(self):
-        return f"ResourcePolicy: {self.name} [{self.groupName}] [action: {self.action}] [type: {self.type}]"
+        return f"ResourcePolicy: {self.name} [{self.groupName}] [action: {self.action}] [type: {self.type}]"