Skip to content

Commit 93618c2

Browse files
milanmajchrakclaude
milanmajchrak
and
claude
committed
Correct every false statement the CCMM crosswalk made, and read the fields it dropped
An adversarial audit over all 2628 live LINDAT records found 30 defects. They fall into three groups, and the first is the one that matters: the feed was stating things the source does not support. False statements, now fixed: - access_rights was taken from others/access-status alone. DefaultAccessStatusHelper derives that from the primary bitstream's READ policies, which on a CLARIN repository does not say whether the resource can be downloaded: the gate is the licence category in dc.rights.label plus others/restrictedAccess. 1216 of 2628 records carried a wrong statement - 1161 anonymously downloadable datasets published as "restricted access", 55 login-gated ones as "open access". Verified against the live repository: 12/12 sampled "restricted" records return HTTP 206 to an anonymous ranged GET, 11/12 "open" ACA/RES records return 401. - Every date_type label was the register's concept id ("Issued") rather than its prefLabel ("Date Issued") - 10512 labels, none correct. - conforms_to_standard identified the profile by this stylesheet's own XML namespace under the label "CCMM 1.1", a string that appears nowhere in the CCMM release. - license/label carried the CLARIN access category (PUB/ACA/RES) instead of the licence name, which was demoted to a description. 1932 records. - Hierarchical vocabulary paths ("People::Masaryk ...") were published verbatim as subject titles; the leaf is now the title and the path a classification_code. - Agent type was hard-coded per source field, so film studios and ANVL placeholders were published as people. Names are now typed from the value, and placeholders produce no agent at all. - publication_year took min() over the approximate years even when the record stated an exact issue date; a comma enumeration ("1920, 1932") became a continuous interval; a "0000" issue date fell through to the ingest timestamp; Created ignored the creation year the record states. - RelationType/Replaces and IsReplacedBy do not exist in the register - the concepts are Obsoletes and IsObsoletedBy. Every emitted IRI and label was rechecked against the register. Fields that were being dropped, now read: the contact person and the repository contact, the owning collection, dc.identifier.other, dc.source.uri, local.demo.uri, every dc.relation qualifier, dc.format, dc.coverage.placeName, given/family name parts, language names, local.sponsor, local.additional.metadata, local.size.info, local.refbox.format and the metashare tree. Bitstreams of the ORIGINAL bundle now become distributions with format, size and checksum. Portability, none of which changed LINDAT output: ISO 639-1 and region-tagged language codes are accepted, the stock DSpace type list is mapped, dates are proved castable before being emitted, the funding guard names the whole grantAgreement prefix, and a repository with no url is no longer described as living at each item's own address. Tests: 70 methods over 9 fixtures, including a stock-DSpace record, every date shape DSpace stores, and the two CLARIN access shapes. Element order is asserted directly - the CCMM schema is an xs:sequence and validating against it would pull xml.xsd and GML over the network at build time - plus closed-set guards so no off-register string can come back. Mutation testing: 45 of 46 single-edit mutants are now caught, against 20 of 72 before. Re-run over all 2628 live records: 2628/2628 valid against CCMM 1.1.0 in XSD 1.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 3552e5f commit 93618c2

9 files changed

Lines changed: 3074 additions & 1255 deletions

File tree

‎dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java‎

Lines changed: 759 additions & 304 deletions
Large diffs are not rendered by default.

‎dspace-oai/src/test/resources/xoai-ccmm-academic-licence-test.xml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
An ACA (academic use) item. DSpace computed open.access from the bitstream policies, but the
4+
download is gated behind the CLARIN identity form, so the feed must not call it open.
5+
-->
6+
<metadata xmlns="http://www.lyncode.com/xoai">
7+
<element name="dc">
8+
<element name="date">
9+
<element name="issued">
10+
<element name="none">
11+
<field name="value">2018-06-01</field>
12+
</element>
13+
</element>
14+
</element>
15+
<element name="rights">
16+
<element name="none">
17+
<field name="value">CLARIN ACA End-User Licence Agreement</field>
18+
</element>
19+
<element name="label">
20+
<element name="none">
21+
<field name="value">ACA</field>
22+
</element>
23+
</element>
24+
</element>
25+
<element name="title">
26+
<element name="none">
27+
<field name="value">An Academic Use Corpus</field>
28+
</element>
29+
</element>
30+
</element>
31+
<element name="bundles">
32+
<element name="bundle">
33+
<field name="name">ORIGINAL</field>
34+
<element name="bitstreams">
35+
<element name="bitstream">
36+
<field name="name">corpus.zip</field>
37+
<field name="format">application/zip</field>
38+
<field name="size">4096</field>
39+
<field name="url">https://lindat.mff.cuni.cz/repository/bitstream/1/3/corpus.zip</field>
40+
</element>
41+
</element>
42+
</element>
43+
</element>
44+
<element name="others">
45+
<field name="handle">11234/1-9</field>
46+
<field name="identifier">oai:lindat.mff.cuni.cz:11234/1-9</field>
47+
<element name="access-status">
48+
<field name="value">open.access</field>
49+
</element>
50+
</element>
51+
<element name="repository">
52+
<field name="name">LINDAT/CLARIAH-CZ</field>
53+
<field name="url">https://lindat.mff.cuni.cz/repository/</field>
54+
</element>
55+
</metadata>

‎dspace-oai/src/test/resources/xoai-ccmm-clarin-access-test.xml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
A CLARIN item whose real downloadability disagrees with others/access-status: the licence is
4+
PUB and the file is fetched anonymously, while DefaultAccessStatusHelper called it restricted.
5+
-->
6+
<metadata xmlns="http://www.lyncode.com/xoai">
7+
<element name="dc">
8+
<element name="date">
9+
<element name="issued">
10+
<element name="none">
11+
<field name="value">2019-01-01</field>
12+
</element>
13+
</element>
14+
</element>
15+
<element name="rights">
16+
<element name="none">
17+
<field name="value">Creative Commons - Attribution-NonCommercial-NoDerivatives 4.0 International</field>
18+
</element>
19+
<element name="label">
20+
<element name="none">
21+
<field name="value">PUB</field>
22+
</element>
23+
</element>
24+
</element>
25+
<element name="title">
26+
<element name="none">
27+
<field name="value">A Publicly Licensed Film</field>
28+
</element>
29+
</element>
30+
</element>
31+
<element name="bundles">
32+
<element name="bundle">
33+
<field name="name">ORIGINAL</field>
34+
<element name="bitstreams">
35+
<element name="bitstream">
36+
<field name="name">film.mp4</field>
37+
<field name="format">video/mp4</field>
38+
<field name="size">1048576</field>
39+
<field name="url">https://lindat.mff.cuni.cz/repository/bitstream/1/2/film.mp4</field>
40+
</element>
41+
</element>
42+
</element>
43+
</element>
44+
<element name="others">
45+
<field name="handle">20.500.12801/1-2</field>
46+
<field name="identifier">oai:lindat.mff.cuni.cz:20.500.12801/1-2</field>
47+
<!-- DSpace computed this from the primary bitstream policies; the file downloads anyway -->
48+
<element name="access-status">
49+
<field name="value">restricted</field>
50+
</element>
51+
</element>
52+
<element name="repository">
53+
<field name="name">LINDAT/CLARIAH-CZ</field>
54+
<field name="url">https://lindat.mff.cuni.cz/repository/</field>
55+
</element>
56+
</metadata>

‎dspace-oai/src/test/resources/xoai-ccmm-date-shapes-test.xml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Date shapes DSpace actually stores, plus two that no format string can parse. Every branch of
4+
FormatDate has to produce a value xs:date accepts, or produce nothing at all.
5+
-->
6+
<metadata xmlns="http://www.lyncode.com/xoai">
7+
<element name="dc">
8+
<element name="date">
9+
<element name="issued">
10+
<element name="none">
11+
<field name="value">2014</field>
12+
</element>
13+
</element>
14+
<element name="accessioned">
15+
<element name="none">
16+
<field name="value">2011-12</field>
17+
<field name="value">no date recorded</field>
18+
</element>
19+
</element>
20+
<element name="available">
21+
<element name="none">
22+
<field name="value">2008-12-01</field>
23+
<field name="value">n.d.</field>
24+
</element>
25+
</element>
26+
</element>
27+
<element name="title">
28+
<element name="none">
29+
<field name="value">Date Shapes</field>
30+
</element>
31+
</element>
32+
</element>
33+
<element name="others">
34+
<field name="handle">123/789</field>
35+
<field name="identifier">oai:repo.example.org:123/789</field>
36+
</element>
37+
<element name="repository">
38+
<field name="name">Example Repository</field>
39+
<field name="url">https://repo.example.org/</field>
40+
</element>
41+
</metadata>

‎dspace-oai/src/test/resources/xoai-ccmm-enumerated-date-test.xml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
"1920, 1932" enumerates two attested years; it is not a continuous range. The migrated
4+
catalogue blob states the year the resource itself was created.
5+
-->
6+
<metadata xmlns="http://www.lyncode.com/xoai">
7+
<element name="dc">
8+
<element name="date">
9+
<element name="issued">
10+
<element name="none">
11+
<field name="value">0000</field>
12+
</element>
13+
</element>
14+
</element>
15+
<element name="title">
16+
<element name="none">
17+
<field name="value">A Compiled Newsreel</field>
18+
</element>
19+
</element>
20+
</element>
21+
<element name="local">
22+
<element name="approximateDate">
23+
<element name="issued">
24+
<element name="none">
25+
<field name="value">1920, 1932</field>
26+
</element>
27+
</element>
28+
</element>
29+
<element name="additional">
30+
<element name="metadata">
31+
<element name="none">
32+
<field name="value">Nid:2307
33+
Begin year of resource creation (field_year):1787
34+
Readily Available (field_resource_available):Yes</field>
35+
</element>
36+
</element>
37+
</element>
38+
</element>
39+
<element name="others">
40+
<field name="handle">20.500.12801/3-4</field>
41+
<field name="identifier">oai:lindat.mff.cuni.cz:20.500.12801/3-4</field>
42+
</element>
43+
<element name="repository">
44+
<field name="name">LINDAT/CLARIAH-CZ</field>
45+
<field name="url">https://lindat.mff.cuni.cz/repository/</field>
46+
</element>
47+
</metadata>

‎dspace-oai/src/test/resources/xoai-ccmm-no-identifier-test.xml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
A record carrying no dc.identifier and no others/handle at all. identifier is 1..unbounded,
4+
so the crosswalk still has to produce one.
5+
-->
6+
<metadata xmlns="http://www.lyncode.com/xoai">
7+
<element name="dc">
8+
<element name="title">
9+
<element name="none">
10+
<field name="value">A Record With No Identifier</field>
11+
</element>
12+
</element>
13+
<element name="date">
14+
<element name="issued">
15+
<element name="none">
16+
<field name="value">2021-02-03</field>
17+
</element>
18+
</element>
19+
</element>
20+
</element>
21+
<element name="others">
22+
<field name="identifier">oai:repo.example.org:no-handle-1</field>
23+
</element>
24+
<element name="repository">
25+
<field name="name">Example Repository</field>
26+
<field name="url">https://repo.example.org/</field>
27+
</element>
28+
</metadata>

‎dspace-oai/src/test/resources/xoai-ccmm-stock-dspace-test.xml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
A stock DSpace item: ISO 639-1 language with a region subtag, a type from the default
4+
submission list, no repository URL, and no access-status plugin. None of the CLARIN-specific
5+
fields are present.
6+
-->
7+
<metadata xmlns="http://www.lyncode.com/xoai">
8+
<element name="dc">
9+
<element name="contributor">
10+
<element name="author">
11+
<element name="none">
12+
<field name="value">Smith, Alice</field>
13+
</element>
14+
</element>
15+
</element>
16+
<element name="date">
17+
<element name="issued">
18+
<element name="none">
19+
<field name="value">2024-05-01</field>
20+
</element>
21+
</element>
22+
</element>
23+
<element name="identifier">
24+
<element name="uri">
25+
<element name="none">
26+
<field name="value">http://hdl.handle.net/123/456</field>
27+
</element>
28+
</element>
29+
</element>
30+
<element name="language">
31+
<element name="iso">
32+
<element name="none">
33+
<field name="value">en_US</field>
34+
</element>
35+
</element>
36+
</element>
37+
<element name="relation">
38+
<element name="none">
39+
<field name="value">info:eu-repo/semantics/openAccess</field>
40+
<!-- five segments, but still not a grant agreement -->
41+
<field name="value">info:eu-repo/semantics/altIdentifier/doi/10.1234/xyz</field>
42+
</element>
43+
</element>
44+
<element name="title">
45+
<element name="none">
46+
<field name="value">A Stock DSpace Item</field>
47+
<field name="value">Second Title Nobody Entered As One</field>
48+
</element>
49+
</element>
50+
<element name="type">
51+
<element name="none">
52+
<field name="value">Article</field>
53+
</element>
54+
</element>
55+
</element>
56+
<element name="others">
57+
<field name="handle">123/456</field>
58+
<field name="identifier">oai:repo.example.org:123/456</field>
59+
</element>
60+
<element name="repository">
61+
<field name="name">Example Repository</field>
62+
</element>
63+
</metadata>

0 commit comments

Comments
 (0)