Request for expert review: law domain items

[vivekkrishna]

Purpose

This issue is for review of the law domain in the Semantic Conflicts Benchmark.

The law domain is intended to capture semantic conflict between statutes, contracts, legal obligations, and doctrinal rules. It is also the domain most likely to be challenged if an item depends on unstated facts, jurisdictional assumptions, enforceability nuances, or issue-spotting rather than a true semantic conflict.

What I’m looking for

Please review law items for the following:

1) Label correctness

• Is the item correctly labeled as conflict, consistent, or ambiguous?

2) Dependence on unstated assumptions

Please flag items where the reasoning depends on:

• facts not present in the passages
• jurisdiction not specified in the item
• legal standards that vary materially by doctrine or forum
• a probabilistic litigation outcome rather than a direct legal conflict

3) Strictness of reasoning chains

The benchmark aims for 100%-certain reasoning steps.

Please flag cases where:

• an item is really “possible unenforceability” rather than direct conflict
• the reasoning is more like legal risk analysis or issue spotting
• the chain depends on balancing tests, multi-factor tests, or contestable doctrine in a way that weakens certainty

4) Ambiguous items

Please check whether ambiguous items are truly underdetermined by the text, rather than just incomplete.

Especially useful kinds of feedback

• “This is a good legal issue, but not a strict semantic conflict.”
• “This item needs jurisdiction or governing-law facts to be valid.”
• “This should be ambiguous instead of conflict.”
• “This doctrine is too fact-dependent for the benchmark’s certainty standard.”

Goal

The goal is to make the law domain more precise, legally defensible, and better aligned with the benchmark’s claimed reasoning standard.

Comments from lawyers, law students, legal researchers, and legally trained engineers are especially welcome.