Skip to content

Commit cc5a0eb

Browse files
mcanuclaude
andauthored
docs: UTC-1178: Document configurable reject options (#9964)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 89ad363 commit cc5a0eb

5 files changed

Lines changed: 43 additions & 18 deletions

File tree

‎docs/source/guide/comments_notifications.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ The Data Manager has several columns to display comment information:
123123
You can configure the project so that annotators and reviewers are required to leave comments after taking specific actions:
124124

125125
* [**Annotation > Annotating Options > Annotators must leave a comment on skip**](project_settings_lse#Annotation)
126-
* [**Review > AReviewers must leave a comment on reject**](project_settings_lse#Review)
126+
* [**Review > Reject Options > Reviewers must leave a comment on reject**](project_settings_lse#reject-options)
127127

128128
## Comment notifications
129129

‎docs/source/guide/project_settings_lse.md‎

Lines changed: 30 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -512,7 +512,9 @@ Tasks are shown in randomized task order while preserving the stable order of an
512512

513513
<dd>
514514

515-
Configure what rejection options are available to reviewers.
515+
Select which reject options are available to reviewers, and which one is the default.
516+
517+
Use the **Available** column to choose the options. At least one option must be available. Use the **Default** column to choose the option that is applied when a reviewer clicks **Reject** without opening the menu.
516518

517519
<table>
518520
<thead>
@@ -524,45 +526,60 @@ Configure what rejection options are available to reviewers.
524526
<tr>
525527
<td>
526528

527-
**Requeue rejected annotations back to annotators**
529+
**Reject and do not require rework**
528530
</td>
529531
<td>
530532

531-
When a reviewer clicks **Reject**, the annotation is reassigned back to the annotator.
533+
The annotation is rejected and is not sent back to the annotator for rework.
534+
535+
This is the default for new projects.
532536

533537
</td>
534538
</tr>
539+
<tr>
535540
<td>
536541

537-
**Remove rejected annotations from labeling queue**
542+
**Reject and return to annotator**
538543
</td>
539544
<td>
540545

541-
When a reviewer clicks **Reject**, the annotation is not reassigned back to the annotator.
546+
The annotation is rejected and sent back to the original annotator for rework. The annotator sees the rejected annotation in their labeling stream and can update it. The updated annotation is then shown to reviewers again in the review stream.
542547

543548
</td>
544549
</tr>
545550
<tr>
546551
<td>
547552

548-
**Allow reviewer to choose: Requeue or Remove**
553+
**Reject and pass to another annotator**
549554
</td>
550555
<td>
551556

552-
Reviewers see the following options:
557+
The annotation is rejected and the task is sent to a different annotator for a new annotation.
553558

554-
* **Accept**
555-
* **Remove** -- When selected, the annotation is rejected and removed from the labeling queue.
556-
* **Requeue** -- When selected, the annotation is rejected and then reassigned back to the annotator.
559+
The rejected annotation no longer counts toward the project's [overlap](#overlap), which opens one slot on the task for another annotator. The original annotator can no longer update the rejected annotation. The annotation, its rejection, and the time spent on it remain in the Data Manager and in dashboards.
557560

558-
For example, a reviewer might decide to requeue an annotation that is nearly correct but just needs a slight change. However, an annotation with numerous errors may be easier to simply reject entirely and remove from the queue.
559-
560-
Note that when you click **Remove**, the annotation is also marked as cancelled/skipped. This is reflected in various metrics (for example, Data Manager columns and dashboards), and differentiates between the two rejection actions in the API with `was_cancelled: true`.
561+
This option is only available when [**Task Assignment**](#distribute-tasks) is set to **Automatic**.
561562

562563
</td>
563564
</tr>
564565
</table>
565566

567+
When only one option is available, reviewers see a single **Reject** button that applies it. When more than one option is available, reviewers can click **Reject** to apply the default option, or open the reject menu to choose another option:
568+
569+
* **No Rework** -- Reject without sending for rework.
570+
* **Return to Annotator** -- Reject and send back to the original annotator for rework.
571+
* **Pass to Another Annotator** -- Reject and send to a different annotator.
572+
573+
![Screenshot of the reject menu](/images/review/reject-menu.png)
574+
575+
For example, a reviewer might return an annotation that is nearly correct and only needs a small change. For an annotation with numerous errors, they might pass the task to another annotator instead.
576+
577+
The same options are available when reviewers reject annotations in bulk from the Data Manager.
578+
579+
Changing the available options only affects future rejections. Annotations that were already rejected keep the outcome that the reviewer chose. For example, annotations that were returned to an annotator can still be updated by that annotator after you make **Reject and return to annotator** unavailable.
580+
581+
Separately, you can enable **Reviewers must leave a comment on reject** to require reviewers to add a comment before they can reject an annotation from the review stream. This does not apply to bulk rejections from the Data Manager.
582+
566583
</dd>
567584

568585
<dt id="data-manager">Data Manager</dt>

‎docs/source/guide/project_states.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ stateDiagram-v2
7979
ANNOTATION_IN_PROGRESS --> COMPLETED: Annotations complete -<br>review not required
8080
NEEDS_REVIEW --> REVIEW_IN_PROGRESS: First review started
8181
REVIEW_IN_PROGRESS --> COMPLETED: All reviews done
82-
REVIEW_IN_PROGRESS --> ANNOTATION_IN_PROGRESS: Annotation rejected<br>(requeue mode)
82+
REVIEW_IN_PROGRESS --> ANNOTATION_IN_PROGRESS: Annotation rejected<br>(returned or passed)
8383
8484
class CREATED neutral
8585
class ANNOTATION_IN_PROGRESS grape
@@ -91,7 +91,7 @@ stateDiagram-v2
9191
| State | Description |
9292
|----------|------------|
9393
| **Initial** | Task has been created but does not have any annotations yet. |
94-
| **Annotating** | First annotation has been submitted, or a rejected annotation has been requeued back to the annotator. <br/><br/>Whether the task lingers in this state long enough for it to be reflected in the Data Manager depends on your project settings. See [Additional notes](#Additional-notes) below.|
94+
| **Annotating** | First annotation has been submitted, or a rejected annotation has been [returned to the annotator or passed to another annotator](project_settings_lse#reject-options). <br/><br/>Whether the task lingers in this state long enough for it to be reflected in the Data Manager depends on your project settings. See [Additional notes](#Additional-notes) below.|
9595
| **Needs Review** | The required number of annotations have been completed and the task is ready for review. <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below. |
9696
| **In Review** | First review has been created for any annotation within the task. <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below.|
9797
| **Done** | All required annotations and reviews have been completed. |

‎docs/source/guide/quality.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,15 @@ After you [assign reviewers to tasks](#Assign-reviewers-to-tasks), they can revi
3535
- If the annotation is correct, click **Accept**.
3636

3737
- If the annotation is mostly correct, you can correct it by selecting a different option, changing the selected region, moving the bounding box, or whichever makes sense for the type of label you're reviewing. After correcting the annotation, click **Fix & Accept**.
38-
- If the annotation is completely incorrect, or you don't want to attempt to correct it at all, click **Reject** to reject the annotation. To place a rejected task back in the Label Stream for annotation, you must delete the annotation. Rejecting an annotation does not return it to annotators to re-label.
38+
- If the annotation is incorrect, or you don't want to attempt to correct it yourself, click **Reject** to reject the annotation.
39+
40+
Clicking **Reject** applies the project's default reject option. If the project has more than one reject option available, you can open the reject menu to choose how to handle the rejection:
41+
42+
- **No Rework** -- The annotation is rejected and is not sent back for rework.
43+
- **Return to Annotator** -- The annotation is sent back to the original annotator so that they can update it.
44+
- **Pass to Another Annotator** -- The task is sent to a different annotator for a new annotation. The original annotator can no longer update the rejected annotation.
45+
46+
Which options you see depends on the project's [**Reject Options**](project_settings_lse#reject-options) settings.
3947

4048
3. Continue reviewing annotated tasks until you've reviewed all annotated tasks. Click **Data Manager** to return to the list of tasks for the project.
4149

@@ -141,7 +149,7 @@ How much access a paused user keeps depends on why they were paused:
141149
A pause only restricts the type of work that triggered it. Annotation limits and annotator evaluation both measure annotation work, so a flexible pause stops new annotation work only. If a paused user is also a reviewer, they can continue reviewing. Only a full pause stops a user from reviewing.
142150

143151
!!! note
144-
A flexible pause acts as a full pause when the project's [**Reject Options**](project_settings_lse#reject-options) are set to **Remove rejected annotations from labeling queue**. In this case, rejected annotations are never returned to the annotator, so there is nothing for them to recover.
152+
A flexible pause acts as a full pause when the project's [**Reject Options**](project_settings_lse#reject-options) do not include **Reject and return to annotator**. In this case, rejected annotations are never returned to the annotator, so there is nothing for them to recover.
145153

146154
### Manually pause an annotator
147155

24.5 KB
Loading

0 commit comments

Comments
 (0)