You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cc5a0eb
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/guide/project_settings_lse.md
+30-13Lines changed: 30 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -512,7 +512,9 @@ Tasks are shown in randomized task order while preserving the stable order of an
512
512
513
513
<dd>
514
514
515
-
Configure what rejection options are available to reviewers.
515
+
Select which reject options are available to reviewers, and which one is the default.
516
+
517
+
Use the **Available** column to choose the options. At least one option must be available. Use the **Default** column to choose the option that is applied when a reviewer clicks **Reject** without opening the menu.
516
518
517
519
<table>
518
520
<thead>
@@ -524,45 +526,60 @@ Configure what rejection options are available to reviewers.
524
526
<tr>
525
527
<td>
526
528
527
-
**Requeue rejected annotations back to annotators**
529
+
**Reject and do not require rework**
528
530
</td>
529
531
<td>
530
532
531
-
When a reviewer clicks **Reject**, the annotation is reassigned back to the annotator.
533
+
The annotation is rejected and is not sent back to the annotator for rework.
534
+
535
+
This is the default for new projects.
532
536
533
537
</td>
534
538
</tr>
539
+
<tr>
535
540
<td>
536
541
537
-
**Remove rejected annotations from labeling queue**
542
+
**Reject and return to annotator**
538
543
</td>
539
544
<td>
540
545
541
-
When a reviewer clicks **Reject**, the annotation is not reassigned back to the annotator.
546
+
The annotation is rejected and sent back to the original annotator for rework. The annotator sees the rejected annotation in their labeling stream and can update it. The updated annotation is then shown to reviewers again in the review stream.
542
547
543
548
</td>
544
549
</tr>
545
550
<tr>
546
551
<td>
547
552
548
-
**Allow reviewer to choose: Requeue or Remove**
553
+
**Reject and pass to another annotator**
549
554
</td>
550
555
<td>
551
556
552
-
Reviewers see the following options:
557
+
The annotation is rejected and the task is sent to a different annotator for a new annotation.
553
558
554
-
***Accept**
555
-
***Remove** -- When selected, the annotation is rejected and removed from the labeling queue.
556
-
***Requeue** -- When selected, the annotation is rejected and then reassigned back to the annotator.
559
+
The rejected annotation no longer counts toward the project's [overlap](#overlap), which opens one slot on the task for another annotator. The original annotator can no longer update the rejected annotation. The annotation, its rejection, and the time spent on it remain in the Data Manager and in dashboards.
557
560
558
-
For example, a reviewer might decide to requeue an annotation that is nearly correct but just needs a slight change. However, an annotation with numerous errors may be easier to simply reject entirely and remove from the queue.
559
-
560
-
Note that when you click **Remove**, the annotation is also marked as cancelled/skipped. This is reflected in various metrics (for example, Data Manager columns and dashboards), and differentiates between the two rejection actions in the API with `was_cancelled: true`.
561
+
This option is only available when [**Task Assignment**](#distribute-tasks) is set to **Automatic**.
561
562
562
563
</td>
563
564
</tr>
564
565
</table>
565
566
567
+
When only one option is available, reviewers see a single **Reject** button that applies it. When more than one option is available, reviewers can click **Reject** to apply the default option, or open the reject menu to choose another option:
568
+
569
+
***No Rework** -- Reject without sending for rework.
570
+
***Return to Annotator** -- Reject and send back to the original annotator for rework.
571
+
***Pass to Another Annotator** -- Reject and send to a different annotator.
572
+
573
+

574
+
575
+
For example, a reviewer might return an annotation that is nearly correct and only needs a small change. For an annotation with numerous errors, they might pass the task to another annotator instead.
576
+
577
+
The same options are available when reviewers reject annotations in bulk from the Data Manager.
578
+
579
+
Changing the available options only affects future rejections. Annotations that were already rejected keep the outcome that the reviewer chose. For example, annotations that were returned to an annotator can still be updated by that annotator after you make **Reject and return to annotator** unavailable.
580
+
581
+
Separately, you can enable **Reviewers must leave a comment on reject** to require reviewers to add a comment before they can reject an annotation from the review stream. This does not apply to bulk rejections from the Data Manager.
REVIEW_IN_PROGRESS --> ANNOTATION_IN_PROGRESS: Annotation rejected<br>(returned or passed)
83
83
84
84
class CREATED neutral
85
85
class ANNOTATION_IN_PROGRESS grape
@@ -91,7 +91,7 @@ stateDiagram-v2
91
91
| State | Description |
92
92
|----------|------------|
93
93
|**Initial**| Task has been created but does not have any annotations yet. |
94
-
|**Annotating**| First annotation has been submitted, or a rejected annotation has been requeued back to the annotator. <br/><br/>Whether the task lingers in this state long enough for it to be reflected in the Data Manager depends on your project settings. See [Additional notes](#Additional-notes) below.|
94
+
|**Annotating**| First annotation has been submitted, or a rejected annotation has been [returned to the annotator or passed to another annotator](project_settings_lse#reject-options). <br/><br/>Whether the task lingers in this state long enough for it to be reflected in the Data Manager depends on your project settings. See [Additional notes](#Additional-notes) below.|
95
95
|**Needs Review**| The required number of annotations have been completed and the task is ready for review. <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below. |
96
96
|**In Review**| First review has been created for any annotation within the task. <br/><br/>Whether the task enters this state depends on your project settings. See [Additional notes](#Additional-notes) below.|
97
97
|**Done**| All required annotations and reviews have been completed. |
Copy file name to clipboardExpand all lines: docs/source/guide/quality.md
+10-2Lines changed: 10 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,15 @@ After you [assign reviewers to tasks](#Assign-reviewers-to-tasks), they can revi
35
35
- If the annotation is correct, click **Accept**.
36
36
37
37
- If the annotation is mostly correct, you can correct it by selecting a different option, changing the selected region, moving the bounding box, or whichever makes sense for the type of label you're reviewing. After correcting the annotation, click **Fix & Accept**.
38
-
- If the annotation is completely incorrect, or you don't want to attempt to correct it at all, click **Reject** to reject the annotation. To place a rejected task back in the Label Stream for annotation, you must delete the annotation. Rejecting an annotation does not return it to annotators to re-label.
38
+
- If the annotation is incorrect, or you don't want to attempt to correct it yourself, click **Reject** to reject the annotation.
39
+
40
+
Clicking **Reject** applies the project's default reject option. If the project has more than one reject option available, you can open the reject menu to choose how to handle the rejection:
41
+
42
+
-**No Rework** -- The annotation is rejected and is not sent back for rework.
43
+
-**Return to Annotator** -- The annotation is sent back to the original annotator so that they can update it.
44
+
-**Pass to Another Annotator** -- The task is sent to a different annotator for a new annotation. The original annotator can no longer update the rejected annotation.
45
+
46
+
Which options you see depends on the project's [**Reject Options**](project_settings_lse#reject-options) settings.
39
47
40
48
3. Continue reviewing annotated tasks until you've reviewed all annotated tasks. Click **Data Manager** to return to the list of tasks for the project.
41
49
@@ -141,7 +149,7 @@ How much access a paused user keeps depends on why they were paused:
141
149
A pause only restricts the type of work that triggered it. Annotation limits and annotator evaluation both measure annotation work, so a flexible pause stops new annotation work only. If a paused user is also a reviewer, they can continue reviewing. Only a full pause stops a user from reviewing.
142
150
143
151
!!! note
144
-
A flexible pause acts as a full pause when the project's [**Reject Options**](project_settings_lse#reject-options)are set to**Remove rejected annotations from labeling queue**. In this case, rejected annotations are never returned to the annotator, so there is nothing for them to recover.
152
+
A flexible pause acts as a full pause when the project's [**Reject Options**](project_settings_lse#reject-options)do not include**Reject and return to annotator**. In this case, rejected annotations are never returned to the annotator, so there is nothing for them to recover.
0 commit comments