Monitoring Annotation Quality

All Sama projects are held to a quality SLA. If a batch of delivered work meets the SLA, the customer approves it. Otherwise, it can be rejected so that tasks can be reworked to fix the mistakes.

To monitor annotation quality, head to the Quality page in the Sama platform. After selecting a project from the project list, navigate to the Quality tab. From here, you can sample delivered tasks and review them.

Follow the instructions below to create a new sample:

• Click on the Create Sample button on the top right of the page.
• In the popup window, set the appropriate filters for the tasks you want to review. For example, you could review tasks which have been delivered in the past five days.
• Click Next
• Select whether to review a percentage or a fixed number of tasks out of the ones returned by your filters. 
• The count of tasks returned by the filters is shown below the sampling strategy options. Some tasks may have been excluded because they are currently getting reviewed in an in-progress (not yet finalized) sample.
• Optionally, you can create multiple samples from the tasks returned. You can either select to create one per last submitter of the tasks, or create a fixed number of samples and distribute the tasks equally within them.
• Click Create Sample

The new sample will now appear highlighted at the top of the sample list. The row contains information about the creation date, sample creator, sample ID, sample size, review progress, average quality score, filters used during creation, and number of pieces of feedback given.

Samples created on tasks in the Completed or Approved states are considered Internal samples, while those created on Delivered or Acknowledged states are considered External. Learn more about filtering for specific samples in the next section.

To find a sample previously created, you can toggle the available filters by clicking on the Filter icon next to the Create sample icon. Enter the appropriate filters to find the sample you are looking for.

To review tasks in a sample, open the sample menu by clicking on the three-dots to the right of the appropriate row in the table. Click on Review. This will open the sampling workspace, where you can enter quality feedback for specific tasks. Below is the flow to give feedback on an annotation.

1. To provide feedback on a specific shape, select the shape from the list and click on the Flag icon.
2. After clicking Flag icon, the feedback panel will open and allow you to select the error committed by the annotation team.
   
   
   1. The error panel allows you to select the Error type, the start, and end frames of the error, and add any comments for the error.
   2. This panel can also be opened by:
      1. Selecting a shape and right-clicking on the shape in the canvas to open the context menu, then clicking “Add feedback”.
      2. Selecting a frame range in the timeline, right-clicking to open the context menu, and clicking “Add feedback”.
   3. Feedback on missing shapes can be added by right-clicking on the canvas to flag that a shape is missing
3. To provide feedback at the output level, click the thumbs down flag and select an error tag.
4. Add a comment at the bottom of the workspace for more details on the overall output. This does not flow to our reporting and is only for instructional guidance. To flag errors, you must use the feedback tools outlined in points 2 and 3 above.
5. Navigate to the next task by using the navigation in the top-center or bottom-center of the workspace.
6. When finished, click Exit at the top left to navigate back to the Quality page.

Note that if a sample has already been finalized, then you cannot edit feedback for a task.

When you are finished with a review of a sample, you should finalize your review. This will ensure that no one (including yourself) can edit your feedback, and will also send the quality data to Sama's data warehouse for review. Click on the three-dots to the right of the sample. Click on Finalize. On the confirmation screen, click Finalize.

After finalization, a green badge will appear next to the kebab menu on the specific sample. This will let Sama's team that you are finished your review so that they can look at the quality feedback given.

Be careful! This cannot be undone. You cannot modify the quality feedback you gave on specific tasks after finalizing a sample and will not be able to delete the sample after finalization.

The detailed quality data for a finalized sample can be seen in Sama's reporting infrastructure. Once Sama's teams have analyzed the quality data, we can decide to rework a sample if needed. It is only possible to rework a sample that is in the finalized state. To do so, click on the three-dots to the right of the sample. Click on Rework. On the confirmation screen, you can select whether this delivery has passed or failed the quality SLA. If it has passed the SLA, then reworking will only reject tasks which have feedback given. If it has failed, then all tasks with feedback and all unreviewed tasks will be rejected to the annotation queue (perfect tasks will not).

Only internal Sama users can rework a sample. Be careful! This cannot be undone.

Deleting a sample

To delete a sample, open the sample menu by clicking on the three-dots to the left of the sample row in the table. Click on Delete. A confirmation popup will appear asking to confirm the deletion. When a sample is deleted, all quality feedback entered will be deleted, and the tasks which were part of that sample will be available to sample again.

Note that samples which were finalized or reworked cannot be deleted. Be careful! This action cannot be undone.