To access the Sample page, go to Projects > [your project] > Sample.

On this page, you can conduct a quality management process called sampling. During this process, you can create a "lot", which is a batch of tasks that you want to review as a group. You can then extract a "sample" from that lot, which simply a small subset of all the tasks in the lot, and review them.

If the tasks in the sample look good, you can approve the entire sample. However, if you find that the tasks in the sample aren't up to scratch, you can reject the entire sample and send it back to the annotator for rework, along with feedback about how they can improve.

Creating a lot

Before you can generate a sample, you must first create a lot. A lot is essentially just the pool of tasks from which you'll eventually pull out your sample.

Lots can either be created by using filters to query all the tasks in the system and include only the tasks that meet certain criteria, or by uploading a CSV to include only the tasks with specified task IDs.

Creating a lot using filters

Using filters can be a good way to search for all the tasks in the system that meet certain criteria. For example, you might want to look at just the tasks that were completed by a new annotator within the last week.

To create a lot using filters:

1. Configure the filters to specify which tasks to include in the lot. Only the tasks that meet ALL the criteria will be included. For descriptions of each field, see the Filter fields section below.
2. Click Create lot. The new lot will appear in the table and will show the filters that were used to create it in the Filters column.

Filter fields

Each of the filter fields is described in detail in the table below.

Creating a lot by CSV upload

A CSV upload can be the way to go if you already know exactly which tasks you want to include in a lot. Additionally, if you would like to create a lot from a wide variety of tasks that don't share common characteristics, you may not be able to configure filters that would return those tasks.

To create a lot by CSV upload:

1. Prepare a CSV file that has one row per task, with only a single column named "task_id" that contains the task ID of each task.
   
   

   🚧 Warning

   If your CSV file contains tasks that cannot be added to a lot, such as In Progress tasks (which are not yet available for review) or Rejected tasks (which have automatically been returned to the queue and are unavailable for review), those tasks will be ignored.

    

2. Click on the arrow next to Create lot and click Create lot by CSV upload.

3. In the Create lot window that appears, click Browse and select your prepared CSV file.
4. Click Create lot. The new lot will appear in the table and will show CSV in the Filters column, to indicate that it was created by a CSV upload instead of filters.

Sampling a lot

From the lot that you created, you will now need to sample a certain percentage of those tasks to get an overview of the quality of the tasks within the lot. For example, if you have 100 samples, you probably don’t want to have to comb through each one individually — looking at just 5% of those tasks could likely give you a good idea of whether the tasks were done well or not.

To sample a lot:

1. In the table, on the right of a row for a lot, click Sample.

2. In the Sample window that appears, configure the settings as desired. For details about each field, see the table below.

3. Click Sample.

The sample will appear as the next row in the table.

Filtering the samples in the table

Several options are available to filter the samples in the table. You can search for a particular sample by its sample ID, or show samples based on who originally created them.

To filter the samples:

1. Click the filter icon at the top right of the table.
2. Configure the filter options as follows:
   • Sample ID: Returns the sample with the specified sample ID.
   • Created by: Me: Returns all samples that were created by you.
   • Created by: My Team: Returns all the samples that were created by your team.
3. Click View.

Viewing and/or reviewing a sample

You can view the tasks in a sample, as well as review each task if desired. By reviewing a task, you can approve or reject it, as well as leave a variety of feedback for the annotators if they need to re-annotate unsatisfactory work.

To view and/or review a sample:

1. In the row for a sample, click View.

2. Review each task by performing one or more of the following actions:
   a. Approve or reject the entire task by clicking Approve or Reject in the When finalized field.
   b. Assign a score to the task by entering a percentage in the Score field.
   c. Provide feedback on individual sections by clicking the thumbs up or thumbs down buttons.
   d. Leave comments in the Comment field.

3. If needed, click Previous or Next to navigate to another task in the sample.
4. Click Exit when done.

Finalizing a lot

Finalizing a lot can be thought of as “submitting” the changes that you made by marking tasks as Approved or Rejected. Once a lot is finalized, the tasks that you marked as Rejected will actually be rejected and sent back to Step A to be re-annotated. The tasks that you marked as Approved will have their statuses changed to Approved.

To finalize a lot:

1. In the row for a sample, click the arrow beside the View button and click Finalize.

2. In the Finalize lot window that appears, you may be prompted about what to do if tasks in the sample were neither approved or rejected.

3. Click Finalize to confirm.