This section provides step-by-step guides showing exactly how to configure different types of annotation projects. Follow these walkthroughs to understand what the platform can do and how to reproduce these examples yourself.
🚀 30-60 minutes → first project → first results
Choose an example below and follow the instructions.
Choose your example
📸 Query selection for image annotation
⏱️ 30-60 min | 💰 ~$3-4 per task| 👥 3000+ experts
Experts evaluate two search queries and select the best match for each image
🔍 Search query generation
⏱️ 30-60 min | 💰 ~$3-4 per task| 👥 3000+ experts
Generate 2 distinct search queries for each image
🖼️ Image classification
⏱️ 30-60 min | 💰 ~$3-4 per task| 👥 3000+ experts
Classify images into predefined categories
🏞 Images comparison
⏱️ 30-60 min | 💰 ~$1 per task| 👥 3400+ experts available
Evaluate two suggested images and select the one that best matches specific criteria
Use Case: Query selection for image annotation
Experts evaluate two suggested search queries for each image and select the one that best matches the image. The selected query should be optimized for searching within a limited dataset with diverse images of similar types but no exact duplicates.
⏱️ 30-60 min | 💰 ~$3-4 per task | 👥 3400+ experts available
Step 1: Prepare your dataset
Actions:
- Create a JSON file with images, metadata, and two suggested queries per image.
- Save as
images_with_queries.json. - Ensure image URLs are publicly accessible.
Or use example files:
Your dataset structure (images_with_queries.json)
[
{
"image_id": "IMG001",
"image_link": "https://example.com/images/chart_bullying_2015.jpg",
"author": "Research Team",
"date_created": "2015-03-15",
"title": "Survey Data on Bullying",
"suggested_query_1": "Find a chart of children being bullied in school in 2015",
"suggested_query_2": "Graph showing bullying prevalence by age group in 2015"
},
{
"image_id": "IMG002",
"image_link": "https://example.com/images/diagram_process_flow.png",
"author": "Design Team",
"date_created": "2023-06-20",
"title": "Customer Onboarding Process",
"suggested_query_1": "Display the workflow diagram showing the customer onboarding process",
"suggested_query_2": "Show me a process flow chart"
}
]
✅ Check: Valid JSON, all fields filled, image URLs accessible
Step 2: Create project and describe your task
Step 2: Create project and describe your task
Actions:
- Log into the platform.
- Create Project Space (e.g., "My First Projects").
-
Click "Create New Project" to Create Project within a Project Space.
- Describe your task to the agent.
What to type:
I need experts to evaluate two suggested search queries for each image and select the one that best matches the image. The dataset contains diverse images of similar types and topics, but no exact duplicates. The selected query should be optimized for searching within this limited dataset. Selection criteria: - Relevant context within dataset (sufficient detail to distinguish from similar images, but not overly specific) - Natural and grammatically correct phrasing (like a realistic user request) - Avoid vagueness (not too broad) - Avoid overspecificity (not excessive details) - Dataset-aware specificity (reasonable detail is enough for accurate retrieval) For each image, experts should select one query and provide a brief reason for their selection.
Or use description from: Project Description File
On the right side, you'll chat with the AI Assistant. After each step, you'll be invited to Accept or modify the generated results shown in the main window on the left. Once you Accept, you'll see the agent's report on the completed workflow. To view the next step's results, you must also accept the configured content.
What you check: ✅
The project goal mentions SELECTING the best query (not generating).
The annotation task emphasizes evaluation and selection.
Selection criteria include relevance, natural phrasing, appropriate specificity (not too vague or overly detailed).
Reason for selection is mentioned.
What you do next: Accept or Ask the agent to add any specific requirements.
Step 3: Upload your dataset
Step 3: Upload your dataset
Actions:
In the chat, click the attachment icon (📎).
Select your
images_with_queries.jsonfile.Wait for the agent to process it.
What the agent responds:
Confirms file upload.
Shows detected fields and structure.
What you check: ✅
Image links are detected correctly.
Metadata fields are recognized.
Both suggested queries are detected.
All fields are present.
What you do next: Accept or ask the agent to fix.
Step 4: Designing datapoint structure
Step 4: Designing datapoint structure
Actions:
Review agent's proposed structure.
-
Define input entities (what experts see):
image_link(image URL);query1(first suggested query);query2(second suggested query).
-
Define output entities (what experts create):
selected_query(the chosen query);selection_reason(brief explanation).
Link output entities to input entities.
What the agent proposes:
Input/output entity structure based on your dataset.
What you check: ✅
All input entities exist (both queries shown as inputs).
All output entities exist (selected query AND selection reasoning).
Entity types correct (image link as IMAGE type).
Extracted data correct (for input entities, e.g. check if the “image link” is extracted correctly).
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Selection accuracy according to selection criteria.
Reasoning quality.
Dataset-aware selection.
What the agent proposes:
Quality Requirements with pass/fail conditions.
-
Criteria aligned with your selection rules.
What you check: ✅
Quality Requirements cover selection accuracy according to the selecting criteria, reasoning quality.
Verify that quality element rules are realistic and achievable.
Generated quality element clearly show pass/fail conditions.
Criteria align with your selection rules.
What you do next: Accept or ask the agent to fix.
What the agent confirms:
✅ Quality Requirements configured. These requirements will be used by LLM QA to automatically check expert work. Experts must meet these standards for their annotations to pass. Ready to proceed to Expert Guidelines?
What you do next: Accept or ask the agent to fix.
Step 6: Building annotation interface
Step 6: Building annotation interface
Actions:
Create user friendly UI for experts.
What the agent proposes
-
Project UI for experts.
What you check: ✅
The annotation flow is logical: the image appears first, search queries follow, comparison criteria are visible, selection options are named consistently, and expectations for the reasoning field are clearly explained.
Search queries are placed side by side.
What you do next: Accept or ask the agent to fix the UI
Fixing example
Review the updates, then Accept;
-
or continue refining until satisfied.
Step 7: Writing annotator guidelines
Step 7: Writing annotator guidelines
Actions:
-
Write clear instructions for experts:
How to evaluate queries.
Selection criteria.
How to write selection reason.
-
Add examples:
Good selection with explanation.
Bad selection with explanation.
Include edge cases and how to handle them
What the agent generates:
What you check: ✅
Guidelines clearly explain the selection task workflow and match the UI flow.
Selection rules are detailed and match your requirements.
Examples show the evaluation and selection process.
Edge cases are covered.
Quality rubrics are provided.
What you do next: Accept or ask the agent to fix the guideline.
Click “Proceed”.
Step 8: Self-check (test your configuration)
Step 9: Set pricing and launch
Step 9: Set pricing and launch
Actions:
Agent recommends you several options with suitable experts. Choose your audience.
Once the audience chosen, proceed to confirm cost estimates by clicking “Save”.
Select number of tasks, estimate task completion time, set the price for completing one task.
Click "Launch" to start labeling.
Step 10: Monitor results
Step 10: Monitor results
What you see:
Tasks sent for annotation
-
Status for each task:
Finding Expert - the task is available on the tasks marketplace
In Expert Work - an expert is working on the task
Quality Check - LLM QA is checking the task
Ready for Review - the task is waiting for your review
Actions:
Wait (experts are working on your tasks - evaluation and selection takes time). It can takes 30-60 min.
-
Track item statuses in dashboard:
Finding Expert → In Expert Work → Quality Check → Ready for Review → Completed.
Review results with status "Ready for Review".
Check LLM QA evaluations.
Monitor progress metrics.
What you check: ✅
Results are coming in.
Each result has a selected query and reason.
Quality checks are passing.
Step 11: Do user quality calibration
Step 11: Do user quality calibration
Actions:
Review results in "Ready for review" status.
Click on a completed item to review it.
Evaluate each Quality Criterion (Pass/Fail).
-
LLM QA evaluation on quality elements (the naming may vary depending on how you configured):
Selection Accuracy on Selected Query.
Reasoning Quality on Selection Reasoning.
Write reasoning for each evaluation.
-
Compare your results with LLM QA assessments.
Check & Provide Quality Feedback: ✅
Check if a search query is selected (radio group is toggled)
Check if the selection reason is provided and explains the choice
-
Review LLM comments on each quality elements (Selection Accuracy on Selected Query and Reasoning Quality on Selection Reasoning)
In early stages, review as many tasks as you want
The review results will be considered in the next project iteration
-
You'll be asked to leave comments if you disagree with the LLM QA verdict
-
If satisfied Download results as JSON
-
If you want to improve quality, complete review and create new project version, Agent will help you to change the settings based on your quality feedback.
What the agent proposes:
The agent will guide you through the next iteration of your project configuration.
Use Case: search query generation
Experts generate two distinct search queries for each image, focusing on natural and implicit user intent rather than literal visual details. Each query should be grammatically correct, context-aware, and tailored for searching within a curated dataset that contains diverse but similar image types without exact duplicates. The queries must avoid being overly vague or excessively specific, striking a balance that helps identify the image’s purpose or use case.
⏱️ 30-60 min | 💰 ~$3-4 per task | 👥 3400+ experts available
Step 1: Prepare your dataset
Step 1: Prepare your dataset
Actions:
Create a JSON file with your images links.
Save it as
image_links.json.Ensure image URLs are publicly accessible.
Or use example files:
Your dataset structure (image_links.json)
[
{
"id": "KmQpXvLa",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-
images-comparison/pair-0001-img-1.png>"
},
{
"id": "RtYnBwEh",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-
images-comparison/pair-0002-img-1.png>"
},
]
✅ Check: Valid JSON, all fields filled, image URLs accessible.
Step 2: Create project and describe your task
Step 2: Create project and describe your task
Actions:
Log into the platform.
Click "Create new project" to create project within a project space.
Describe your task to the agent What to type:
I need experts to generate two distinct search queries for each image that would realistically help a user find that specific image within a curated dataset. The dataset contains diverse images of similar types and topics (such as dashboards, data visualizations, and analytics interfaces), but no exact duplicates. Each query should be crafted as if a real user were searching for the image, balancing specificity and naturalness. Query requirements: - Natural and grammatically correct – phrased like a realistic user request, not a list of keywords - Implicit rather than literal – describe the image’s purpose, concept, or use-case rather than listing visible UI elements or colors - Dataset-aware specificity – include enough detail to distinguish from similar images in the dataset, but avoid excessive or made-up details - Not too vague – generic terms like “dashboard” or “analytics” alone are insufficient - Not overly specific – avoid describing exact layouts, specific percentages, or granular visual components For each image, experts should provide two queries that approach the image from different angles (e.g., business context vs. user persona), ensuring both would plausibly retrieve the target image from the dataset.
Or use description from: Project description file.
On the right side, you'll chat with the AI Assistant. After each step, you'll be invited to Accept or modify the generated results shown in the main window on the left. Once you Accept, you'll see the agent's report on the completed workflow. To view the next step's results, you must also accept the configured content.
What you check: ✅
The project goal mentions GENERATION of queries.
The annotation task emphasizes query generation.
Inputs and outputs are mentioned.
Deliverables.
Generated queries criteria are listed.
What You Do Next:
Accept or Ask the agent to add any specific requirements.
In this case, since we just need two generated queries, the agent was informed accordingly.
Accept when satisfied.
Step 3: Upload your dataset
Step 3: Upload your dataset
Actions:
In the chat, click the attachment icon (📎).
Select your
image_links.jsonfile.Wait for the agent to process.
What the agent responds:
Confirms file upload
Shows detected fields and structure
What you check: ✅
Image links are detected correctly.
Number of items is correct.
What you do next: Accept or ask the agent to fix.
Step 4: Designing datapoint structure
Step 4: Designing datapoint structure
Actions:
Review agent's proposed structure.
Define input entities (what experts see).
Define output entities (what experts create).
Link output entities to input entities.
What the agent proposes:
Input/output entity structure based on your dataset
What you check: ✅
Check if all input and output entities exist, e.g. image link and two queries.
Check input and output entities types, e.g., “image link” is configured as IMAGE entity type or you can ask why some entities have a certain type.
Check extracted data for input entities, e.g. check if the “image link” is extracted correctly.
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Implicit style, naturalness, specificity balance, and word count of generated queries.
Distinctness between the two queries.
What the agent proposes:
Quality Requirements with pass/fail conditions.
Criteria aligned with your selection rules.
What you check: ✅
Quality Requirements address the criteria for good queries.
Query distinctness is addressed.
Generated quality element clearly show pass/fail conditions.
What the agent confirms:
✅ Quality Requirements configured. These requirements will be used by LLM QA to automatically check expert work. Experts must meet these standards for their annotations to pass. Ready to proceed to Expert Guidelines?
What you do next: Accept or ask the agent to fix.
Step 6: Building annotation interface
Step 6: Building annotation interface
Actions:
Create user friendly UI for experts.
What the agent generates:
Project UI for experts.
What you check: ✅
The workflow is logical: the image appears next to expected search queries, the criteria are visible.
The UI is user-friendly and easy to navigate.
What you do next: Accept or ask the agent to fix the UI.
Fixing Example
-
For example, are all criteria for queries displayed for visibility.
Review options and choose if you agree.
Review the updates, then accept or continue refining until satisfied.
Step 7: Writing annotator guidelines
Step 7: Writing annotator guidelines
Actions:
Write clear instructions for experts.
Add examples.
Include edge cases and how to handle them.
What the agent generates:
What You Check: ✅
Guidelines clearly explain the selection task workflow and match the UI flow.
Quality criteria explained.
Examples are provided.
The guideline sections are logical and easy to understand.
What you do next: Accept or ask the agent to fix the guideline. Click “Proceed”.
Step 8: Self-check (test your configuration)
Step 8: Self-check (test your configuration)
Actions:
Complete at least 1-2 tasks from 10 sample tasks yourself.
Test the interface, guidelines, and quality criteria.
Review LLM QA evaluations.
What You check: ✅
Interface usability.
Task completeness.
LLM QA responses.
Guideline clarity.
What you do:
Click "Proceed" to navigate to the Self-check step.
The system shows you 10 sample tasks from your dataset.
Expand the task to review the UI.
Check that all inputs are extracted correctly in the provided samples: images display properly, fields for queries are shown, etc.
-
Complete the first task yourself:
View the image.
Write queries.
Submit your answer by clicking on the ”Submit to LLM QA”.
LLM QA evaluation will take 30-60 sec.
Verify that the LLM evaluation and comments align with the pass or fail criteria. If not, go to Project Configuration step and improve project settings.
You can download completed test samples with status “Ready for Review” by selecting them.
-
You can test as many cases as you want.
Test very short reasoning that you can limit before LLM QA.
Repeating the query in the reasoning.
What You Do Next: Complete 2-3 more test tasks with different image types to ensure everything works correctly.
If not, go to the Project Configuration step and improve the settings.
If yes, go to the Contract section by clicking “Proceed”
Step 9: Set pricing and launch
Step 9: Set pricing and launch
Actions:
Agent recommends you several options with suitable experts. Choose your audience.
Once the audience chosen, proceed to confirm cost estimates by clicking “Save”.
Select number of tasks, estimate task completion time, set the price for completing one task.
Click "Launch" to start labeling
Step 10: Monitor results
Step 10: Monitor results
What you see:
Tasks sent for annotation.
-
Status for each task:.
Finding Expert - the task is available on the tasks marketplace.
In Expert Work - an expert is working on the task.
Quality Check - LLM QA is checking the task.
Ready for Review - the task is waiting for your review.
Actions:
Wait (experts are working on your tasks - evaluation and selection takes time). It can takes 30-60 min.
-
Track item statuses in dashboard:
Finding Expert → In Expert Work → Quality Check → Ready for Review → Completed
Review results with status "Ready for Review".
Check LLM QA evaluations.
Monitor progress metrics.
What you check: ✅
Results are coming in.
Each result has a selected query and reason.
Quality checks are passing.
Step 11: Do user quality calibration
Step 11: Do user quality calibration
What you do:
Review results in "Ready for Review" status.
-
Evaluate each Quality Criterion (Pass/Fail).
LLM QA evaluation on quality elements (the naming may vary depending on how you configured).
Write reasoning for each evaluation.
Compare your results with LLM QA assessments.
Check & provide quality feedback: ✅
Check if a search query is selected (radio group is toggled).
Check if the selection reason is provided and explains the choice.
-
Review LLM comments on each quality elements.
In early stages, review as many tasks as you want.
The review results will be considered in the next project iteration.
You'll be asked to leave comments if you disagree with the LLM QA verdict.
If satisfied Download results as JSON.
If you want to improve quality, complete review and create new project version, Agent will help you to change the settings based on your quality feedback.
What the agent proposes:
The agent will guide you through the next iteration of your project configuration
Use Case: Image classification
This project classifies images into predefined categories based on their visual intent and communicative purpose. Inputs include image URLs and IDs. The categories include: Graphs and Charts (data visualizations), Diagrams (technical schematics, categorization structures, timelines, product instructions), Data and Process Flows (workflows, pipelines, decision pathways), Infographics (rich visual summaries and dashboards), and Multi-Figure Images (composite panels from reports or studies). Images that do not fit any category are labeled as Uncategorized. The output is a JSON file pairing each image’s link and ID with its assigned category, enabling consistent organization and downstream analysis of visual content.
⏱️ 30-60 min | 💰 ~$3-4 per task | 👥 3400+ experts available
Step 1: Prepare your dataset
Step 1: Prepare your dataset
What you do:
Create a JSON file with your images links.
Save it as
image_links.json.Ensure image URLs are publicly accessible.
Or use example files:
Your dataset structure (image_links.json).
[
{
"id": "KmQpXvLa",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images-
comparison/pair-0001-img-1.png>"
},
{
"id": "RtYnBwEh",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images-
comparison/pair-0002-img-1.png>"
},
]
Step 2: Create project and describe your task
Step 2: Create project and describe your task
Actions:
Log into the platform.
Create Project Space (e.g., "My First Projects").
Click "Create New Project" to Create Project within a Project Space.
Describe your task to the agent What you type:
I need experts to classify a list of images into predefined categories based on their visual content and purpose. The input consists of image links and image IDs. Each image should be analyzed and assigned to exactly one of the following categories: Classification Categories: - Graphs and Charts – Visual representations of data highlighting trends, patterns, and relationships through graphical elements like lines, bars, and pie charts. - Diagrams – Including the following subtypes: * Technical diagrams: Network architecture, system components, and detailed schematic representations of complex systems, equipment, or processes, often used in engineering, manufacturing, and technical fields * Categorization diagrams: Illustrations that organize and classify information into distinct groups or hierarchies, such as concept maps, taxonomies, or organizational structures * Hierarchical and timeline diagrams: Visuals showing relationships, structures, or chronological order, including organizational charts, family trees, and project timelines * Product diagrams/images from user manuals: Illustrations or photos depicting components, features, or usage of a product, typically found in user guides, instruction manuals, or product documentation - Data and Process Flow – Diagrams portraying the movement and transformation of data or the sequence of steps in a workflow or process, including decision trees; commonly used in software engineering, business analysis, and process improvement. - Infographics – Visually engaging graphics combining data, information, and design elements to communicate concepts, trends, or detailed information in a digestible format, including data visualizations and dashboards. - Multi-Figure Images – Composed of multiple related figures, such as side-by-side panels or labeled figures from documentation or reports. - Uncategorized – Any image not matching the above categories. Classification requirements: Mutually exclusive – each image should be assigned to only one category, even if it contains elements from multiple categories; choose the most dominant or primary purpose Content-driven – base the classification on the image’s visual content and intended purpose, not on file names or metadata Subtype-aware – for images classified as “Diagrams,” identify the most appropriate subtype based on the image’s structure and use-case Consistent application – apply the same criteria uniformly across all images to ensure classification consistency When in doubt – if an image does not clearly fit any category, label it as “Uncategorized” rather than forcing a fit Output: For each image, experts should provide a JSON entry containing: Image link Image ID Assigned category (and subtype if applicable for Diagrams
Or use description from: Project description file.
On the right side, you'll chat with the AI Assistant. After each step, you'll be invited to Accept or modify the generated results shown in the main window on the left. Once you Accept, you'll see the agent's report on the completed workflow. To view the next step's results, you must also accept the configured content.
What you check: ✅
The project goal mentions image classification.
Inputs and outputs are mentioned.
Image categories are listed.
Deliverables.
Quality expectations.
What you do next: Accept or ask the agent to add any specific requirements.
Step 3: Upload your dataset
Step 3: Upload your dataset
Actions:
In the chat, click the attachment icon (📎).
Select your
image_links.jsonfile.Wait for the agent to process.
What the agent responds:
Confirms file upload.
Shows detected fields and structure.
What you check: ✅
Image links are detected correctly.
Number of items is correct.
What you do next: Accept or ask the agent to fix.
Step 4: Designing datapoint structure
Step 4: Designing datapoint structure
Actions:
Review agent's proposed structure.
Define input entities (what experts see).
Define output entities (what experts create).
Link output entities to input entities.
What the agent proposes:
Input/output entity structure based on your dataset.
What you check: ✅
Check if all input and output entities exist, e.g. image link and two queries.
Check input and output entities types, e.g., “image link” is configured as IMAGE entity type or you can ask why some entities have a certain type.
Check extracted data for input entities, e.g. check if the “image link” is extracted correctly.
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Classification accuracy.
Reasoning quality.
What the agent proposes:
Quality Requirements with pass/fail conditions.
Criteria aligned with your selection rules.
What you check: ✅
Check if proposed quality criteria are not redundant.
Request the agent to edit criteria.
Quality Requirements address classification accuracy.
Generated quality element clearly show pass/fail conditions.
What you do next: Accept or ask the agent to fix.
What the agent confirms:
✅ Quality Requirements configured. These requirements will be used by LLM QA to automatically check expert work. Experts must meet these standards for their annotations to pass. Ready to proceed to Expert Guidelines?
Step 6: Building annotation interface
Step 6: Building annotation interface
Actions:
Create user friendly UI for experts.
What the agent proposes
-
Project UI for experts.
What you check: ✅
The workflow is logical: the image appears next to the categories options, explanation text field is available, category definition are available.
The UI is user-friendly and easy to navigate.
What you do next: Accept or ask the agent to fix the UI.
Step 7: Writing annotator guidelines
Step 7: Writing annotator guidelines
Actions:
Write clear instructions for experts.
Add examples.
Include edge cases and how to handle them.
What the agent generates:
What you check: ✅
Guidelines clearly explain the selection task workflow and match the UI flow
Category definitions are provided.
Quality criteria explained.
Examples are provided.
The guideline sections are logical and easy to understand.
What you do next: Accept or ask the agent to fix the guideline. Click “Proceed”.
Step 8: Self-check (test your configuration)
Step 8: Self-check (test your configuration)
Actions:
Complete at least 1-2 tasks from 10 sample tasks yourself.
Test the interface, guidelines, and quality criteria.
Review LLM QA evaluations.
What you check: ✅
Interface usability.
Task completeness.
LLM QA responses.
Guideline clarity.
What you do:
Click "Proceed" to navigate to the Self-Check step.
The system shows you 10 sample tasks from your dataset.
Expand the task to review the UI.
Check that all inputs are extracted correctly in the provided samples: images display properly, fields for queries are shown, etc.
-
Complete the first task yourself based on the UI:
View the image.
Write queries.
Submit your answer by clicking on the ”Submit to LLM QA”.
LLM QA evaluation will take 30-60 sec.
Verify that the LLM evaluation and comments align with the pass or fail criteria. If not, go to Project Configuration step and improve project settings.
You can download completed test samples with status “Ready for Review” by selecting them.
What you do next: Complete 2-3 more test tasks with different image types to ensure everything works correctly.
If not, go to the Project Configuration step and improve the settings;
If yes, go to the Contract section by clicking “Proceed”.
Step 9: Set pricing and launch
Step 9: Set pricing and launch
Actions:
Agent recommends you several options with suitable experts. Choose your audience.
Once the audience chosen, proceed to confirm cost estimates by clicking “Save”.
Select number of tasks, estimate task completion time, set the price for completing one task.
Click "Launch" to start labeling.
Step 10: Monitor results
Step 10: Monitor results
What you see:
Tasks sent for annotation.
-
Status for each task:
Finding Expert - the task is available on the tasks marketplace.
In Expert Work - an expert is working on the task.
Quality Check - LLM QA is checking the task.
Ready for Review - the task is waiting for your review.
Actions:
Wait (experts are working on your tasks - evaluation and selection takes time). It can takes 30-60 min.
-
Track item statuses in dashboard:
Finding Expert → In Expert Work → Quality Check → Ready for Review → Completed
Review results with status "Ready for Review".
Check LLM QA evaluations.
Monitor progress metrics.
What you check: ✅
Results are coming in.
Each result has a selected query and reason.
Quality checks are passing.
Step 11: Do user quality calibration
Step 11: Do user quality calibration
Actions:
-
Review results in "Ready for Review" status.
Click on a completed item to review it.
-
Evaluate each Quality Criterion (Pass/Fail).
LLM QA evaluation on quality elements (the naming may vary depending on how you configured).
Write reasoning for each evaluation.
Compare your results with LLM QA assessments.
Check & provide quality feedback: ✅
Review LLM comments on each quality elements (Selection Accuracy on Selected Query and Reasoning Quality on Selection Reasoning).
In early stages, review as many tasks as you want.
The review results will be considered in the next project iteration.
You'll be asked to leave comments if you disagree with the LLM QA verdict.
If satisfied Download results as JSON
If you want to improve quality, complete review and create new project version, Agent will help you to change the settings based on your quality feedback
The agent will guide you through the next iteration of your project configuration.
Use Case: Images comparison
Experts evaluate two suggested images and select the one that best matches specific criteria.
⏱️ 30-60 min | 💰 ~$1 per task| 👥 3400+ experts available
Step 1: Prepare your dataset
Step 1: Prepare your dataset
Actions:
Create a JSON file with images, metadata, and two suggested queries per image.
Save as
images_to_compare.json.Ensure image URLs are publicly accessible.
Or use example files:
-
Your dataset structure (
images_to_compare.json).[ { "pair_id": "pair-0001", "image_1": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images- comparison/pair-0001-img-1.png>", "image_2": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images- comparison/pair-0001-img-2.png>" }, { "pair_id": "pair-0002", "image_1": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images- comparison/pair-0002-img-1.png>", "image_2": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images- comparison/pair-0002-img-2.png>" } ]
✅ Check: Valid JSON, all fields filled, image URLs accessible.
Step 2: Create project and describe your task
Step 2: Create project and describe your task
Actions:
Log into the platform.
Create Project Space (e.g., "My First Projects").
-
Click "Create New Project" to Create Project within a Project Space.
Describe your task to the agent.
What to type:
I need experts who can assess and compare two given images based on standardized evaluation metrics. The goal is to measure clarity, design quality, completeness, and overall communication effectiveness. The image receiving the highest total score will be declared the winner. Evaluation Metrics (0–5 points each) 1. Title & Labelling Measures how clear and descriptive the image's title and labels are. - 0-2: Missing title and/or labels - 3-4: Present but unclear/incomplete - 5: Clear, descriptive title and well-labelled components 2. Grammar & Language Assesses spelling, grammar, and professionalism of text in the image. - 0: Multiple grammar/spelling errors - 3-4: Minor errors not hindering understanding - 5: Grammatically correct with professional wording 3. Clarity of Process Depiction Evaluates how clearly the image presents steps or concepts in logical order. - 0–2: Steps unclear or missing - 3–4: Structure mostly clear with minor gaps - 5: Fully clear, chronologically correct, and visually guided 4. Standalone Understanding Judges whether the image can be understood without external explanations. - 0–2: Needs heavy external context - 3–4: Understandable with some background knowledge - 5: Understandable to a layperson 5. Visual Design & Readability Considers layout, color contrast, and visual appeal. - 0-2: Poor design and color use - 3-4: Readable but slightly messy or inconsistent - 5: Clean, appealing, and visually consistent 6. Completeness & Key Content Coverage Looks at whether all major elements are included. - 0–2: Many missing elements - 3–4: Most concepts covered but incomplete - 5: All major items included accurately Scoring System - Each metric scores 0–5 points - Maximum possible score per image: 30 points - The image with the highest score wins - Tiebreaker: Overall clarity and/or human preference The experts evaluate both images against all metrics and record the scores in a comparison table. Calculate totals to determine the winner, then assign a descriptive title to the winning image.
Or use description from: Project Description File.
On the right side, you'll chat with the AI Assistant. After each step, you'll be invited to Accept or modify the generated results shown in the main window on the left. Once you Accept, you'll see the agent's report on the completed workflow. To view the next step's results, you must also accept the configured content.
What you check: ✅
The project goal mentions SELECTING the best image.
The annotation task emphasizes evaluation and selection.
Evaluation criteria are mentioned.
Scoring system is clearly explained.
What you do next: Accept or Ask the agent to add any specific requirements.
Step 3: Upload your dataset
Step 3: Upload your dataset
Actions:
In the chat, click the attachment icon (📎).
-
Select your
images_to_compare.jsonfile. Wait for the agent to process it.
What the agent responds:
Confirms file upload.
Shows detected fields and structure.
What you check: ✅
Images links are detected correctly.
What you do next: Accept or ask the agent to fix.
Step 4: Designing datapoint structure
Step 4: Designing datapoint structure
Actions:
Review agent's proposed structure.
Define input entities (what experts see).
Define output entities (what experts create).
What the agent proposes:
Input/output entity structure based on your dataset.
What you check: ✅
All input entities exist (both queries shown as inputs).
All output entities exist (selected query AND selection reasoning).
Entity types correct (image link as IMAGE type).
Extracted data correct (for input entities, e.g. check if the “image link” is extracted correctly).
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Evaluation metrics.
Reasoning quality.
What the agent proposes:
Quality requirements with pass/fail conditions.
Criteria aligned with your selection rules.
What you check: ✅
Quality Requirements cover all metrics.
Request the agent to edit criteria.
Verify that quality element rules are realistic and achievable.
Generated quality element clearly shows pass/fail conditions.
Criteria align with your selection rules.
What you do next: Accept or ask the agent to fix.
What the agent confirms:
🔎 **Review the criteria** to verify pass/fail thresholds align with your evaluation standards. Approve to continue → **Building annotation interface** — or suggest changes
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Evaluation metrics.
Reasoning quality.
What the agent proposes:
Quality requirements with pass/fail conditions.
Criteria aligned with your selection rules.
What you check: ✅
Quality Requirements cover all metrics.
Request the agent to edit criteria.
Verify that quality element rules are realistic and achievable.
Generated quality element clearly shows pass/fail conditions.
Criteria align with your selection rules.
What you do next: Accept or ask the agent to fix.
What the agent confirms:
🔎 **Review the criteria** to verify pass/fail thresholds align with your evaluation standards. Approve to continue → **Building annotation interface** — or suggest changes
What you do next: Accept or ask the agent to fix.
Step 7: Writing annotator guidelines
Step 7: Writing annotator guidelines
Actions:
-
Write clear instructions for experts:
Workflow description
Evaluation metrics and scoring rubrics
Reasoning expectations
-
Add examples:
Pass example
Fail example
What the agent generates:
What you check: ✅
Guidelines clearly explain the selection task workflow and match the UI flow.
Evaluation metrics and scoring rubrics are explained in detail.
Examples show the evaluation and selection process.
Quality rubrics are provided.
What you do next: Accept or ask the agent to fix the guideline. Click “Proceed”.
Step 8: Self-check (test your configuration)
Step 8: Self-check (test your configuration)
Actions:
Complete at least 1-2 tasks from 10 sample tasks yourself.
Test the interface, guidelines, and quality criteria.
Review LLM QA evaluations.
What you check: ✅
Interface usability.
Task completeness.
LLM QA responses.
Guideline clarity.
What you do:
Click "Proceed" to navigate to the Self-Check step.
The system shows you 10 sample tasks from your dataset.
Expand the task to review the UI.
Check that all inputs are extracted correctly in the provided samples: images display properly, evaluation metric fields are available, etc.
-
Complete the first task yourself:
Submit your answer by clicking on the ”Submit to LLM QA”.
LLM QA evaluation will take 30-60 sec.
Verify that the LLM evaluation and comments align with the pass or fail criteria. If not, go to Project Configuration step and improve project settings.
You can download completed test samples with status “Ready for Review” by selecting them.
You can test as many cases as you want.
What you do next: Complete 2-3 more test tasks with different image types to ensure everything works correctly.
If not, go to the Project Configuration step and improve the settings.
If yes, go to the Contract section by clicking “Proceed”.
Step 9: Set pricing and launch
Step 9: Set pricing and launch
Actions:
Agent recommends you several options with suitable experts. Choose your audience.
Once the audience chosen, proceed to confirm cost estimates by clicking “Save”.
Select number of tasks, estimate task completion time, set the price for completing one task.
Click "Launch" to start labeling.
Step 10: Monitor results
Step 10: Monitor results
What you see:
Tasks sent for annotation.
-
Status for each task:
Finding Expert - the task is available on the tasks marketplace.
In Expert Work - an expert is working on the task.
Quality Check - LLM QA is checking the task.
Ready for Review - the task is waiting for your review.
Actions:
Wait (experts are working on your tasks - evaluation and selection takes time). It can takes 30-60 min.
-
Track item statuses in dashboard:
Finding Expert → In Expert Work → Quality Check → Ready for Review → Completed
Review results with status "Ready for Review".
Check LLM QA evaluations.
Monitor progress metrics.
What you check: ✅
Results are coming in.
Each result has a selected image and reason.
Quality checks are passing.
Step 11: Do user quality calibration
Step 11: Do user quality calibration
Actions:
-
Review results in "Ready for Review" status.
-
Click on a completed item to review it.
-
-
Evaluate each Quality Criterion (Pass/Fail).
LLM QA evaluation on quality elements (the naming may vary depending on how you configured).
Write reasoning for each evaluation.
Compare your results with LLM QA assessments.
Check & Provide Quality Feedback: ✅
Check if a search query is selected (radio group is toggled).
Check if the selection reason is provided and explains the choice.
Review LLM comments on each quality elements (Selection Accuracy on Selected Query and Reasoning Quality on Selection Reasoning).
In early stages, review as many tasks as you want.
The review results will be considered in the next project iteration.
You'll be asked to leave comments if you disagree with the LLM QA verdict.
If satisfied Download results as JSON.
-
If you want to improve quality, complete review and create new project version, Agent will help you to change the settings based on your quality feedback.
The agent will guide you through the next iteration of your project configuration.
Use Case: Image-based question and answer generation
Experts analyze provided images and create one question-answer pair per image. Each question is categorized by type: Lookup (finding visible information), Summarization (describing content), or Analytical (reasoning/calculation based on image data). The goal is to generate diverse, high-quality Q&A pairs that test different levels of image understanding.
⏱️ 30-60 min | 💰 ~$3 per task| 👥 3400+ experts available
Step 1: Prepare your dataset
Step 1: Prepare your dataset
Actions:
Create a JSON file with images, metadata, and two suggested queries per image.
Save as
image_links.json.Ensure image URLs are publicly accessible.
Or use example files:
Your Dataset Structure (image_links.json)
[
{
"id": "KmQpXvLa",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images
-comparison/pair-0001-img-1.png>"
},
{
"id": "RtYnBwEh",
"link": "<https://prodtlkcsafiles.blob.core.windows.net/self-service-images
-comparison/pair-0002-img-1.png>"
}
]
✅ Check: Valid JSON, all fields filled, image URLs accessible
Step 2: Create project and describe your task
Step 2: Create project and describe your task
Actions:
Log into the platform.
Create Project Space (e.g., "My First Projects").
Click "Create New Project" to Create Project within a Project Space.
Describe your task to the agent What you type:
I need experts to create high-quality question-answer pairs based on provided images. For each image link, experts should generate one question and its corresponding answer. For each image, I need experts to provide: 1. Question Type (select one): - Lookup – A question that requires finding specific, directly visible information in the image - Summarization – A question that requires describing, interpreting, or summarizing the image content - Analytical/Reasoning – A question that requires calculation, inference, or logical reasoning based on image data 2. Question – A clear, well-formed question about the image 3. Answer – The accurate answer to the question
Or use description from: Project description file.
On the right side, you'll chat with the AI Assistant. After each step, you'll be invited to Accept or modify the generated results shown in the main window on the left. Once you Accept, you'll see the agent's report on the completed workflow. To view the next step's results, you must also accept the configured content.
What you check: ✅
The project goal mentions question and answer generation.
Inputs and outputs are mentioned.
Question categories are listed.
Quality expectations are provided.
What you do next: Accept or Ask the agent to add any specific requirements.
Step 3: Upload your dataset
Step 3: Upload your dataset
Actions:
In the chat, click the attachment icon (📎).
Select your
image_links.jsonfile.Wait for the agent to process.
What the agent responds:
Confirms file upload.
Shows detected fields and structure.
What you check: ✅
Image links are detected correctly
Number of items is correct
What You Do Next: Accept or ask the agent to fix
Step 4: Designing datapoint structure
Step 4: Designing datapoint structure
Actions:
Review agent's proposed structure.
Define input entities (what experts see).
Define output entities (what experts create).
Link output entities to input entities.
What the agent proposes:
Input/output entity structure based on your dataset.
What you check: ✅
-
Check if all input and output entities exist.
-
Check if you need an explanation output.
-
Check input and output entities types, e.g., “image link” is configured as IMAGE entity type or you can ask why some entities have a certain type.
You may ask about entity types for your outputs and adjust them.
-
Set word limits for outputs.
-
Check extracted data for input entities, e.g. check if the “image link” is extracted correctly.
What you do next: Accept or ask the agent to fix.
Step 5: Defining quality criteria
Step 5: Defining quality criteria
Actions:
Review agent's proposed quality criteria.
-
Verify criteria cover:
Question type classification accuracy.
Question and answer relevance.
Question and answer clarity.
What the agent proposes:
Quality Requirements with pass/fail conditions.
Criteria aligned with your generation rules.
What you check: ✅
Check if proposed quality criteria are not redundant.
Request the agent to edit criteria if necessary.
Quality Requirements address classification accuracy.
Generated quality element clearly show pass/fail conditions.
What You Do Next: Accept or ask the agent to fix.
What the agent confirms:
✅ Quality Requirements configured. These requirements will be used by LLM QA to automatically check expert work. Experts must meet these standards for their annotations to pass. Ready to proceed to Expert UI?
Step 6: Building annotation interface
Step 6: Building annotation interface
Actions:
Create user friendly UI for experts.
What the agent proposes
Project UI for experts.
What you check: ✅
The workflow is logical: the image appears next to the question and answer fields.
The UI is user-friendly and easy to navigate.
What you do next: Accept or ask the agent to fix the UI.
Step 7: Writing annotator guidelines
Step 7: Writing annotator guidelines
Actions:
-
Write clear instructions for experts:
Workflow description.
Evaluation metrics and scoring rubrics.
Reasoning expectations.
-
Add examples:
Pass example.
Fail example.
What the agent generates:
What you check: ✅
Guidelines clearly explain the selection task workflow and match the UI flow.
Question types are listed.
Examples show the evaluation and selection process.
Quality rubrics are provided.
What you do next: Accept or ask the agent to fix the guideline. Click “Proceed”.
Step 8: Self-check (test your configuration)
Step 8: Self-check (test your configuration)
Actions:
Complete at least 1-2 tasks tasks yourself.
Test the interface, guidelines, and quality criteria.
Review LLM QA evaluations.
What you check: ✅
Interface usability.
Task completeness.
LLM QA responses.
Guideline clarity.
What you do:
Click "Proceed" to navigate to the Self-check step.
-
The system shows you 10 sample tasks from your dataset.
Open the task to review the UI.
Check that all inputs are extracted correctly in the provided samples: image displays properly, fields for question and answer are shown, etc.
-
Complete the first task yourself based on the UI:
View the image.
Write question and answer.
Submit your answer by clicking on the ”Submit to LLM QA”.
LLM QA evaluation will take 30-60 sec.
-
Verify that the LLM evaluation and comments align with the pass or fail criteria. If not, go to Project Configuration step and improve project settings.
You can download completed test samples with status “Ready for Review” by selecting them.
What you do next: Complete 2-3 more test tasks with different image types to ensure everything works correctly.
If not, go to the Project Configuration step and improve the settings.
If yes, go to the Contract section by clicking “Proceed”.
Step 9: Set pricing and launch
Step 9: Set pricing and launch
Actions:
Agent recommends you several options with suitable experts. Choose your audience.
Once the audience chosen, proceed to confirm cost estimates by clicking “Save”.
Select number of tasks, estimate task completion time, set the price for completing one task.
Click "Launch" to start labeling.
Step 10: Monitor results
Step 10: Monitor results
What you see:
Tasks sent for annotation.
-
Status for each task:
Finding Expert - the task is available on the tasks marketplace.
In Expert Work - an expert is working on the task.
Quality Check - LLM QA is checking the task.
Ready for Review - the task is waiting for your review.
Actions:
Wait (experts are working on your tasks - evaluation and selection takes time). It can takes 30-60 min.
-
Track item statuses in dashboard:
Finding Expert → In Expert Work → Quality Check → Ready for Review → Completed
Review results with status "Ready for Review".
Check LLM QA evaluations.
Monitor progress metrics.
What you check: ✅
Results are coming in.
Each result has a selected query and reason.
Quality checks are passing.
Step 11: Do user quality calibration
Step 11: Do user quality calibration
Actions:
-
Review results in "Ready for Review" status.
Click on a completed item to review it.
-
Evaluate each Quality Criterion (Pass/Fail).
LLM QA evaluation on quality elements (the naming may vary depending on how you configured).
Write reasoning for each evaluation.
Compare your results with LLM QA assessments.
Check & provide quality feedback: ✅
Review LLM comments on each quality elements (Selection Accuracy on Selected Query and Reasoning Quality on Selection Reasoning).
In early stages, review as many tasks as you want.
The review results will be considered in the next project iteration.
You'll be asked to leave comments if you disagree with the LLM QA verdict.
If satisfied Download results as JSON.
If you want to improve quality, complete review and create new project version, Agent will help you to change the settings based on your quality feedback.
The agent will guide you through the next iteration of your project configuration.
Best practices
Start small: Test with 10-20 images before scaling to thousands.
Do self check.
Be specific about selection criteria: Clearly explain to the agent what makes your data "best" (relevant context, natural language, appropriate specificity).
Provide example selections: Share examples of good/bad results with reasons to help the agent understand your expectations.
Review early results: Pay special attention in early results to ensure selections follow your rules (not too vague, not too specific).
Iterate on quality requirements: Use Quality Feedback to refine project settings.