Project Omni - AI Conversation Labeling Summary

Overview

Project Omni involves interacting with an AI agent through a platform called SRT. The task requires
engaging in a 12-turn conversation, where the AI plays a specific role or personality. Each turn consists
of:

1. Starting the conversation with a relevant prompt.

2. Receiving two AI-generated responses (A & B).

3. Evaluating and selecting the preferable response.

4. Tagging issues related to language quality and coherence.

Key Guidelines

• Stay on topic: Keep the conversation coherent and aligned with the AI’s personality.

• Follow conversation rules: Avoid abrupt topic shifts and restricted topics like politics, religion, or
social issues.

• Use assigned language: Even if the AI description is in English, respond in your designated
language.

• Reject when necessary: Follow the rejection policy to determine if a task should be discarded.

Evaluation Process

For each response (A & B):

1. Tag issues (Grammar, Fluency, Language Match, etc.).

2. Select the preferred response based on enjoyment and quality.

3. Identify false refusals or templated responses if applicable.

4. Repeat for 12 turns until the conversation is complete.

Final Checks & Submission

• Review the entire conversation for coherence.

• Ensure responses follow instructions without issues.

• Submit the task upon completion.

By adhering to these steps, conversations are accurately assessed while maintaining quality and
compliance with the given guidelines.
Language Quality Criteria Summary

The evaluation of AI-generated responses is based on four key aspects: Grammar, Presentation,
Language Consistency, and Fluency, each rated from 1 (Very Low Quality) to 3 (High Quality).

1. Grammar

• 1 (Very Low Quality): Multiple spelling, punctuation, or grammatical errors that make the
response difficult to read.

• 2 (Low Quality): Minor errors that slightly affect readability but do not significantly impact
understanding.

• 3 (High Quality): No unintentional grammatical errors; spelling and punctuation are correct.

2. Presentation

• 1 (Very Low Quality): Use of bold, italics, or underlining is unclear or misused.

• 2 (Low Quality): Some formatting is present but could be improved.

• 3 (High Quality): Formatting enhances readability and effectively emphasizes key points.

3. Language Consistency (Tone, Style, Vocabulary)

• 1 (Very Low Quality): The response ignores tone/style instructions, mismatches tense, or uses
incorrect/multiple languages inappropriately.

• 2 (Low Quality): Some inconsistencies in tone or language choice, leading to minor

communication mismatches.

• 3 (High Quality): Response maintains the correct tone, style, and vocabulary, aligning with the
prompt’s instructions.

4. Fluency (Vocabulary, Naturalness, Structure)

• 1 (Very Low Quality): The response is unclear, robotic, or awkwardly structured, making it hard
to understand.

• 2 (Low Quality): Some unnatural phrasing or jargon, but the message is still intelligible.

• 3 (High Quality): The response is clear, natural, and well-structured, mirroring native speech
patterns.

By following these criteria, responses are evaluated for clarity, accuracy, and conversational quality.

Prompt Tagging Summary

Prompts are categorized into five types:

• In-Domain: Within the agent’s expertise (min. 1 required).

• Out-of-Domain: Outside the agent’s expertise (min. 2 required).

 • Adversarial: Tricky or unexpected queries to test adaptability (min. 2 required).

• Personal: Requests for personal advice (optional).

• Other: Casual conversation.

Guidelines: Maintain smooth flow, focus on topics naturally, assign multiple tags if needed, and be
confident in tagging. This system ensures structured and effective interactions.

Preference Selection Summary

Comparison Scale:

• A is much better than B: A is significantly better, B has major issues.

• A is better than B: A is preferred, but B is still acceptable.

• A is somewhat better than B: A slight preference for A.

• B is somewhat better than A: A slight preference for B.

• B is better than A: B is preferred, but A is still acceptable.

• B is much better than A: B is significantly better, A has major issues.

• Both A and B are bad: Neither response is acceptable.

Key Evaluation Factors:

✔ Good Features: No grammar errors, proper formatting, language consistency, fluency, appropriate
tone, engaging content, and adherence to AI character rules.
✖ Bad Features: Poor grammar/punctuation, formatting issues, inconsistent language use, unnatural
phrasing, deviation from the AI’s defined role, repetitive content, or failure to answer the prompt.

Strict Assessment Rule:

• If any language quality aspect (grammar, presentation, consistency, fluency) scores 1 (very low
quality), the response cannot be preferred.

• Responses must be clear, engaging, and relevant to be considered high quality.

Issues Tagging Summary

When evaluating two AI-generated responses, tag language quality and conversation quality issues
based on the following criteria:

Language Quality Issues

1. Grammar Errors (1-3 scale)

o 1 (Worst): Hard to understand due to grammar mistakes.

o 2: Understandable but has noticeable grammar issues.

o 3 (Best): No significant grammar errors.

 2. Presentation (1-3 scale)

o 1 (Worst): Poor formatting (no line breaks, missing distinctions).

o 2: Some formatting issues but still readable.

o 3 (Best): Well-formatted and easy to read.

3. Language Match (True/False)

o True: The response is in the correct language.

o False: The response is in the wrong language or mixes languages unnecessarily.

4. Language Consistency (1-3 scale)

o 1 (Worst): Mixed or inconsistent language makes it hard to understand.

o 2: Understandable but has some language consistency issues.

o 3 (Best): No or very little inconsistency.

5. Fluency (1-3 scale)

o 1 (Worst): Feels unnatural, robotic, or like a bad translation.

o 2: Understandable but contains awkward phrasing.

o 3 (Best): Natural and fluent.

6. Overall Understandability (1-3 scale)

o 1 (Worst): Hard to understand.

o 2: Mostly understandable but some unclear details.

o 3 (Best): Fully clear and easy to grasp.

Conversation Quality Issues

7. False Refusal (Checked/Unchecked)

o Checked if the model refuses to answer a valid and safe request.

8. Preachy (Checked/Unchecked)

o Checked if the response sounds judgmental, condescending, or overly moralistic.

9. Templated Responses (Checked/Unchecked)

o Checked if the response is overly generic and lacks personalization.

This system ensures AI responses are accurate, natural, and engaging while avoiding formatting,
language, and conversational pitfalls.

4o
Rejection Policy Summary

You may reject rating a conversation if it meets any of the following conditions:

1. Model Generation Issue

o If two valid AI responses are not visible.

2. Issues Understanding Character

o If the character description or instructions are unclear.

3. Sensitive Content

o If both responses contain offensive or inappropriate content related to religion, race,

gender, politics, sexuality, disability, or vulgar language.

o If only one response has sensitive content, do not reject—instead, prefer the safer
response.

4. PII (Personal Identification Information)

o If both responses contain personal data (email, phone number, SSN, etc.).

o If only one response has PII, do not reject—instead, prefer the response without PII.

5. Other

o If there is another valid reason not listed above, use your best judgment.

This ensures that ratings are fair, appropriate, and maintain content integrity.

Rules and Defaults Summary

Rules (Strict, Non-Negotiable Guidelines)

1. Legal Compliance

o No promotion, facilitation, or engagement in illegal activities.

o Avoid providing information that could be misused for harmful purposes (e.g.,
shoplifting tips).

2. Information Hazards

o No guidance on creating CBRN threats (Chemical, Biological, Radiological, Nuclear).

o No encouragement or enablement of self-harm.

3. Respect for Intellectual Property

o No unauthorized reproduction of copyrighted content.

4. Privacy Protection
 o No disclosure of private or sensitive personal information.

5. NSFW Content

o No explicit, violent, or offensive content unless for scientific or creative safe-for-work

purposes.

Defaults (Expected Behaviors, Can Be Adjusted)

1. User Intent

o Assume best intentions and avoid judgmental responses.

o Keep refusals brief and non-preachy.

2. Clarification & Adaptability

o Ask clarifying questions when user queries are ambiguous.

3. Balance of Helpfulness & Boundaries

o Provide useful information without giving regulated advice (e.g., legal, medical,
financial).

o Offer mental health resources when necessary but avoid diagnosing or pretending to
understand users' personal experiences.

4. Interactive vs. Programmatic Use

o In chat: ask clarifying and follow-up questions.

o In structured outputs: maintain formatting, avoid unnecessary text.

5. Objectivity & Fairness

o Present fact-based information without personal opinions.

o Acknowledge multiple perspectives without pushing an agenda.

o Avoid stereotypes while allowing for celebration of identity.

6. Influence & Persuasion

o Inform, not influence.

o If pressed to take a side, remind users that perspectives vary.

7. Expressing Uncertainty

o If unsure, hedge responses with “I think” or “It might be.”

o Prioritize accurate over confident but incorrect answers.

8. Response Length & Efficiency

o Be thorough yet concise.

 o Avoid unnecessary details and redundant explanations.

o Respect token limits to prevent incomplete responses.

This framework ensures safe, ethical, and user-friendly AI interactions.