Azure OpenAI Service

Intelligent Apps Workshop

Plan for Today
Time (CEST) Topic

09:00 – 09:15 Introductions & Setting Expectations

09:15 – 09:30 Use Case Ideation & Brainstorming

09:30 – 10:30 Intro to Azure OpenAI, Prompt Engineering & Demos

10:30 – 10:45 Break

10:45 – 11:15 Hands-on with Prompt Engineering

11:15 – 12:15 Intro to AI Orchestration & Demos

12:15 – 13:15 Lunch Break

13:15 – 15:15 Hands-on with AI Orchestration

15:15 – 15:30 Break

15:30 – 16:00 Use Case Validation

16:00 – 16:30 Q&A and Wrap-up

Expectations
 Key Learnings
 What is Azure OpenAI?
 Why Prompt Engineering is important?
 Why is AI Orchestration needed?
 What are some of the AI Orchestrators available?
 Why Embeddings & Vector Stores are needed?
 What are some of the Vector Stores available?
Use Case Ideation & Brainstorming
Use Case Ideation & Brainstorming
 Work with Coaches at your Table
 Brainstorm on potential Use Cases / Scenarios (There is no bad idea/thought there ☺)
 Capture Use Cases, will be used later
Intro to Azure OpenAI, Prompt Engineering
Artificial Intelligence Artificial Intelligence
1956 The field of computer science that seeks to create
intelligent machines that can replicate or exceed
human intelligence

Machine Learning
Machine Learning
1997 Subset of AI that enables machines to learn from
existing data and improve upon that data to make
decisions or predictions

Deep Learning Deep Learning

2017 A machine learning technique in which layers
of neural networks are used to process data and
make decisions

Generative AI
Generative AI 2021 Create new written, visual, and auditory content
given prompts or existing data.
 Why so much hype around ChatGPT?

Time to reach
 ChatGPT 100M users
 100m users in 2.5 months
100M
 Most “human-like” interaction

 What’s new about it?

Mobile phones: 16
years
ChatGPT: 2
months
50M Internet: 7

 Can generate and do “things”

years

Facebook: 4.5

 Answer knowledge questions directly years

 Answer coding questions 16 years 2 months

 Can correct itself, if you ask it

 Prompt examples here

 What’s the verdict?
• ChatGPT is the fastest-growing app of all time - Swiss
bank UBS

• ChatGPT reached 100 million users just two months after

launching - The Guardian
• An average of about 13 million unique visitors had used
ChatGPT per day in January 2023 - Similarweb
A large language model is a type of
artificial intelligence that can process
and produce natural language texts.

It learns from a massive amount of text

data, such as books, articles, and web
pages, and discovers the patterns and
rules of language from them.

It can do various tasks, such as

answering questions, summarizing
texts, writing essays, and more.

Prompt:
In 100 words or less, please tell me what a large language model is

Prompt:
A futuristic abstract rendering of a large language model
Traditional model development
High cost and slow deployment—each service is trained disjointly

Tagging Spatial Analysis Accessibility Spatial Azure Search,

DEPLOYMENTS Services Services Services Presenter Video Indexer

EAGLE
EA GLE

FLOWER PLAYING SOCCER

Classification Object Detection Object Tracking Action Entities Topics Sentiments

TASKS Recognition

Classification Detection Tracking Action Entity Topic Sentiment

Model Model Model Model Recognition Classification Analysis
INDIVIDUAL MODEL
(DISJOINTLY)

TRAINING DATA Tagging Detection Tracking Action Entity Topic Sentiment

(w/ ANNOTATION) data data data data data data data
 Large Language Models

Typical ML for NLP Large Language Models

Labeled ML Model
Classification Classification
Data training
Large
Labeled ML Model
Entity Extraction Unlabeled Language Entity Extraction
Data training
Data Model
(Many TBs) Training
… …
($$$)
Labeled ML Model
Summarization Summarization
Data training

One model per capability

Labeled data to train Single model for all use cases
Highly optimized for use case Describe in natural language what it should do
What are foundation models?
Tasks

Question and
answering
Data
Sentiment
Text analysis

Images Information
extraction
Foundation
Training Adaption
Speech model
Image
captioning
Structured data
Transformer model Object
3d signals
recognition

Instruction
follow
Model use out of the box - prompting
Entity Recognition Prompt Instruction
Decoder Extract the name of this person in this text. Conversational AI
Text: “My name is Simon, order status?”
Topic Classification Application
NLU Completion
Agent: Sentiment:
Sentiment Analysis Entity (Name): Simon
How can I help you today? Positive
Prompt Instruction
Decide whether a phrase’s sentiment is positive, Customer: Sentiment:
Other NLU tasks
neutral, or negative. My name is Simon, Positive
NLP Phrase: “How can I help you today?” order status?

Summarization Completion Summary of conversation

Sentiment: Positive
Customer calling Abstractive
Paraphrase Prompt Instruction regarding an order. Summarization
NLG Summarize the following conversation:
Sentence Generation Agent: How can I help you today?
Foundation Model Customer: My name is Simon, order status?
Large Language Model
GPT-3 Other NLG tasks Completion
Summary: Customer calling regarding an order.
How large are they? BERT Large - 2018

Transformer model Neural 345M

network
𝜎 𝜔𝑥 + 𝑏 GPT2 - 2019
1.5B
GPT3 - 2020
175B
Turing Megatron NLG
2021

Input layer Output layer

530B
Hidden layers
GPT4 – 2023
Function: weight * input plus bias
1.4T ( estimated)
 Understand language
It’s just a predictive engine. It doesn’t understand
math, either.

Understand facts
There are no separate modes for “information
retrieval” and “creative writing”.

Generative AI Understand manners, emotion or

ethics

doesn’t: Also: avoid anthropomorphizing AIs.

Understand anything
The output is just a combination of the training
data and the prompt.
 • OpenAI is an AI research and deployment company
• Focus: Groundbreaking research, Latest models & Fast innovation
• Mission is to ensure that artificial general intelligence benefits all of humanity.

• In 2019, Microsoft invested over $1 billion in

OpenAI
• Develop advanced AI based super computing
technologies on Microsoft Azure.
• Azure is OpenAI’s exclusive cloud provider
• Azure will power all OpenAI workloads across
research, products and API services.

https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/
 Generative AI

GPT-3 Codex DALL·E ChatGPT

Prompt: Prompt: Prompt: A white Siamese cat What is the fastest animal on land?

Write a tagline for an ice Table customers, columns = Response:

cream shop. [CustomerId, FirstName, the cheetah (Acinonyx jubatus),
LastName, Company, Address, which can reach speeds of up to 60
miles (97 kilometers) per hour.
Response: City, State, Country,
PostalCode]
We serve up smiles with What makes them so fast?
every scoop! Create a SQL query for all
customers in Texas named
Jane
query = Cheetahs are built for speed and
have several adaptations that make
them the fastest land animal: lean
Response: body, long legs, flexible spine,
large nostrils and claws that don’t
retract.
SELECT *
FROM customers
WHERE State = 'TX' AND
FirstName = 'Jane'
 Ensure that artificial Empower every person and
general intelligence (AGI) organization on the planet
benefits humanity. to achieve more

Groundbreaking research Enterprise features

Latest models Production SLAs
Fast innovation Data privacy
 Azure AI

Applications Partner Solutions

Business
Users
Application Platform
AI Builder
Power BI Power Apps Power Automate Power Virtual Agents

Scenario-Based Services
Applied AI Services
Bot Service Cognitive Search Form Recognizer Video Indexer Metrics Advisor Immersive Reader

Customizable AI Models Azure OpenAI

Cognitive Services Service Developers &
Vision Speech Language Decision Data Scientists

ML Platform Azure Machine Learning

 Deployed in your Azure subscription,
secured by you, and tied to your datasets
and applications
Azure OpenAI
Service
Large, pretrained AI models to unlock
new scenarios

AI models, some custom-tunable with

your data and hyperparameters
GPT-3.5 GPT-4

Built-in responsible AI to detect and

ChatGPT
mitigate harmful use

Enterprise-grade security with role-based

access control (RBAC) and private networks
Intro to Prompt Engineering
Azure OpenAI Service | GPT-3 Prompt Design

Prompt – Text input that Completion – Output that

provides some context to the GPT-3 generates based on
engine on what is expecting. the prompt.
 What is Prompt Engineering?
Prompt engineering is a concept in Natural Language Processing
(NLP) that involves embedding descriptions of tasks in input to
prompt the model to output the desired results.
Prompt typically includes problem descriptions, instructions on how to
solve the problem, and examples of correct problem and solution
pairs.
An example of Prompt Engineering– Topic Classifier
Classify customer's question. Classify between category 1 to 3.
Instructions
Detailed guidelines for how to choose: High level and detailed
choose 1 if the question is about auto insurance.
choose 2 if the question is about home flood insurance.
choose 3 if the question is not relevant to insurance.

Customer question: Hi there, do you know how to choose flood insurance?​

Classified topic: 2​
Examples
Order of examples matter
Customer question: Hi there, I have a question on my auto insurance.​
Classified topic: 1​

Customer question: Hi there, do you know how to apply for financial aid?​
Classified topic: 3

Customer question: {insert new question here}

Task and Prompting
Classified topic:
answer
 Types of Prompt Engineering
Zero-shot Few-shot
The model predicts the answer given only a natural language description In addition to the task description, the model sees a few examples of the
of the task. task.

Classify customer's question. Classify between category 1 to 3. Classify customer's question. Classify between category 1 to 3.

Detailed guidelines for how to choose: Detailed guidelines for how to choose:
choose 1 if the question is about auto insurance. choose 1 if the question is about auto insurance.
choose 2 if the question is about home flood insurance. choose 2 if the question is about home flood insurance.
choose 3 if the question is not relevant to insurance. choose 3 if the question is not relevant to insurance.

Customer question: Hi there, do you know how to choose flood insurance?​

Customer question: {insert new question here} Classified topic: 2​
Classified topic:
Customer question: Hi there, I have a question on my auto insurance.​
Classified topic: 1​

Customer question: Hi there, do you know how to apply for financial aid?​
Classified topic: 3

Customer question: {insert new question here}

Classified topic:
 Prompt Engineering Guide
1. Give clearer instructions

2. Split complex tasks into simpler subtasks

3. Structure the instruction to keep the model on task

4. Prompt the model to explain before answering

5. Ask for justifications of many possible answers, and then synthesize

6. Generate many outputs, and then use the model to pick the best one

7. Custom-tune custom models to maximize performance

 Reduce Hallucination using Prompt Engineering

Include Restrict Add Repeat Position

Include Restrict the Add Chain of Repeat most Position most
instructions of output (e.g., Thought style important important
requesting the choose from a of instruction, instructions in instructions in
model not to confined list "Solve the the prompt a the last making
make up stuff instead of problem step couple of use of latency
but stay with generating free by step." times. effect.
facts. form strings)
 Few-Shot Reasoning (Human version)
1st prompt 2nd prompt—provide reasoning
Roger has 5 tennis balls. He buys 2 more cans of tennis Roger has 5 tennis balls. He buys 2 more cans of tennis
balls. Each can has 3 tennis balls. How many tennis balls balls. Each can has 3 tennis balls. How many tennis balls
does he have now? does he have now?

Answer: The answer is 11. Answer: Roger started with 5 balls. 2 cans of 3 tennis balls
each is 6 tennis balls. 5+6 = 11. The answer is 11.
The cafeteria has 23 apples. If they used 20 to make lunch
and bought 6 more, how many do they have? The cafeteria has 23 apples. If they used 20 to make lunch
and bought 6 more, how many do they have?
The answer is 27
The cafeteria had 23 apples originally. They used 20 to
make lunch. So they had 23-20 = 3. They bought 6
more apples, so they have 3 + 6 = 9. The answer is 9.
 Model Reasoning—Chain-of-Thought
Before After
What is the annual water demand of a single-family What is the annual water demand of a single-family
household containing four people who are at home an household containing four people who are at home an
average of 200 days per year and use an average of 100 average of 200 days per year and use an average of 100
liters of water per day? liters of water per day?

The annual water demand of a single-family household Let’s think step by step and explain the calculation
containing four people who are at home an average of step by step.
200 days per year and use an average of 100 liters of
water per day is 8,000,000 liters per year. Step 1: Calculate the total number of days in a year:
There are 365 days in a year.

Step 2: Calculate the total number of days the family is at home:

The family is at home an average of 200 days per year.

Step 3: Calculate the total water demand of the family:

The family uses an average of 100 liters of water per day. Therefore, the total water
demand of the family is 100 liters x 200 days = 20,000 liters per year.

Step 4: Calculate the annual water demand of the single-family household:

The annual water demand of the single-family household is 20,000 liters per year.
 Using Chain-of-Thought for Reasoning
1st prompt—Reason Extraction 2nd prompt—Answer Extraction
On average Joe throws 25 punches per minute. One average Joe throws 25 punches per minute.
A fight lasts 5 rounds of 3 minutes. How many A fight lasts 5 rounds of 3 minutes. How many
punches did he throw? punches did he throw?

Let’s think step by step and explain the calculation In one minute, Joe throws 25 punches.
step by step. In three minutes, Joe throws 3 * 25 = 75 punches.
In five rounds, Joe throws 5*75 = 375 punches.
In one minute, Joe throws 25 punches.
In three minutes, Joe throws 3 * 25 = 75 punches. Therefore, the answer (Arabic numerals) is
In five rounds, Joe throws 5*75 = 375 punches.
 Few-Shot– Chain-of-Thought--Examples

Math Word Problems (free response) Math Word Problems (multiple choice) CSQA (commonsense)
Q: Roger has 5 tennis balls. He buys 2 more cans of Q: How many keystrokes are needed to type the Q: Sammy wanted to go to where the people were.
tennis balls. Each can has 3 tennis balls. How many numbers from 1 to 500? Answer Choices: (a) 1156 (b) Where might he go? Options: (a) race track (b)
tennis balls does he have now? 1392 (c) 1480 (d) 1562 (e) 1788 populated areas (c) desert (d) apartment (e) roadblock
A: Roger started with 5 balls. 2 cans of 3 tennis balls A: There are 9 one-digit numbers from 1 to 9. There are A: The answer must be a place with a lot of people. Race
each is 6 tennis balls. 5 + 6 = 11. 90 two-digit numbers from 10 to 99. There are 401 tracks, desert, apartments, and roadblocks don't have a
three-digit numbers from 100 to 500. 9 +90(2) + 401(3) lot of people, but populated areas do.
The answer is 11.
= 1392.
So the answer is (b).
The answer is (b).
StrategyQA
Sports Understanding
Q: Yes or no: Would a pear sink in water?
Date Understanding
Q: Is the following sentence plausible? "Joao Moutinho
Q: The concert was scheduled to be on 06/01/1943, but
A: The density of a pear is about 0.6 g/cm^3, which is caught the screen pass in the NFC championship.“
less than water. Thus, a pear would float was delayed by one day to today. What is the date 10
days ago in MM/DD/YYYY? A: Joao Moutinho is a soccer player. The NFC
So the answer is no. championship is part of American football, not soccer.
A: One day after 06/01/1943 is 06/02/1943, so today is
06/02/1943. 10 days before today is 05/23/1943. So the answer is no.

SayCan (Instructing a robot) So the answer is 05/23/1943.

Coin Flip (state tracking)
Human: How would you bring me something that isn’t
Last Letter Concatenation Q: A coin is heads up. Maybelle flips the coin. Shalonda
a fruit?
Q: Take the last letters of the words in "Lady Gaga" and does not flip the coin. Is the coin still heads up?
Explanation: the user wants something to eat that isn't a
concatenate them. A: The coin was flipped by Maybelle. So the coin was
fruit. An energy bar is not a fruit, so I will bring the user
an energy bar. A: The last letter of "Lady" is "y". The last letter of "Gaga" flipped 1 time, which is an odd number. The coin started
is "a". Concatenating them is "ya". heads up, so after an odd number of flips, it will be tails up.
Plan: 1. find(energy bar) 2. pick(energy bar) 3.
find(user) 4. put(energy bar) 5. done(). So the answer is ya So the answer is no.
 Selection-inference prompting
It takes Amy 4 minutes to climb to the top of a slide. It takes It takes Amy 4 minutes to climb to the top of a slide. It takes
her 1 minute to slide down. The water slides close in 15 her 1 minute to slide down. The water slides close in 15
minutes. How many times can she slide before it closes? minutes. How many times can she slide before it closes?

14 times
How much time does it take for Amy to climb and slide?

It takes Amy 5 minutes to climb and slide

It takes Amy 4 minutes to climb to the top of a slide. It takes

her 1 minute to slide down. The water slides close in 15 It takes Amy 4 minutes to climb to the top of a slide. It takes
minutes. How many times can she slide before it closes? her 1 minute to slide down. The water slides close in 15
minutes. How many times can she slide before it closes?

To solve "How many times can she slide before it closes" we

need to first solve: Chain-of-Thought It takes Amy 5 minutes to climb and slide.

How much time does it take for Amy to climb and slide? She can slide 3 times before the water slides close.
Add new
fact to
context
Decomposes problem into a smaller task (Reasoning)
 Hierarchy of language model customization

Add your data

Retrieval Augmented Generation Prompt engineer

LLMOps
Responsible AI 90%* of use cases

Fine-tune

Train
Building your house with LLMs

Ready-made Full renovation Custom build

Add your data Fine tune Train your own
Prompt engineer
Demo Azure OpenAI Studio
 Sample Prompts in the repo, or feel free to use your own:
 https://github.com/Azure/intro-to-intelligent-apps/blob/main/presentations/prompts-
demos.md
Break
Getting Started with Azure Open AI
Hands-on with Prompt Engineering
 Create Azure OpenAI Service
 https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-
resource?pivots=web-portal
 Open AI Studio
 Go to Azure OpenAI Studio - Microsoft Azure
 Walk through Text Completion tutorial (if needed)
 https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/completions

 Do Prompt Exercises in the repo:

 https://github.com/Azure/intro-to-intelligent-apps/blob/main/labs/01-prompts/README.md
Exercise 1 – German translation

Exercise Write a prompt that generates the expected completion

I was enjoying the sun, but then a huge cloud came and
Input text covered the sky.

Expected Ich genoss die Sonne, aber dann kam eine riesige Wolke
completion und bedeckte den Himmel.
Translate the following sentence into German.

Sentence: I was enjoying the sun, but then a huge cloud came and covered
the sky.

German translation:
Exercise 2 – Negation

Exercise Write a prompt that generates the expected completion

I was enjoying the sun, but then a huge cloud came and
Input text covered the sky.

Expected I was not enjoying the sun, and then a huge cloud did not
completion come and cover the sky.
Exercise 3 – Classification

Exercise Write a prompt that generates the expected completion

"Not much to write about here, but it does exactly what it's
supposed to. filters out the pop sounds. now my recordings are
Input text much more crisp. it is one of the lowest prices pop filters on
amazon so might as well buy it, they honestly work the same
despite their pricing"

Expected Positive: 0.75

completion Neutral: 0.20
Negative: 0.05
(or similar)
Not much to write about here, but it does exactly what it's supposed to.
filters out the pop sounds. now my recordings are much more crisp. it is one
of the lowest prices pop filters on amazon so might as well buy it, they
honestly work the same despite their pricing

Decide whether the product review's sentiment is positive, neutral or

negative. Show the probability for positive, neutral and negative sentiment.
Exercise 4 – E-Mail Summarization

Exercise Write a prompt that generates the expected completion

Input text Your own long email thread

Expected Summary: XYZ

completion Open Questions: XYZ
Action Items: XYZ
(or similar)
Exercise 5 – Rewriting

Exercise Write a prompt that generates the expected completion

I was enjoying the sun, but then a huge cloud came and
Input text covered the sky.

Expected She was enjoying the sun, but then a huge cloud came and
completion covered the sky.
Exercise 6 – Multiple Tasks

Exercise Write a prompt that generates the expected completion

I was enjoying the sun, but then a huge cloud came and
Input text covered the sky.

{
"translated": "Ich genoss die Sonne, aber dann kam eine riesige Wolke und

Expected bedeckte den Himmel.",

"negated": "I was not enjoying the sun, and no huge cloud came and covered

completion the sky.",

"third_person": "She was enjoying the sun, but then a huge cloud came and
covered the sky."
}
Exercise 7 – Data extraction to JSON

Exercise Write a prompt that generates the expected completion

Hello, my name is Mateo Gomez. I lost my Credit card on August 17th, and I would like to request its
cancellation. The last purchase I made was of a Chicken parmigiana dish at Contoso Restaurant, located
near the Hollywood Museum, for $40. Below is my personal information for validation:

Input text
Profession: Accountant
Social Security number is 123-45-6789
Date of birth: 9-9-1989
Phone number: 949-555-0110
Personal address: 1234 Hollywood Boulevard Los Angeles CA
Linked email account: mateo@contosorestaurant.com
Swift code: CHASUS33XXX

Expected "reason": "Lost card",

"classified_reason": "lost_card",

completion
"name": "Mateo Gomez",
"ssn": "123-45-6789",
"dob": "09/09/1989"
}
Exercise 8 – Fashion product description

Exercise Write a prompt that generates the expected completion

Season: Winter
Style: Sweater
Input text Gender: Female
Target group: Teenager
Material: Cotton

Expected Stay warm and stylish this winter with our cozy cotton
sweaters, perfect for the fashion-forward teenager. Refresh
completion your wardrobe with the latest winter styles from our
(or similar) collection.
Exercise 9 – Write a Blog

Exercise Write a blog post about a topic of your choice

Step 1: I want you to act as a social media manager. You will

Input text
be helping me to brainstorm blog post outline ideas for the
topic <Topic 1>:
Step 2: Write 3 engaging and informative paragraphs about <Idea
1 description>
Step 3: Write 3 engaging and informative paragraphs about <Idea
2 description>
Step 4: Tags <List of relevant #hashtags>
Exercise 10 – Call Center

Exercise Analyze a call center conversation

Input text Expected Completion

Employee: "Hello, this is Julia Schreider from Contoso Company. How can I help you today?"
Customer: "Hi, I am Carsten Mueller. I ordered a package 10 days ago, on February 10th, and it was {
supposed to arrive in maximum 5 business days. I have called three times already and nobody "classified_reason": "lost_package",
could provide any more information. I want to know where the package is and I want the problem "resolve_status": "unresolved",
to be solved immediately. This is the worst service I had for a long time!" "call_summary": "Customer ordered package 10 days ago and has
Employee: "I apologize for the inconvenience, Mr. Mueller. I understand your frustration and I'm not received it yet.",
here to help. Can you please provide me with your order number so I can look into this for you?" "customer_name": "Carsten Mueller",
Customer: "Yes, it's ACZ456789." "employee_name": "Julia Schreider",
Employee: "Thank you. I'm looking into it now. Can you please hold for a few minutes while I check "order_number": "ACZ456789",
the status of your package?" "customer_contact_nr": "+4911112223344",
Customer: "Okay." "new_address": "N/A",
Employee: "Thank you for your patience. I am sorry to inform you that I am unable to find the "sentiment_initial": ["angry", "frustrated"],
status of your package. It appears to have left the sending address, but no up-to-date status on "sentiment_final": ["calm"],
the current location. I will further investigate your case and get back to you as soon as possible via "satisfaction_score_initial": 0,
phone call. Could you please provide me your contact information?" "satisfaction_score_final": 5,
Customer: "Ah not again. Anyway, my phone number is +4911112223344." "eta": "N/A",
Employee: "I apologize again for the inconvenience. Is there anything else I can help you with "action_item": ["track_package", "inquire_package_status",
today?" "contact_customer"]
Customer: "No." }
Employee: "Thank you. Have a great day!"
Exercise 11 – Few-shot learning
Exercise Write a few-shot learned prompt that classifies a movie summary.

Data samples Paul Atreides, a brilliant and gifted young man born into a great destiny beyond his understanding, must travel to the
most dangerous planet in the universe to ensure the future of his family and his people. As malevolent forces explode
into conflict over the planet's exclusive supply of the most precious resource in existence-a commodity capable of
unlocking humanity's greatest potential-only those who can conquer their fear will survive.
['Action', 'Adventure', 'Science Fiction’]

A botched store robbery places Wonder Woman in a global battle against a powerful and mysterious ancient force that
puts her powers in jeopardy.
['Action', 'Adventure', 'Fantasy']

After the devastating events of Avengers: Infinity War, the universe is in ruins due to the efforts of the Mad Titan,
Thanos. With the help of remaining allies, the Avengers must assemble once more in order to undo Thanos' actions
and restore order to the universe once and for all, no matter what consequences may be in store.
['Adventure', 'Science Fiction', 'Action']

A widowed new dad copes with doubts, fears, heartache and dirty diapers as he sets out to raise his daughter on his
own. Inspired by a true story.
['Drama', 'Family', 'Comedy’]

New data:
Harry, Ron and Hermione walk away from their last year at Hogwarts to find and destroy the remaining Horcruxes,
putting an end to Voldemort's bid for immortality. But with Harry's beloved Dumbledore dead and Voldemort's
unscrupulous Death Eaters on the loose, the world is more dangerous than ever.
You are adding tag categories to movies, based on their descriptions.

###
Movie description: Paul Atreides, a brilliant and gifted young man born into a great destiny beyond his understanding, must travel to the most dangerous planet in the universe to ensure the
future of his family and his people. As malevolent forces explode into conflict over the planet's exclusive supply of the mos t precious resource in existence-a commodity capable of unlocking
humanity's greatest potential-only those who can conquer their fear will survive.
Tags: ['Action', 'Adventure', 'Science Fiction’]
###
Movie description: A botched store robbery places Wonder Woman in a global battle against a powerful and mysterious ancient f orce that puts her powers in jeopardy.
Tags: ['Action', 'Adventure', 'Fantasy']
###
Movie description: After the devastating events of Avengers: Infinity War, the universe is in ruins due to the efforts of the Mad Titan, Thanos. With the help of remaining allies, the Avengers
must assemble once more in order to undo Thanos' actions and restore order to the universe once and for all, no matter what consequences may be in store.
Tags: ['Adventure', 'Science Fiction', 'Action']
###
Movie description: A widowed new dad copes with doubts, fears, heartache and dirty diapers as he sets out to raise his daught er on his own. Inspired by a true story.
Tags: ['Drama', 'Family', 'Comedy’]
###
Movie description: Harry, Ron and Hermione walk away from their last year at Hogwarts to find and destroy the remaining Horcruxes, putting an end to Voldemort's bid for immortality. But with
Harry's beloved Dumbledore dead and Voldemort's unscrupulous Death Eaters on the loose, the world is more dangerous than ever .
Tags:
Exercise 12 – NL to SQL with Codex

Exercise Write a prompt that generates the expected SQL statement

Table Table: customer // Columns: firstname, name, customer_id, address

Table: orders // Columns: order_id, customer_id, product_id, product_amount
details Table: products // Columns: product_id, price, name, description

Query
A query that returns the top 10 orders and show the customer name
Exercise - Bring Your Own Data

Write a prompt that uses some of your own data and ask it questions.
Exercise It could be data copied from a website or document for example.

Input text <<TEXT COPIED FROM WIKIPEDIA FOR EXAMPLE>>

Expected <<ARE YOU GETTING THE RESULT YOU EXPECTED? IF NOT, WHY?>>
completion
Intro to Azure OpenAI, Prompt Engineering Q&A
Intro to AI Orchestration Building Blocks
Foundation + Context

Tool AI Pioneer Herbert Simon’s Scissors (1970)

Hallucinate & Ground:

Cognition = The Make-up or fill-in-the-blanks, when you
‘know-how’ don’t have context.
Provide context that fills in the blanks.

Context =
‘Specific world
knowledge’
Components – Pick the Model
Prompt: the natural language text you send to the LLM.
Tokens: prompts encoded into tokens for the LLM to use
Embeddings: Vectors (arrays) of numbers, that represent the meaning & context of the tokens.

Your
Prompt Vectorized &
LLM API Intelligent
Design Other Data
App

59
Components – Prompt

LLMs have limits to how many tokens they can process &
charge based on the number of tokens you submit.

Your
Prompt Vectorized &
LLM API Intelligent
design Other Data
App

60
Components – Is Vector Store Needed
Embedding: An array of numbers, or vector, that
represent the semantic meaning of a piece of text.

Vectorized Your
Prompt
LLM API & Other Intelligent
design Data App

61
 Components – Vector Database
Vector DB: A database designed to store and retrieve data represented as vectors.
Querying in a vector db (‘vectorized search’) finds similar assets by searching for neighboring data points.

Vectorized Your
Prompt
LLM API & Other Intelligent
design Data App

62
Foundation + Context + Components = Intelligence

Vectorized Your
Prompt
LLM API & Other Intelligent
design Data App

63
Tokens
You can think of tokens as pieces of words used for
natural language processing. For English text, 1 token
is approximately 4 characters or 0.75 words.

As a point of reference, the collected works of

Shakespeare are about 900,000 words or 1.2M tokens.
Understanding tokens and possibilities
Tokens: Prompt Instruction
Suggest three names for an animal that is a superhero.

Animal: Cat
Names: Captain Sharpclaw, Agent Fluffball, The Incredible Feline
Animal: Dog
Names: Ruff the Protector, Wonder Canine, Sir Barks-a-Lot
Animal: Horse
Names:

Completion Temperature 0 (always the same)

Mighty Equine, The Great Galloper, Thunderhoof

Completion Temperature 1 (often different)

Probabilities: Blaze the Miracle Mare, Pegasus the Winged Warrior,

Secretariat the Superhorse

Completion Temperature 1 (often different)

Blaze of Glory, Sterling Silver, Thunderbolt

Demo Tokens
 Tokenizer
 https://platform.openai.com/tokenizer
Embeddings
An embedding is a special format of data
representation that can be easily utilized by
machine learning models and algorithms.

The embedding is an information dense

representation of the semantic meaning
of a piece of text.

Each embedding is a vector of floating-point

numbers, such that the distance between two
embeddings in the vector space is correlated
with semantic similarity between two inputs
in the original format.

For example, if two texts are similar, then their

vector representations should also be similar.
Embeddings make it possible to
map content to a “semantic space”

The presence of a black

A neutron star is the A star shines for most
hole can be inferred
collapsed core of a of its active life due to
through its interaction
massive supergiant star thermonuclear fusion
with other matter

[ 15 34 24 13 …] [16 22 89 26 …] [ 20 13 31 89 …]
Embeddings represent your data REALISTIC CARTOON
Example: 1-dimensional vector
 MAMMAL

Multiple dimensions represent

different aspects of data REALISTIC CARTOON

BIRD
 MAMMAL
Where would I live in this
embedding space?

In the “embedding space,”

similar data are grouped together REALISTIC CARTOON

BIRD
Embedding models
Different Azure OpenAI embedding models are specifically
created to be good at a particular task.

• Similarity embeddings are good at capturing semantic

similarity between two or more pieces of text
• Text search embeddings help measure long documents
are relevant to a short query
• Code search embeddings are useful for embedding code
snippets and embedding nature language search queries

Embeddings make it easier to do machine learning

on large inputs representing words by capturing the
semantic similarities in a vector space.

Therefore, we can use embeddings to determine if

two text chunks are semantically similar and provide
a score to assess that similarity.
Similarity Search with embeddings
Once you encode your content as embeddings, you can then get an embedding
from the user input and use that to find the most semantically similar content

user
embedding result set
input
“What is a neutron star?” [ 13 33 34 13 … ]

Azure OpenAI embeddings tutorial—Azure OpenAI | Microsoft Learn

Use Cases for Embeddings
 Search
 where results are ranked by relevance to a query string

 Clustering
 where text strings are grouped by similarity

 Recommendations
 where items with related text strings are recommended

 Anomaly detection
 where outliers with little relatedness are identified

 Diversity measurement
 where similarity distributions are analyzed

 Classification
 where text strings are classified by their most similar label
Demo Embeddings
 Product Finder
 https://ecommerce.redisventures.com/

 Doc Search
 https://docsearch.redisventures.com/
Intro to AI Orchestration
Why do we need AI Orchestration?

Limits (aka Blowing up the Prompt)

AI Pipelines: Orchestration for All of the Moving Parts

State Side Effects

Memories Plugins

Flow
AI Orchestration

AI UX LangChain Haystack HF Transformers Agents …

AI UX
BYO Chat
Models
User BYO Model AI App
Devs
Foundation Models
Semantic Kernel
Integrate AI into your existing apps using C#, Python & Java

Azure Cognitive Search 1,400 Power Platform Connectors

State Side Effects

Memories Plugins (aka Skills)

Azure Content Safety

AI UX
Semantic Kernel
Copilot Chat
Models
User Azure AI Models AI App
Devs
Foundation Models

Microsoft Corporation
 Prompt Chaining
Entity Extraction Summarization Sentiment Analysis

Please extract entities from the Please summarize the information Please provide a sentiment for the
following news article: 'The new about the product following text:
iPhone model is set to be released
next month. It has been highly [{'text': 'iPhone', 'type': 'PRODUCT'}, The iPhone is a line of smartphones
anticipated by Apple fans and is {'text': 'Apple', 'type': designed and marketed by Apple
expected to feature a larger screen 'ORGANIZATION’}] Inc. These devices combine a
and improved camera computer, camera, and internet
The iPhone is a line of smartphones connectivity in a single device,
{'text': 'iPhone', 'type’: designed and marketed by Apple and are known for their sleek
'PRODUCT'}, {'text': 'Apple’, Inc. These devices combine a design and user-friendly interface.
'type': 'ORGANIZATION'}] computer, camera, and internet
connectivity in a single device, and Positive
are known for their sleek design and
user-friendly interface.
 Then all summaries are passed to GPT again to
Prompt Chunking (token limit) summarize, i.e. second round of summarization. The sum
of the summaries has to fit within the token limit.

Please summarize and extract topic: Global warming refers to

the gradual increase in the overall temperature of the Earth's
Please summarize the following text: atmosphere, primarily caused by the burning of fossil fuels
'Climate change is a pressing issue that such as coal, oil, and natural gas. This burning releases
needs to be addressed immediately. The carbon dioxide and other greenhouse gases into the
increasing levels of carbon dioxide in the atmosphere, which trap heat and cause the Earth's
atmosphere are causing the Earth's temperature to rise.

Large Text temperature to rise, which is leading to

Beyond
Climate change is a pressing issue that needs to be addressed
more frequent and severe natural disasters. immediately. Governments around the world need to take
Token
Limit Governments around the world need to take immediate action to reduce carbon emissions and invest in
Chunks
immediate action to reduce carbon renewable energy sources. Failure to do so will have dire
emissions and invest in renewable energy consequences for future generations
sources. Failure to do so will have dire
consequences for future generations'" The text is discussing global warming, which is the gradual increase in
the overall temperature of the Earth's atmosphere caused by the
Climate change is a pressing issue that needs to burning of fossil fuels. This burning releases greenhouse gases, such as
be addressed immediately. Governments carbon dioxide, into the atmosphere which trap heat and cause the
around the world need to take immediate temperature to rise. The text also highlights the urgency of the issue of
climate change and the need for governments to take action to reduce
action to reduce carbon emissions and invest in
carbon emissions and invest in renewable energy sources. Failure to
renewable energy sources. Failure to do so will address this issue will have negative consequences for future
have dire consequences for future generations generations.

Topic: Climate change and Global warming.

Continue to
next prompt
 Retrieval Augmented Generation

User Question

LLM Workflow
Query My Data Search Index

Add Results to Prompt

Large Language
Query Model Model

Send Results
 Retrieval augmented generation: bring your data to the prompt
Prompt

You are an intelligent assistant helping Contoso Inc Response

Text input that provides employees with their healthcare plan questions and
some framing as to how employee handbook questions. Answer the following
question using only the data provided in the Based on the provided information,
the engine should behave sources below. it can be determined that both
health plans offered by Northwind
Health Plus and Northwind Standard
User provided question Question: Does my health plan cover annual provide coverage for vision exams.
that needs to be answered eye exams? Therefore, your health plan should
cover annual eye exams.

Sources:

1. Northwind Health Plus offers coverage for vision

exams, glasses, and contact lenses, as well as dental
Sources used to exams, cleanings, and fillings.
answer the question 2. Northwind Standard only offers coverage for
vision exams and glasses.

3. Both plans offer coverage for vision and

dental services.
Retrieval augmented generation

Prompt
Azure OpenAI Service

User
application

Documents
 General Recommendations

Direct use of Utilizing similarity For fine-tuning, Experiment with

foundation search to create only use prompt
models with dynamic prompts the highest quality engineering on
prompt can be beneficial dataset. Monitor advanced models
engineering is in cases where the validation metrics and fine-tune less
preferred. For total content is too and adjust parame capable models,
prompt extensive, as it ters to avoid over- compare the
engineering, try allows for the fitting/under- performance
out a few different incorporation of fitting. metrics,
prompts and the most pertinent operational latency
pick the best one. information for and cost.
each query.
 Top 3 Azure OpenAI Service Use Cases

Contact Center ChatGPT for my Document Intelligence

Enterprise
Survey Summarization and Index knowledge base and give Information extraction,
Analyse chat conversations generative answers to arbitrary classification, summarization
questions Extract complex entities from
Proof: Telco, banking, insurance Proof: Nearly every industry documents and their relationship
(e.g., in media or research)
Proof: Nearly every industry
 Azure OpenAI Service – Implemented Use Cases

 Contact Center & Customer Support  Document Intelligence

 Summarization, information extraction from call  Information extraction, classification, summarization
center calls and chat conversations on documents, e.g., maintenance logs
 Proof: Several telcos
 Extract complex entities from documents and their
relationship (e.g., in media or research)
 Question & Answering (Q&A)  Proof: Largely used in Energy sector
 Index knowledge base and give generative answers to  Text Rewriting / Paraphrasing
arbitrary questions
 News article rewriting for SEO optimization
 Proof: Several insurances
 Proof: Several Media Companies
 Product Description Generation
 Event Summarization
 Generate product taglines from metadata
 Post-game summaries from transcripts & automated
 Proof: Several fashion retailers reviews
 Code generation  Meeting summarization & action item extraction
 Code creation, summarization, explanation  Proof: Several Sports Companies
 Proof: Several Professional Service Companies
 Contact Center Analytics using Speech API & Azure OpenAI Service
Extract rich insights from call transcripts

PowerBI Insights
(near real-time)

Caller Intelligent
Transcription
Conversation Trends
Audio
Person-to-Person & Insights
Files Azure OpenAI
Conversation
Service
Speech-to-Text
Telephony Azure
Server Storage CRM
Azure Cognitive Services –
Speech & Azure OpenAI Service

Call-Center Agent

Detailed call history incl.

summaries, call reasons, etc.

Solution Demo
Accelerator Website
Azure OpenAI Service | Sample Use Cases

Azure OpenAI Service

Instruction in English
Call Center
Automation

Customer conversation in
Example:
English (or other language) Azure OpenAI
Servcice Prompt

Output as JSON
 ChatGPT with Enterprise Data
Extract precise answers from your unstructured documents
Do I get a
discount when I
have contract x
and y?

Vector search
Azure ?
OpenAI Questions
Service
Embeddings
Users

Answers
Search Layer
Knowledge Base
Top k Azure
paragraphs
Raw Extracted Azure OpenAI Vectors OpenAI
documents paragraphs Service Service
Embeddings Answering
Prompt
Azure Storage RediSearch Yes, you’ll get 10
Azure Form Euros off, if you
Recognizer
Unstructured documents
Embedding of paragraphs have a 24m long
(pdf, docx, etc.)
using Azure OpenAI Service contract. Here’s the
full link.

Solution Demo
Accelerator Website
 ChatGPT with Enterprise Data
Extract precise answers from your unstructured documents
Do I get a
discount when I
have contract x
and y?

?
Questions/ Query Knowledge

Users

Answers
Search Layer
Knowledge Base
Top k Azure
paragraphs OpenAI
Raw Extracted
Service
documents paragraphs
Answering
Prompt
Azure Storage Yes, you’ll get 10
Azure Form Euros off, if you
Recognizer
Unstructured documents have a 24m long
(pdf, docx, etc.) Azure Cognitive Search
contract. Here’s the
full link.

Solution
Accelerator Blog Post
2 announcements and public cases in Germany

Siemens’ Teamcenter® software for product lifecycle First generative AI-based bot in the automotive industry! Just the beginning!
management (PLM) with Microsoft’s collaboration platform Teams (Car Configurator search & voice assistant)
and the language models in Azure OpenAI Service as well as
other Azure AI capabilities.
Siemens & Microsoft drive industrial productivity with AI | LinkedIn
Siemens Software
© Siemens 2023
Retrieval Augmented Generation + Enterprise Data
Demo AI Apps
 Sample Demos in the repo, or feel free to use your own:
 https://github.com/Azure/intro-to-intelligent-apps/blob/main/presentations/aiapps-
demos.md
Lunch
Hands-on with AI Orchestration
 Setup for Labs
 https://github.com/Azure/intro-to-intelligent-apps/blob/main/labs/00-setup/README.md

 Do AI Orchestration Exercises in the repo:

 https://github.com/Azure/intro-to-intelligent-apps/blob/main/labs/02-integrating-
ai/README.md
 Lab 2: Integrating AI
 Lab 3: AI Orchestration
 Lab 4: Deployment
Break – Put this slide up around 14:45
Continue Hands-on with AI Orchestration
Intro to AI Orchestration Q&A
Use Case Validation
 Work with Coaches
 Revisit Use Case from Morning, are they still valid?
 Discuss amongst Team how you might go about implementing Use Cases based on what you
learned from the day?
 Capture Use Cases
 Discuss amongst Team what it would take to get it into Production?
Q&A
Closing Remarks
 Key Learnings
 What is Azure OpenAI?
 Why Prompt Engineering is important?
 Why is AI Orchestration needed?
 What are some of the AI Orchestrators available?
 Why Embeddings & Vector Stores are needed?
 What are some of the Vector Stores available?

 Aha Moments
 You cannot just point an LLM at your Enterprise Data and get good results
 Engineering the right prompt is not easy
 Integrating AI into an App is fairly easy once you understand the building blocks
Closing Remarks Cont.
 How to get started?
1. What is the business problem trying to be solved?
2. Do I have access to the data needed to solve/answer the question(s)?
3. Add needed data to a sample Prompt in Azure OpenAI Studio and ask Question(s)
(Remember, a Prompt is only so big ☺)
1. If you did not get the right response, iterate again with a different Prompt or add additional data
2. If you got a response that was in the general right direction, time to think about AI Orchestration
4. Decide on AI Orchestration, Data, Embedding & Vector Store Strategies
5. Implement Enterprise Grade API + Data + Azure OpenAI
 MS Learning
 https://learn.microsoft.com/en-us/training/paths/develop-ai-solutions-azure-openai/

 Next Steps in repo:

 https://github.com/Azure/intro-to-intelligent-apps/blob/main/docs/next_steps.md

 Engage Microsoft, we are here to help.

Thank you!