code/06.E2E/E2E_Phi-4-RAG-Azure-AI-Search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phi-4 RAG with Azure AI Search\n",
    "\n",
    "This notebook demonstrates how to use Phi-4-mini and Phi-4-multimodal for Retrieval Augmented Generation (RAG) with Azure AI Search. It covers both single-modality (text-only) and multi-modality (text and image) scenarios.\n",
    "\n",
    "**Prerequisites:**\n",
    "*   Azure AI Search vector index (follow [these instructions](https://learn.microsoft.com/azure/search/search-get-started-portal-import-vectors?tabs=sample-data-storage%2Cmodel-aoai%2Cconnect-data-storage) to create one)\n",
    "*   Phi-4-mini OR Phi-4-multimodal endpoints deployed in Azure AI Foundry\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install azure-ai-inference azure-search-documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text-Only RAG with Phi-4-mini\n",
    "\n",
    "This section demonstrates how to use Phi-4-mini as a chat completion model for RAG, using only text as input.  It involves connecting to Azure AI Foundry Inference and Azure AI Search, retrieving relevant documents, and generating an answer using the retrieved context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Connect to Azure AI Foundry Inference & Azure AI Search\n",
    "import os\n",
    "from azure.ai.inference import ChatCompletionsClient\n",
    "from azure.ai.inference.models import UserMessage\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "from azure.search.documents import SearchClient\n",
    "from azure.search.documents.models import VectorizableTextQuery\n",
    "\n",
    "chat_client = ChatCompletionsClient(\n",
    "    endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"], # Phi-4-mini endpoint\n",
    "    credential=AzureKeyCredential(os.environ[\"AZURE_INFERENCE_CREDENTIAL\"]),\n",
    ")\n",
    "\n",
    "search_client = SearchClient(\n",
    "    endpoint=os.environ[\"AZURE_SEARCH_ENDPOINT\"],\n",
    "    index_name=os.environ[\"AZURE_SEARCH_INDEX\"],\n",
    "    credential=AzureKeyCredential(os.environ[\"AZURE_SEARCH_KEY\"])\n",
    ")\n",
    "\n",
    "# Step 2: Retrieve relevant documents from Azure AI Search\n",
    "def retrieve_documents(query: str, top: int = 10):\n",
    "    vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=top, fields=\"text_vector\")\n",
    "    results = search_client.search(search_text=query,vector_queries=[vector_query],select=[\"content\"],top=top)\n",
    "    return [doc[\"content\"] for doc in results]\n",
    "\n",
    "# Step 3: Generate answer using RAG!\n",
    "def generate_rag_response(query: str):\n",
    "    docs = retrieve_documents(query)\n",
    "    context = \"\\n---\\n\".join(docs)\n",
    "    prompt = f\"\"\"You are a helpful assistant. Use only the following context to answer the question. If the answer isn't in the context, say 'I don't know'.\n",
    "    Context: {context} Question: {query} Answer:\"\"\"\n",
    "    response = chat_client.complete(messages=[UserMessage(content=prompt)])\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "# Example usage\n",
    "user_query = \"What is the capital of France?\"\n",
    "answer = generate_rag_response(user_query)\n",
    "print(f\"Q: {user_query}\\nA: {answer}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-Modal RAG with Phi-4-multimodal\n",
    "\n",
    "This section demonstrates how to use Phi-4-multimodal as a chat completion model for RAG, incorporating both text and image input.  It covers connecting to Azure AI Inference and Azure AI Search, retrieving relevant documents, and generating a multimodal response.\n",
    "\n",
    "**Note:** You can also perform a multi-vector query if you have both `text_vector` and `image_vector` fields in your Azure AI Search index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Connect to Azure AI Inference & Azure AI Search\n",
    "from azure.ai.inference import ChatCompletionsClient\n",
    "from azure.ai.inference.models import (\n",
    "    SystemMessage,\n",
    "    UserMessage,\n",
    "    TextContentItem,\n",
    "    ImageContentItem,\n",
    "    ImageUrl,\n",
    "    ImageDetailLevel,\n",
    ")\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "from azure.search.documents import SearchClient\n",
    "from azure.search.documents.models import VectorizableTextQuery\n",
    "\n",
    "\n",
    "chat_client = ChatCompletionsClient(\n",
    "    endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"], #Phi-4-multimodal serverless endpoint\n",
    "    credential=AzureKeyCredential(os.environ[\"AZURE_INFERENCE_CREDENTIAL\"]),\n",
    ")\n",
    "\n",
    "search_client = SearchClient(\n",
    "    endpoint=os.environ[\"AZURE_SEARCH_ENDPOINT\"],\n",
    "    index_name=os.environ[\"AZURE_SEARCH_INDEX\"],\n",
    "    credential=AzureKeyCredential(os.environ[\"AZURE_SEARCH_KEY\"])\n",
    ")\n",
    "\n",
    "# Step 2: Retrieve relevant documents from Azure AI Search\n",
    "def retrieve_documents(query: str, top: int = 3):\n",
    "    vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=top, fields=\"text_vector\")\n",
    "    results = search_client.search(search_text=query, vector_queries=[vector_query], select=[\"content\"], top=top)    \n",
    "    return [doc[\"content\"] for doc in results]\n",
    "\n",
    "## Example for Muli-modal search if you have a text_vector AND image_vector field in your vector_index\n",
    "## NOTE, image vectorization is in preview at the time of writing this code, please use azure-search-documents pypi version >11.6.0b6 \n",
    "# def retrieve_documents_multimodal(query: str, image_url: str, top: int = 3):\n",
    "#     text_vector_query = VectorizableTextQuery(\n",
    "#         text=query,\n",
    "#         k_nearest_neighbors=top,\n",
    "#         fields=\"text_vector\",\n",
    "#         weight=0.5  # Adjust weight as needed\n",
    "#     )\n",
    "#     image_vector_query = VectorizableImageUrlQuery(\n",
    "#         url=image_url,\n",
    "#         k_nearest_neighbors=top,\n",
    "#         fields=\"image_vector\",\n",
    "#         weight=0.5  # Adjust weight as needed\n",
    "#     )\n",
    "\n",
    "#     results = search_client.search(\n",
    "#         search_text=query,  \n",
    "#         vector_queries=[text_vector_query, image_vector_query],\n",
    "#         select=[\"content\"],\n",
    "#         top=top\n",
    "#     )\n",
    "#     return [doc[\"content\"] for doc in results]\n",
    "\n",
    "\n",
    "# Step 3: Generate a multimodal RAG-based answer using retrieved text and an image input\n",
    "def generate_multimodal_rag_response(query: str, image_url: str):\n",
    "    # Retrieve text context from search\n",
    "    docs = retrieve_documents(query)\n",
    "    context = \"\\n---\\n\".join(docs)\n",
    "\n",
    "    # Build a prompt that combines the retrieved context with the user query\n",
    "    prompt = f\"\"\"You are a helpful assistant. Use only the following context to answer the question. If the answer isn't in the context, say 'I don't know'.\n",
    "    Context: {context} Question: {query} Answer:\"\"\"\n",
    "    # Create a chat request that includes both text and image input\n",
    "    response = chat_client.complete(\n",
    "        messages=[\n",
    "            SystemMessage(content=\"You are a helpful assistant that can process both text and images.\"),\n",
    "            UserMessage(\n",
    "                content=[\n",
    "                    TextContentItem(text=prompt),\n",
    "                    ImageContentItem(image_url=ImageUrl(url=image_url, detail=ImageDetailLevel.HIGH)),\n",
    "                ]\n",
    "            ),\n",
    "        ]\n",
    "    )\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "# Example usage\n",
    "user_query = \"Can you search for similar items to this shoe?\"\n",
    "sample_image_url = \"https://images.unsplash.com/photo-1542291026-7eec264c27ff?q=80&w=1770&auto=format&fit=crop&ixlib=rb-4.0.3\"\n",
    "answer = generate_multimodal_rag_response(user_query, sample_image_url)\n",
    "print(f\"Q: {user_query}\\nA: {answer}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}