Llama Stack: from Zero to Hero

Llama Stack defines and standardizes the set of core building blocks needed to bring generative AI applications to market. These building blocks are presented in the form of interoperable APIs with a broad set of Providers providing their implementations. These building blocks are assembled into Distributions which are easy for developers to get from zero to production.

This guide will walk you through an end-to-end workflow with Llama Stack with Ollama as the inference provider and ChromaDB as the VectorIO provider. Please note the steps for configuring your provider and distribution will vary depending on the services you use. However, the user experience will remain universal - this is the power of Llama-Stack.

If you're looking for more specific topics, we have a Zero to Hero Guide that covers everything from 'Tool Calling' to 'Agents' in detail. Feel free to skip to the end to explore the advanced topics you're interested in.

If you'd prefer not to set up a local server, explore our notebook on tool calling with the Together API. This notebook will show you how to leverage together.ai's Llama Stack Server API, allowing you to get started with Llama Stack without the need for a locally built and running server.

Table of Contents

Llama Stack: from Zero to Hero

Setup ollama

Download Ollama App:
- Go to https://ollama.com/download.
- Follow instructions based on the OS you are on. For example, if you are on a Mac, download and unzip Ollama-darwin.zip.
- Run the Ollama application.
Download the Ollama CLI: Ensure you have the ollama command line tool by downloading and installing it from the same website.
Start ollama server: Open the terminal and run:
```
ollama serve
```
Run the model: Open the terminal and run:
```
ollama run llama3.2:3b-instruct-fp16 --keepalive -1m
```
Note:
- The supported models for llama stack for now is listed in here
- keepalive -1m is used so that ollama continues to keep the model in memory indefinitely. Otherwise, ollama frees up memory and you would have to run ollama run again.

Install Dependencies and Set Up Environment

Install uv: Install uv for managing dependencies:

# macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Install ChromaDB: Install chromadb using uv:
```
uv pip install chromadb
```

Run ChromaDB: Start the ChromaDB server:

chroma run --host localhost --port 8000 --path ./my_chroma_data

Build, Configure, and Run Llama Stack

Install dependencies:

llama stack list-deps starter | xargs -L1 uv pip install

Start the distribution:
```
llama stack run starter
```

Set the ENV variables by exporting them to the terminal:

export OLLAMA_URL="http://localhost:11434"
export LLAMA_STACK_PORT=8321
export INFERENCE_MODEL="meta-llama/Llama-3.2-3B-Instruct"
export SAFETY_MODEL="meta-llama/Llama-Guard-3-1B"

Run the Llama Stack: Run the stack using uv:
```
INFERENCE_MODEL=$INFERENCE_MODEL \
SAFETY_MODEL=$SAFETY_MODEL \
OLLAMA_URL=$OLLAMA_URL \
uv run --with llama-stack llama stack run starter \
   --port $LLAMA_STACK_PORT
```
Note: Every time you run a new model with ollama run, you will need to restart the llama stack. Otherwise it won't see the new model.

The server will start and listen on http://localhost:8321.

Test with `llama-stack-client` CLI

After setting up the server, open a new terminal window and configure the llama-stack-client.

Configure the CLI to point to the llama-stack server.

uv run --with llama-stack-client llama-stack-client configure --endpoint http://localhost:8321

Expected Output:

Done! You can now use the Llama Stack Client CLI with endpoint http://localhost:8321

Test the CLI by running inference:

uv run --with llama-stack-client llama-stack-client inference chat-completion --message "Write me a 2-sentence poem about the moon"

Expected Output:

OpenAIChatCompletion(
   id='chatcmpl-950',
   choices=[
      OpenAIChatCompletionChoice(
            finish_reason='stop',
            index=0,
            message=OpenAIChatCompletionChoiceMessageOpenAIAssistantMessageParam(
               role='assistant',
               content='...The moon casts silver threads through the velvet night, a silent bard of shadows, ancient and bright.',
               name=None,
               tool_calls=None,
               refusal=None,
               annotations=None,
               audio=None,
               function_call=None
            ),
            logprobs=None
      )
   ],
   created=1759240813,
   model='meta-llama/Llama-3.2-3B-Instruct',
   object='chat.completion',
   service_tier=None,
   system_fingerprint='fp_ollama',
   usage={
      'completion_tokens': 479,
      'prompt_tokens': 19,
      'total_tokens': 498,
      'completion_tokens_details': None,
      'prompt_tokens_details': None
   },
)

Test with `curl`

After setting up the server, open a new terminal window and verify it's working by sending a POST request using curl:

curl http://localhost:$LLAMA_STACK_PORT/v1/chat/completions
-H "Content-Type: application/json"
-d @- <<EOF
{
    "model": "$INFERENCE_MODEL",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write me a 2-sentence poem about the moon"}
    ],
      "temperature": 0.7,
      "seed": 42,
      "max_tokens": 512
   }
}
EOF

You can check the available models with the command uv run --with llama-stack-client llama-stack-client models list.

Expected Output:

{
    ...
    "content": "... The moon glows softly in the midnight sky,\nA beacon of wonder, as it catches the eye.",
    ...
}

Test with Python

You can also interact with the Llama Stack server using a simple Python script. Below is an example:

1. Create Python Script (`test_llama_stack.py`)

touch test_llama_stack.py

2. Create a Chat Completion Request in Python

In test_llama_stack.py, write the following code:

import os
from llama_stack_client import LlamaStackClient

# Get the model ID from the environment variable
INFERENCE_MODEL = os.environ.get("INFERENCE_MODEL")

# Check if the environment variable is se
if INFERENCE_MODEL is None:
    raise ValueError("The environment variable 'INFERENCE_MODEL' is not set.")

# Initialize the clien
client = LlamaStackClient(base_url="http://localhost:8321")

# Create a chat completion request
response = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Write a two-sentence poem about llama."},
    ],
    model=INFERENCE_MODEL,
)

# Print the response
print(response.choices[0].message.content)

3. Run the Python Script

uv run --with llama-stack-client python test_llama_stack.py

Expected Output:

The moon glows softly in the midnight sky,
A beacon of wonder, as it catches the eye.

With these steps, you should have a functional Llama Stack setup capable of generating text using the specified model. For more detailed information and advanced configurations, refer to some of our documentation below.

This command initializes the model to interact with your local Llama Stack instance.

Next Steps

Explore Other Guides: Dive deeper into specific topics by following these guides:

Understanding Distribution
Inference 101
Local and Cloud Model Toggling 101
Prompt Engineering
Chat with Image - LlamaStack Vision API
Tool Calling: How to and Details
Memory API: Show Simple In-Memory Retrieval
Using Safety API in Conversation
Agents API: Explain Components

Explore Client SDKs: Utilize our client SDKs for various languages to integrate Llama Stack into your applications:

Advanced Configuration: Learn how to customize your Llama Stack distribution by referring to the Building a Llama Stack Distribution guide.

Explore Example Apps: Check out llama-stack-apps for example applications built using Llama Stack.