ExecuTorch LLM Android Demo App

This app serves as a valuable resource to inspire your creativity and provide foundational code that you can customize and adapt for your particular use case.

Please dive in and start exploring our demo app today! We look forward to any feedback and are excited to see your innovative ideas.

Key Concepts

From this demo app, you will learn many key concepts such as:

How to prepare Llama models, build the ExecuTorch library, and model inferencing across delegates
Expose the ExecuTorch library via JNI layer
Familiarity with current ExecuTorch app-facing capabilities

The goal is for you to see the type of support ExecuTorch provides and feel comfortable with leveraging it for your use cases.

Supporting Models

As a whole, the models that this app supports are (varies by delegate):

Llama 3.2 Quantized 1B/3B
Llama 3.2 1B/3B in BF16
Llama Guard 3 1B
Llama 3.1 8B
Llama 3 8B
Llama 2 7B
LLaVA-1.5 vision model (only XNNPACK)
Qwen 3 0.6B, 1.7B, and 4B
Voxtral Mini 3B
Gemma 3 4B

Building the APK

First it’s important to note that by default, the app depends on ExecuTorch library on Maven Central. It uses the latest org.pytorch:executorch-android package, which comes with all the default kernel libraries (portable, quantized, optimized), LLM customized libraries, and XNNPACK backend.

No modification is needed if you want to use the pre-built ExecuTorch library.

However, you can build your own ExecuTorch Android library (an AAR file). Copy the file to app/libs/executorch.aar. In gradle.properties file, add a line useLocalAar=true so that gradle uses the local AAR file.

This page contains the documentation for building the ExecuTorch Android library.

Currently ExecuTorch provides support across 4 delegates. Once you identify the delegate of your choice, select the README link to get a complete end-to-end instructions for environment set-up to exporting the models to build ExecuTorch libraries and apps to run on device:

Delegate	Resource
XNNPACK (CPU-based library)	link
QNN (Qualcomm AI Accelerators)	link
MediaTek (MediaTek AI Accelerators)	link
Vulkan	link

How to Use the App

This section will provide the main steps to use the app, along with a code snippet of the ExecuTorch API.

For loading the app, development, and running on device we recommend Android Studio:

Open Android Studio and select "Open an existing Android Studio project" to open the directory containing this README.md file.
Run the app (^R). This builds and launches the app on the phone.

Opening the App

Below are the UI features for the app.

Select the settings widget to get started with picking a model, its parameters and any prompts.

Select Models and Parameters

Once you've selected the model, tokenizer, and model type you are ready to click on "Load Model" to have the app load the model and go back to the main Chat activity.

Optional Parameters:

Temperature: Defaulted to 0, you can adjust the temperature for the model as well. The model will reload upon any adjustments.
System Prompt: More for the advanced user, without any formatting, you can enter in a system prompt. For example, "you are a travel assistant" or "give me a response in a few sentences".
User Prompt: More for the advanced user, if you would like to manually input a prompt then you can do so by modifying the {{user prompt}}. You can also modify the special tokens as well. Once changed then go back to the main Chat activity to send.

ExecuTorch App API

// Upon returning to the Main Chat Activity
mModule = new LlmModule(
            ModelUtils.getModelCategory(mCurrentSettingsFields.getModelType()),
            modelPath,
            tokenizerPath,
            temperature,
            dataPath);
int loadResult = mModule.load();

modelCategory: Indicate whether it’s a text-only or vision model
modePath: path to the .pte file
tokenizerPath: path to the tokenizer file
temperature: model parameter to adjust the randomness of the model’s output
dataPath: path to one or a list of .ptd files

User Prompt

Once model is successfully loaded then enter any prompt and click the send (i.e. generate) button to send it to the model.

You can provide it more follow-up questions as well.

ExecuTorch App API

mModule.generate(prompt,sequence_length, MainActivity.this);

prompt: User formatted prompt
sequence_length: Number of tokens to generate in response to a prompt
MainActivity.this: Indicate that the callback functions (OnResult(), OnStats()) are present in this class.

[LLaVA-1.5: Only for XNNPACK delegate]

For LLaVA-1.5 implementation, select the exported LLaVA .pte and tokenizer file in the Settings menu and load the model. After this you can send an image from your gallery or take a live picture along with a text prompt to the model.

Output Generated

To show completion of the follow-up question, here is the complete detailed response from the model.

Example Output

Llama 3.2 1B

llama-android.mp4

Llava - Llama 2 7b

llava-android.mp4

Gemma 3 4B

gemma-android.mp4

Voxtral Mini 3B

voxtral-S24.mp4

ExecuTorch App API

Ensure you have the following functions in your callback class that you provided in the mModule.generate(). For this example, it is MainActivity.this.

  @Override
  public void onResult(String result) {
    //...result contains token from response
    //.. onResult will continue to be invoked until response is complete
  }

  @Override
  public void onStats(String stats) {
    //... will be a json. See extension/llm/stats.h for the field definitions
  }

Instrumentation Test

You can run the instrumentation test for sanity check. The test loads a model pte file and tokenizer.bin file under /data/local/tmp/llama.

Model preparation

You need to install executorch python package first.

curl -C - -Ls "https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.pt" --output stories110M.pt
curl -C - -Ls "https://raw.githubusercontent.com/karpathy/llama2.c/master/tokenizer.model" --output tokenizer.model
# Create params.json file
touch params.json
echo '{"dim": 768, "multiple_of": 32, "n_heads": 12, "n_layers": 12, "norm_eps": 1e-05, "vocab_size": 32000}' > params.json
python -m executorch.extension.llm.export.export_llm base.checkpoint=stories110M.pt base.params=params.json model.dtype_override="fp16" export.output_name=stories110m_h.pte model.use_kv_cache=True
python -m pytorch_tokenizers.tools.llama2c.convert -t tokenizer.model -o tokenizer.bin

Push model

adb shell mkdir -p /data/local/tmp/llama
adb push stories110m_h.pte /data/local/tmp/llama
adb push tokenizer.bin /data/local/tmp/llama

Run test

./gradlew connectedAndroidTest

Reporting Issues

If you encountered any bugs or issues following this tutorial please file a bug/issue here on Github, or join our discord here.