Add Llama API OAI compatible endpoint support #6442

WuhanMonkey · 2025-04-30T16:00:01Z

Why are these changes needed?

To add the latest support for using Llama API offerings with AutoGen

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

Add Llama API Experimental

SongChiYoung · 2025-04-30T23:31:28Z

Llama support is cool!
could you adding register_transformer at _message_transfrom to?
https://github.com/microsoft/autogen/blob/main/python/packages/autogen-ext/src/autogen_ext/models/openai/_message_transform.py

You could start with __BASE_TRANSFORMER_MAP
or build __LLAMA_TRANSFROMER_MAP same as __BASE_TRANSFORMER_MAP for future modified.

autogen/python/packages/autogen-ext/src/autogen_ext/models/openai/_message_transform.py

Lines 413 to 432 in c7757de

    
           # === Transformers === 
        
           __BASE_TRANSFORMER_MAP: TransformerMap = { 
        
               SystemMessage: build_transformer_func( 
        
                   funcs=system_message_transformers, 
        
                   message_param_func=ChatCompletionSystemMessageParam, 
        
               ), 
        
               UserMessage: build_conditional_transformer_func( 
        
                   funcs_map=user_transformer_funcs, 
        
                   message_param_func_map=user_transformer_constructors, 
        
                   condition_func=user_condition, 
        
               ), 
        
               AssistantMessage: build_conditional_transformer_func( 
        
                   funcs_map=assistant_transformer_funcs, 
        
                   message_param_func_map=assistant_transformer_constructors, 
        
                   condition_func=assistant_condition, 
        
               ), 
        
               FunctionExecutionResultMessage: function_execution_result_message, 
        
           }

and...

autogen/python/packages/autogen-ext/src/autogen_ext/models/openai/_message_transform.py

Lines 470 to 492 in c7757de

    
           # set openai models to use the transformer map 
        
           total_models = get_args(ModelFamily.ANY) 
        
           __openai_models = [model for model in total_models if ModelFamily.is_openai(model)] 
        
           __claude_models = [model for model in total_models if ModelFamily.is_claude(model)] 
        
           __gemini_models = [model for model in total_models if ModelFamily.is_gemini(model)] 
        
           __unknown_models = list(set(total_models) - set(__openai_models) - set(__claude_models) - set(__gemini_models)) 
        
           for model in __openai_models: 
        
               register_transformer("openai", model, __BASE_TRANSFORMER_MAP) 
        
           for model in __claude_models: 
        
               register_transformer("openai", model, __CLAUDE_TRANSFORMER_MAP) 
        
           for model in __gemini_models: 
        
               register_transformer("openai", model, __GEMINI_TRANSFORMER_MAP) 
        
           for model in __unknown_models: 
        
               register_transformer("openai", model, __BASE_TRANSFORMER_MAP) 
        
           register_transformer("openai", "default", __BASE_TRANSFORMER_MAP)

I think you are expert of Llama. If you know about Llama's message limited, could support more well with _message_transfrom.

In Anthropic case, Claude does not support empty message, so this message transformer remove empty messages.

Adding links to Llama API website for sign-up

ekzhu · 2025-05-06T21:33:44Z

@WuhanMonkey thanks for the PR. Could you address @SongChiYoung 's comments?

Set Llama models to use base message transformer. It is fully compatible with OAI

WuhanMonkey · 2025-05-06T22:50:51Z

@WuhanMonkey thanks for the PR. Could you address @SongChiYoung 's comments?

Hey yes, just updated it.

We are still pending on CLA review from our legal side.

WuhanMonkey · 2025-05-14T19:35:37Z

@microsoft-github-policy-service agree company="Meta"

WuhanMonkey · 2025-05-14T19:45:48Z

We finally get the CLA approved and signed. @SongChiYoung and @ekzhu would you mind help me review and approve this PR? Thanks

WuhanMonkey · 2025-05-14T19:48:13Z

One more question, @SongChiYoung, does AutoGen allows extra headers in the request for customized x-title or http-referer for tracking purpose?

ekzhu · 2025-05-15T21:32:45Z

One more question, @SongChiYoung, does AutoGen allows extra headers in the request for customized x-title or http-referer for tracking purpose?

You can pass in default_headers as part of the OpenAIChatCompletionClient. https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.OpenAIChatCompletionClient

Fixed lint error

Fix issue during rebase.

codecov · 2025-05-15T22:58:59Z

Codecov Report

Attention: Patch coverage is 76.47059% with 4 lines in your changes missing coverage. Please review.

Project coverage is 79.52%. Comparing base (1eb7f93) to head (5413c51).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...xt/src/autogen_ext/models/openai/_openai_client.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6442      +/-   ##
==========================================
- Coverage   79.53%   79.52%   -0.01%     
==========================================
  Files         225      225              
  Lines       16644    16661      +17     
==========================================
+ Hits        13237    13249      +12     
- Misses       3407     3412       +5

Flag	Coverage Δ
unittests	`79.52% <76.47%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Fix mypy

ekzhu · 2025-05-15T23:50:19Z

@WuhanMonkey I will bring it to finish. Thanks. For local checks you can see python/README.md for guides.

WuhanMonkey · 2025-05-16T00:38:23Z

@WuhanMonkey I will bring it to finish. Thanks. For local checks you can see python/README.md for guides.

Thanks appreciate it.

WuhanMonkey added 4 commits April 24, 2025 16:11

Add Llama API support

e817fb7

Update for vision support in Llama 4

9b368d4

Update the model name

066a447

Merge branch 'microsoft:main' into main

671b708

WuhanMonkey marked this pull request as draft April 30, 2025 16:01

WuhanMonkey added 2 commits April 30, 2025 09:25

Update models.ipynb

853be08

Add Llama API Experimental

Merge branch 'main' of github.com:WuhanMonkey/autogen

c5bebd7

WuhanMonkey marked this pull request as ready for review April 30, 2025 16:28

Update models.ipynb

b8e979d

Adding links to Llama API website for sign-up

Update _message_transform.py

3bbe3e7

Set Llama models to use base message transformer. It is fully compatible with OAI

Merge branch 'main' into main

be9dc09

WuhanMonkey added 2 commits May 15, 2025 14:39

Update _message_transform.py

771179b

Fixed lint error

Update _model_client.py

7bb28f6

Fix issue during rebase.

Update models.ipynb

b0167e9

Fix mypy

ekzhu approved these changes May 15, 2025

View reviewed changes

formatting

5413c51

ekzhu merged commit 9d29731 into microsoft:main May 15, 2025
63 of 64 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Llama API OAI compatible endpoint support #6442

Add Llama API OAI compatible endpoint support #6442

Uh oh!

WuhanMonkey commented Apr 30, 2025

Uh oh!

SongChiYoung commented Apr 30, 2025 •

edited

Loading

Uh oh!

ekzhu commented May 6, 2025

Uh oh!

WuhanMonkey commented May 6, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

ekzhu commented May 15, 2025

Uh oh!

codecov bot commented May 15, 2025 •

edited

Loading

Uh oh!

ekzhu commented May 15, 2025

Uh oh!

Uh oh!

WuhanMonkey commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add Llama API OAI compatible endpoint support #6442

Add Llama API OAI compatible endpoint support #6442

Uh oh!

Conversation

WuhanMonkey commented Apr 30, 2025

Why are these changes needed?

Checks

Uh oh!

SongChiYoung commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ekzhu commented May 6, 2025

Uh oh!

WuhanMonkey commented May 6, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

WuhanMonkey commented May 14, 2025

Uh oh!

ekzhu commented May 15, 2025

Uh oh!

codecov bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ekzhu commented May 15, 2025

Uh oh!

Uh oh!

WuhanMonkey commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SongChiYoung commented Apr 30, 2025 •

edited

Loading

codecov bot commented May 15, 2025 •

edited

Loading