Microsoft.Extensions.AI.Evaluation - API Usage Examples

Examples.sln contains a set of examples that demonstrate various concepts and APIs that are included in the following NuGet packages:

Microsoft.Extensions.AI.Evaluation: Defines core abstractions and types for supporting evaluation.
Microsoft.Extensions.AI.Evaluation.Quality: Contains evaluators that can be used to evaluate the quality of AI responses in your projects including Relevance, Truth, Completeness, Fluency, Coherence, Retrieval, Equivalence and Groundedness.
Microsoft.Extensions.AI.Evaluation.Safety: Contains a set of evaluators that are built atop the Azure AI Foundry Evaluation service that can be used to evaluate the content safety of AI responses in your projects including Protected Material, Groundedness Pro, Ungrounded Attributes, Hate and Unfairness, Self Harm, Violence, Sexual, Code Vulnerability and Indirect Attack.
Microsoft.Extensions.AI.Evaluation.Reporting: Contains support for caching LLM responses, storing the results of evaluations and generating reports from that data.
Microsoft.Extensions.AI.Evaluation.Reporting.Azure: Supports the Microsoft.Extensions.AI.Evaluation.Reporting library with an implementation for caching LLM responses and storing the evaluation results in an Azure Storage container.

Sample Structure

All examples in Examples.sln are structured as unit tests, and each unit test is a self-contained example that demonstrates a specific concept or API.

Each unit test is defined in its own separate .cs file. However, in order to make it easier to share the same test setup code across multiple unit tests, all tests in a given project are defined within the same (partial) test class.

The examples are split across the following two projects:

Name Description

Evaluation API Examples A set of examples that demonstrate core concepts (such as evaluators, evaluation results, diagnostics and result interpretation) that are included in the Microsoft.Extensions.AI.Evaluation and Microsoft.Extensions.AI.Evaluation.Quality packages above.

The examples in this project demonstrate how to perform evaluations without bringing in any of the reporting functionality present in the Microsoft.Extensions.AI.Evaluation.Reporting package. This can be useful in contexts where you either don't need the response caching, result storage and report generation functionality, or where you need to store and report the results in other ways. An example of this may be the case where you need to perform 'online' evaluation of LLM responses within your deployed production code, and report the evaluation results via your product's existing telemetry pipeline.

Note: The examples included in this project invoke the LLM to perform an evaluation each time the corresponding unit test is executed. These examples do not leverage the response caching functionality described below, and will therefore generally run slower than the examples included in the Reporting API Examples below.

Reporting API Examples A set of examples that demonstrate how concepts that are included in the Microsoft.Extensions.AI.Evaluation and Microsoft.Extensions.AI.Evaluation.Quality packages above can be used in conjunction with concepts (such as response caching, result storage and report generation) that are available in the Microsoft.Extensions.AI.Evaluation.Reporting package.

The examples in this project demonstrate how to orchestrate 'offline' evaluation by leveraging unit testing frameworks, the test runners available within IDEs (to perform evaluations locally as part of your development inner loop) and tools such as dotnet test (to perform evaluations as part of your CI/CD pipelines).

These examples also demonstrate how you can improve performance, and control the costs incurred, when using LLMs to perform frequent offline evaluations (in your CI\CD pipelines, for example). By leveraging the response caching functionality available in the Microsoft.Extensions.AI.Evaluation.Reporting package, you can store previously fetched LLM responses in an (on-prem) cache, and use the cached LLM responses for subsequent requests, as long as the request parameters (such as model and endpoint invoked, request prompts, included context, etc.) remain unchanged.

Finally, these examples also demonstrate how the reporting functionality can be leveraged to store evaluation results (across multiple executions over time), and generate reports based on the stored results.

Note: Because the examples included in this project leverage the response caching functionality, the corresponding unit tests will invoke the LLM only the first time they are executed. The corresponding LLM responses will be cached during this first run. Subsequent runs of the same test will fetch LLM responses from the cache and therefore execute faster (as long as the request parameters remain unchanged). Note that by default, cached responses would expire after 14 days. When you run the examples, any cached responses that have expired will be automatically refreshed by fetching a new response from the LLM.

Running the examples

See INSTRUCTIONS.md