Description

Changes:

1. Fixed BindCapacityScheduler to pass peft_cache_manager to the CPP binding.
   1.1. Fixed BindCapacityScheduler constructions accordingly.
2. Fixed PeftCacheManager.free_resources to call mark_request_done
3. Changed LoRA "prevent serialization of entire LoRA adapters in each request" optimization to check whether an adapter was loaded in the LoRA CPU cache directly, instead of the additional python dict adapter cache in LoraManager.
   3.1. Removed support for this optimization for non-torch flow.
   3.2. Added LoraManager.is_adapter_in_cpu_cache method.
   3.3. Added optional cpp_peft_cache_manager argument to LoraManager constructor, used by its newly added is_adapter_in_cpu_cache method.
   3.4. Changed GenerationExecutorWorker in torch flow only, to get the CPP peft cache manager and pass it to LoraManager constructor.
4. Added tests of lora eviction to test_llm.py and to test_llm_pytorch.py.
5. Refactored existing LoRA tests in test_llm.py and test_llm_pytorch.py to a separate file, saving duplication.
6. Added a detailed "not supported note" in the thrown exception in the unsupported flow of handling a LoRA request that has no LoRA weights/config and not found in LoRA CPU cache that was introduced in [TRTLLM-5921][feat] Prevent serialization of entire LoRA adapters in each request #5080.
   6.1. For pytorch flow - Changed PeftCacheManager::determineNumPages to throw a PeftTaskNotCachedException with a detailed "not supported note" when the request has no lora weights, no lora config, and its lora adapter is not found in cache.
   6.2. For TRT flow - Appended a detailed "not supported note" to the error message thrown in PeftCacheManager::addRequestPeft when the request has no lora weights or no lora config and not lora adapter not found in cache. REVERTED, as the optimization was disabled for non-torch flow.

Test Coverage

• test_llm.py::test_llama_7b_multi_lora_evict_load_new_adapters
• test_llm_pytorch.py::test_llama_7b_multi_lora_evict_load_new_adapters
• test_llm_pytorch.py::test_llama_7b_multi_lora_load_previously_cpu_cache_evicted_adapter_fails

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Summary by CodeRabbit

• New Features

  • Added ability to check if a LoRA adapter is present in the CPU cache.
  • Enhanced LoRA adapter management for both PyTorch and TensorRT backends, improving cache integration and resource handling.
  • Introduced new utility functions and context managers for test subprocess management and list operations.

• Bug Fixes

  • Improved error messages and exception handling when LoRA adapters are missing or evicted from cache.

• Tests

  • Replaced and expanded LoRA adapter tests to cover adapter eviction, cache reloading, and failure scenarios.
  • Added new test utilities for multi-LoRA adapter scenarios and subprocess execution with output capture.

• Documentation

  • Added explanatory comments to clarify backend-specific LoRA behavior and cache limitations.