KEMBAR78
Guidance API extensions added by danikhan632 · Pull Request #2637 · oobabooga/text-generation-webui · GitHub
Skip to content

Conversation

danikhan632
Copy link

this is an extension to microsoft/guidance to allow text-generation-webui models to be used which allow for updated models, easy setup, mutiple GPUs etc to be used with gudiance

@danikhan632
Copy link
Author

corrensponds with this pr

guidance-ai/guidance#221

@Chrusciki
Copy link

im sorry, but i don't see any code in the folder, just a new sub module.

@danikhan632
Copy link
Author

just fixed the submodule issue and it should show a new extension

@romansop
Copy link

several issues found after trying the patch:

  1. port 9000 conflicts with ports 9000-9004 used by vscode python extensions, better to use more random port e.g. 9555
  2. extension can be loaded only after LLM model is already loaded, otherwise error in GuidanceGenerator.init on line self.bos_token= shared.tokenizer.bos_token, where self.tokenizer is None
  3. print_type is not defined but used in several places, triggering runtime error

@danikhan632 danikhan632 force-pushed the main branch 2 times, most recently from 740664b to 909d8c6 Compare June 16, 2023 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants