cc @WoosukKwon

Does this PR overlap with #589?

I didn't look too deeply but I think they both should fix the issue!

Marking as draft until benchmarking is complete, but it does look like it brings extra overhead @WoosukKwon

@WoosukKwon I have reworked the code to combine the correctness of TGI approach with high performance of vLLM approach. The new iteration of this PR is not only more correct than the current vLLM implementation but also more performant. PTAL!

@Yard1 Could you please update the test as well? I got errors in the test:

>           output_tokens = prev_output_tokens + new_tokens
E           TypeError: unsupported operand type(s) for +: 'int' and 'list'

I found that aftering applying this pr, some models will produce some unexpected tokens when generating non-ascii texts (especially spaces and </s>). For examples, with the default generation params except max_tokens=256 and temperature=0.0:

1. baichuan-inc/Baichuan-13B-Chat
   prompt:
   <reserved_102>你是谁<reserved_103>
   before:
   我的名字叫百川大模型，是由百川智能的工程师们创造的人工智能大语言模型，我拥有回答问题、聊天互动、文本创作、逻辑推理、数学计算、代码生成等技能，您可以随时向我提问。\n您需要什么帮助吗？
   after:
   我的名字叫百川大模型，是由百川智能的工程师们创造的人工智能大语言模型，我拥有回答问题、聊天互动、文本创作、逻辑推理、数学计算、代码生成等技能，您可以随时向我提问。\n您需要什么帮助吗？？</s>
   problem:
   an extra space at the front, and extra ？ and </s> at the end.

2. baichuan-inc/Baichuan2-13B-Chat
   prompt:
   <reserved_102>你是谁<reserved_103>
   before:
   我是百川大模型，是由百川智能的工程师们创造的大语言模型，我可以和人类进行自然交流、解答问题、协助创作，帮助大众轻松、普惠的获得世界知识和专业服务。如果你有任何问题，可以随时向我提问
   after:
   我是百川大模型，是由百川智能的工程师们创造的大语言模型，我可以和人类进行自然交流、解答问题、协助创作，帮助大众轻松、普惠的获得世界知识和专业服务。如果你有任何问题，可以随时向我提问问</s>
   problem:
   extra 问 and </s> at the end.

Do you have any ideas?

Let me check- I believe the space up front is actually correct, but not sure about the characters at the end