Does this explain the difference between HF and vLLM? I.e. if you enable the same CPU initialization with greedy sampling, we get same outputs?

@casper-hansen Not always. Because floating-point arithmetics is not associative, different kernel implementations might lead to different outputs. The difference is more significant when reduction operation is involved. Therefore, our custom CUDA kernels for attention and RMS normalization do not produce exactly the same outputs as the original HF implementation, and thus outputs of vLLM models can be different from the HF models. We've checked that the outputs usually match when using FP32 and greedy sampling, but there are some cases where the outputs do not match. However, please note that this does not hurt task accuracy, as vLLM's implementation is mathematically equivalent to HF's.