Asaf Gardin and Yuval Belfer investigate rare gibberish from a hybrid Mamba and attention model served through vLLM. Lowering GPU memory utilization made the first failure reproducible under concurrent requests, while a comparison with Hugging Face Transformers exposed token probability differences. The failure did not appear as a crash or warning.
Tracing a failing request ID into the model forward pass revealed that the scheduler could send a new request to Mamba's decode path before prefill. That path read stale state from earlier requests. The correction classified requests with no computed tokens as prefill work before they reached the kernels.
A separate training issue produced periodic log-probability spikes between rollout and training calculations despite unchanged weights and inputs. Increasing rollouts per prompt shifted the failure earlier; shrinking the state buffer made it disappear. The cause was a 32-bit cache index wrapping at large offsets, fixed by using a wider size type. The broader debugging method is to compare token probabilities against a baseline, change parameters that alter the failure pattern, and carry request identity deep enough to inspect state.
Watch on YouTube
