Width vs. Depth: Speculating on the Margin

The article discusses the benefits of speculative decoding in large language models, specifically the trade-off between width (batching multiple sequences) and depth (speculating on a single sequence), and proposes a confidence-gated speculation approach to optimize throughput.