"topic:attention-sink" — Search

2 results for “topic:attention-sink”

[ICLR 2025] When Attention Sink Emerges in Language Models: An Empirical View (Spotlight)

attention-mechanismattention-sinklanguage-modellarge-language-models

kelvindelrosario/flash-attention-with-sink

🐙 Implements Flash Attention with sink for gpt-oss-20b; includes test.py. WIP backward pass, varlen support, and community sync to return softmax_lse only.

Python20Updated 1 hour ago

attentionattention-sinkattention-withattention-with-sinkflashflash-attentionflash-attention-sinkflash-attention-withflash-attention-with-sinkflash-sinkflash-withflash-with-sinksinkwithwith-sink