Sliding window attention hits 100% recall at 1024 tokens. From 512 tokens, adding linear attention to sliding window attention (i.e., Based) provides no benefit for recall. Has anyone tried striped…

Posters: georgejrjrjr. georgejrjrjr

3 likes recorded by the source corpus.