one of my other (crack) theories is that the low-active-parameter MoE arch really did hurt v3 in the way gwern describes. but the reasoning unhobbles the experts by letting them dump into KV and…

Posters: voooooogel. voooooogel

5 likes recorded by the source corpus.