On steps-to-reach-loss, FLOPs-to-reach-benchmark, inference speed, etc, the mixture-of-experts paradigm gets between 2x~5x improvement over a dense baseline. Fundamentally, MoE is unfriendly to low…

Posters: gleech. gleech

2 likes recorded by the source corpus.