Neural Architecture Search

Morphing into Hybrid Attention Models

Talon and Wildflower discuss FlashMorph, a new method for choosing which Transformer layers should keep full attention when converting pretrained LLMs into hybrid attention models.