"Can you figure out what the experts in a Mixture of Experts model are each specialized in?" Yes, this is touched on in the Mixtral paper (2024) and discussed quite extensively in the ST-MoE paper (2022), section 7. Also summarized in hf.co/blog/moe#what-does-an-… People's intuition is that MoE experts should specialize in a specific topic (e.g., chemistry). That is not the case (except for FrankenMoEs/MoErges or a new pattern called shared experts) One thing to remember is that the MoEs gate/router aims to load balance. If you use topic-specialized experts, then you risk all tokens of topic X being sent to the same expert and end up having unused experts and filling the capacity of popular experts. This won't work well as then for every type of context, just one set of experts will be used. Hence, the MoE loss function includes an auxiliary loss that encourages giving all experts equal importance. Due to this auxiliary loss function and the routing, you end up having experts specializing in different types of tokens (e.g., verbs, proper names, numbers, etc), which makes sure that the tokens are well distributed across experts. People are asking for human descriptions for all experts, and I don't think that is exactly possible. That's like asking what each attention map in Vision Transformers means. We have nice tools for getting the attention maps and visualizing; we can see some patterns, but trying to expect pretrained MoE experts to have topic specialization is somewhat inaccurate If you want to dive more into it, I suggest looking at DeepSeekMoE paper (2024), which has some very interesting ideas about knowledge hybridity and redundancy and uses shared experts to capture common knowledge across varying contexts. I hope this was useful 🤗 Reads: - ST-MoE (2022) arxiv.org/abs/2202.08906 - Mixtral (2024) arxiv.org/abs/2401.04088 - DeepSeekMoE (2024) arxiv.org/abs/2401.06066 - My comprehensive reading list of MoE papers huggingface.co/collections/o… - MoEs blog post hf.co/blog/moe cc @natfriedman @maximelabonne @dwarkesh_sp