ترجمه فارسی مقاله فراتر از تعداد پارامترها: تعصب ضمنی در ترکیب نرم متخصصان

• کتاب صدها نکته فارسی (خودمونی) – نسخه PDF — زبان ساده و کاربردی
  مشاهده نمونه نسخه نکات ساده
• کتاب صدها نکته رسمی فارسی – نسخه PDF — نگارش استاندارد و علمی
  مشاهده نمونه نسخه نکات رسمی
• کتاب صدها پرسش و پاسخ تشریحی – نسخه PDF
  — هر سؤال همراه با پاسخ کامل برای درک عمیق مفاهیم
  مشاهده نمونه نسخه پرسش و پاسخ
• کتاب پرسش و پاسخ چهارگزینه‌ای – نسخه یادگیری سریع
  — پاسخ‌ها بلافاصله بعد از سؤال برای مرور سریع
  مشاهده نمونه نسخه کوییز سریع
• کتاب پرسش و پاسخ چهارگزینه‌ای – نسخه خودآزمایی
  — پاسخ‌ها در انتهای بخش‌ها برای سنجش واقعی یادگیری
  مشاهده نمونه نسخه آزمونی

🎯 این بسته یک دورهٔ آموزشی کامل و چندلایه است؛ شامل ویدیوهای آموزشی، کتاب‌ها، تمرین‌ها و خودآزمایی.

چکیده

The traditional viewpoint on Sparse Mixture of Experts (MoE) models is that instead of training a single large expert, which is computationally expensive, we can train many small experts. The hope is that if the total parameter count of the small experts equals that of the singular large expert, then we retain the representation power of the large expert while gaining computational tractability and promoting expert specialization. The recently introduced Soft MoE replaces the Sparse MoE's discrete routing mechanism with a differentiable gating function that smoothly mixes tokens. While this smooth gating function successfully mitigates the various training instabilities associated with Sparse MoE, it is unclear whether it induces implicit biases that affect Soft MoE's representation power or potential for expert specialization. We prove that Soft MoE with a single arbitrarily powerful expert cannot represent simple convex functions. This justifies that Soft MoE's success cannot be explained by the traditional viewpoint of many small experts collectively mimicking the representation power of a single large expert, and that multiple experts are actually necessary to achieve good representation power (even for a fixed total parameter count). Continuing along this line of investigation, we introduce a notion of expert specialization for Soft MoE, and while varying the number of experts yet fixing the total parameter count, we consider the following (computationally intractable) task. Given any input, how can we discover the expert subset that is specialized to predict this input's label? We empirically show that when there are many small experts, the architecture is implicitly biased in a fashion that allows us to efficiently approximate the specialized expert subset. Our method can be easily implemented to potentially reduce computation during inference.

چکیده به فارسی (ترجمه ماشینی)