Great interview! Angela was a bit misleading in the "open source" section though, by omitting what is not open-sourced and instead focusing on what is open-sourced. Namely none of their models are open sourced (nor datasets to the best of my knowledge), meaning you can't use any of them for commercial applications and you have to retrain the models from scratch (due to the CC-BY-NC 4.0 license). Which means you have to have 100+ GPUs if you want to train 54B MoE in any reasonable timeframe. With 100 A100 GPUs you need roughly a month. :) (assuming of course you have MT experts and a crystal ball at hand otherwise a lot of additional inefficiencies creep in)
Great interview! Angela was a bit misleading in the "open source" section though, by omitting what is not open-sourced and instead focusing on what is open-sourced. Namely none of their models are open sourced (nor datasets to the best of my knowledge), meaning you can't use any of them for commercial applications and you have to retrain the models from scratch (due to the CC-BY-NC 4.0 license). Which means you have to have 100+ GPUs if you want to train 54B MoE in any reasonable timeframe. With 100 A100 GPUs you need roughly a month. :) (assuming of course you have MT experts and a crystal ball at hand otherwise a lot of additional inefficiencies creep in)
Love hearing what Angela Fan has to say!
Very nice and inspiring. Thanks Jakub.
Very impressive work and great interview too thanks a lot
Very great contribution to humanity
Very nice Angela, Miss you