Hi all,

In the age of decoder-only LLMs, I'll like to ask if there's any competitive encoder-decoder architectures that are known to scale well for multilingual seq2seq tasks?

- https://huggingface.co/docs/transformers/en/model_doc/mt5 - https://huggingface.co/facebook/m2m100_418M - https://huggingface.co/google-bert/bert-base-multilingual-cased + https://www.kaggle.com/code/alvations/neural-plasticity-bert2bert-on-wmt14 - https://huggingface.co/Helsinki-NLP/opus-mt-en-mul + https://huggingface.co/Helsinki-NLP/opus-mt-mul-en - https://huggingface.co/docs/transformers/en/model_doc/umt5

There's these that reported state-of-the-art NLI scores but they were not known to be multilingual

- https://huggingface.co/google/ul2 - https://huggingface.co/docs/transformers/en/model_doc/flan-t5 - https://huggingface.co/docs/transformers/en/model_doc/byt5

There's some ideas on doing encoder with mamba https://github.com/state-spaces/mamba/issues/78 but it looks like an open question.

Other than the above, are there any competitive encoder-decoder architectures that are known to scale well for multilingual seq2seq tasks?

Thank you in advance for the pointers!

Regards, Liling