What is Colle?

Colle is a multidisciplinary French Natural Language Understanding benchmark ( NLU ). It takes inspiration from its predecessors GLUE and SuperGLUE to build a benchmark capable of evaluating models in the French language on multiple topics of language understanding. See our paper for more information.

The Colle benchmark is built with multiple goals in mind. First, it aims to provide a solid and complete French alternative for benchmarking models on NLU tasks. Second, it provides the user with multiple datasets, all usable through HuggingFace’s libraries, to train or fine-tune models on specific tasks.

We have made the choice to hide test labels to discourage cheating or overfitting on test data. To get results on your test data, you may send us your results as explained in our guide.