clarinsi4ai
Evaluation and benchmarks
← Overview · Part of CLARIN.SI resources for AI development · Data as of 22 October 2026
Slovenian has a usable LLM evaluation suite: SuperGLUE and SQuAD 2.0 translations, SI-NLI, SloWiC, KE-WSC, a bias set (EEC-SL) and, since 2026, legal NLI, pragmatics and ethics sets. The other South Slavic languages are covered more thinly: COPA for Croatian, Serbian, Macedonian and dialects, PragMega and CoSimLex for Croatian, AspectBench for Bosnian, Croatian and Serbian, and the multilingual word-analogy sets. The SloBENCH framework, listed under models and tools, hosts leaderboards for several of the Slovenian sets.
A multilingual dataset counts once for each of its languages.
Sections: LLM benchmarks · Task and lexical test sets
LLM benchmarks
WALS-Bench and GramQA (2026) are multilingual and test metalinguistic and grammatical knowledge across many languages.
Slovenian, including multilingual resources with Slovenian
Other languages
| Resource | Languages | Size | Licence | Year |
|---|---|---|---|---|
| Choice of plausible alternatives dataset in Croatian COPA-HR | hr | 1k items | CC BY-SA | 2021 |
| Choice of plausible alternatives dataset in Macedonian COPA-MK | mk | 1k items | CC BY-SA | 2022 |
| Choice of plausible alternatives dataset in Serbian COPA-SR | sr | 1k items | CC BY-SA | 2022 |
| A multilingual benchmark for evaluating metalinguistic knowledge WALS-Bench 1.0 | en, mul | 76.7k entries | CC BY | 2026 |
| Corpus-grounded evaluation dataset for grammatical question answering GramQA 1.0 | mul | 1.9k entries | CC BY | 2026 |
| English translation of the Slovene Natural Language Inference Dataset SI-NLI-en 1.0 | en | 5.9k units | CC BY-NC-SA | 2024 |
Task and lexical test sets
SYNDIST and COLLDIST (2025) test LLM lexical knowledge with distractors; Šolar-Eval and DASSLE target grammatical error correction.
Other stages: Pre-training data · Speech and multimodal data · Post-training data · Evaluation and benchmarks · Lexical and knowledge resources · Models and tools