Icelandic LLM leaderboard


🟦	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF	80.31	92.83	71.5	90.79	93.89	93.94	52.66


🟦	gemini-2.5-pro-exp-03-25	82.6	92.83	71.5	90.79	93.89	93.94	52.66
🟦	o1-preview	80.31	92.83	74.5	84.38	92.22	93.43	44.47
🟦	claude-3-5-sonnet-20241022	80.21	91.27	72.5	88.75	92.22	91.3	45.24
🟦	claude-3-5-sonnet-20240620	79.32	90.44	70	89.21	92	89.59	44.66
🟦	claude-3-7-sonnet-20250219	78.9	89.71	67	88.25	92.44	90.36	45.66
🟦	gemini-2.0-pro-exp-02-05	77.99	90.07	60	89.67	93.11	92.58	42.5
🟦	gpt-4o-2024-08-06	76.68	85.39	68	87.75	90.44	90.44	38.03
🟦	gpt-4.1-2025-04-14	76.52	87.41	66.5	84.58	91.78	91.55	37.29
🟦	gpt-4o-2024-05-13	75.77	83.46	65	87.83	90.78	89.76	37.79
🟦	claude-3-opus-20240229	75.03	86.95	63.5	84.21	91.33	85.24	38.92
🟦	gemini-2.0-flash	74.47	84.38	62.5	86.58	91.56	90.78	31
🟦	o4-mini-2025-04-16	72.34	83.73	62.5	73.12	91.78	93	29.92
🟦	gpt-4-turbo	72	85.85	60.5	76.62	89.33	88.74	30.97
🟦	deepseek-ai/DeepSeek-R1	69.85	81.43	59	68.5	90.33	90.61	29.24
🟦	deepseek-ai/DeepSeek-V3	67.5	75.92	57	77.29	87.89	79.69	27.24
🟦	gpt-4.1-mini-2025-04-14	66.62	70.86	59	69.58	89.11	87.2	23.97
🟦	claude-3-5-haiku-20241022	63.92	64.43	61.5	70.62	86.44	77.99	22.55
🟦	gpt-4-0314	63.92	81.16	55.5	54.62	84.89	82.59	24.76
🟦	gpt-4-0613	63.54	83.55	53	50.67	86.56	83.7	23.76
🟦	NousResearch/Hermes-3-Llama-3.1-405B-FP8	62.97	70.59	53.5	61.83	86.11	72.01	33.76
🟦	o3-mini-2025-01-31	62.66	77.85	58	46.04	88.33	89.16	16.58
🟦	gpt-4o-mini-2024-07-18	62.06	65.35	57	63.92	86.11	81.48	18.47
🟦	meta-llama/Llama-3.3-70B-Instruct	58.62	68.66	52.5	48.21	86	74.66	21.71
🟦	meta-llama/Meta-Llama-3.1-70B-Instruct	58.38	70.13	53	46.33	84.67	74.49	21.66
🟦	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF	58.31	70.04	52.5	45.96	84.78	74.57	22
🟦	o1-mini-2024-09-12	58.02	72.61	51	36.54	87.89	86.09	13.97
?	deepseek-ai/Deepseek-R1-Distill-Llama-70B	53.67	75.28	55	21.46	80.67	78.41	11.18
🟦	gpt-4.1-nano-2025-04-14	50.23	52.02	57	39.88	71.11	70.39	11
🟦	claude-2.1	47.64	55.06	52.5	55.21	42.11	59.9	21.05
🟦	meta-llama/Meta-Llama-3-70B-Instruct	47.4	60.02	49	23.75	73.89	60.75	16.97
🟦	gpt-3.5-turbo	44.48	52.02	52	39.12	59.22	49.49	15.03
⭕	Qwen/Qwen2-72B-Instruct	41.66	61.4	48.5	4.46	73.78	55.97	5.87
🟦	meta-llama/Meta-Llama-3.1-8B-Instruct	36.22	51.93	50	17.33	59.44	33.87	4.71
⭕	mistralai/Mistral-Nemo-Instruct-2407	36.06	55.06	53	27.17	31.67	41.89	7.58
🟦	meta-llama/Meta-Llama-3.1-8B	33.1	52.48	50	18.83	42.67	30.97	3.66
🟦	meta-llama/Llama-3.2-3B-Instruct	30.71	49.72	49	10.17	44.11	29.44	1.82
🟦	meta-llama/Meta-Llama-3-8B-Instruct	30.02	49.08	52	7.17	40.11	27.47	4.32
🟢	LumiOpen/Viking-13B	28.48	49.08	49.5	21.46	22.78	22.35	5.74
⭕	AI-Sweden-Models/gpt-sw3-20b-instruct	28.08	49.82	50	17.71	18	22.53	10.42
🟦	meta-llama/Llama-3.2-1B-Instruct	27.58	50.92	50	9.71	28.33	26.28	0.24
⭕	AI-Sweden-Models/gpt-sw3-6.7b-v2-instruct	27.14	49.36	51.5	13.46	20.22	22.44	5.84
🟢	LumiOpen/Viking-33B	27.07	54.32	51.5	16.92	13.11	21.84	4.74
⭕	Qwen/Qwen2-7B-Instruct	26.85	49.63	51	3.25	39.67	15.96	1.61
⭕	AI-Sweden-Models/gpt-sw3-1.3b-instruct	26.84	52.11	50.5	10.25	22.89	22.44	2.84
⭕	AI-Sweden-Models/gpt-sw3-356m-instruct	26.24	56.53	48	14.12	22.78	15.02	1
🟢	LumiOpen/Viking-7B	25.62	48.99	40	15.83	21.67	22.53	4.68
⭕	google/gemma-2-9b-it	24.76	52.85	51	11.04	25.56	7.51	0.61
⭕	AI-Sweden-Models/gpt-sw3-126m-instruct	19.73	53.86	51	12.83	0	0	0.68
🔶	Sigurdur/icebreaker	17.54	56.89	48	0	0.33	0	0


🟦	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF	80.31	92.83	71.5	90.79	93.89	93.94	52.66	instruction-tuned	DeepseekV3ForCausalLM	bfloat16	apache-2.0	684.53	null	false	main


🟦	gemini-2.5-pro-exp-03-25	82.6	92.83	71.5	90.79	93.89	93.94	52.66	RL-tuned	?	?	null	null	null	false
🟦	o1-preview	80.31	92.83	74.5	84.38	92.22	93.43	44.47	RL-tuned	?	?	null	null	null	false
🟦	claude-3-5-sonnet-20241022	80.21	91.27	72.5	88.75	92.22	91.3	45.24	RL-tuned	?	?	null	null	null	false
🟦	claude-3-5-sonnet-20240620	79.32	90.44	70	89.21	92	89.59	44.66	RL-tuned	?	?	null	null	null	false
🟦	claude-3-7-sonnet-20250219	78.9	89.71	67	88.25	92.44	90.36	45.66	RL-tuned	?	?	null	null	null	false
🟦	gemini-2.0-pro-exp-02-05	77.99	90.07	60	89.67	93.11	92.58	42.5	RL-tuned	?	?	null	null	null	false
🟦	gpt-4o-2024-08-06	76.68	85.39	68	87.75	90.44	90.44	38.03	RL-tuned	?	?	null	null	null	false
🟦	gpt-4.1-2025-04-14	76.52	87.41	66.5	84.58	91.78	91.55	37.29	RL-tuned	?	?	null	null	null	false
🟦	gpt-4o-2024-05-13	75.77	83.46	65	87.83	90.78	89.76	37.79	RL-tuned	?	?	null	null	null	false
🟦	claude-3-opus-20240229	75.03	86.95	63.5	84.21	91.33	85.24	38.92	RL-tuned	?	?	null	null	null	false
🟦	gemini-2.0-flash	74.47	84.38	62.5	86.58	91.56	90.78	31	RL-tuned	?	?	null	null	null	false
🟦	o4-mini-2025-04-16	72.34	83.73	62.5	73.12	91.78	93	29.92	RL-tuned	?	?	null	null	null	false
🟦	gpt-4-turbo	72	85.85	60.5	76.62	89.33	88.74	30.97	RL-tuned	?	?	null	null	null	false
🟦	deepseek-ai/DeepSeek-R1	69.85	81.43	59	68.5	90.33	90.61	29.24	RL-tuned	?	?	null	null	null	false
🟦	deepseek-ai/DeepSeek-V3	67.5	75.92	57	77.29	87.89	79.69	27.24	RL-tuned	DeepseekV3ForCausalLM	?	null	684.53	1739	true	main
🟦	gpt-4.1-mini-2025-04-14	66.62	70.86	59	69.58	89.11	87.2	23.97	RL-tuned	?	?	null	null	null	false
🟦	claude-3-5-haiku-20241022	63.92	64.43	61.5	70.62	86.44	77.99	22.55	RL-tuned	?	?	null	null	null	false
🟦	gpt-4-0314	63.92	81.16	55.5	54.62	84.89	82.59	24.76	RL-tuned	?	?	null	null	null	false
🟦	gpt-4-0613	63.54	83.55	53	50.67	86.56	83.7	23.76	RL-tuned	?	?	null	null	null	false
🟦	NousResearch/Hermes-3-Llama-3.1-405B-FP8	62.97	70.59	53.5	61.83	86.11	72.01	33.76	RL-tuned	LlamaForCausalLM	?	llama3	405.87	18	true
🟦	o3-mini-2025-01-31	62.66	77.85	58	46.04	88.33	89.16	16.58	RL-tuned	?	?	null	null	null	false
🟦	gpt-4o-mini-2024-07-18	62.06	65.35	57	63.92	86.11	81.48	18.47	RL-tuned	?	?	null	null	null	false
🟦	meta-llama/Llama-3.3-70B-Instruct	58.62	68.66	52.5	48.21	86	74.66	21.71	RL-tuned	LlamaForCausalLM	bfloat16	llama3.3	70.55	893	true	main
🟦	meta-llama/Meta-Llama-3.1-70B-Instruct	58.38	70.13	53	46.33	84.67	74.49	21.66	RL-tuned	LlamaForCausalLM	bfloat16	llama3.1	70.55	380	true	main
🟦	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF	58.31	70.04	52.5	45.96	84.78	74.57	22	RL-tuned	LlamaForCausalLM	bfloat16	llama3.1	70.55	338	true	main
🟦	o1-mini-2024-09-12	58.02	72.61	51	36.54	87.89	86.09	13.97	RL-tuned	?	?	null	null	null	false
?	deepseek-ai/Deepseek-R1-Distill-Llama-70B	53.67	75.28	55	21.46	80.67	78.41	11.18		?	?	?	0	0	false
🟦	gpt-4.1-nano-2025-04-14	50.23	52.02	57	39.88	71.11	70.39	11	RL-tuned	?	?	null	null	null	false
🟦	claude-2.1	47.64	55.06	52.5	55.21	42.11	59.9	21.05	RL-tuned	?	?	null	null	null	false
🟦	meta-llama/Meta-Llama-3-70B-Instruct	47.4	60.02	49	23.75	73.89	60.75	16.97	RL-tuned	LlamaForCausalLM	bfloat16	llama3	70.55	1276	true	main
🟦	gpt-3.5-turbo	44.48	52.02	52	39.12	59.22	49.49	15.03	RL-tuned	?	?	null	null	null	false
⭕	Qwen/Qwen2-72B-Instruct	41.66	61.4	48.5	4.46	73.78	55.97	5.87	instruction-tuned	Qwen2ForCausalLM	bfloat16	other	72.71	503	true	main
🟦	meta-llama/Meta-Llama-3.1-8B-Instruct	36.22	51.93	50	17.33	59.44	33.87	4.71	RL-tuned	LlamaForCausalLM	bfloat16	llama3.1	8.03	1922	true	main
⭕	mistralai/Mistral-Nemo-Instruct-2407	36.06	55.06	53	27.17	31.67	41.89	7.58	instruction-tuned	MistralForCausalLM	bfloat16	apache-2.0	12.25	961	true	main
🟦	meta-llama/Meta-Llama-3.1-8B	33.1	52.48	50	18.83	42.67	30.97	3.66	RL-tuned	LlamaForCausalLM	bfloat16	llama3.1	8.03	575	true	main
🟦	meta-llama/Llama-3.2-3B-Instruct	30.71	49.72	49	10.17	44.11	29.44	1.82	RL-tuned	LlamaForCausalLM	bfloat16	llama3.2	3.21	362	true	main
🟦	meta-llama/Meta-Llama-3-8B-Instruct	30.02	49.08	52	7.17	40.11	27.47	4.32	RL-tuned	LlamaForCausalLM	bfloat16	llama3	8.03	2905	true	main
🟢	LumiOpen/Viking-13B	28.48	49.08	49.5	21.46	22.78	22.35	5.74	pretrained	LlamaForCausalLM	bfloat16	apache-2.0	14.03	8	true	main
⭕	AI-Sweden-Models/gpt-sw3-20b-instruct	28.08	49.82	50	17.71	18	22.53	10.42	instruction-tuned	GPT2LMHeadModel	bfloat16	other	20.92	12	true	main
🟦	meta-llama/Llama-3.2-1B-Instruct	27.58	50.92	50	9.71	28.33	26.28	0.24	RL-tuned	LlamaForCausalLM	bfloat16	llama3.2	1.24	376	true	main
⭕	AI-Sweden-Models/gpt-sw3-6.7b-v2-instruct	27.14	49.36	51.5	13.46	20.22	22.44	5.84	instruction-tuned	GPT2LMHeadModel	bfloat16	other	7.11	10	true	main
🟢	LumiOpen/Viking-33B	27.07	54.32	51.5	16.92	13.11	21.84	4.74	pretrained	LlamaForCausalLM	bfloat16	apache-2.0	33.12	15	true	main
⭕	Qwen/Qwen2-7B-Instruct	26.85	49.63	51	3.25	39.67	15.96	1.61	instruction-tuned	Qwen2ForCausalLM	bfloat16	apache-2.0	7.62	366	true	main
⭕	AI-Sweden-Models/gpt-sw3-1.3b-instruct	26.84	52.11	50.5	10.25	22.89	22.44	2.84	instruction-tuned	GPT2LMHeadModel	bfloat16	other	1.44	2	true	main
⭕	AI-Sweden-Models/gpt-sw3-356m-instruct	26.24	56.53	48	14.12	22.78	15.02	1	instruction-tuned	GPT2LMHeadModel	bfloat16	other	0.47	0	true	main
🟢	LumiOpen/Viking-7B	25.62	48.99	40	15.83	21.67	22.53	4.68	pretrained	LlamaForCausalLM	bfloat16	apache-2.0	7.55	36	true	main
⭕	google/gemma-2-9b-it	24.76	52.85	51	11.04	25.56	7.51	0.61	instruction-tuned	Gemma2ForCausalLM	bfloat16	gemma	9.24	239	true	main
⭕	AI-Sweden-Models/gpt-sw3-126m-instruct	19.73	53.86	51	12.83	0	0	0.68	instruction-tuned	GPT2LMHeadModel	bfloat16	other	0.19	2	true	main
🔶	Sigurdur/icebreaker	17.54	56.89	48	0	0.33	0	0	fine-tuned	GPT2LMHeadModel	float32	null	0.11	0	true	main

New submissions

Do you want your model to be included on the leaderboard? Open a discussion on this repository with the details of your model and we will get back to you.

Benchmark tasks

The Icelandic LLM leaderboard evaluates models on several tasks. All of them are set up as generation tasks, where the model's output is compared to the expected output. This means that models that have not been instruction fine-tuned might perform poorly on these tasks.

The following tasks are evaluated:

WinoGrande-IS

The Icelandic WinoGrande task is a human-translated and localized version of the ~1000 test set examples in the WinoGrande task in English. Each example consists of a sentence with a blank, and two answer choices for the blank. The task is to choose the correct answer choice using coreference resolution. The benchmark is designed to test the model's ability to use knowledge and common sense reasoning in Icelandic. For this benchmark, we use 3-shot evaluation. The Icelandic WinoGrande dataset is described in more detail in the IceBERT paper (https://aclanthology.org/2022.lrec-1.464.pdf).

Link to dataset: https://huggingface.co/datasets/mideind/icelandic-winogrande

GED

This is a benchmark for binary sentence-level Icelandic grammatical error detection, adapted from the Icelandic Error Corpus (IEC) and contains 200 examples. Each example consists of a sentence that may contain one or more grammatical errors, and the task is to predict whether the sentence contains an error.

Link to dataset: https://huggingface.co/datasets/mideind/icelandic-sentences-gec

Inflection benchmark

The inflection benchmark tests models' ability to generate inflected forms of 300 Icelandic adjective-noun pairs for all four cases, singular and plural.

Link to dataset: https://huggingface.co/datasets/mideind/icelandic-inflection-all-flat

Belebele (IS)

This is the Icelandic subset (900 examples) of the Belebele benchmark, a multiple-choice reading comprehension task. The task is to answer questions about a given passage.

Link to dataset: https://huggingface.co/datasets/facebook/belebele

ARC-Challenge-IS

A machine-translated version of the ARC-Challenge multiple-choice question-answering dataset. For this benchmark, we use the test set which contains 1.23k examples.

Link to dataset: https://huggingface.co/datasets/mideind/icelandic-arc-challenge

WikiQA-IS

The Icelandic WikiQA dataset is a collection of 1.9k question-answer pairs from the Icelandic Wikipedia, meant to evaluate models' knowledge of Icelandic culture and history. They were collected by making GPT-4o generate questions and anwswers given Icelandic Wikipedia articles as context. All examples were then manually verified and corrected where necessary. For evaluation, we prompt GPT-4o to compare the generated answer to the original answer for semantic similarity and rate the answer on the following scale: (0, "poor"), (1, "fair"), (2, "excellent").

Link to dataset: https://huggingface.co/datasets/mideind/icelandic_wiki_qa