Skip to content

Commit 6eb09b1

Browse files
authored
Merge pull request #369 from ARBML/add-aradice_arabicmmlu_egy
Adding AraDICE-ArabicMMLU-egy to the catalogue
2 parents 86f70d9 + 3a10396 commit 6eb09b1

File tree

1 file changed

+60
-0
lines changed

1 file changed

+60
-0
lines changed
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
{
2+
"Name": " AraDICE-ArabicMMLU-egy",
3+
"Subsets": [],
4+
"HF Link": "https://huggingface.co/datasets/QCRI/AraDICE-ArabicMMLU-egy",
5+
"Link": "https://huggingface.co/datasets/QCRI/AraDICE-ArabicMMLU-egy",
6+
"License": "CC BY-NC-SA 4.0",
7+
"Year": 2024,
8+
"Language": "ar",
9+
"Dialect": "Egypt",
10+
"Domain": [
11+
"public datasets"
12+
],
13+
"Form": "text",
14+
"Collection Style": [
15+
"machine annotation",
16+
"human annotation"
17+
],
18+
"Description": "Within the AraDiCE collection, this particular subset is designated as ArabicMMLU - Egyptian Dialect.",
19+
"Volume": 14459.0,
20+
"Unit": "sentences",
21+
"Ethical Risks": "Low",
22+
"Provider": [
23+
"QCRI"
24+
],
25+
"Derived From": [
26+
"ArabicMMLU"
27+
],
28+
"Paper Title": "ARADICE: Benchmarks for Dialectal and Cultural Capabilities in LLMs",
29+
"Paper Link": "https://arxiv.org/pdf/2409.11404",
30+
"Script": "Arab",
31+
"Tokenized": false,
32+
"Host": "HuggingFace",
33+
"Access": "Free",
34+
"Cost": "",
35+
"Test Split": false,
36+
"Tasks": [
37+
"multiple choice question answering"
38+
],
39+
"Venue Title": "arXiv",
40+
"Venue Type": "preprint",
41+
"Venue Name": "arXiv",
42+
"Authors": [
43+
"Basel Mousi",
44+
"Nadir Durrani",
45+
"Fatema Ahmad",
46+
"Md. Arid Hasan",
47+
"Maram Hasanain",
48+
"Tameem Kabbani",
49+
"Fahim Dalvi",
50+
"Shammur Absar Chowdhury",
51+
"Firoj Alam"
52+
],
53+
"Affiliations": [
54+
"Qatar Computing Research Institute",
55+
"University of New Brunswick",
56+
"American University of Sharjah"
57+
],
58+
"Abstract": "Arabic, with its rich diversity of dialects, re-\nmains significantly underrepresented in Large\nLanguage Models, particularly in dialectal vari-\nations. We address this gap by introducing\nseven synthetic datasets in dialects alongside\nModern Standard Arabic (MSA), created us-\ning Machine Translation (MT) combined with\nhuman post-editing. We present AraDiCE, a\nbenchmark for Arabic Dialect and Cultural\nEvaluation. We evaluate LLMs on dialect com-\nprehension and generation, focusing specifi-\ncally on low-resource Arabic dialects. Addi-\ntionally, we introduce the first-ever fine-grained\nbenchmark designed to evaluate cultural aware-\nness across the Gulf, Egypt, and Levant re-\ngions, providing a novel dimension to LLM\nevaluation. Our findings demonstrate that while\nArabic-specific models like Jais and AceGPT\noutperform multilingual models on dialectal\ntasks, significant challenges persist in dialect\nidentification, generation, and translation. This\nwork contributes \u224845K post-edited samples, a\ncultural benchmark, and highlights the impor-\ntance of tailored training to improve LLM per-\nformance in capturing the nuances of diverse\nArabic dialects and cultural contexts. We have\nreleased the dialectal translation models and\nbenchmarks developed in this study.",
59+
"Added By": "Zaid Alyafeai"
60+
}

0 commit comments

Comments
 (0)