Benchmarks-for-NLP

• Benchmarks-for-NLP
  • Open-Source LLM (10-35B, Chinese & English)
  • GLUE Benchmark (NLU Tasks, English)
  • Named Entity Recognition
    • LLM
    • Flat NER
    • Nested NER
    • Biomedical NER
  • Relation Extraction
    • Document-level
    • Sentence-level
    • Biomedical
  • Joint Entity and Relation Extraction
  • Key Information Extraction
  • Link Prediction
    • WN18RR

Open-Source LLM (10-35B, Chinese & English)

Leaderboard from OpenCompass

Model	Org.	Param.	Examination	Language	Knowledge	Understanding	Reasoning
GPT-4	OpenAI	N/A	77.2	62	73.5	70	74.4
ChatGLM3-6B-Base	ZhipuAI	6B	67.2	52.4	62	70.3	67.4
Qwen-14B	Alibaba	14B	71.3	52.7	56.1	68.8	60.1
Yi-34B	01.AI	34B	78.1	48.9	64.5	69.2	55.5
Qwen-14B-Chat	Alibaba	14B	71.2	52.1	61.2	68.2	54.9
InternLM-20B	Shanghai AI Lab & SenseTime	20B	62.5	55	60.1	67.3	54.9
Aquila2-34B	BAAI	34B	70	47.2	59.2	66.9	50.1
Baichuan2-13B-Chat	Baichuan Intelligent Technology	13B	59.8	51.5	51.9	63.1	50.1

GLUE Benchmark (NLU Tasks, English)

Single-task single models on dev

Model	MNLI	QNLI	QQP	RTE	SST	MRPC	CoLA	STS
BERT-large	86.6	92.3	91.3	70.4	93.2	88.0	60.6	90.0
XLNet-large	89.8	93.9	91.8	83.8	95.6	89.2	63.6	91.8
RoBERTa-large	90.2	94.7	92.2	86.6	96.4	90.9	68.0	92.4
ALBERT-xxlarge	90.8	95.3	92.2	89.2	96.9	90.9	71.4	93.0

其它语言模型

1. ERNIE 3.0：自回归与自编码结合的框架
2. BART：编码器mask+解码器自回归
3. PERT：基于文本乱序自监督的预训练编码器
4. MacBERT：针对中文提出Whole Word Masking和N-gram masking技术

About GLUE

1. CoLA：单句二分类任务，判断是否符合语法。
2. SST：单句二分类任务，判断情感积极/消极。
3. MRPC：双句释义二分类，判断两句话是否相同含义。
4. STS：双句相似性回归任务，1-5之间打分。
5. QQP：双句相似性分类任务，判断两句问句含义是否相同。
6. MNLI：三分类任务，判断两句关系：包含、矛盾和中立。输入句子对，一个为条件，另一个为假设。
7. QNLI：二分类，判断问题和句子之间是否为包含关系。
8. RTE：二分类，判断句子对是否互为包含关系。

Named Entity Recognition

F1 for all

LLM

Method	Param.	Org.	Year	CoNLL03	OntoNotes	ACE05	BC5CDR	NCBI
UniversalNER	7B	University of Southern California, Microsoft Research	2023	93.30	89.91	86.69	89.34	86.96

Flat NER

Method	Model	Org.	Year	CoNLL03(Eng)	OntoNotes
ACE	BiLSTM-CRF + BiLSTM-Biaffine	ShanghaiTech, UCAS, DAMO	ACL2021	94.6
CL-KL	Bio-BERT + CRF	DAMO	ACL2021	93.56
BOPN	BERT-Large+BiLSTM+CLN+3DConv	CAS, UCAS, Meituan	EMNLP2023	93.19	91.16
PromptNER	RoBERTa-large	ZJU	ACL 2023	93.08
PIQN	BERT+2BiLSTM	ZJU, DAMO	ACL2022	92.87	90.96
BART NER	BART-large	FDU	ACL2021	93.24	90.38
Locate and Label	BERT-large-cased	ZJU, USTC	ACL2021	92.94
Named Entity Recognition as Dependency Parsing	BERT-Large + fastText embeddings	Queen Mary University, Google Research	ACL2020	92.5	89.83

Nested NER

Method	Model	Org.	Year	ACE04	ACE05	Genia	KBP17
BOPN	BERT-Large+BiLSTM+CLN+3DConv	CAS, UCAS, Meituan	EMNLP2023	89.26	90.39	82.14
PromptNER	RoBERTa-large	ZJU	ACL 2023	88.72	88.26
PIQN	BERT+2BiLSTM	ZJU, DAMO	ACL2022	88.14	87.42	81.77	84.50
Locate and Label	BERT-large-cased	ZJU, USTC	ACL2021	87.41	86.67	80.54	84.05
BART NER	BART-large	FDU	ACL2021	86.84	84.74	79.23
Named Entity Recognition as Dependency Parsing	BERT-Large + fastText embeddings	Queen Mary University, Google Research	ACL2020	85.67	84.61	78.87
Neural Architectures for Nested NER through Linearization	seq2seq+BERT+Flair	Charles University	ACL2019	84.40	84.33	78.31

Biomedical NER

Method	Model	Org.	Year	BC5CDR	NCBI-disease
BINDER	PubMedBERT	Microsoft Research	ICLR 2023	91.9	90.9
ConNER	BioBERT/BioLM	Korea University	Oxford University Press 2022	91.3	89.9
CL-KL	Bio-BERT + CRF	DAMO	ACL2021	90.93	88.96
BioFLAIR	BioFLAIR (V1)+BioELMo	Manipal Institute of Technology, Elsevier Labs	Arxiv2019	89.42	88.85
BioBERT	BioBERT	Korea University, Clova AI	Oxford University Press 2020		87.70(token-level F1)

Relation Extraction

RE F1 for all
DocRED is a large scale dataset constructed from Wikipedia and Wikidata.
CDR(Chemical-Disease Reactions) is a biomedical dataset constructed using PubMed abstracts.
GDA(Gene-Disease Associations) is also a binary relation classification task that identify Gene and Disease concepts interactions.

Document-level

Reproduced	Method	Model	Org.	Year	DocRED-Test	Re-DocRED-Test
	DocRE-CLiP	BERT	Indraprastha Institute of Information Technology	AAAI2024	68.51	81.55
✅	AA-RE	RoBERTa-large	Beihang University	EMNLP2023	64.98	81.20
	DREEAM	RoBERTa-large	Tokyo Institute of Technology	ACL2023	64.27	80.73
	SAIS	RoBERTa-large	CMU, Stanford	NAACL 2022	65.11	-
	EIDER	RoBERTa-large	University of Illinois at Urbana-Champaign	ACL2022	64.79	-
	KD-DocRE	RoBERTa-large	DAMO, National University of Singapore	ACL2022	64.28	78.65
	SSAN	RoBERTa-large	USTC, Baidu	AAAI2021	61.42	-

Sentence-level

Method	Model	Org.	Year	TACRED	SemEval
SuRE	PEGASUS	University of Southern California	EMNLP 2022	75.1	89.7
GPT-RE	GPT3	Kyoto University	EMNLP 2023	70.97	91.82
NLI	DeBERTa	Ixa NLP Group,UPV/EHU	EMNLP 2021	73.9	-

Biomedical

Method	Model	Org.	Year	CDR	GDA
SAIS	SciBERT	CMU, Stanford	NAACL 2022	79.0	87.1
EIDER	SciBERT	University of Illinois at Urbana-Champaign	ACL2022	70.63	84.54
SSAN	SciBERT	USTC, Baidu	AAAI2021	68.7	83.7

Joint Entity and Relation Extraction

Reproduced	Method	Encoder	Year		ACE05			ACE04			SciERC
				Ent	Rel	Rel+	Ent	Rel	Rel+	Ent	Rel	Rel+
✅	HGERE	BERT/SciBERT	EMNLP 2023	90.2	70.7	67.5	89.9	68.2	64.2	74.9	55.7	43.6
✅	HGERE	ALBERT	EMNLP 2023	91.9	73.5	70.8	91.9	71.9	68.3
	GCN	BERT	ACL 2019	90.2	69.6	66.5	90	67.6	63.5	74.1	54.8	42.9
	GCN	ALBERT	ACL 2019	91.7	73.1	69.9	92	71.5	67.9
	Nguyen et al., 2021	BERT	ACL 2021	88.9	68.9
	DyGIE++	BERT	ACL 2019	88.6	63.4					67.5	48.4
	DyGIE++	albert	ACL 2019	89.5	67.6	64.3	88.6	63.3	59.6
	PURE	BERT	ACL 2021	90.1	67.7	64.8	89.2	63.9	60.1	68.9	50.1	36.8
	PURE	ALBERT	ACL 2021	90.9	69.4	67.0	90.3	66.1	62.2
	MFVI	BERT		90.2	69.7	67.1	89.7	67.4	63.4	73.3	54.7	42.5
	MFVI	ALBERT		91.6	72.7	70.1	89.9	68.5	65.1
✅	PL-Marker(re)*	BERT	ACL 2022	90	69.8	66.7	89.5	66.6	62.1	71.3	52.3	40.2
✅	PL-Marker(re)*	ALBERT	ACL 2022	91.5	72.9	70.2	91.6	70.2	66.6

Key Information Extraction

SROIE : Scanned receipts OCR and information extraction
CORD is a Consolidated Receipt Dataset for Post-OCR Parsing.
FUNSD is a Form Understanding in Noisy Scanned Documents (FUNSD) comprises 199 real, fully annotated, scanned forms.
F1 for all.
default: BASE-MODEL

Reproduced	Model	Org.	Year	Modality	SROIE	CORD	FUNSD
✅	GeoLayoutLM-Large	DAMO	CVPR 2023	T+L+V	-	97.97	92.86
	LayoutLMv3	Sun Yat-sen University, Microsoft Research Asia	ACM MM 2022	T+L+V	95.30	96.56	90.29
	LILT	SCUT	ACL 2022	T+L+V	97.65	95.11	-
✅	GenKIE	University of Michigan, National University of Defense Technology	EMNLP 2023	T+L+V	97.40	95.75	83.45
	DocFormer	AWS AI	ICCV2021	T+L+V	-	96.33	83.34
	StrucText	Baidu	ACM MM 2021	T+L+V	96.88	-	83.09

Link Prediction

MRR: mean reciprocal rank
Hits@k: H@k for brevity

WN18RR

Reproduced	Model	Method	Org.	Year	MRR	H@1	H@3	H@10
❌	MoCoSA	description-based	Kuaishou Technology	2023	69.6	62.4	73.7	82.0
	PMD	description-based	Tsinghua University	2024	67.8	58.8	73.7	82.0
✅	SimKGC	description-based	Microsoft Research Asia, Yuanfudao AI Lab	ACL 2022	67.1	58.7	73.1	81.7
	CSPromp-KG	structure-based	Nanyang Technological University	ACL 2023	57.5	52.2	59.6	67.8
	KG-S2S	description-based	Nanyang Technological University	COLING 2022	57.4	53.1	59.5	66.1
✅	NBFNet	structure-based	Mila - Québec AI Institute	NeurIPS 2021	55.1	49.7	57.3	66.6
	C-LMKE	description-based	Fudan University	IJCAI 2022	59.8	48.0	67.5	80.6
	TuckER	structure-based	University of Edinburgh, Samsung AI Centre	COLING 2019	47.0	44.3	48.2	52.6
	KICGPT	description-based	South University of Science and Technology	EMNLP 2024	56.4	47.8	61.2	67.7