Dataset Card for EurlexResources: A Corpus Covering the Largest EURLEX Resources

Dataset Summary

This dataset contains large text resources (~179GB in total) from EURLEX that can be used for pretraining language models.

Use the dataset like this:

from datasets import load_dataset
config = "de_caselaw" # {lang}_{resource}
dataset = load_dataset("joelito/eurlex_resources", config, split='train', streaming=True) 

Supported Tasks and Leaderboards

The dataset supports the task of masked language modeling.

Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

Dataset Structure

Data Instances

The file format is jsonl.xz and there is one split available ("train"). The following resource types are supported: caselaw, decision, directive, intagr, proposal, recommendation, regulation

More information about the resource types can be found here:

• Caselaw: EU
• Decision: EU , Wikipedia
• Directive: EU , Wikipedia
• Recommendation: EU , Wikipedia
• Regulation: EU , Wikipedia
• Intagr: EU , Wikipedia
• Proposal: No resource found

Source	Size (MB)	Words	Documents	Words/Document
all_all	180668	12106556233	8306749	1457
all_caselaw	34939	3413551598	2487794	1372
all_decision	28519	1698585620	1267402	1340
all_directive	4786	368577940	104187	3537
all_intagr	11421	743271516	274485	2707
all_proposal	26526	2087989530	702392	2972
all_recommendation	1886	164979037	80277	2055
all_regulation	72590	3629600992	3390212	1070
bg_all	7819	398067053	348691	1141
bg_caselaw	1588	109749174	104434	1050
bg_decision	1248	58817972	54075	1087
bg_directive	263	15731608	4388	3585
bg_intagr	603	31292848	11581	2702
bg_proposal	1083	60674956	29251	2074
bg_recommendation	89	5588991	3321	1682
bg_regulation	2943	116211504	141641	820
cs_all	8360	471961631	449793	1049
cs_caselaw	1163	110005022	104519	1052
cs_decision	1102	58921128	54075	1089
cs_directive	186	13951134	4388	3179
cs_intagr	449	28106332	11581	2426
cs_proposal	840	61838692	29252	2113
cs_recommendation	64	5416549	3323	1630
cs_regulation	4557	193722774	242655	798
da_all	8932	671484862	332500	2019
da_caselaw	1746	185589641	88234	2103
da_decision	1356	89498535	54085	1654
da_directive	207	17525792	4388	3994
da_intagr	506	35596169	11582	3073
da_proposal	1399	119759476	29257	4093
da_recommendation	100	9463897	3352	2823
da_regulation	3618	214051352	141602	1511
de_all	9607	695512401	348290	1996
de_caselaw	1930	193232441	104228	1853
de_decision	1449	93688222	53980	1735
de_directive	218	17337760	4385	3953
de_intagr	531	36791153	11580	3177
de_proposal	1556	126987454	29219	4346
de_recommendation	109	9608034	3318	2895
de_regulation	3813	217867337	141580	1538
el_all	12469	696216541	349667	1991
el_caselaw	2951	202027703	105138	1921
el_decision	1823	94919886	54150	1752
el_directive	321	19411959	4390	4421
el_intagr	701	38965777	11584	3363
el_proposal	2085	128005737	29290	4370
el_recommendation	145	9344866	3357	2783
el_regulation	4443	203540613	141758	1435
en_all	9217	769465561	348641	2207
en_caselaw	1846	222891827	104422	2134
en_decision	1504	114626013	54054	2120
en_directive	204	18860876	4388	4298
en_intagr	499	39029843	11581	3370
en_proposal	1538	140781768	29242	4814
en_recommendation	97	10091809	3320	3039
en_regulation	3530	223183425	141634	1575
es_all	8588	725125274	348443	2081
es_caselaw	1870	220621730	104312	2115
es_decision	1334	98163499	54001	1817
es_directive	221	21484479	4385	4899
es_intagr	516	41841805	11581	3612
es_proposal	1366	133674486	29224	4574
es_recommendation	82	8864018	3319	2670
es_regulation	3199	200475257	141621	1415
et_all	6090	328068754	349615	938
et_caselaw	1074	93096396	105111	885
et_decision	1069	50752324	54159	937
et_directive	177	11555930	4390	2632
et_intagr	436	24018147	11584	2073
et_proposal	810	51600852	29283	1762
et_recommendation	61	4451369	3355	1326
et_regulation	2464	92593736	141733	653
fi_all	7346	404265224	349633	1156
fi_caselaw	1596	126525296	105119	1203
fi_decision	1227	59659475	54163	1101
fi_directive	204	12766491	4389	2908
fi_intagr	463	25392311	11584	2192
fi_proposal	1075	69198401	29288	2362
fi_recommendation	73	5070392	3356	1510
fi_regulation	2707	105652858	141734	745
fr_all	9937	828959218	348295	2380
fr_caselaw	2158	246262666	104228	2362
fr_decision	1473	108648744	53981	2012
fr_directive	222	20308801	4385	4631
fr_intagr	536	41986012	11580	3625
fr_proposal	1592	149134298	29218	5104
fr_recommendation	112	11510415	3318	3469
fr_regulation	3845	251108282	141585	1773
ga_all	1028	65030095	349778	185
ga_caselaw	11	696305	105205	6
ga_decision	87	4415457	54189	81
ga_directive	18	1512027	4390	344
ga_intagr	19	1820723	11586	157
ga_proposal	289	26106889	29298	891
ga_recommendation	10	902390	3361	268
ga_regulation	594	29576304	141749	208
hr_all	4594	258816068	348691	742
hr_caselaw	617	62432734	104434	597
hr_decision	596	31911903	54075	590
hr_directive	156	10855913	4388	2474
hr_intagr	450	24962086	11581	2155
hr_proposal	552	33437815	29251	1143
hr_recommendation	40	3612247	3321	1087
hr_regulation	2183	91603370	141641	646
hu_all	6653	375253894	349605	1073
hu_caselaw	1278	110179375	105144	1047
hu_decision	1147	57108172	54156	1054
hu_directive	200	13568304	4389	3091
hu_intagr	470	27258501	11586	2352
hu_proposal	912	60882750	29291	2078
hu_recommendation	70	5312868	3357	1582
hu_regulation	2576	100943924	141682	712
it_all	9586	768605772	333631	2303
it_caselaw	1889	206117726	89560	2301
it_decision	1445	102848859	53983	1905
it_directive	217	19687773	4385	4489
it_intagr	528	40134330	11580	3465
it_proposal	1533	140713925	29218	4816
it_recommendation	109	10923431	3318	3292
it_regulation	3865	248179728	141587	1752
lt_all	6400	364361783	200565	1816
lt_caselaw	1137	101808706	105477	965
lt_decision	1096	55850308	21990	2539
lt_directive	185	13078983	3239	4037
lt_intagr	452	27009631	7481	3610
lt_proposal	850	58553579	29272	2000
lt_recommendation	64	5121089	3363	1522
lt_regulation	2617	102939487	29743	3460
lv_all	6349	363239195	349919	1038
lv_caselaw	1153	103456811	105242	983
lv_decision	1103	55512944	54224	1023
lv_directive	186	13023024	4392	2965
lv_intagr	452	26693107	11630	2295
lv_proposal	96	58176216	29298	1985
lv_recommendation	64	5074494	3361	1509
lv_regulation	2545	101302599	141772	714
mt_all	6540	367834815	350292	1050
mt_caselaw	1164	100423543	105479	952
mt_decision	1109	55239141	54280	1017
mt_directive	203	14355266	4392	3268
mt_intagr	470	27701991	11675	2372
mt_proposal	878	59749277	29274	2041
mt_recommendation	65	5039600	3363	1498
mt_regulation	2650	105325997	141829	742
nl_all	9586	770312808	349407	2204
nl_caselaw	1847	206271837	105005	1964
nl_decision	1456	104060901	54152	1921
nl_directive	217	19529361	4388	4450
nl_intagr	529	40247634	11584	3474
nl_proposal	1540	141258274	29279	4824
nl_recommendation	111	11002405	3355	3279
nl_regulation	3886	247942396	141644	1750
pl_all	6677	406648795	350349	1160
pl_caselaw	1231	115824759	105479	1098
pl_decision	1125	60407576	54287	1112
pl_directive	197	14672157	4392	3340
pl_intagr	466	28543668	11680	2443
pl_proposal	886	64728230	29317	2207
pl_recommendation	68	5769893	3363	1715
pl_regulation	2703	116702512	141831	822
pt_all	8450	675152149	348449	1937
pt_caselaw	1763	198084937	104312	1898
pt_decision	1327	93278293	54007	1727
pt_directive	217	19831549	4385	4522
pt_intagr	504	37999753	11581	3281
pt_proposal	1361	127461782	29224	4361
pt_recommendation	81	8396661	3319	2529
pt_regulation	3197	190099174	141621	1342
ro_all	6315	415038571	350300	1184
ro_caselaw	1110	114780999	105516	1087
ro_decision	1047	59479553	54281	1095
ro_directive	206	16101628	4392	3666
ro_intagr	481	31497000	11675	2697
ro_proposal	805	62130419	29274	2122
ro_recommendation	63	5977913	3363	1777
ro_regulation	2603	125071059	141799	882
sk_all	6484	392235510	350570	1118
sk_caselaw	1160	110125141	105608	1042
sk_decision	1111	59576875	54349	1096
sk_directive	188	14132755	4393	3217
sk_intagr	458	28298155	11676	2423
sk_proposal	859	63726047	29290	2175
sk_recommendation	66	5654790	3364	1680
sk_regulation	2642	110721747	141890	780
sl_all	6222	394814289	350574	1126
sl_caselaw	1071	111238184	105608	1053
sl_decision	1075	59454906	54349	1093
sl_directive	176	13908097	4393	3165
sl_intagr	441	28239078	11676	2418
sl_proposal	812	63391970	29290	2164
sl_recommendation	62	5628775	3364	1673
sl_regulation	2585	112953279	141894	796
sv_all	7419	500085970	351051	1424
sv_caselaw	1585	162108645	105980	1529
sv_decision	1213	71744934	54357	1319
sv_directive	195	15386273	4393	3502
sv_intagr	463	29845462	11676	2556
sv_proposal	1059	86016237	29292	2936
sv_recommendation	79	7152141	3366	2124
sv_regulation	2825	127832278	141987	900

Data Fields

[More Information Needed]

Data Splits

[More Information Needed]

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

The data has been downloaded using the R package eurlex between June and August 2022.

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

CC BY 4.0 see also the legal notice

Citation Information

[More Information Needed]

Contributions

Thanks to @JoelNiklaus for adding this dataset.