数据集:

joelito/MultiLegalPileWikipediaFiltered

任务:

填充掩码

语言:

计算机处理:

multilingual

大小:

10M<n<100M

语言创建人:

found

批注创建人:

other

源数据集:

original

许可:

cc-by-4.0

数据集介绍文件清单

中文

Dataset Card for MultiLegalPileWikipediaFiltered: A filtered version of the MultiLegalPile dataset, together with wikipedia articles

Dataset Summary

The Multi_Legal_Pile is a large-scale multilingual legal dataset suited for pretraining language models. It spans over 24 languages and four legal text types.

Supported Tasks and Leaderboards

The dataset supports the tasks of fill-mask.

Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

Dataset Structure

It is structured in the following format: {language} {text_type} {shard}.jsonl.xz

text_type is one of the following:

caselaw
contracts
legislation
other
wikipedia

Use the dataset like this:

from datasets import load_dataset

config = 'en_contracts' # {language}_{text_type}
dataset = load_dataset('joelito/Multi_Legal_Pile', config, split='train', streaming=True)

'config' is a combination of language and text_type, e.g. 'en_contracts' or 'de_caselaw'. To load all the languages or all the text_types, use 'all' instead of the language or text_type (e.g., ' all_legislation').

Data Instances

The file format is jsonl.xz and there is a train and validation split available. Since some configurations are very small or non-existent, they might not contain a train split or not be present at all.

The complete dataset consists of five large subsets:

Language	Source	Size (MB)	Words	Documents	Words/Document
all	all	1.29761e+06	81214262514	57305071	1417
all	caselaw	695837	44372248995	30085886	1474
all	contracts	122599	7964531030	1785686	4460
all	legislation	189135	10879386581	3601518	3020
all	other	126570	8780080882	3358073	2614
all	wikipedia	163468	9218015026	18473908	498
bg	all	14028	535256525	355650	1505
bg	caselaw	2897	109634090	52648	2082
bg	contracts	748	31292877	7107	4403
bg	legislation	8015	308946116	82777	3732
bg	other	0	0	0	0
bg	wikipedia	2368	85383442	213118	400
cs	all	21818	1123000335	839914	1337
cs	caselaw	11151	574336489	296652	1936
cs	contracts	492	28106428	7383	3806
cs	legislation	6288	333850509	88731	3762
cs	other	0	0	0	0
cs	wikipedia	3887	186706909	447148	417
da	all	16024	970954498	576256	1684
da	caselaw	3469	210730560	89702	2349
da	contracts	559	35592407	10827	3287
da	legislation	10736	653153146	265868	2456
da	other	0	0	0	0
da	wikipedia	1259	71478385	209859	340
de	all	63887	3512253170	3216030	1092
de	caselaw	31527	1785439383	596800	2991
de	contracts	614	36786772	11041	3331
de	legislation	8934	512840663	276034	1857
de	other	0	0	0	0
de	wikipedia	22812	1177186352	2332155	504
el	all	23167	800722723	457553	1750
el	caselaw	6007	203770918	85496	2383
el	contracts	1050	38963772	10266	3795
el	legislation	12906	455240770	171356	2656
el	other	0	0	0	0
el	wikipedia	3204	102747263	190435	539
en	all	712173	47279626514	21112650	2239
en	caselaw	380976	25561971376	10240724	2496
en	contracts	71360	7260323438	1594942	4552
en	legislation	36587	2537696894	657805	3857
en	other	126570	8780080882	3358073	2614
en	wikipedia	51053	3139553924	5261106	596
es	all	23657	1515689548	1567527	966
es	caselaw	3299	220506573	83872	2629
es	contracts	594	41840328	10048	4164
es	legislation	6837	462661276	149368	3097
es	other	0	0	0	0
es	wikipedia	12928	790681371	1324239	597
et	all	7446	372896353	261641	1425
et	caselaw	1835	92951578	58736	1582
et	contracts	433	24017402	7371	3258
et	legislation	4200	210952455	63922	3300
et	other	0	0	0	0
et	wikipedia	978	44974918	131612	341
fi	all	11501	513990484	592986	866
fi	caselaw	2854	126368889	77882	1622
fi	contracts	504	25386705	8894	2854
fi	legislation	5532	252344531	103907	2428
fi	other	0	0	0	0
fi	wikipedia	2610	109890359	402303	273
fr	all	47186	2936056985	2734954	1073
fr	caselaw	18313	1170335690	435569	2686
fr	contracts	633	41983091	11071	3792
fr	legislation	9297	600170792	243313	2466
fr	other	0	0	0	0
fr	wikipedia	18942	1123567412	2045001	549
ga	all	1209	72041312	30064	2396
ga	caselaw	11	676795	835	810
ga	contracts	29	1820765	365	4988
ga	legislation	1048	62513018	5983	10448
ga	other	0	0	0	0
ga	wikipedia	122	7030734	22881	307
hr	all	5377	315295665	211151	1493
hr	caselaw	1026	62358456	31322	1990
hr	contracts	395	24957774	6552	3809
hr	legislation	2906	171415656	36365	4713
hr	other	0	0	0	0
hr	wikipedia	1050	56563779	136912	413
hu	all	12351	564082537	495822	1137
hu	caselaw	2376	110034426	59074	1862
hu	contracts	534	27258352	7385	3691
hu	legislation	5744	264572303	86862	3045
hu	other	0	0	0	0
hu	wikipedia	3697	162217456	342501	473
it	all	26744	1658638775	1615301	1026
it	caselaw	6483	406520336	156630	2595
it	contracts	597	40131223	10985	3653
it	legislation	8332	542579039	227968	2380
it	other	0	0	0	0
it	wikipedia	11332	669408177	1219718	548
lt	all	7772	399310081	264537	1509
lt	caselaw	1992	101672069	59485	1709
lt	contracts	475	27009922	7473	3614
lt	legislation	4550	235543873	64106	3674
lt	other	0	0	0	0
lt	wikipedia	755	35084217	133473	262
lv	all	7701	386833125	211244	1831
lv	caselaw	2082	103311512	58992	1751
lv	contracts	481	26692972	7429	3593
lv	legislation	4621	233088284	64087	3637
lv	other	0	0	0	0
lv	wikipedia	518	23740357	80736	294
mt	all	7180	370558634	122056	3035
mt	caselaw	2016	100309542	52942	1894
mt	contracts	486	27701852	6937	3993
mt	legislation	4620	239708644	57979	4134
mt	other	0	0	0	0
mt	wikipedia	58	2838596	4198	676
nl	all	17674	1112460059	1200534	926
nl	caselaw	3227	206147113	87170	2364
nl	contracts	604	40245662	11027	3649
nl	legislation	8484	550788527	232204	2372
nl	other	0	0	0	0
nl	wikipedia	5360	315278757	870133	362
pl	all	14762	773692198	1160849	666
pl	caselaw	2141	115695709	59649	1939
pl	contracts	489	28543526	7478	3817
pl	legislation	5459	299334705	89264	3353
pl	other	0	0	0	0
pl	wikipedia	6672	330118258	1004458	328
pt	all	210656	13466463586	18173061	741
pt	caselaw	196919	12611760973	17251236	731
pt	contracts	571	37997495	9897	3839
pt	legislation	6853	439066783	148176	2963
pt	other	0	0	0	0
pt	wikipedia	6313	377638335	763752	494
ro	all	14794	808799454	481763	1678
ro	caselaw	1960	114665535	53092	2159
ro	contracts	495	31496978	7202	4373
ro	legislation	10464	559092153	215694	2592
ro	other	0	0	0	0
ro	wikipedia	1874	103544788	205775	503
sk	all	8700	463447112	262638	1764
sk	caselaw	2072	109996398	59383	1852
sk	contracts	489	28298113	7470	3788
sk	legislation	5208	280182047	76760	3650
sk	other	0	0	0	0
sk	wikipedia	931	44970554	119025	377
sl	all	9345	561775614	277497	2024
sl	caselaw	1816	111097741	59193	1876
sl	contracts	432	28238938	7475	3777
sl	legislation	6057	365513763	88651	4123
sl	other	0	0	0	0
sl	wikipedia	1041	56925172	122178	465
sv	all	12457	700417227	1083393	646
sv	caselaw	2806	161956844	78802	2055
sv	contracts	491	29844238	9061	3293
sv	legislation	5456	308130634	104338	2953
sv	other	0	0	0	0
sv	wikipedia	3704	200485511	891192	224

Data Fields

[More Information Needed]

Data Splits

There are two splits: train and validation. The validation split contains 1000 examples and the training split contains the rest of the data.

Data Size

$ xz --list data/*.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1    167.6 MiB  3’276.3 MiB  0.051  CRC64   data/bg_caselaw_train.0.jsonl.xz
    1       1    502.3 KiB  9’398.0 KiB  0.053  CRC64   data/bg_caselaw_validation.0.jsonl.xz
    1       1     33.4 MiB    700.3 MiB  0.048  CRC64   data/bg_contracts_train.0.jsonl.xz
    1       1  5’989.6 KiB    123.0 MiB  0.048  CRC64   data/bg_contracts_validation.0.jsonl.xz
    1       1    418.5 MiB  8’931.0 MiB  0.047  CRC64   data/bg_legislation_train.0.jsonl.xz
    1       1  5’029.4 KiB    103.1 MiB  0.048  CRC64   data/bg_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/bg_other_validation.0.jsonl.xz
    1       1    192.2 MiB  2’488.6 MiB  0.077  CRC64   data/bg_wikipedia_train.0.jsonl.xz
    1       1  1’757.8 KiB     22.9 MiB  0.075  CRC64   data/bg_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  4’126.1 MiB  0.116  CRC64   data/cs_caselaw_train.0.jsonl.xz
    1       1    259.8 MiB  2’556.9 MiB  0.102  CRC64   data/cs_caselaw_train.1.jsonl.xz
    1       1    420.1 KiB  3’370.3 KiB  0.125  CRC64   data/cs_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    237.9 MiB  0.105  CRC64   data/cs_contracts_train.0.jsonl.xz
    1       1  4’412.1 KiB     41.7 MiB  0.103  CRC64   data/cs_contracts_validation.0.jsonl.xz
    1       1    361.2 MiB  3’488.9 MiB  0.104  CRC64   data/cs_legislation_train.0.jsonl.xz
    1       1     10.3 MiB     91.6 MiB  0.112  CRC64   data/cs_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/cs_other_validation.0.jsonl.xz
    1       1    390.6 MiB  1’939.4 MiB  0.201  CRC64   data/cs_wikipedia_train.0.jsonl.xz
    1       1  2’604.7 KiB     12.2 MiB  0.209  CRC64   data/cs_wikipedia_validation.0.jsonl.xz
    1       1    252.5 MiB  1’529.7 MiB  0.165  CRC64   data/da_caselaw_train.0.jsonl.xz
    1       1    555.9 KiB  3’227.1 KiB  0.172  CRC64   data/da_caselaw_validation.0.jsonl.xz
    1       1     30.1 MiB    233.9 MiB  0.129  CRC64   data/da_contracts_train.0.jsonl.xz
    1       1  2’897.6 KiB     23.6 MiB  0.120  CRC64   data/da_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’325.8 MiB  0.143  CRC64   data/da_legislation_train.0.jsonl.xz
    1       1    237.3 MiB  1’444.5 MiB  0.164  CRC64   data/da_legislation_train.1.jsonl.xz
    1       1  3’232.5 KiB     60.6 MiB  0.052  CRC64   data/da_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/da_other_validation.0.jsonl.xz
    1       1    128.8 MiB    512.1 MiB  0.252  CRC64   data/da_wikipedia_train.0.jsonl.xz
    1       1  1’514.1 KiB  5’476.3 KiB  0.276  CRC64   data/da_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  2’803.8 MiB  0.170  CRC64   data/de_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  2’821.4 MiB  0.169  CRC64   data/de_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  2’720.2 MiB  0.175  CRC64   data/de_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  2’704.1 MiB  0.176  CRC64   data/de_caselaw_train.3.jsonl.xz
    1       1    460.5 MiB  2’504.5 MiB  0.184  CRC64   data/de_caselaw_train.4.jsonl.xz
    1       1    594.0 KiB  3’416.4 KiB  0.174  CRC64   data/de_caselaw_validation.0.jsonl.xz
    1       1     32.0 MiB    255.8 MiB  0.125  CRC64   data/de_contracts_train.0.jsonl.xz
    1       1  3’037.7 KiB     24.7 MiB  0.120  CRC64   data/de_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’386.0 MiB  0.141  CRC64   data/de_legislation_train.0.jsonl.xz
    1       1     93.3 MiB    592.3 MiB  0.158  CRC64   data/de_legislation_train.1.jsonl.xz
    1       1  3’265.9 KiB     20.5 MiB  0.156  CRC64   data/de_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/de_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’883.7 MiB  0.253  CRC64   data/de_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  1’891.6 MiB  0.252  CRC64   data/de_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  1’893.7 MiB  0.252  CRC64   data/de_wikipedia_train.2.jsonl.xz
    1       1    476.9 MiB  1’894.1 MiB  0.252  CRC64   data/de_wikipedia_train.3.jsonl.xz
    1       1    407.9 MiB  1’622.0 MiB  0.251  CRC64   data/de_wikipedia_train.4.jsonl.xz
    1       1  1’172.5 KiB  4’210.2 KiB  0.278  CRC64   data/de_wikipedia_validation.0.jsonl.xz
    1       1    344.7 MiB  6’908.3 MiB  0.050  CRC64   data/el_caselaw_train.0.jsonl.xz
    1       1    870.4 KiB     14.3 MiB  0.060  CRC64   data/el_caselaw_validation.0.jsonl.xz
    1       1     49.7 MiB  1’083.8 MiB  0.046  CRC64   data/el_contracts_train.0.jsonl.xz
    1       1  4’701.3 KiB    101.6 MiB  0.045  CRC64   data/el_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB     10.2 GiB  0.046  CRC64   data/el_legislation_train.0.jsonl.xz
    1       1    203.0 MiB  3’994.0 MiB  0.051  CRC64   data/el_legislation_train.1.jsonl.xz
    1       1  9’744.3 KiB    186.6 MiB  0.051  CRC64   data/el_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/el_other_validation.0.jsonl.xz
    1       1    246.4 MiB  3’465.7 MiB  0.071  CRC64   data/el_wikipedia_train.0.jsonl.xz
    1       1  2’591.7 KiB     35.6 MiB  0.071  CRC64   data/el_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  2’188.6 MiB  0.218  CRC64   data/en_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  2’416.1 MiB  0.197  CRC64   data/en_caselaw_train.10.jsonl.xz
    1       1    477.2 MiB  2’688.1 MiB  0.178  CRC64   data/en_caselaw_train.11.jsonl.xz
    1       1    476.9 MiB  2’865.9 MiB  0.166  CRC64   data/en_caselaw_train.12.jsonl.xz
    1       1    476.9 MiB  2’494.1 MiB  0.191  CRC64   data/en_caselaw_train.13.jsonl.xz
    1       1    476.9 MiB  2’126.6 MiB  0.224  CRC64   data/en_caselaw_train.14.jsonl.xz
    1       1    476.9 MiB  2’440.9 MiB  0.195  CRC64   data/en_caselaw_train.15.jsonl.xz
    1       1    476.9 MiB  3’822.2 MiB  0.125  CRC64   data/en_caselaw_train.16.jsonl.xz
    1       1    476.9 MiB  3’831.4 MiB  0.124  CRC64   data/en_caselaw_train.17.jsonl.xz
    1       1    476.9 MiB  3’812.2 MiB  0.125  CRC64   data/en_caselaw_train.18.jsonl.xz
    1       1    476.9 MiB  2’233.5 MiB  0.214  CRC64   data/en_caselaw_train.19.jsonl.xz
    1       1    476.9 MiB  2’195.9 MiB  0.217  CRC64   data/en_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  2’185.8 MiB  0.218  CRC64   data/en_caselaw_train.20.jsonl.xz
    1       1    476.9 MiB  2’634.9 MiB  0.181  CRC64   data/en_caselaw_train.21.jsonl.xz
    1       1    476.9 MiB  2’670.8 MiB  0.179  CRC64   data/en_caselaw_train.22.jsonl.xz
    1       1    476.9 MiB  2’762.0 MiB  0.173  CRC64   data/en_caselaw_train.23.jsonl.xz
    1       1    476.9 MiB  2’153.6 MiB  0.221  CRC64   data/en_caselaw_train.24.jsonl.xz
    1       1    476.9 MiB  2’152.0 MiB  0.222  CRC64   data/en_caselaw_train.25.jsonl.xz
    1       1    476.9 MiB  2’205.0 MiB  0.216  CRC64   data/en_caselaw_train.26.jsonl.xz
    1       1    476.9 MiB  2’141.0 MiB  0.223  CRC64   data/en_caselaw_train.27.jsonl.xz
    1       1    476.9 MiB  2’145.1 MiB  0.222  CRC64   data/en_caselaw_train.28.jsonl.xz
    1       1    476.9 MiB  2’137.9 MiB  0.223  CRC64   data/en_caselaw_train.29.jsonl.xz
    1       1    476.9 MiB  2’189.0 MiB  0.218  CRC64   data/en_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  2’150.9 MiB  0.222  CRC64   data/en_caselaw_train.30.jsonl.xz
    1       1    476.9 MiB  2’142.7 MiB  0.223  CRC64   data/en_caselaw_train.31.jsonl.xz
    1       1    476.9 MiB  2’203.4 MiB  0.216  CRC64   data/en_caselaw_train.32.jsonl.xz
    1       1    476.9 MiB  2’205.4 MiB  0.216  CRC64   data/en_caselaw_train.33.jsonl.xz
    1       1    476.9 MiB  2’206.0 MiB  0.216  CRC64   data/en_caselaw_train.34.jsonl.xz
    1       1    476.9 MiB  2’164.9 MiB  0.220  CRC64   data/en_caselaw_train.35.jsonl.xz
    1       1    476.9 MiB  2’810.3 MiB  0.170  CRC64   data/en_caselaw_train.36.jsonl.xz
    1       1    476.9 MiB  2’854.1 MiB  0.167  CRC64   data/en_caselaw_train.37.jsonl.xz
    1       1    476.9 MiB  3’109.2 MiB  0.153  CRC64   data/en_caselaw_train.38.jsonl.xz
    1       1    476.9 MiB  3’323.6 MiB  0.143  CRC64   data/en_caselaw_train.39.jsonl.xz
    1       1    476.9 MiB  2’155.3 MiB  0.221  CRC64   data/en_caselaw_train.3.jsonl.xz
    1       1    476.9 MiB  2’881.5 MiB  0.165  CRC64   data/en_caselaw_train.40.jsonl.xz
    1       1    476.9 MiB  2’157.1 MiB  0.221  CRC64   data/en_caselaw_train.41.jsonl.xz
    1       1    477.0 MiB  2’530.2 MiB  0.189  CRC64   data/en_caselaw_train.42.jsonl.xz
    1       1    476.8 MiB  2’540.1 MiB  0.188  CRC64   data/en_caselaw_train.43.jsonl.xz
    1       1    476.9 MiB  2’182.2 MiB  0.219  CRC64   data/en_caselaw_train.44.jsonl.xz
    1       1    476.9 MiB  2’163.2 MiB  0.220  CRC64   data/en_caselaw_train.45.jsonl.xz
    1       1    476.9 MiB  2’213.3 MiB  0.215  CRC64   data/en_caselaw_train.46.jsonl.xz
    1       1    476.9 MiB  2’241.5 MiB  0.213  CRC64   data/en_caselaw_train.47.jsonl.xz
    1       1    476.9 MiB  2’203.6 MiB  0.216  CRC64   data/en_caselaw_train.48.jsonl.xz
    1       1    476.9 MiB  2’480.6 MiB  0.192  CRC64   data/en_caselaw_train.49.jsonl.xz
    1       1    476.9 MiB  2’176.7 MiB  0.219  CRC64   data/en_caselaw_train.4.jsonl.xz
    1       1    476.9 MiB  2’214.7 MiB  0.215  CRC64   data/en_caselaw_train.50.jsonl.xz
    1       1    476.9 MiB  2’128.0 MiB  0.224  CRC64   data/en_caselaw_train.51.jsonl.xz
    1       1    476.9 MiB  2’151.0 MiB  0.222  CRC64   data/en_caselaw_train.52.jsonl.xz
    1       1    476.9 MiB  2’173.6 MiB  0.219  CRC64   data/en_caselaw_train.53.jsonl.xz
    1       1    476.9 MiB  2’773.8 MiB  0.172  CRC64   data/en_caselaw_train.54.jsonl.xz
    1       1    476.9 MiB  2’806.2 MiB  0.170  CRC64   data/en_caselaw_train.55.jsonl.xz
    1       1    476.9 MiB  3’920.9 MiB  0.122  CRC64   data/en_caselaw_train.56.jsonl.xz
    1       1    476.9 MiB  2’517.2 MiB  0.189  CRC64   data/en_caselaw_train.57.jsonl.xz
    1       1    477.5 MiB  2’844.0 MiB  0.168  CRC64   data/en_caselaw_train.58.jsonl.xz
    1       1    476.9 MiB  2’810.7 MiB  0.170  CRC64   data/en_caselaw_train.59.jsonl.xz
    1       1    476.9 MiB  2’160.4 MiB  0.221  CRC64   data/en_caselaw_train.5.jsonl.xz
    1       1    476.9 MiB  3’033.0 MiB  0.157  CRC64   data/en_caselaw_train.60.jsonl.xz
    1       1    476.9 MiB  2’255.1 MiB  0.211  CRC64   data/en_caselaw_train.61.jsonl.xz
    1       1    476.9 MiB  2’110.1 MiB  0.226  CRC64   data/en_caselaw_train.62.jsonl.xz
    1       1    476.9 MiB  2’130.3 MiB  0.224  CRC64   data/en_caselaw_train.63.jsonl.xz
    1       1    476.9 MiB  2’133.2 MiB  0.224  CRC64   data/en_caselaw_train.64.jsonl.xz
    1       1     44.8 MiB    199.6 MiB  0.225  CRC64   data/en_caselaw_train.65.jsonl.xz
    1       1    476.9 MiB  2’153.3 MiB  0.221  CRC64   data/en_caselaw_train.6.jsonl.xz
    1       1    476.9 MiB  2’130.8 MiB  0.224  CRC64   data/en_caselaw_train.7.jsonl.xz
    1       1    476.9 MiB  2’152.2 MiB  0.222  CRC64   data/en_caselaw_train.8.jsonl.xz
    1       1    476.9 MiB  2’173.3 MiB  0.219  CRC64   data/en_caselaw_train.9.jsonl.xz
    1       1  2’977.4 KiB     12.9 MiB  0.226  CRC64   data/en_caselaw_validation.0.jsonl.xz
    1       1    476.9 MiB  3’016.6 MiB  0.158  CRC64   data/en_contracts_train.0.jsonl.xz
    1       1    476.9 MiB  3’015.3 MiB  0.158  CRC64   data/en_contracts_train.10.jsonl.xz
    1       1    476.9 MiB  3’012.5 MiB  0.158  CRC64   data/en_contracts_train.11.jsonl.xz
    1       1    477.0 MiB  3’002.5 MiB  0.159  CRC64   data/en_contracts_train.12.jsonl.xz
    1       1    476.9 MiB  2’962.4 MiB  0.161  CRC64   data/en_contracts_train.13.jsonl.xz
    1       1    476.9 MiB  3’019.4 MiB  0.158  CRC64   data/en_contracts_train.14.jsonl.xz
    1       1    124.1 MiB    781.2 MiB  0.159  CRC64   data/en_contracts_train.15.jsonl.xz
    1       1    476.9 MiB  2’994.0 MiB  0.159  CRC64   data/en_contracts_train.1.jsonl.xz
    1       1    476.8 MiB  3’084.9 MiB  0.155  CRC64   data/en_contracts_train.2.jsonl.xz
    1       1    476.9 MiB  3’123.4 MiB  0.153  CRC64   data/en_contracts_train.3.jsonl.xz
    1       1    476.9 MiB  3’120.7 MiB  0.153  CRC64   data/en_contracts_train.4.jsonl.xz
    1       1    477.0 MiB  3’094.2 MiB  0.154  CRC64   data/en_contracts_train.5.jsonl.xz
    1       1    476.9 MiB  3’010.9 MiB  0.158  CRC64   data/en_contracts_train.6.jsonl.xz
    1       1    476.9 MiB  3’015.0 MiB  0.158  CRC64   data/en_contracts_train.7.jsonl.xz
    1       1    476.9 MiB  2’995.7 MiB  0.159  CRC64   data/en_contracts_train.8.jsonl.xz
    1       1    476.9 MiB  3’017.9 MiB  0.158  CRC64   data/en_contracts_train.9.jsonl.xz
    1       1  9’980.4 KiB     63.7 MiB  0.153  CRC64   data/en_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’040.8 MiB  0.157  CRC64   data/en_legislation_train.0.jsonl.xz
    1       1    476.9 MiB  3’047.3 MiB  0.156  CRC64   data/en_legislation_train.1.jsonl.xz
    1       1    476.9 MiB  3’351.5 MiB  0.142  CRC64   data/en_legislation_train.2.jsonl.xz
    1       1    478.7 MiB  3’408.4 MiB  0.140  CRC64   data/en_legislation_train.3.jsonl.xz
    1       1    372.5 MiB  2’620.0 MiB  0.142  CRC64   data/en_legislation_train.4.jsonl.xz
    1       1  2’733.5 KiB     13.8 MiB  0.193  CRC64   data/en_legislation_validation.0.jsonl.xz
    1       1    476.9 MiB  4’782.4 MiB  0.100  CRC64   data/en_other_train.0.jsonl.xz
    1       1    476.9 MiB  4’347.1 MiB  0.110  CRC64   data/en_other_train.10.jsonl.xz
    1       1    477.1 MiB  3’044.6 MiB  0.157  CRC64   data/en_other_train.11.jsonl.xz
    1       1    477.1 MiB  2’147.8 MiB  0.222  CRC64   data/en_other_train.12.jsonl.xz
    1       1    477.0 MiB  2’182.8 MiB  0.219  CRC64   data/en_other_train.13.jsonl.xz
    1       1     33.3 MiB    151.7 MiB  0.219  CRC64   data/en_other_train.14.jsonl.xz
    1       1    476.9 MiB  4’883.8 MiB  0.098  CRC64   data/en_other_train.1.jsonl.xz
    1       1    476.9 MiB  4’646.7 MiB  0.103  CRC64   data/en_other_train.2.jsonl.xz
    1       1    476.9 MiB  4’542.8 MiB  0.105  CRC64   data/en_other_train.3.jsonl.xz
    1       1    476.9 MiB  4’574.8 MiB  0.104  CRC64   data/en_other_train.4.jsonl.xz
    1       1    476.9 MiB  4’622.5 MiB  0.103  CRC64   data/en_other_train.5.jsonl.xz
    1       1    476.9 MiB  4’520.7 MiB  0.105  CRC64   data/en_other_train.6.jsonl.xz
    1       1    476.9 MiB  2’942.4 MiB  0.162  CRC64   data/en_other_train.7.jsonl.xz
    1       1    476.9 MiB  2’544.0 MiB  0.187  CRC64   data/en_other_train.8.jsonl.xz
    1       1    476.9 MiB  4’515.4 MiB  0.106  CRC64   data/en_other_train.9.jsonl.xz
    1       1  2’165.8 KiB     19.6 MiB  0.108  CRC64   data/en_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’803.2 MiB  0.264  CRC64   data/en_wikipedia_train.0.jsonl.xz
    1       1    441.1 MiB  1’670.5 MiB  0.264  CRC64   data/en_wikipedia_train.10.jsonl.xz
    1       1    476.9 MiB  1’803.6 MiB  0.264  CRC64   data/en_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  1’802.5 MiB  0.265  CRC64   data/en_wikipedia_train.2.jsonl.xz
    1       1    476.9 MiB  1’805.0 MiB  0.264  CRC64   data/en_wikipedia_train.3.jsonl.xz
    1       1    476.9 MiB  1’804.3 MiB  0.264  CRC64   data/en_wikipedia_train.4.jsonl.xz
    1       1    476.9 MiB  1’804.0 MiB  0.264  CRC64   data/en_wikipedia_train.5.jsonl.xz
    1       1    476.9 MiB  1’804.1 MiB  0.264  CRC64   data/en_wikipedia_train.6.jsonl.xz
    1       1    476.9 MiB  1’803.6 MiB  0.264  CRC64   data/en_wikipedia_train.7.jsonl.xz
    1       1    476.9 MiB  1’805.2 MiB  0.264  CRC64   data/en_wikipedia_train.8.jsonl.xz
    1       1    476.9 MiB  1’804.3 MiB  0.264  CRC64   data/en_wikipedia_train.9.jsonl.xz
    1       1  1’004.9 KiB  3’492.7 KiB  0.288  CRC64   data/en_wikipedia_validation.0.jsonl.xz
    1       1    216.4 MiB  1’458.0 MiB  0.148  CRC64   data/es_caselaw_train.0.jsonl.xz
    1       1    586.4 KiB  3’537.8 KiB  0.166  CRC64   data/es_caselaw_validation.0.jsonl.xz
    1       1     29.0 MiB    244.0 MiB  0.119  CRC64   data/es_contracts_train.0.jsonl.xz
    1       1  3’826.2 KiB     31.2 MiB  0.120  CRC64   data/es_contracts_validation.0.jsonl.xz
    1       1    401.8 MiB  3’054.9 MiB  0.132  CRC64   data/es_legislation_train.0.jsonl.xz
    1       1  8’217.6 KiB     56.6 MiB  0.142  CRC64   data/es_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/es_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’017.9 MiB  0.236  CRC64   data/es_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  2’025.0 MiB  0.235  CRC64   data/es_wikipedia_train.1.jsonl.xz
    1       1    308.8 MiB  1’305.6 MiB  0.237  CRC64   data/es_wikipedia_train.2.jsonl.xz
    1       1  1’339.7 KiB  5’265.5 KiB  0.254  CRC64   data/es_wikipedia_validation.0.jsonl.xz
    1       1    132.5 MiB    831.3 MiB  0.159  CRC64   data/et_caselaw_train.0.jsonl.xz
    1       1    387.2 KiB  2’310.9 KiB  0.168  CRC64   data/et_caselaw_validation.0.jsonl.xz
    1       1     22.9 MiB    179.6 MiB  0.128  CRC64   data/et_contracts_train.0.jsonl.xz
    1       1  3’164.3 KiB     26.8 MiB  0.115  CRC64   data/et_contracts_validation.0.jsonl.xz
    1       1    255.2 MiB  1’908.2 MiB  0.134  CRC64   data/et_legislation_train.0.jsonl.xz
    1       1  9’239.2 KiB     64.7 MiB  0.140  CRC64   data/et_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/et_other_validation.0.jsonl.xz
    1       1    100.5 MiB    408.8 MiB  0.246  CRC64   data/et_wikipedia_train.0.jsonl.xz
    1       1  1’352.2 KiB  4’921.0 KiB  0.275  CRC64   data/et_wikipedia_validation.0.jsonl.xz
    1       1    194.5 MiB  1’359.0 MiB  0.143  CRC64   data/fi_caselaw_train.0.jsonl.xz
    1       1    604.1 KiB  3’656.1 KiB  0.165  CRC64   data/fi_caselaw_validation.0.jsonl.xz
    1       1     26.0 MiB    219.8 MiB  0.118  CRC64   data/fi_contracts_train.0.jsonl.xz
    1       1  2’971.2 KiB     27.4 MiB  0.106  CRC64   data/fi_contracts_validation.0.jsonl.xz
    1       1    334.7 MiB  2’599.3 MiB  0.129  CRC64   data/fi_legislation_train.0.jsonl.xz
    1       1  7’476.3 KiB     53.9 MiB  0.136  CRC64   data/fi_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/fi_other_validation.0.jsonl.xz
    1       1    255.6 MiB  1’118.0 MiB  0.229  CRC64   data/fi_wikipedia_train.0.jsonl.xz
    1       1  2’464.2 KiB      9.9 MiB  0.242  CRC64   data/fi_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  3’128.1 MiB  0.152  CRC64   data/fr_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  3’104.4 MiB  0.154  CRC64   data/fr_caselaw_train.1.jsonl.xz
    1       1    350.2 MiB  2’194.9 MiB  0.160  CRC64   data/fr_caselaw_train.2.jsonl.xz
    1       1    603.0 KiB  3’778.7 KiB  0.160  CRC64   data/fr_caselaw_validation.0.jsonl.xz
    1       1     31.9 MiB    278.3 MiB  0.115  CRC64   data/fr_contracts_train.0.jsonl.xz
    1       1  3’034.4 KiB     26.6 MiB  0.111  CRC64   data/fr_contracts_validation.0.jsonl.xz
    1       1    477.0 MiB  3’721.8 MiB  0.128  CRC64   data/fr_legislation_train.0.jsonl.xz
    1       1     89.3 MiB    670.9 MiB  0.133  CRC64   data/fr_legislation_train.1.jsonl.xz
    1       1  3’185.5 KiB     22.6 MiB  0.138  CRC64   data/fr_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/fr_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’150.5 MiB  0.222  CRC64   data/fr_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  2’151.4 MiB  0.222  CRC64   data/fr_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  2’151.2 MiB  0.222  CRC64   data/fr_wikipedia_train.2.jsonl.xz
    1       1    384.8 MiB  1’736.1 MiB  0.222  CRC64   data/fr_wikipedia_train.3.jsonl.xz
    1       1    937.8 KiB  3’777.6 KiB  0.248  CRC64   data/fr_wikipedia_validation.0.jsonl.xz
    1       1    721.9 KiB  5’663.9 KiB  0.127  CRC64   data/ga_caselaw_validation.0.jsonl.xz
    1       1  1’246.1 KiB     15.6 MiB  0.078  CRC64   data/ga_contracts_validation.0.jsonl.xz
    1       1     41.2 MiB    419.0 MiB  0.098  CRC64   data/ga_legislation_train.0.jsonl.xz
    1       1     14.9 MiB    123.2 MiB  0.121  CRC64   data/ga_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/ga_other_validation.0.jsonl.xz
    1       1     11.0 MiB     52.9 MiB  0.207  CRC64   data/ga_wikipedia_train.0.jsonl.xz
    1       1    782.4 KiB  3’438.9 KiB  0.228  CRC64   data/ga_wikipedia_validation.0.jsonl.xz
    1       1     72.7 MiB    460.3 MiB  0.158  CRC64   data/hr_caselaw_train.0.jsonl.xz
    1       1    359.9 KiB  2’214.8 KiB  0.162  CRC64   data/hr_caselaw_validation.0.jsonl.xz
    1       1     21.2 MiB    158.3 MiB  0.134  CRC64   data/hr_contracts_train.0.jsonl.xz
    1       1  3’785.9 KiB     26.6 MiB  0.139  CRC64   data/hr_contracts_validation.0.jsonl.xz
    1       1    160.6 MiB  1’258.7 MiB  0.128  CRC64   data/hr_legislation_train.0.jsonl.xz
    1       1     11.2 MiB     86.1 MiB  0.130  CRC64   data/hr_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/hr_other_validation.0.jsonl.xz
    1       1    110.3 MiB    425.5 MiB  0.259  CRC64   data/hr_wikipedia_train.0.jsonl.xz
    1       1  1’743.8 KiB  6’170.1 KiB  0.283  CRC64   data/hr_wikipedia_validation.0.jsonl.xz
    1       1    150.6 MiB  1’320.5 MiB  0.114  CRC64   data/hu_caselaw_train.0.jsonl.xz
    1       1    423.8 KiB  3’496.6 KiB  0.121  CRC64   data/hu_caselaw_validation.0.jsonl.xz
    1       1     26.9 MiB    266.0 MiB  0.101  CRC64   data/hu_contracts_train.0.jsonl.xz
    1       1  3’532.6 KiB     36.1 MiB  0.096  CRC64   data/hu_contracts_validation.0.jsonl.xz
    1       1    337.6 MiB  3’129.4 MiB  0.108  CRC64   data/hu_legislation_train.0.jsonl.xz
    1       1  3’913.7 KiB     94.8 MiB  0.040  CRC64   data/hu_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/hu_other_validation.0.jsonl.xz
    1       1    364.2 MiB  1’835.0 MiB  0.198  CRC64   data/hu_wikipedia_train.0.jsonl.xz
    1       1  1’719.5 KiB  8’000.8 KiB  0.215  CRC64   data/hu_wikipedia_validation.0.jsonl.xz
    1       1    459.8 MiB  2’742.8 MiB  0.168  CRC64   data/it_caselaw_train.0.jsonl.xz
    1       1    577.8 KiB  3’194.2 KiB  0.181  CRC64   data/it_caselaw_validation.0.jsonl.xz
    1       1     31.2 MiB    240.4 MiB  0.130  CRC64   data/it_contracts_train.0.jsonl.xz
    1       1  3’068.9 KiB     24.0 MiB  0.125  CRC64   data/it_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’362.3 MiB  0.142  CRC64   data/it_legislation_train.0.jsonl.xz
    1       1     38.9 MiB    238.7 MiB  0.163  CRC64   data/it_legislation_train.1.jsonl.xz
    1       1  3’211.3 KiB     25.3 MiB  0.124  CRC64   data/it_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/it_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’864.5 MiB  0.256  CRC64   data/it_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  1’864.8 MiB  0.256  CRC64   data/it_wikipedia_train.1.jsonl.xz
    1       1    184.6 MiB    726.2 MiB  0.254  CRC64   data/it_wikipedia_train.2.jsonl.xz
    1       1  1’334.0 KiB  4’843.5 KiB  0.275  CRC64   data/it_wikipedia_validation.0.jsonl.xz
    1       1    136.6 MiB    975.7 MiB  0.140  CRC64   data/lt_caselaw_train.0.jsonl.xz
    1       1    397.0 KiB  2’660.9 KiB  0.149  CRC64   data/lt_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    211.8 MiB  0.118  CRC64   data/lt_contracts_train.0.jsonl.xz
    1       1  3’275.5 KiB     26.1 MiB  0.123  CRC64   data/lt_contracts_validation.0.jsonl.xz
    1       1    274.0 MiB  2’174.1 MiB  0.126  CRC64   data/lt_legislation_train.0.jsonl.xz
    1       1  9’780.7 KiB     73.4 MiB  0.130  CRC64   data/lt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/lt_other_validation.0.jsonl.xz
    1       1     72.6 MiB    349.5 MiB  0.208  CRC64   data/lt_wikipedia_train.0.jsonl.xz
    1       1  1’251.2 KiB  5’369.5 KiB  0.233  CRC64   data/lt_wikipedia_validation.0.jsonl.xz
    1       1    141.0 MiB  1’106.7 MiB  0.127  CRC64   data/lv_caselaw_train.0.jsonl.xz
    1       1    410.3 KiB  3’004.0 KiB  0.137  CRC64   data/lv_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    224.5 MiB  0.111  CRC64   data/lv_contracts_train.0.jsonl.xz
    1       1  3’629.0 KiB     33.6 MiB  0.106  CRC64   data/lv_contracts_validation.0.jsonl.xz
    1       1    271.5 MiB  2’377.4 MiB  0.114  CRC64   data/lv_legislation_train.0.jsonl.xz
    1       1     10.5 MiB     87.5 MiB  0.120  CRC64   data/lv_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/lv_other_validation.0.jsonl.xz
    1       1     47.5 MiB    254.7 MiB  0.186  CRC64   data/lv_wikipedia_train.0.jsonl.xz
    1       1    984.1 KiB  4’559.4 KiB  0.216  CRC64   data/lv_wikipedia_validation.0.jsonl.xz
    1       1    132.2 MiB    956.6 MiB  0.138  CRC64   data/mt_caselaw_train.0.jsonl.xz
    1       1    396.1 KiB  2’680.0 KiB  0.148  CRC64   data/mt_caselaw_validation.0.jsonl.xz
    1       1     25.6 MiB    201.0 MiB  0.127  CRC64   data/mt_contracts_train.0.jsonl.xz
    1       1  4’178.4 KiB     34.0 MiB  0.120  CRC64   data/mt_contracts_validation.0.jsonl.xz
    1       1    270.7 MiB  2’121.7 MiB  0.128  CRC64   data/mt_legislation_train.0.jsonl.xz
    1       1     11.4 MiB     84.2 MiB  0.135  CRC64   data/mt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/mt_other_validation.0.jsonl.xz
    1       1  4’608.3 KiB     19.5 MiB  0.231  CRC64   data/mt_wikipedia_train.0.jsonl.xz
    1       1  1’405.0 KiB  5’754.4 KiB  0.244  CRC64   data/mt_wikipedia_validation.0.jsonl.xz
    1       1    223.1 MiB  1’338.9 MiB  0.167  CRC64   data/nl_caselaw_train.0.jsonl.xz
    1       1    566.0 KiB  3’152.2 KiB  0.180  CRC64   data/nl_caselaw_validation.0.jsonl.xz
    1       1     31.6 MiB    242.3 MiB  0.130  CRC64   data/nl_contracts_train.0.jsonl.xz
    1       1  2’663.9 KiB     22.4 MiB  0.116  CRC64   data/nl_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’311.9 MiB  0.144  CRC64   data/nl_legislation_train.0.jsonl.xz
    1       1     41.1 MiB    268.7 MiB  0.153  CRC64   data/nl_legislation_train.1.jsonl.xz
    1       1  3’678.8 KiB     72.9 MiB  0.049  CRC64   data/nl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/nl_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’856.9 MiB  0.257  CRC64   data/nl_wikipedia_train.0.jsonl.xz
    1       1     59.9 MiB    236.4 MiB  0.253  CRC64   data/nl_wikipedia_train.1.jsonl.xz
    1       1    979.4 KiB  3’414.8 KiB  0.287  CRC64   data/nl_wikipedia_validation.0.jsonl.xz
    1       1    147.9 MiB  1’034.1 MiB  0.143  CRC64   data/pl_caselaw_train.0.jsonl.xz
    1       1    416.2 KiB  2’737.2 KiB  0.152  CRC64   data/pl_caselaw_validation.0.jsonl.xz
    1       1     24.8 MiB    208.9 MiB  0.119  CRC64   data/pl_contracts_train.0.jsonl.xz
    1       1  4’241.9 KiB     34.6 MiB  0.120  CRC64   data/pl_contracts_validation.0.jsonl.xz
    1       1    325.0 MiB  2’646.2 MiB  0.123  CRC64   data/pl_legislation_train.0.jsonl.xz
    1       1  3’593.0 KiB     29.0 MiB  0.121  CRC64   data/pl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/pl_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’144.7 MiB  0.222  CRC64   data/pl_wikipedia_train.0.jsonl.xz
    1       1    189.5 MiB    864.0 MiB  0.219  CRC64   data/pl_wikipedia_train.1.jsonl.xz
    1       1  1’233.2 KiB  4’965.9 KiB  0.248  CRC64   data/pl_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  3’494.2 MiB  0.136  CRC64   data/pt_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  3’392.1 MiB  0.141  CRC64   data/pt_caselaw_train.10.jsonl.xz
    1       1    476.9 MiB  3’505.3 MiB  0.136  CRC64   data/pt_caselaw_train.11.jsonl.xz
    1       1    476.9 MiB  3’524.1 MiB  0.135  CRC64   data/pt_caselaw_train.12.jsonl.xz
    1       1    476.9 MiB  3’458.4 MiB  0.138  CRC64   data/pt_caselaw_train.13.jsonl.xz
    1       1    476.9 MiB  3’602.9 MiB  0.132  CRC64   data/pt_caselaw_train.14.jsonl.xz
    1       1    476.9 MiB  4’923.4 MiB  0.097  CRC64   data/pt_caselaw_train.15.jsonl.xz
    1       1    476.9 MiB  6’648.8 MiB  0.072  CRC64   data/pt_caselaw_train.16.jsonl.xz
    1       1    476.9 MiB  7’461.0 MiB  0.064  CRC64   data/pt_caselaw_train.17.jsonl.xz
    1       1    476.9 MiB  6’866.4 MiB  0.069  CRC64   data/pt_caselaw_train.18.jsonl.xz
    1       1    476.9 MiB  3’455.7 MiB  0.138  CRC64   data/pt_caselaw_train.19.jsonl.xz
    1       1    476.9 MiB  3’513.7 MiB  0.136  CRC64   data/pt_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  3’477.3 MiB  0.137  CRC64   data/pt_caselaw_train.20.jsonl.xz
    1       1    476.9 MiB  3’492.8 MiB  0.137  CRC64   data/pt_caselaw_train.21.jsonl.xz
    1       1    476.9 MiB  3’528.6 MiB  0.135  CRC64   data/pt_caselaw_train.22.jsonl.xz
    1       1     94.1 MiB    694.3 MiB  0.135  CRC64   data/pt_caselaw_train.23.jsonl.xz
    1       1    476.9 MiB  3’436.5 MiB  0.139  CRC64   data/pt_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  3’527.9 MiB  0.135  CRC64   data/pt_caselaw_train.3.jsonl.xz
    1       1    476.9 MiB  3’492.2 MiB  0.137  CRC64   data/pt_caselaw_train.4.jsonl.xz
    1       1    476.9 MiB  3’554.8 MiB  0.134  CRC64   data/pt_caselaw_train.5.jsonl.xz
    1       1    476.9 MiB  3’494.7 MiB  0.136  CRC64   data/pt_caselaw_train.6.jsonl.xz
    1       1    476.9 MiB  3’439.1 MiB  0.139  CRC64   data/pt_caselaw_train.7.jsonl.xz
    1       1    476.9 MiB  3’625.6 MiB  0.132  CRC64   data/pt_caselaw_train.8.jsonl.xz
    1       1    476.9 MiB  3’726.4 MiB  0.128  CRC64   data/pt_caselaw_train.9.jsonl.xz
    1       1    798.9 KiB  4’820.6 KiB  0.166  CRC64   data/pt_caselaw_validation.0.jsonl.xz
    1       1     28.4 MiB    243.2 MiB  0.117  CRC64   data/pt_contracts_train.0.jsonl.xz
    1       1  3’899.7 KiB     32.6 MiB  0.117  CRC64   data/pt_contracts_validation.0.jsonl.xz
    1       1    406.2 MiB  3’217.5 MiB  0.126  CRC64   data/pt_legislation_train.0.jsonl.xz
    1       1  8’350.4 KiB     58.4 MiB  0.140  CRC64   data/pt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/pt_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’050.4 MiB  0.233  CRC64   data/pt_wikipedia_train.0.jsonl.xz
    1       1    140.6 MiB    617.4 MiB  0.228  CRC64   data/pt_wikipedia_train.1.jsonl.xz
    1       1  1’480.0 KiB  6’344.8 KiB  0.233  CRC64   data/pt_wikipedia_validation.0.jsonl.xz
    1       1    124.9 MiB    956.9 MiB  0.131  CRC64   data/ro_caselaw_train.0.jsonl.xz
    1       1    400.4 KiB  2’785.0 KiB  0.144  CRC64   data/ro_caselaw_validation.0.jsonl.xz
    1       1     24.6 MiB    210.5 MiB  0.117  CRC64   data/ro_contracts_train.0.jsonl.xz
    1       1  3’886.3 KiB     34.3 MiB  0.111  CRC64   data/ro_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  4’496.4 MiB  0.106  CRC64   data/ro_legislation_train.0.jsonl.xz
    1       1     97.6 MiB  1’053.6 MiB  0.093  CRC64   data/ro_legislation_train.1.jsonl.xz
    1       1  3’691.3 KiB     33.4 MiB  0.108  CRC64   data/ro_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/ro_other_validation.0.jsonl.xz
    1       1    179.7 MiB    833.0 MiB  0.216  CRC64   data/ro_wikipedia_train.0.jsonl.xz
    1       1  2’089.4 KiB  9’053.5 KiB  0.231  CRC64   data/ro_wikipedia_validation.0.jsonl.xz
    1       1    143.6 MiB  1’094.2 MiB  0.131  CRC64   data/sk_caselaw_train.0.jsonl.xz
    1       1    415.8 KiB  3’012.4 KiB  0.138  CRC64   data/sk_caselaw_validation.0.jsonl.xz
    1       1     25.9 MiB    226.7 MiB  0.114  CRC64   data/sk_contracts_train.0.jsonl.xz
    1       1  3’933.6 KiB     35.2 MiB  0.109  CRC64   data/sk_contracts_validation.0.jsonl.xz
    1       1    322.4 MiB  2’745.5 MiB  0.117  CRC64   data/sk_legislation_train.0.jsonl.xz
    1       1  3’735.8 KiB     31.7 MiB  0.115  CRC64   data/sk_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sk_other_validation.0.jsonl.xz
    1       1     91.2 MiB    435.3 MiB  0.210  CRC64   data/sk_wikipedia_train.0.jsonl.xz
    1       1  1’724.4 KiB  7’568.3 KiB  0.228  CRC64   data/sk_wikipedia_validation.0.jsonl.xz
    1       1    131.9 MiB    815.8 MiB  0.162  CRC64   data/sl_caselaw_train.0.jsonl.xz
    1       1    392.8 KiB  2’328.2 KiB  0.169  CRC64   data/sl_caselaw_validation.0.jsonl.xz
    1       1     22.9 MiB    172.4 MiB  0.133  CRC64   data/sl_contracts_train.0.jsonl.xz
    1       1  3’493.7 KiB     27.2 MiB  0.125  CRC64   data/sl_contracts_validation.0.jsonl.xz
    1       1    388.1 MiB  2’732.3 MiB  0.142  CRC64   data/sl_legislation_train.0.jsonl.xz
    1       1  3’429.8 KiB     24.3 MiB  0.138  CRC64   data/sl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sl_other_validation.0.jsonl.xz
    1       1    104.6 MiB    425.6 MiB  0.246  CRC64   data/sl_wikipedia_train.0.jsonl.xz
    1       1  1’392.8 KiB  5’004.9 KiB  0.278  CRC64   data/sl_wikipedia_validation.0.jsonl.xz
    1       1    189.5 MiB  1’325.4 MiB  0.143  CRC64   data/sv_caselaw_train.0.jsonl.xz
    1       1    581.2 KiB  3’566.7 KiB  0.163  CRC64   data/sv_caselaw_validation.0.jsonl.xz
    1       1     25.3 MiB    211.7 MiB  0.119  CRC64   data/sv_contracts_train.0.jsonl.xz
    1       1  2’890.6 KiB     26.0 MiB  0.108  CRC64   data/sv_contracts_validation.0.jsonl.xz
    1       1    324.5 MiB  2’570.4 MiB  0.126  CRC64   data/sv_legislation_train.0.jsonl.xz
    1       1  6’984.8 KiB     50.1 MiB  0.136  CRC64   data/sv_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sv_other_validation.0.jsonl.xz
    1       1    333.4 MiB  1’668.1 MiB  0.200  CRC64   data/sv_wikipedia_train.0.jsonl.xz
    1       1  1’088.6 KiB  4’372.9 KiB  0.249  CRC64   data/sv_wikipedia_validation.0.jsonl.xz
-------------------------------------------------------------------------------
  374     351     90.1 GiB    579.9 GiB  0.155  CRC64   374 files

Dataset Creation

This dataset has been created by combining the following datasets: Native Multi Legal Pile, Eurlex Resources, MC4 Legal, Pile of Law, EU Wikipedias. It has been filtered to remove short documents (less than 64 whitespace-separated tokens) and documents with more than 30% punctuation or numbers (see prepare_legal_data.py for more details).

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

[More Information Needed]

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

TODO add citation

Contributions

Thanks to @JoelNiklaus for adding this dataset.

作者:

joelito

数据集大小:

8.96 GB