[tex-hyphen] Patgen

[Keno Wehr]

Patgen 2.4 (TeX Live 2019) has some problems with huge input lists.
I have a input list with over 11.000.000 entries, which I need to 
prepare new hyphenation patterns for classical Latin, a heavily 
inflected language.
The percentages output by patgen during the run are erroneous. The sum 
of good and missed hyphens should be 100 %, but this is not the case. 
Even negative percentages occur.
Furthermore, patgen aborts in the seventh run with a "PATGEN capacity 
exceeded" message.
The patgen logs of the first and the seventh run are attached.

Is it possible to adapt patgen for such huge lists?

Keno
-------------- next part --------------
This is PATGEN, Version 2.4 (TeX Live 2019)
left_hyphen_min = 2, right_hyphen_min = 2, 52 letters
0 patterns read in
pattern trie has 310 nodes, trie_max = 415, 0 outputs
hyph_start, hyph_finish: pat_start, pat_finish: good weight, bad weight, threshold: processing dictionary with pat_len = 1, pat_dot = 0
 
0 good, 0 bad, 47946702 missed
0.00 %, 0.00 %, 10.42 %
0 patterns, 256 nodes in count trie, triec_max = 256
13 good and 11 bad patterns added (more to come)
finding 25785534 good and 5520626 bad hyphens, efficiency = 2.34
pattern trie has 310 nodes, trie_max = 415, 2 outputs
processing dictionary with pat_len = 1, pat_dot = 1
 
25785534 good, 5520626 bad, 22161168 missed
-35.80 %, 11.51 %, -43.36 %
0 patterns, 256 nodes in count trie, triec_max = 256
7 good and 9 bad patterns added (more to come)
finding 1588369 good and 572374 bad hyphens, efficiency = 1.39
pattern trie has 310 nodes, trie_max = 415, 7 outputs
processing dictionary with pat_len = 2, pat_dot = 1
 
27373903 good, 6093000 bad, 20572799 missed
-32.49 %, 12.71 %, 42.91 %
821 patterns, 1077 nodes in count trie, triec_max = 2259
162 good and 340 bad patterns added (more to come)
finding 11417689 good and 1391412 bad hyphens, efficiency = 4.10
pattern trie has 812 nodes, trie_max = 1631, 8 outputs
processing dictionary with pat_len = 2, pat_dot = 0
 
38791592 good, 7484412 bad, 9155110 missed
-8.67 %, 15.61 %, 19.09 %
978 patterns, 1234 nodes in count trie, triec_max = 2704
101 good and 527 bad patterns added (more to come)
finding 3899144 good and 893632 bad hyphens, efficiency = 2.18
pattern trie has 1194 nodes, trie_max = 2683, 11 outputs
processing dictionary with pat_len = 2, pat_dot = 2
 
42690736 good, 8378044 bad, 5255966 missed
-0.54 %, 17.47 %, 10.96 %
1211 patterns, 1467 nodes in count trie, triec_max = 3684
36 good and 688 bad patterns added (more to come)
finding 1364129 good and 546586 bad hyphens, efficiency = 1.25
pattern trie has 1479 nodes, trie_max = 3440, 24 outputs
processing dictionary with pat_len = 3, pat_dot = 1
4K 8K 12K  
44054865 good, 8924630 bad, 3891837 missed
2.31 %, 18.61 %, 8.12 %
3415 patterns, 3974 nodes in count trie, triec_max = 13858
494 good and 1769 bad patterns added (more to come)
finding 2181175 good and 453104 bad hyphens, efficiency = 2.41
pattern trie has 3895 nodes, trie_max = 9533, 24 outputs
processing dictionary with pat_len = 3, pat_dot = 2
4K 8K 12K  
46236040 good, 9377734 bad, 1710662 missed
6.85 %, 19.56 %, 3.57 %
3657 patterns, 4398 nodes in count trie, triec_max = 14460
158 good and 2646 bad patterns added (more to come)
finding 393484 good and 81547 bad hyphens, efficiency = 2.41
pattern trie has 6391 nodes, trie_max = 16337, 24 outputs
processing dictionary with pat_len = 3, pat_dot = 0
4K 8K 12K 16K 20K  
46629524 good, 9459281 bad, 1317178 missed
7.67 %, 19.73 %, 2.75 %
6300 patterns, 6906 nodes in count trie, triec_max = 21708
278 good and 4466 bad patterns added (more to come)
finding 410710 good and 66292 bad hyphens, efficiency = 3.09
pattern trie has 9949 nodes, trie_max = 25973, 33 outputs
processing dictionary with pat_len = 3, pat_dot = 3
4K 8K 12K 16K 20K 24K 28K 32K  
47040234 good, 9525573 bad, 906468 missed
8.53 %, 19.87 %, 1.89 %
9126 patterns, 10708 nodes in count trie, triec_max = 36149
119 good and 8317 bad patterns added (more to come)
finding 394167 good and 137547 bad hyphens, efficiency = 1.43
pattern trie has 15005 nodes, trie_max = 39374, 62 outputs
13041 nodes and 40 outputs deleted
total of 1368 patterns at hyph_level 1
hyphenate word list? writing pattmp.1
 
47434401 good, 9663120 bad, 512301 missed
9.35 %, 20.15 %, 1.07 %
-------------- next part --------------
PATGEN capacity exceeded, sorry [5000000 count trie nodes].
= 2, right_hyphen_min = 2, 52 letters
24463 patterns read in
pattern trie has 33232 nodes, trie_max = 56427, 281 outputs
hyph_start, hyph_finish: pat_start, pat_finish: good weight, bad weight, threshold: processing dictionary with pat_len = 7, pat_dot = 3
4K 8K 12K 16K 20K 24K 28K 32K 36K 40K 44K 48K 52K 56K 60K 64K 68K 72K 76K 80K 84K 88K 92K 96K 100K 104K 108K 112K 116K 120K 124K 128K 132K 136K 140K 144K 148K 152K 156K 160K 164K 168K 172K 176K 180K 184K 188K 192K 196K 200K 204K 208K 212K 216K 220K 224K 228K 232K 236K 240K 244K 248K 252K 256K 260K 264K 268K 272K 276K 280K 284K 288K 292K 296K 300K 304K 308K 312K 316K 320K 324K 328K 332K 336K 340K 344K 348K 352K 356K 360K 364K 368K 372K 376K 380K 384K 388K 392K 396K 400K 404K 408K 412K 416K 420K 424K 428K 432K 436K 440K 444K 448K 452K 456K 460K 464K 468K 472K 476K 480K 484K 488K 492K 496K 500K 504K 508K 512K 516K 520K 524K 528K 532K 536K 540K 544K 548K 552K 556K 560K 564K 568K 572K 576K 580K 584K 588K 592K 596K 600K 604K 608K 612K 616K 620K 624K 628K 632K 636K 640K 644K 648K 652K 656K 660K 664K 668K 672K 676K 680K 684K 688K 692K 696K 700K 704K 708K 712K 716K 720K 724K 728K 732K 736K 740K 744K 748K 752K 756K 760K 764K 768K 772K 776K 780K 784K 788K 792K 796K 800K 804K 808K 812K 816K 820K 824K 828K 832K 836K 840K 844K 848K 852K 856K 860K 864K 868K 872K 876K 880K 884K 888K 892K 896K 900K 904K 908K 912K 916K 920K 924K 928K 932K 936K 940K 944K 948K 952K 956K 960K 964K 968K 972K 976K 980K 984K 988K 992K 996K 1000K 1004K 1008K 1012K 1016K 1020K 1024K 1028K 1032K 1036K 1040K 1044K 1048K 1052K 1056K 1060K 1064K 1068K 1072K 1076K 1080K 1084K 1088K 1092K 1096K 1100K 1104K 1108K 1112K 1116K 1120K 1124K 1128K 1132K 1136K 1140K 1144K 1148K 1152K 1156K 1160K 1164K 1168K 1172K 1176K 1180K 1184K 1188K 1192K 1196K 1200K 1204K 1208K 1212K 1216K 1220K 1224K 1228K 1232K 1236K 1240K 1244K 1248K 1252K 1256K 1260K 1264K 1268K 1272K 1276K 1280K 1284K 1288K 1292K 1296K 1300K 1304K 1308K 1312K 1316K 1320K 1324K 1328K 1332K 1336K 1340K 1344K 1348K 1352K 1356K 1360K 1364K 1368K 1372K 1376K 1380K 1384K 1388K 1392K 1396K 1400K 1404K 1408K 1412K 1416K 1420K 1424K 1428K 1432K 1436K 1440K 1444K 1448K 1452K 1456K 1460K 1464K 1468K 1472K 1476K 1480K 1484K 1488K 1492K 1496K 1500K 1504K 1508K 1512K 1516K 1520K 1524K 1528K 1532K 1536K 1540K 1544K 1548K 1552K 1556K 1560K 1564K 1568K 1572K 1576K 1580K 1584K 1588K 1592K 1596K 1600K 1604K 1608K 1612K 1616K 1620K 1624K 1628K 1632K 1636K 1640K 1644K 1648K 1652K 1656K 1660K 1664K 1668K 1672K 1676K 1680K 1684K 1688K 1692K 1696K 1700K 1704K 1708K 1712K 1716K 1720K 1724K 1728K 1732K 1736K 1740K 1744K 1748K 1752K 1756K 1760K 1764K 1768K 1772K 1776K 1780K 1784K 1788K 1792K 1796K 1800K 1804K 1808K 1812K 1816K 1820K 1824K 1828K 1832K 1836K 1840K 1844K 1848K 1852K 1856K 1860K 1864K 1868K 1872K 1876K 1880K 1884K 1888K 1892K 1896K 1900K 1904K 1908K 1912K 1916K 1920K 1924K 1928K 1932K 1936K 1940K 1944K 1948K 1952K 1956K 1960K 1964K 1968K 1972K 1976K 1980K 1984K 1988K 1992K 1996K 2000K 2004K 2008K 2012K 2016K 2020K 2024K 2028K 2032K 2036K 2040K 2044K 2048K 2052K 2056K 2060K 2064K 2068K 2072K 2076K 2080K 2084K 2088K 2092K 2096K 2100K 2104K 2108K 2112K 2116K 2120K 2124K 2128K 2132K 2136K 2140K 2144K 2148K 2152K 2156K 2160K 2164K 2168K 2172K 2176K 2180K 2184K 2188K 2192K 2196K 2200K 2204K 2208K 2212K 2216K 2220K 2224K 2228K 2232K 2236K 2240K 2244K 2248K 2252K 2256K 2260K 2264K 2268K 2272K 2276K 2280K 2284K 2288K 2292K 2296K 2300K 2304K 2308K 2312K 2316K 2320K 2324K 2328K 2332K 2336K 2340K 2344K 2348K 2352K 2356K 2360K 2364K 2368K 2372K 2376K 2380K 2384K 2388K 2392K 2396K 2400K 2404K 2408K 2412K 2416K 2420K 2424K 2428K 2432K 2436K 2440K 2444K 2448K 2452K 2456K 2460K 2464K 2468K 2472K 2476K 2480K 2484K 2488K 2492K 2496K 2500K 2504K 2508K 2512K 2516K 2520K 2524K 2528K 2532K 2536K 2540K 2544K 2548K 2552K 2556K 2560K 2564K 2568K 2572K 2576K 2580K 2584K 2588K 2592K 2596K 2600K 2604K 2608K 2612K 2616K 2620K 2624K 2628K 2632K 2636K 2640K 2644K 2648K 2652K 2656K 2660K 2664K 2668K 2672K 2676K 2680K 2684K 2688K 2692K 2696K 2700K 2704K 2708K 2712K 2716K 2720K 2724K 2728K 2732K 2736K 2740K 2744K 2748K 2752K 2756K 2760K 2764K 2768K 2772K 2776K 2780K 2784K 2788K 2792K 2796K 2800K 2804K 2808K 2812K 2816K 2820K 2824K 2828K 2832K 2836K 2840K 2844K 2848K 2852K 2856K 2860K 2864K 2868K 2872K 2876K 2880K 2884K 2888K 2892K 2896K 2900K 2904K 2908K 2912K 2916K 2920K 2924K 2928K 2932K 2936K 2940K 2944K 2948K 2952K 2956K 2960K 2964K 2968K 2972K 2976K 2980K 2984K 2988K 2992K 2996K 3000K 3004K 3008K 3012K 3016K 3020K 3024K 3028K 3032K 3036K 3040K 3044K 3048K 3052K 3056K 3060K 3064K 3068K 3072K 3076K 3080K 3084K 3088K 3092K 3096K 3100K 3104K 3108K 3112K 3116K 3120K 3124K 3128K 3132K 3136K 3140K 3144K 3148K 3152K 3156K 3160K 3164K 3168K 3172K 3176K 3180K 3184K 3188K 3192K 3196K 3200K 3204K 3208K 3212K 3216K 3220K 3224K 3228K 3232K 3236K 3240K 3244K 3248K 3252K 3256K 3260K 3264K 3268K 3272K 3276K 3280K 3284K 3288K 3292K 3296K 3300K 3304K 3308K 3312K 3316K 3320K 3324K 3328K 3332K 3336K 3340K 3344K 3348K 3352K 3356K 3360K 3364K 3368K 3372K 3376K 3380K 3384K 3388K 3392K 3396K 3400K 3404K 3408K 3412K 3416K 3420K 3424K 3428K 3432K 3436K 3440K 3444K 3448K 3452K 3456K 3460K 3464K 3468K 3472K 3476K 3480K 3484K 3488K 3492K 3496K 3500K 3504K 3508K 3512K 3516K 3520K 3524K 3528K 3532K 3536K 3540K 3544K 3548K 3552K 3556K 3560K 3564K 3568K 3572K 3576K 3580K 3584K 3588K 3592K 3596K 3600K 3604K 3608K 3612K 3616K 3620K 3624K 3628K 3632K 3636K 3640K 3644K 3648K 3652K 3656K 3660K 3664K 3668K 3672K 3676K 3680K 3684K 3688K 3692K 3696K 3700K 3704K 3708K 3712K 3716K 3720K 3724K 3728K 3732K 3736K 3740K 3744K 3748K 3752K 3756K 3760K 3764K 3768K 3772K 3776K 3780K 3784K 3788K 3792K 3796K 3800K 3804K 3808K 3812K 3816K 3820K 3824K 3828K 3832K 3836K 3840K 3844K 3848K 3852K 3856K 3860K 3864K 3868K 3872K 3876K 3880K 3884K 3888K 3892K 3896K 3900K 3904K 3908K 3912K 3916K 3920K 3924K 3928K 3932K 3936K 3940K 3944K 3948K 3952K 3956K 3960K 3964K 3968K 3972K 3976K 3980K 3984K 3988K 3992K 3996K 4000K 4004K 4008K 4012K 4016K 4020K 4024K 4028K 4032K 4036K 4040K 4044K 4048K 4052K 4056K 4060K 4064K 4068K 4072K 4076K 4080K 4084K 4088K 4092K 4096K 4100K 4104K 4108K 4112K 4116K 4120K 4124K 4128K 4132K 4136K 4140K 4144K 4148K 4152K 4156K 4160K 4164K 4168K 4172K 4176K 4180K 4184K 4188K 4192K 4196K 4200K 4204K 4208K 4212K 4216K 4220K 4224K 4228K 4232K 4236K 4240K 4244K 4248K 4252K 4256K 4260K 4264K 4268K 4272K 4276K 4280K 4284K 4288K 4292K 4296K 4300K 4304K 4308K 4312K 4316K 4320K 4324K 4328K 4332K 4336K 4340K 4344K 4348K 4352K 4356K 4360K 4364K 4368K 4372K 4376K 4380K 4384K 4388K 4392K 4396K 4400K 4404K 4408K 4412K 4416K 4420K 4424K 4428K 4432K 4436K 4440K 4444K 4448K 4452K 4456K 4460K 4464K 4468K 4472K 4476K 4480K 4484K 4488K 4492K 4496K 4500K 4504K 4508K 4512K 4516K 4520K 4524K 4528K 4532K 4536K 4540K 4544K 4548K 4552K 4556K 4560K 4564K 4568K 4572K 4576K 4580K 4584K 4588K 4592K 4596K 4600K 4604K 4608K 4612K 4616K 4620K 4624K 4628K 4632K 4636K 4640K 4644K 4648K 4652K 4656K 4660K 4664K 4668K 4672K 4676K 4680K 4684K 4688K 4692K 4696K 4700K 4704K 4708K 4712K 4716K 4720K 4724K 4728K 4732K 4736K 4740K 4744K 4748K 4752K 4756K 4760K 4764K 4768K 4772K 4776K 4780K 4784K 4788K 4792K 4796K 4800K 4804K 4808K 4812K 4816K 4820K 4824K 4828K 4832K 4836K 4840K 4844K 4848K 4852K 4856K 4860K 4864K 4868K 4872K 4876K 4880K