Results

Plots and tables of our experimental results. Please refer to the paper for explanations regarding the methods. This page also contains additional results that we have omitted from the paper in order to save space.

Please note that data on restarts is sometimes missing, and this can result in an NA value in tables below.

PUP Single Fault

PUP Single Fault Results 1

PUP Single Fault Results 2

Statistics

Size Alpha Method Choices Learned Restarts
6 90/10 λ = 0.01 65000 4533 8
6 90/10 λ = 0.1 65000 4533 8
6 90/10 λ = 0.3 65000 4533 8
6 90/10 λ = 0.6 65000 4533 8
6 90/10 λ = 1 65000 4533 8
6 90/10 λ = 0.1 (C) 345000 77603 439
6 90/10 λ = 1 (C) 255000 44393 NA
6 90/10 PS 65000 4514 8
6 90/10 Restart 1477957 206585 251
6 80/20 λ = 0.01 65000 4465 8
6 80/20 λ = 0.1 65000 4465 8
6 80/20 λ = 0.3 65000 4465 8
6 80/20 λ = 0.6 65000 4465 8
6 80/20 λ = 1 65000 4465 8
6 80/20 λ = 0.1 (C) 380000 87087 NA
6 80/20 λ = 1 (C) 280000 50903 535
6 80/20 PS 65000 4446 8
6 80/20 Restart 3022123 451440 591
6 60/40 λ = 0.01 65000 4252 8
6 60/40 λ = 0.1 65000 4252 8
6 60/40 λ = 0.3 65000 4252 8
6 60/40 λ = 0.6 65000 4252 8
6 60/40 λ = 1 65000 4252 8
6 60/40 λ = 0.1 (C) 380000 87087 NA
6 60/40 λ = 1 (C) 285000 56123 NA
6 60/40 PS 65000 4233 NA
6 60/40 Restart 4587825 708215 1079
7 90/10 λ = 0.01 120000 10261 16
7 90/10 λ = 0.1 120000 10261 16
7 90/10 λ = 0.3 120000 10261 16
7 90/10 λ = 0.6 120000 10261 16
7 90/10 λ = 1 120000 10261 16
7 90/10 λ = 0.1 (C) 1045000 231933 1491
7 90/10 λ = 1 (C) 715000 138383 1485
7 90/10 PS 100000 5180 4
7 90/10 Restart 5785176 1843802 4954
7 80/20 λ = 0.01 95000 4172 2
7 80/20 λ = 0.1 95000 4172 2
7 80/20 λ = 0.3 95000 4172 2
7 80/20 λ = 0.6 95000 4172 2
7 80/20 λ = 1 95000 4172 2
7 80/20 λ = 0.1 (C) 970000 204197 1385
7 80/20 λ = 1 (C) 770000 154045 1524
7 80/20 PS 95000 4722 NA
7 80/20 Restart 11981298 3914982 10366
7 60/40 λ = 0.01 95000 4585 2
7 60/40 λ = 0.1 95000 4585 2
7 60/40 λ = 0.3 95000 4585 2
7 60/40 λ = 0.6 95000 4585 2
7 60/40 λ = 1 95000 4585 2
7 60/40 λ = 0.1 (C) 970000 215836 1321
7 60/40 λ = 1 (C) 905000 187585 1839
7 60/40 PS 130000 17126 29
7 60/40 Restart 18653893 6215195 16617
8 90/10 λ = 0.01 125000 3127 5
8 90/10 λ = 0.1 125000 3127 5
8 90/10 λ = 0.3 125000 3127 5
8 90/10 λ = 0.6 125000 3127 5
8 90/10 λ = 1 125000 3127 5
8 90/10 λ = 0.1 (C) 1485000 305774 1683
8 90/10 λ = 1 (C) 1000000 184175 1520
8 90/10 PS 125000 3127 5
8 90/10 Restart 7843206 2070220 9021
8 80/20 λ = 0.01 130000 3073 5
8 80/20 λ = 0.1 130000 3073 5
8 80/20 λ = 0.3 130000 3073 5
8 80/20 λ = 0.6 130000 3073 5
8 80/20 λ = 1 130000 3073 5
8 80/20 λ = 0.1 (C) 1530000 304796 1713
8 80/20 λ = 1 (C) 1210000 231523 1995
8 80/20 PS 130000 3007 5
8 80/20 Restart 15797074 4188771 18266
8 60/40 λ = 0.01 130000 3938 5
8 60/40 λ = 0.1 130000 3938 5
8 60/40 λ = 0.3 130000 3938 5
8 60/40 λ = 0.6 130000 3938 5
8 60/40 λ = 1 125000 3828 5
8 60/40 λ = 0.1 (C) 1355000 276749 1398
8 60/40 λ = 1 (C) 1250000 244173 2053
8 60/40 PS 125000 3748 6
8 60/40 Restart 24623964 6684041 28778
9 90/10 λ = 0.01 180000 6419 6
9 90/10 λ = 0.1 180000 6419 6
9 90/10 λ = 0.3 180000 6419 6
9 90/10 λ = 0.6 180000 6419 6
9 90/10 λ = 1 180000 6419 6
9 90/10 λ = 0.1 (C) 2275000 378586 2719
9 90/10 λ = 1 (C) 1665000 261281 2657
9 90/10 PS 180000 6419 6
9 90/10 Restart 12898145 3334677 11061
9 80/20 λ = 0.01 180000 6318 6
9 80/20 λ = 0.1 180000 6318 6
9 80/20 λ = 0.3 180000 6318 6
9 80/20 λ = 0.6 180000 6318 6
9 80/20 λ = 1 180000 6318 6
9 80/20 λ = 0.1 (C) 1965000 310478 2339
9 80/20 λ = 1 (C) 1655000 261233 2605
9 80/20 PS 180000 6318 6
9 80/20 Restart 26580720 6966754 23534
9 60/40 λ = 0.01 180000 6196 6
9 60/40 λ = 0.1 180000 6196 6
9 60/40 λ = 0.3 180000 6196 6
9 60/40 λ = 0.6 180000 6196 6
9 60/40 λ = 1 180000 6196 6
9 60/40 λ = 0.1 (C) 1985000 326978 2161
9 60/40 λ = 1 (C) 1805000 289169 2949
9 60/40 PS 180000 6195 NA
9 60/40 Restart 41023835 10960457 36736
10 90/10 λ = 0.01 200000 5835 5
10 90/10 λ = 0.1 200000 5835 5
10 90/10 λ = 0.3 200000 5835 5
10 90/10 λ = 0.6 200000 5835 5
10 90/10 λ = 1 200000 5835 NA
10 90/10 λ = 0.1 (C) 3125000 460063 3363
10 90/10 λ = 1 (C) 2905000 417612 4470
10 90/10 PS 200000 5819 5
10 90/10 Restart 17553971 4078438 16765
10 80/20 λ = 0.01 200000 7097 5
10 80/20 λ = 0.1 200000 7097 5
10 80/20 λ = 0.3 200000 7097 5
10 80/20 λ = 0.6 200000 7097 5
10 80/20 λ = 1 200000 7097 5
10 80/20 λ = 0.1 (C) 3430000 519589 3783
10 80/20 λ = 1 (C) 2780000 396750 NA
10 80/20 PS 200000 6736 5
10 80/20 Restart 34811524 8028074 33331
10 60/40 λ = 0.01 210000 5911 5
10 60/40 λ = 0.1 210000 5911 5
10 60/40 λ = 0.3 210000 5911 5
10 60/40 λ = 0.6 210000 5911 5
10 60/40 λ = 1 205000 5793 5
10 60/40 λ = 0.1 (C) 3230000 454116 3488
10 60/40 λ = 1 (C) 2705000 379432 3950
10 60/40 PS 205000 5173 NA
10 60/40 Restart 50207355 11318204 47320
11 90/10 λ = 0.01 260000 4289 2
11 90/10 λ = 0.1 260000 4289 2
11 90/10 λ = 0.3 260000 4289 2
11 90/10 λ = 0.6 260000 4289 2
11 90/10 λ = 1 260000 4289 2
11 90/10 λ = 0.1 (C) 5120000 686916 5235
11 90/10 λ = 1 (C) 4585000 606611 6376
11 90/10 PS 260000 4289 2
11 90/10 Restart 17020358 2706572 8504
11 80/20 λ = 0.01 260000 4160 2
11 80/20 λ = 0.1 260000 4160 2
11 80/20 λ = 0.3 260000 4160 2
11 80/20 λ = 0.6 260000 4160 2
11 80/20 λ = 1 260000 4160 2
11 80/20 λ = 0.1 (C) 5005000 653851 5389
11 80/20 λ = 1 (C) 4620000 631315 6504
11 80/20 PS 260000 4146 2
11 80/20 Restart 34956603 5760416 18849
11 60/40 λ = 0.01 260000 4406 2
11 60/40 λ = 0.1 260000 4406 2
11 60/40 λ = 0.3 260000 4406 2
11 60/40 λ = 0.6 260000 4406 2
11 60/40 λ = 1 260000 4406 2
11 60/40 λ = 0.1 (C) 5310000 737985 5278
11 60/40 λ = 1 (C) 4655000 632591 6555
11 60/40 PS 260000 4407 2
11 60/40 Restart 53294175 8946087 29164

PUP Multi Fault

PUP Multi Fault Results 1

PUP Multi Fault Results 2

Statistics

Size Alpha Method Choices Learned Restarts
6 90/10 λ = 0.01 65000 4631 8
6 90/10 λ = 0.1 65000 4631 8
6 90/10 λ = 0.3 65000 4631 8
6 90/10 λ = 0.6 65000 4631 8
6 90/10 λ = 1 65000 4631 NA
6 90/10 λ = 0.1 (C) 365000 81457 510
6 90/10 λ = 1 (C) 245000 42996 417
6 90/10 PS 65000 4612 8
6 90/10 Restart 1422579 177801 223
6 80/20 λ = 0.01 65000 4542 8
6 80/20 λ = 0.1 65000 4542 8
6 80/20 λ = 0.3 65000 4542 8
6 80/20 λ = 0.6 65000 4542 8
6 80/20 λ = 1 65000 4542 8
6 80/20 λ = 0.1 (C) 365000 81457 510
6 80/20 λ = 1 (C) 290000 58000 540
6 80/20 PS 65000 4523 8
6 80/20 Restart 2902838 391614 509
6 60/40 λ = 0.01 75000 4467 15
6 60/40 λ = 0.1 75000 4467 15
6 60/40 λ = 0.3 75000 4467 15
6 60/40 λ = 0.6 75000 4467 15
6 60/40 λ = 1 75000 4467 15
6 60/40 λ = 0.1 (C) 365000 85681 NA
6 60/40 λ = 1 (C) 315000 64895 587
6 60/40 PS 75000 6952 15
6 60/40 Restart 4492905 665018 1048
7 90/10 λ = 0.01 120000 10257 16
7 90/10 λ = 0.1 120000 10257 NA
7 90/10 λ = 0.3 120000 10257 16
7 90/10 λ = 0.6 120000 10257 16
7 90/10 λ = 1 120000 10257 16
7 90/10 λ = 0.1 (C) 870000 187177 1205
7 90/10 λ = 1 (C) 840000 171355 1764
7 90/10 PS 100000 5368 4
7 90/10 Restart 5499736 1703259 4684
7 80/20 λ = 0.01 95000 4174 2
7 80/20 λ = 0.1 95000 4174 2
7 80/20 λ = 0.3 95000 4174 2
7 80/20 λ = 0.6 95000 4174 2
7 80/20 λ = 1 95000 4174 2
7 80/20 λ = 0.1 (C) 950000 200408 1418
7 80/20 λ = 1 (C) 770000 153619 1584
7 80/20 PS 95000 4739 2
7 80/20 Restart 11224306 3541894 9491
7 60/40 λ = 0.01 90000 4587 2
7 60/40 λ = 0.1 90000 4587 2
7 60/40 λ = 0.3 90000 4587 2
7 60/40 λ = 0.6 90000 4587 2
7 60/40 λ = 1 90000 4587 2
7 60/40 λ = 0.1 (C) 920000 192283 1336
7 60/40 λ = 1 (C) 570000 100454 1004
7 60/40 PS 90000 4824 2
7 60/40 Restart 17580321 5699432 15280
8 90/10 λ = 0.01 125000 3148 5
8 90/10 λ = 0.1 125000 3148 5
8 90/10 λ = 0.3 125000 3148 5
8 90/10 λ = 0.6 125000 3148 5
8 90/10 λ = 1 125000 3148 5
8 90/10 λ = 0.1 (C) 1540000 323008 NA
8 90/10 λ = 1 (C) 1210000 236780 1992
8 90/10 PS 130000 3183 5
8 90/10 Restart 7972663 2118722 9167
8 80/20 λ = 0.01 130000 3093 5
8 80/20 λ = 0.1 130000 3093 5
8 80/20 λ = 0.3 130000 3093 5
8 80/20 λ = 0.6 130000 3093 5
8 80/20 λ = 1 130000 3093 5
8 80/20 λ = 0.1 (C) 1420000 291627 1564
8 80/20 λ = 1 (C) 1220000 239177 1962
8 80/20 PS 130000 3045 5
8 80/20 Restart 15794718 4183756 18049
8 60/40 λ = 0.01 125000 4116 5
8 60/40 λ = 0.1 125000 4116 5
8 60/40 λ = 0.3 125000 4116 5
8 60/40 λ = 0.6 125000 4116 5
8 60/40 λ = 1 125000 4116 5
8 60/40 λ = 0.1 (C) 1410000 292849 1522
8 60/40 λ = 1 (C) 1170000 219855 1862
8 60/40 PS 150000 13757 43
8 60/40 Restart 24218085 6516890 27679
9 90/10 λ = 0.01 180000 6521 6
9 90/10 λ = 0.1 180000 6521 NA
9 90/10 λ = 0.3 180000 6521 6
9 90/10 λ = 0.6 180000 6521 6
9 90/10 λ = 1 180000 6521 6
9 90/10 λ = 0.1 (C) 1930000 311293 2156
9 90/10 λ = 1 (C) 1985000 314511 3295
9 90/10 PS 180000 6521 6
9 90/10 Restart 12541666 3230118 9899
9 80/20 λ = 0.01 180000 6383 6
9 80/20 λ = 0.1 180000 6383 NA
9 80/20 λ = 0.3 180000 6383 6
9 80/20 λ = 0.6 180000 6383 6
9 80/20 λ = 1 180000 6383 6
9 80/20 λ = 0.1 (C) 1980000 329929 NA
9 80/20 λ = 1 (C) 1630000 253106 2566
9 80/20 PS 180000 6383 6
9 80/20 Restart 25720651 6686569 21479
9 60/40 λ = 0.01 180000 6285 6
9 60/40 λ = 0.1 180000 6285 6
9 60/40 λ = 0.3 180000 6285 6
9 60/40 λ = 0.6 180000 6285 6
9 60/40 λ = 1 180000 6285 6
9 60/40 λ = 0.1 (C) 2120000 348081 2399
9 60/40 λ = 1 (C) 1810000 287849 2882
9 60/40 PS 180000 6291 6
9 60/40 Restart 39424926 10385017 33904
10 90/10 λ = 0.01 200000 8450 5
10 90/10 λ = 0.1 200000 8450 5
10 90/10 λ = 0.3 200000 8450 5
10 90/10 λ = 0.6 200000 8450 5
10 90/10 λ = 1 200000 8450 5
10 90/10 λ = 0.1 (C) 3265000 473642 3683
10 90/10 λ = 1 (C) 2590000 363001 3891
10 90/10 PS 200000 6950 5
10 90/10 Restart 17573311 4104372 16379
10 80/20 λ = 0.01 205000 5794 5
10 80/20 λ = 0.1 205000 5794 5
10 80/20 λ = 0.3 205000 5794 5
10 80/20 λ = 0.6 205000 5794 5
10 80/20 λ = 1 205000 5794 5
10 80/20 λ = 0.1 (C) 3045000 448035 3302
10 80/20 λ = 1 (C) 2655000 381765 3919
10 80/20 PS 205000 5794 5
10 80/20 Restart 34709121 8030389 32449
10 60/40 λ = 0.01 210000 5420 5
10 60/40 λ = 0.1 210000 5420 5
10 60/40 λ = 0.3 210000 5420 5
10 60/40 λ = 0.6 210000 5420 5
10 60/40 λ = 1 205000 5309 5
10 60/40 λ = 0.1 (C) 3265000 486868 3465
10 60/40 λ = 1 (C) 2685000 378809 3944
10 60/40 PS 205000 5859 5
10 60/40 Restart 50059941 11307031 45825
11 90/10 λ = 0.01 260000 4399 2
11 90/10 λ = 0.1 260000 4399 2
11 90/10 λ = 0.3 260000 4399 2
11 90/10 λ = 0.6 260000 4399 2
11 90/10 λ = 1 260000 4399 2
11 90/10 λ = 0.1 (C) 5260000 706119 5461
11 90/10 λ = 1 (C) 4260000 569669 5840
11 90/10 PS 260000 4399 2
11 90/10 Restart 16518742 2512360 7589
11 80/20 λ = 0.01 260000 4095 2
11 80/20 λ = 0.1 260000 4095 2
11 80/20 λ = 0.3 260000 4095 2
11 80/20 λ = 0.6 260000 4095 2
11 80/20 λ = 1 260000 4095 2
11 80/20 λ = 0.1 (C) 4895000 669926 4815
11 80/20 λ = 1 (C) 4605000 613107 6508
11 80/20 PS 260000 4080 2
11 80/20 Restart 34107799 5451517 17420
11 60/40 λ = 0.01 260000 4333 2
11 60/40 λ = 0.1 260000 4333 2
11 60/40 λ = 0.3 260000 4333 2
11 60/40 λ = 0.6 260000 4333 2
11 60/40 λ = 1 260000 4333 2
11 60/40 λ = 0.1 (C) 5285000 720655 5373
11 60/40 λ = 1 (C) 4175000 543569 5761
11 60/40 PS 260000 4334 2
11 60/40 Restart 51574107 8363779 26646

PUP Switch Mutation

Note that our experiments using the Switch Mutations were done at an earlier time, and follow different naming conventions. “Naive” corresponds to a learning rate of 1.0, “Learning” corresponds to a learning rate of 0.1. The -C variants use only clause learning, i.e. progress saving has been disabled for the experiment. The -PS variant uses only progress saving, as all clauses remain frozen.

PUP Switch Results 1

PUP Switch Results 2

Statistics

Size Alpha Method Choices Learned Restarts
6 90/10 Naive 75000 5486 NA
6 90/10 Learning 75000 5486 17
6 90/10 Learning-C 365000 86989 451
6 90/10 Learning-PS 75000 5631 13
6 90/10 Restart 1640820 331064 456
6 80/20 Naive 100000 17903 NA
6 80/20 Learning 105000 21563 64
6 80/20 Learning-C 365000 86989 451
6 80/20 Learning-PS 140000 40352 109
6 80/20 Restart 3209985 623691 878
6 60/40 Naive 110000 23454 NA
6 60/40 Learning 125000 31593 102
6 60/40 Learning-C 310000 68251 NA
6 60/40 Learning-PS 140000 39655 118
6 60/40 Restart 4714954 886053 1374
7 90/10 Naive 170000 41869 NA
7 90/10 Learning 130000 18650 41
7 90/10 Learning-C 950000 211231 1319
7 90/10 Learning-PS 200000 50784 114
7 90/10 Restart 5379598 1741803 4223
7 80/20 Naive 160000 30221 107
7 80/20 Learning 240000 69974 240
7 80/20 Learning-C 870000 183980 NA
7 80/20 Learning-PS 295000 93223 272
7 80/20 Restart 10397510 3292407 8062
7 60/40 Naive 200000 49136 207
7 60/40 Learning 250000 72235 300
7 60/40 Learning-C 885000 196722 1126
7 60/40 Learning-PS 255000 74024 NA
7 60/40 Restart 15013663 4649331 11618
8 90/10 Naive 250000 54505 244
8 90/10 Learning 245000 51395 204
8 90/10 Learning-C 1485000 301168 1718
8 90/10 Learning-PS 260000 57011 142
8 90/10 Restart 8060330 2198217 8651
8 80/20 Naive 270000 62506 282
8 80/20 Learning 360000 98114 448
8 80/20 Learning-C 1440000 295696 NA
8 80/20 Learning-PS 390000 112833 NA
8 80/20 Restart 15313313 4067652 15919
8 60/40 Naive 390000 110383 569
8 60/40 Learning 395000 117715 425
8 60/40 Learning-C 1300000 263945 NA
8 60/40 Learning-PS 425000 121296 464
8 60/40 Restart 22173432 5794543 22334
9 90/10 Naive 345000 67107 341
9 90/10 Learning 395000 90202 437
9 90/10 Learning-C 2060000 347750 2390
9 90/10 Learning-PS 235000 24477 77
9 90/10 Restart 11731230 2957798 8737
9 80/20 Naive 400000 91734 490
9 80/20 Learning 505000 132643 610
9 80/20 Learning-C 2080000 345295 NA
9 80/20 Learning-PS 515000 126650 528
9 80/20 Restart 23288057 5860076 17751
9 60/40 Naive 665000 187840 1171
9 60/40 Learning 830000 257217 1139
9 60/40 Learning-C 1950000 315042 2057
9 60/40 Learning-PS 795000 231315 1057
9 60/40 Restart 34339719 8589846 26451
10 90/10 Naive 525000 118568 622
10 90/10 Learning 340000 51580 244
10 90/10 Learning-C 3045000 439523 3193
10 90/10 Learning-PS 600000 136916 686
10 90/10 Restart 16270527 3892068 12908
10 80/20 Naive 720000 190290 1157
10 80/20 Learning 480000 98696 464
10 80/20 Learning-C 3300000 480289 3582
10 80/20 Learning-PS 565000 121158 554
10 80/20 Restart 31263711 7228219 24632
10 60/40 Naive 855000 232827 1495
10 60/40 Learning 745000 189568 945
10 60/40 Learning-C 3010000 409506 3379
10 60/40 Learning-PS 870000 231808 960
10 60/40 Restart 44904488 9891474 34736
11 90/10 Naive 545000 99140 NA
11 90/10 Learning 635000 125838 592
11 90/10 Learning-C 4765000 638755 4814
11 90/10 Learning-PS 745000 162100 701
11 90/10 Restart 16114767 2536538 7631
11 80/20 Naive 775000 177128 1072
11 80/20 Learning 1380000 375875 NA
11 80/20 Learning-C 4760000 626787 4786
11 80/20 Learning-PS 595000 108133 528
11 80/20 Restart 33153551 5376263 17571
11 60/40 Naive 1100000 280006 1743
11 60/40 Learning 1405000 383291 2013
11 60/40 Learning-C 5360000 710795 5233
11 60/40 Learning-PS 1380000 363600 NA
11 60/40 Restart 50074243 8189602 27184

N-Queens

N-Queens Results

Statistics

Size Method Choices Learned Restarts
14 λ = 0.01 25000 369 0
14 λ = 0.1 25000 369 NA
14 λ = 0.3 25000 369 0
14 λ = 0.6 25000 369 0
14 λ = 1 25000 369 0
14 λ = 0.1 (C) 25000 385 NA
14 λ = 1 (C) 25000 385 NA
14 PS 30000 1720 NA
14 Restart 4677 2007 0
18 λ = 0.01 40000 1225 0
18 λ = 0.1 40000 1225 NA
18 λ = 0.3 40000 1225 0
18 λ = 0.6 40000 1225 0
18 λ = 1 40000 1225 NA
18 λ = 0.1 (C) 40000 1065 NA
18 λ = 1 (C) 40000 1065 NA
18 PS 45000 4449 NA
18 Restart 13563 7786 0
22 λ = 0.01 75000 4301 23
22 λ = 0.1 75000 4301 NA
22 λ = 0.3 75000 4301 23
22 λ = 0.6 75000 4301 23
22 λ = 1 75000 4301 23
22 λ = 0.1 (C) 80000 6203 38
22 λ = 1 (C) 80000 6197 39
22 PS 75000 5548 NA
22 Restart 13094 4920 0
26 λ = 0.01 100000 4889 13
26 λ = 0.1 100000 5096 NA
26 λ = 0.3 100000 5062 13
26 λ = 0.6 100000 4889 11
26 λ = 1 100000 4890 13
26 λ = 0.1 (C) 105000 5596 NA
26 λ = 1 (C) 105000 5538 NA
26 PS 105000 6142 NA
26 Restart 33632 18064 0
30 λ = 0.01 125000 7002 8
30 λ = 0.1 125000 7457 10
30 λ = 0.3 125000 7591 7
30 λ = 0.6 125000 7831 8
30 λ = 1 125000 7277 11
30 λ = 0.1 (C) 130000 7451 35
30 λ = 1 (C) 130000 8050 49
30 PS 125000 7272 3
30 Restart 40451 22527 73