Update: Hedges Comparison
ant-1.4
[32, 31, 32, 30]
[54, 48, 44, 32]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 51 | 0.03 | 0
ant-1.5
[46, 44, 45, 43]
[90, 96, 59, 48]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 52 | 0.29 | 0
ant-1.6
[38, 36, 37, 36, 38, 36, 37, 35]
[48, 39, 41, 36, 50, 50, 57, 30]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.22 | 0
ant-1.7
[43, 45, 44, 44, 43, 44, 45, 43]
[114, 210, 76, 68, 59, 88, 77, 53]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 50 | 0.04 | 0
camel-1.2
[42, 43, 41, 44, 43, 42, 43, 41]
[97, 75, 65, 89, 59, 63, 117, 43]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 50 | 0.37 | 0
camel-1.4
[77, 76, 75, 77, 77, 75, 76, 75]
[106, 119, 127, 119, 109, 100, 137, 55]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 51 | 0.91 | 1
camel-1.6
[112, 106, 105, 114, 110, 108, 109, 108]
[137, 114, 138, 134, 129, 107, 153, 53]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.51 | 1
ivy-1.4
[29, 27, 28, 27]
[82, 58, 67, 34]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
42 | 52 | 1.68 | 2
ivy-2.0
[61, 60, 61, 59]
[116, 76, 114, 46]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.10 | 0
jedit-4.0
[35, 34, 35, 33, 35, 33, 34, 33]
[42, 45, 35, 37, 38, 40, 52, 17]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
57 | 55 | 0.22 | 0
jedit-4.1
[40, 38, 39, 38, 39, 37, 38, 37]
[41, 36, 34, 43, 38, 48, 56, 16]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 52 | 0.39 | 1
jedit-4.2
[40, 39, 40, 38, 40, 38, 39, 38]
[46, 46, 36, 48, 48, 59, 63, 21]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 52 | 0.10 | 0
jedit-4.3
[47, 46, 45, 46, 47, 45, 46, 45]
[106, 56, 48, 59, 51, 56, 102, 14]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 49 | 0.04 | 0
log4j-1.1
[35, 33, 34, 33]
[27, 21, 35, 26]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
62 | 57 | 0.35 | 0
log4j-1.2
[28, 27, 28, 26]
[65, 45, 64, 31]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
64 | 60 | 0.20 | 0
lucene-2.2
[50, 48, 49, 48]
[71, 66, 79, 31]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 54 | 0.80 | 1
lucene-2.4
[32, 31, 62, 62, 60]
[50, 50, 86, 102, 52]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 51 | 0.44 | 1
synapse-1.1
[40, 39, 40, 38]
[70, 46, 61, 45]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
54 | 52 | 0.25 | 0
synapse-1.2
[58, 54, 56, 54]
[76, 49, 77, 54]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 52 | 0.79 | 1
velocity-1.5
[50, 49, 49, 48]
[74, 45, 80, 15]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 52 | 0.39 | 0
velocity-1.6
[55, 53, 54, 52]
[76, 48, 89, 16]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
52 | 49 | 1.15 | 2
xalan-2.5
[93, 89, 91, 89, 91, 90, 91, 89]
[124, 103, 89, 86, 92, 105, 129, 75]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 50 | 0.03 | 0
xalan-2.6
[102, 100, 101, 99, 101, 100, 101, 99]
[115, 94, 87, 128, 119, 106, 149, 87]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
47 | 50 | 0.36 | 0
xalan-2.7
[113, 111, 112, 111, 111, 109, 110, 108]
[130, 117, 106, 102, 132, 129, 112, 81]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
73 | 63 | 0.64 | 1
xerces-1.3
[58, 54, 60, 50, 56, 54, 55, 53]
[75, 60, 73, 31, 54, 64, 62, 34]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.16 | 0
xerces-1.4
[59, 55, 61, 52, 58, 56, 57, 55]
[104, 77, 96, 48, 70, 84, 84, 25]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
65 | 57 | 0.58 | 1
[32, 31, 32, 30]
[54, 48, 44, 32]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 51 | 0.03 | 0
ant-1.5
[46, 44, 45, 43]
[90, 96, 59, 48]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 52 | 0.29 | 0
ant-1.6
[38, 36, 37, 36, 38, 36, 37, 35]
[48, 39, 41, 36, 50, 50, 57, 30]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.22 | 0
ant-1.7
[43, 45, 44, 44, 43, 44, 45, 43]
[114, 210, 76, 68, 59, 88, 77, 53]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 50 | 0.04 | 0
camel-1.2
[42, 43, 41, 44, 43, 42, 43, 41]
[97, 75, 65, 89, 59, 63, 117, 43]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 50 | 0.37 | 0
camel-1.4
[77, 76, 75, 77, 77, 75, 76, 75]
[106, 119, 127, 119, 109, 100, 137, 55]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 51 | 0.91 | 1
camel-1.6
[112, 106, 105, 114, 110, 108, 109, 108]
[137, 114, 138, 134, 129, 107, 153, 53]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.51 | 1
ivy-1.4
[29, 27, 28, 27]
[82, 58, 67, 34]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
42 | 52 | 1.68 | 2
ivy-2.0
[61, 60, 61, 59]
[116, 76, 114, 46]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.10 | 0
jedit-4.0
[35, 34, 35, 33, 35, 33, 34, 33]
[42, 45, 35, 37, 38, 40, 52, 17]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
57 | 55 | 0.22 | 0
jedit-4.1
[40, 38, 39, 38, 39, 37, 38, 37]
[41, 36, 34, 43, 38, 48, 56, 16]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 52 | 0.39 | 1
jedit-4.2
[40, 39, 40, 38, 40, 38, 39, 38]
[46, 46, 36, 48, 48, 59, 63, 21]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 52 | 0.10 | 0
jedit-4.3
[47, 46, 45, 46, 47, 45, 46, 45]
[106, 56, 48, 59, 51, 56, 102, 14]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 49 | 0.04 | 0
log4j-1.1
[35, 33, 34, 33]
[27, 21, 35, 26]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
62 | 57 | 0.35 | 0
log4j-1.2
[28, 27, 28, 26]
[65, 45, 64, 31]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
64 | 60 | 0.20 | 0
lucene-2.2
[50, 48, 49, 48]
[71, 66, 79, 31]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
51 | 54 | 0.80 | 1
lucene-2.4
[32, 31, 62, 62, 60]
[50, 50, 86, 102, 52]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 51 | 0.44 | 1
synapse-1.1
[40, 39, 40, 38]
[70, 46, 61, 45]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
54 | 52 | 0.25 | 0
synapse-1.2
[58, 54, 56, 54]
[76, 49, 77, 54]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
55 | 52 | 0.79 | 1
velocity-1.5
[50, 49, 49, 48]
[74, 45, 80, 15]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
53 | 52 | 0.39 | 0
velocity-1.6
[55, 53, 54, 52]
[76, 48, 89, 16]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
52 | 49 | 1.15 | 2
xalan-2.5
[93, 89, 91, 89, 91, 90, 91, 89]
[124, 103, 89, 86, 92, 105, 129, 75]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
50 | 50 | 0.03 | 0
xalan-2.6
[102, 100, 101, 99, 101, 100, 101, 99]
[115, 94, 87, 128, 119, 106, 149, 87]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
47 | 50 | 0.36 | 0
xalan-2.7
[113, 111, 112, 111, 111, 109, 110, 108]
[130, 117, 106, 102, 132, 129, 112, 81]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
73 | 63 | 0.64 | 1
xerces-1.3
[58, 54, 60, 50, 56, 54, 55, 53]
[75, 60, 73, 31, 54, 64, 62, 34]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
49 | 50 | 0.16 | 0
xerces-1.4
[59, 55, 61, 52, 58, 56, 57, 55]
[104, 77, 96, 48, 70, 84, 84, 25]
Cluster pD,pF mean AUC | Straw Cluster mean AUC | Hedges | Level
65 | 57 | 0.58 | 1
Explanation:
Results:
This doesn't always help us, (sometimes it hurts) but for some clusters, it REALLY works.
Key: (too verbose for each graph)
Optimized Results for each cluster == colored Solid line with Stars
Default Param Results for each cluster == colored Dashed line with Circles
No clustering, just optimize N-1, test N == Double-dashed Black line with Diamonds
The AUCs:
ant-1.4 pD, pF AUCs (optimized/default)
No Clustering: 52
Cluster 0 50/49
Cluster 1 45/46
Cluster 2 49/51
Cluster 3 57/56
===============================================
ant-1.5 pD, pF AUCs (optimized/default)
No Clustering: 58
Cluster 0 40/50
Cluster 1 50/51
Cluster 2 55/53
Cluster 3 57/55
===============================================
ant-1.6 pD, pF AUCs (optimized/default)
No Clustering: 65
Cluster 0 50/50
Cluster 1 49/49
Cluster 2 49/49
Cluster 3 57/46
Cluster 4 46/47
Cluster 5 50/52
Cluster 6 49/47
Cluster 7 57/58
===============================================
ant-1.7 pD, pF AUCs (optimized/default)
No Clustering: 66
Cluster 0 53/52
Cluster 1 50/49
Cluster 2 44/48
Cluster 3 53/50
Cluster 4 54/51
Cluster 5 49/48
Cluster 6 46/50
Cluster 7 56/53
===============================================
camel-1.2 pD, pF AUCs (optimized/default)
No Clustering: 53
Cluster 0 50/50
Cluster 1 49/49
Cluster 2 50/50
Cluster 3 50/50
Cluster 4 50/50
Cluster 5 53/50
Cluster 6 55/54
Cluster 7 54/53
===============================================
camel-1.4 pD, pF AUCs (optimized/default)
No Clustering: 60
Cluster 0 60/52
Cluster 1 53/49
Cluster 2 50/48
Cluster 3 63/54
Cluster 4 53/56
Cluster 5 52/49
Cluster 6 55/52
Cluster 7 50/48
===============================================
camel-1.6 pD, pF AUCs (optimized/default)
No Clustering: 55
Cluster 0 51/49
Cluster 1 46/48
Cluster 2 47/49
Cluster 3 51/54
Cluster 4 47/50
Cluster 5 50/50
Cluster 6 54/52
Cluster 7 47/50
===============================================
ivy-1.4 pD, pF AUCs (optimized/default)
No Clustering: 58
Cluster 0 46/51
Cluster 1 33/61
Cluster 2 42/53
Cluster 3 39/43
===============================================
ivy-2.0 pD, pF AUCs (optimized/default)
No Clustering: 58
Cluster 0 47/49
Cluster 1 41/49
Cluster 2 49/48
Cluster 3 56/54
===============================================
jedit-4.0 pD, pF AUCs (optimized/default)
No Clustering: 65
Cluster 0 47/48
Cluster 1 50/49
Cluster 2 49/50
Cluster 3 52/47
Cluster 4 67/65
Cluster 5 69/58
Cluster 6 56/59
Cluster 7 69/62
===============================================
jedit-4.1 pD, pF AUCs (optimized/default)
No Clustering: 68
Cluster 0 48/48
Cluster 1 52/46
Cluster 2 52/47
Cluster 3 45/46
Cluster 4 42/49
Cluster 5 66/57
Cluster 6 57/60
Cluster 7 68/61
===============================================
jedit-4.2 pD, pF AUCs (optimized/default)
No Clustering: 68
Cluster 0 60/48
Cluster 1 47/46
Cluster 2 47/48
Cluster 3 50/59
Cluster 4 54/55
Cluster 5 61/58
Cluster 6 52/52
Cluster 7 56/52
===============================================
jedit-4.3 pD, pF AUCs (optimized/default)
No Clustering: 61
Cluster 0 49/49
Cluster 1 48/49
Cluster 2 47/48
Cluster 3 45/48
Cluster 4 40/46
Cluster 5 51/45
Cluster 6 58/54
Cluster 7 60/57
===============================================
log4j-1.1 pD, pF AUCs (optimized/default)
No Clustering: 75
Cluster 0 39/48
Cluster 1 52/47
Cluster 2 73/65
Cluster 3 73/69
===============================================
log4j-1.2 pD, pF AUCs (optimized/default)
No Clustering: 57
Cluster 0 47/54
Cluster 1 54/51
Cluster 2 58/56
Cluster 3 94/80
===============================================
lucene-2.2 pD, pF AUCs (optimized/default)
No Clustering: 57
Cluster 0 49/53
Cluster 1 54/51
Cluster 2 56/54
Cluster 3 55/56
===============================================
lucene-2.4 pD, pF AUCs (optimized/default)
No Clustering: 58
Cluster 0 51/52
Cluster 1 43/46
Cluster 2 54/52
Cluster 3 58/53
Cluster 4 52/50
===============================================
synapse-1.1 pD, pF AUCs (optimized/default)
No Clustering: 55
Cluster 0 49/51
Cluster 1 44/53
Cluster 2 48/51
Cluster 3 64/54
===============================================
synapse-1.2 pD, pF AUCs (optimized/default)
No Clustering: 62
Cluster 0 58/52
Cluster 1 51/50
Cluster 2 49/51
Cluster 3 54/55
===============================================
velocity-1.5 pD, pF AUCs (optimized/default)
No Clustering: 49
Cluster 0 50/50
Cluster 1 57/53
Cluster 2 57/55
Cluster 3 59/48
===============================================
velocity-1.6 pD, pF AUCs (optimized/default)
No Clustering: 58
Cluster 0 44/48
Cluster 1 54/50
Cluster 2 49/50
Cluster 3 53/50
===============================================
xalan-2.5 pD, pF AUCs (optimized/default)
No Clustering: 55
Cluster 0 50/50
Cluster 1 50/50
Cluster 2 51/49
Cluster 3 52/49
Cluster 4 48/50
Cluster 5 52/53
Cluster 6 47/47
Cluster 7 59/54
===============================================
xalan-2.6 pD, pF AUCs (optimized/default)
No Clustering: 56
Cluster 0 51/51
Cluster 1 53/49
Cluster 2 50/48
Cluster 3 56/54
Cluster 4 47/49
Cluster 5 47/49
Cluster 6 47/51
Cluster 7 40/47
===============================================
xalan-2.7 pD, pF AUCs (optimized/default)
No Clustering: 72
Cluster 0 53/51
Cluster 1 63/54
Cluster 2 58/40
Cluster 3 79/59
Cluster 4 81/64
Cluster 5 69/70
Cluster 6 88/79
Cluster 7 98/89
===============================================
xerces-1.3 pD, pF AUCs (optimized/default)
No Clustering: 48
Cluster 0 39/46
Cluster 1 61/49
Cluster 2 46/52
Cluster 3 33/50
Cluster 4 49/55
Cluster 5 46/48
Cluster 6 54/53
Cluster 7 52/51
===============================================
xerces-1.4 pD, pF AUCs (optimized/default)
No Clustering: 63
Cluster 0 50/50
Cluster 1 54/51
Cluster 2 60/54
Cluster 3 56/52
Cluster 4 67/54
Cluster 5 73/58
Cluster 6 68/58
Cluster 7 97/83
===============================================
No comments:
Post a Comment