Parameter Tuning

Update 5/13 B: Looks about the same with heaven rankings

Closeness definition: (ranges from 0: bad to 1: good)

Graphs; Beware: they alternate Heaven+Hell, pD+pF

Update 5/13: Looking good against default parameters

The learners:

Learner	# of Parameters	# of Param Permutations
SKL Gaussian Bayes	0	1
SKL Multinomial Bayes	2	14
SKL Bernoulli Bayes	3	98
Total		113

"Unoptimized" learners are the following default parameters:

• SKL Gaussian Bayes: {}
• SKL Multinomial Bayes {'alpha': 1.0, 'fit_prior': True}
• SKL Bernoulli Bayes {'binarize': 0.5, 'alpha': 1.0, 'fit_prior': True}

Getting away from the whole "top ranked" reporting, I'm now reporting top 10 mean g.

• This still isn't the best, considering it's a summary statistic
• How do you feel about non-dominated sort on pD, 1-pF?

 Update 4/08: Interfaced with JMOO, nothing too fancy yet.

Update: More Evals, New Sets, Corrected Stats

These results generated with the following Xval/permutation params: (this took about 1.5hrs)

default_cross_val_folds=4
default_cross_val_repeats=4
default_param_permutations=5
default_percent_test=1.0/default_cross_val_folds

Stats generated with:

self.metrics={

'accuracy':metrics.accuracy_score(self.actual,self.predicted),

'precision':metrics.precision_score(self.actual,self.predicted),

'recall':metrics.recall_score(self.actual,self.predicted),

'confusion matrix':metrics.confusion_matrix(self.actual,self.predicted),

'Jaccard Score':metrics.jaccard_similarity_score(self.actual,self.predicted),

'ROC-AUC':metrics.accuracy_score(self.actual,self.predicted),

'class precision':metrics.precision_score(self.actual,self.predicted, average=None)

}

cm=self.metrics['confusion matrix']

d=cm[1][1]                                               #######from here out, it's basically your ABCD code, but I had to flip the CM

b=cm[1][0]

c=cm[0][1]

a=cm[0][0]

pd = 1.0*d / (b+d)

pf = 1.0*c / (a+c)

pn = 1.0*(b+d) / (a+c)

prec = 1.0*d / (c+d)

g = 2.0*(1-pf)*pd / (1-pf+pd)

f = 2.0*prec*pd/(prec+pd)

assert(sum(cm[1])==sum(self.actual))

self.metrics.update({

'pD':pd,

'pF':pf,

'g':g,

'f':f,

'pN':pn
})

Tabular results:
http://unbox.org/things/var/ben/consolidated/pres/results_summary_3-25-14.txt
http://unbox.org/things/var/ben/consolidated/pres/full_tuning_results_3-25-14.txt

Note about the charts: I switched to 1-pF so that heaven, hell are always (1,1) and (0,0).

Results from train on N, test on N+1

http://unbox.org/things/var/ben/consolidated/