Scoring JPL

Ideas implemented:

1. Changed criterion to gini and checked results. Result: Minor changes to CART tree.
2. Train on N nearest to centroid. What should N be? I am planning to try out range of values and check the result.
3. Prune leaves with sd greater than 0.3 of root. -->Very different clusters. Result: 3 clusterers eliminated.

Ideas to be implemented:

1. Join two clusters that are statistically insignificantly different. Problem: Needs major changes to structure, Trying to figure a smart way out.

Scoring:

For every cluster:

• Compare with all other clusters.
• It is said to be "better" over other when:
• Both clusters are statistically significantly different by both tests.
• Centroid is better(more or less based on dependent variables) than other.
• It is said to be similar when:
• first if a12 returns as similar.
• if a12 returns as different--bootstrap test is done to check that.
• Similar if Both tests returns similar.
• It is said to be worse when:
• if it is different as stated by both tests.
• Centroid is not better than other.
• If cluster is better on one and is worse on none, then score of that cluster is incremented.

Scores of each cluster based on training in nearest n = 20 to centroid:

http://unbox.org/things/var/nave/lpj/out/dats/jpl_scored_n20.dat

when nearest n is not used:

http://unbox.org/things/var/nave/lpj/out/dats/jpl_scored_ninf.dat

To do:

As structures for each branch are already done, categorize clusters based on scores into "good" and "bad". Pick each cluster from "bad", find the nearest branch in CART that has good and difference the conditions.

Run these conditions through xomo to generate new dataset and compare result.

Ideas for "good" and "bad":
Sort clusters based on scores and pick top n^0.5 and say they are "good". Others are "bad".

Flow Chart and List of Ideas

Flow chart:

• Build clusters recursing spectrally using Darren's data.
• Run CART on those clusters to find and list conditions that are different between clusters.
• For each cluster, run a12 and bootstrap to find others that are better. Best on one dep variable and worse on none.
• Find the differences in conditions when we traverse from one cluster to another using CART branching.
• Generate new dataset and repeat to see if those conditions actually matter.

Ideas:

1. ***Label clusters with how close they are.
2. ***Round the floating numbers to .2 decimal points 
3. ***For each dependent variable Run a12 and then bootstrap to see if clusters are actually different.
4. ***Generate conditions for each branch and find difference between them.
5. Change criterion to "gini" and check results.
6. Don't split the clusters if the neighbors have statistically insignificantly different scores.
7. Train on N things nearest to the centroids of each cluster and test on all.

Note: *** are done.

Current work:

Working on spectrally generated clusters to use WHICH to see if rules of variables between clusters are same as ones produced by CART. To check and see if CART is right.

JPL clustered line graph

Up Next:

• Ribbon?
• Score Clusters and represent them as in http://goo.gl/nqJdtu 

Things to do:

• For each cluster: 

1. Find other cluster that is better
2. One that it "envies"

• Find contrast between two cluster
• See how different they are-a12 predicate

Learner Noise Oddities

While re-generating some plots for the data-quality paper I stumbled into some oddities in how PEEKER handles some forms of noise in some types of data. Below are the results of a small experiment I ran comparing PEEKER's performance to Naive Bayes on the same sets using the same methods as before, but with all 10 Data Sets on one plot for each performance category.

PEEKER:
Noise Methods from here

Note: NB Results are nearly identical, they are not included to save space.

PEEKER:

Random Swap

NB:

Random Swap

Based on these results there seems to be something strange about the way PEEKER handles the "Random Swap" case of noise for certain data. This is particularly odd because it handles the other forms of noise with ease as we can see in the first set of results--most of which are far more complex types of noise than simply swapping the class value, or so one would think...

It is possible the culprit could be as simple as the set becoming is too heavily weighted in one direction or the other (defective/non-defective) after Feature/Instance Selection causing the decline in performance for some of the data-sets. However as of right now I cannot determine the cause of the performance decline for certain.

This didn't seem evident before we started comparing the results in this fashion. It is worthwhile to note however that the decline seems to kick into effect around the 30%+ Noise mark, which is still at least 10% higher than the study linked above suggests in the most generous of estimates of their data. However their reported figures are somewhat unclear and unspecific.

The results for RF and KNN mirror the results for NB so they were not included in this post. They can be found here... KNN  RF

500 sample Cocomo/Coqualmo/Sced-Risk runs using constraints from Figure 4 in http://menzies.us/pdf/08drastic.pdf (JPL Flight).

Output here: http://unbox.org/things/var/darren/coco/jplflight.csv

Business Case Temporal Error Prediction

Some bests from a for-loop tune Feb 11, 2014

http://unbox.org/things/var/ben/consolidated/logs/log_1%20copy.txt

Results 0 - Feb 4, 2014

Here is what we want:

results: https://www.dropbox.com/sh/0ubpqtah1g4u504/bAriPfgj8Q/Results%20Feb4

Data Quality Paper Progress

The paper can be viewed here:

http://unbox.org/things/var/brian/2013/papers/dataQuality13/dataQuality13.pdf

Major changes since last update:

• Updated plots to display all 10 Data Sets
• A12 stats table in Learning Section (will probably replace with wilcoxon)
• Revamped intro
• Fixed some of the tables to be more readable
• Added corners of the box example to Exemplar Theory section (maybe move this to MicroSampling section?)
• Various minor changes such as spelling, spacing, adding just a few sentences here or there, ect..

TODO:

• Fix code samples
• Get higher rez random-pull plot from Fayola
• Conclusion
• Abstract
• Future Work
• Write
• Rewrite
• Write
• Rewrite
• Find more references

POM4

List of changes in POM4 compared to POM3:

* Replaced Requirements with User Stories
* Added Sprints
* Changed Execution
* Models time of completion (Velocity)
* No product Backlog as mentioned earlier--Not relevant


Sample output:

[Cost, Score, completion, idle, velocity]

[2409.671458440433, 0.8841628909936482, 0.29411764705882354, 0.7058823529411764, 28.1]
[2308.2960550738408, 0.7867703286312365, 0.3660130718954248, 0.6339869281045751, 28.791666666666668]
[2570.022284498054, 0.8192890455541242, 0.3492063492063492, 0.6507936507936508, 28.15]
[1776.6221769857425, 0.8934120231082071, 0.06578947368421052, 0.9305555555555556, 18.25]
[2663.5466581317396, 0.8377502677420756, 0.20610687022900764, 0.7923076923076923, 24.583333333333332]
[2873.7679677836973, 0.8005907722786698, 0.30708661417322836, 0.6929133858267716, 28.0625]
[2974.0503086683807, 0.9211912145791122, 0.13821138211382114, 0.8571428571428572, 21.375]
[2254.083309388589, 0.8483742963781609, 0.22875816993464052, 0.7712418300653595, 27.125]
[2252.9218006369865, 0.8187803230043464, 0.10588235294117647, 0.8881987577639752, 24.125]
[1970.4420192421026, 0.8980781700913005, 0.10382513661202186, 0.8716216216216216, 28.083333333333332]
[2239.8372694910295, 0.7732764195049752, 0.4065040650406504, 0.5934959349593496, 28.041666666666668]
[2490.8178434146753, 0.8548533334449776, 0.30158730158730157, 0.6984126984126984, 27.6875]
[2090.459851323212, 0.906146585636779, 0.09595959595959595, 0.8983957219251337, 28.875]
[2487.32661497613, 0.8025235096809521, 0.310126582278481, 0.689873417721519, 28.541666666666668]
[1999.257417423182, 0.7881805570666411, 0.15263157894736842, 0.8473684210526315, 28.166666666666668]
[2659.3639169614175, 0.8136230090693194, 0.44715447154471544, 0.5528455284552846, 31.416666666666668]
[2518.5831627767234, 0.7714916788706024, 0.3984375, 0.6015625, 28.458333333333332]
[2653.9829039013243, 0.7942550808491925, 0.3898305084745763, 0.6101694915254237, 27.916666666666668]
[2520.328542653908, 0.8374772551439275, 0.29545454545454547, 0.7045454545454546, 28.4375]
[2544.763860934486, 0.8723176429348501, 0.11711711711711711, 0.8828828828828829, 26.0]
[2628.456723197506, 0.8203655792518135, 0.39316239316239315, 0.6068376068376069, 28.8]
[2573.503127321302, 0.7884237959629854, 0.3142857142857143, 0.6857142857142857, 26.55]
[1961.7933461746904, 0.8152854455707702, 0.27350427350427353, 0.7264957264957265, 27.875]
[2316.8948214132224, 0.8217888256150522, 0.21951219512195122, 0.7804878048780488, 28.583333333333332]
[2480.6806520346904, 0.8747529690927919, 0.16923076923076924, 0.8211382113821138, 26.583333333333332]
[2659.449863697794, 0.9241150906277656, 0.10285714285714286, 0.8971428571428571, 21.75]
[2670.630413055988, 0.8185772070791713, 0.44660194174757284, 0.5533980582524272, 26.833333333333332]
[2261.9032345883297, 0.880074563709406, 0.3673469387755102, 0.6326530612244898, 26.5]
[2803.9527726963315, 0.8112949960654825, 0.4020618556701031, 0.5760869565217391, 24.4]
[1789.2637140962051, 0.9134638787851941, 0.1320754716981132, 0.8571428571428572, 26.25]
[2551.9635529278703, 0.8700612111658858, 0.2184873949579832, 0.7777777777777778, 27.583333333333332]
[2715.4994139204637, 0.8218098445599464, 0.19718309859154928, 0.8028169014084507, 25.8125]
[2792.223819921654, 0.8980444641536334, 0.25510204081632654, 0.7448979591836735, 28.583333333333332]
[2484.5482551716327, 0.8902994085435698, 0.2127659574468085, 0.7872340425531915, 25.333333333333332]
[2723.6434810973956, 0.7986069896029132, 0.3389830508474576, 0.6610169491525424, 29.125]
[2927.417113886656, 0.8386515663739715, 0.3037037037037037, 0.6962962962962963, 28.9375]
[2295.532964721932, 0.8358459503202905, 0.20437956204379562, 0.7956204379562044, 28.25]
[2312.3109954731426, 0.7936932169797876, 0.3, 0.7, 27.625]
[2043.7151661914083, 0.8710077212863819, 0.1721311475409836, 0.7835051546391752, 23.25]
[2331.391170948724, 0.7928084070500105, 0.296875, 0.703125, 31.25]
[3078.8501033528346, 0.8888925374805372, 0.2549019607843137, 0.7450980392156863, 24.166666666666668]
[3100.958248463834, 0.7515307494535662, 0.23893805309734514, 0.7610619469026548, 26.9375]
[2199.850852366886, 0.8303671112167473, 0.21022727272727273, 0.7897727272727273, 26.875]
[2232.2895282162535, 0.904199293106172, 0.10559006211180125, 0.8944099378881988, 27.25]
[2795.6273106821627, 0.8801933995960874, 0.18115942028985507, 0.8188405797101449, 23.0]
[3066.9233408587424, 0.8898168997188672, 0.16822429906542055, 0.8317757009345794, 23.75]
[2887.73100680117, 0.9125038606759316, 0.12149532710280374, 0.8785046728971962, 22.25]
[2848.9219718849586, 0.9314492852738758, 0.11564625850340136, 0.8721804511278195, 25.75]
[2423.8773329190794, 0.8759079531570052, 0.20353982300884957, 0.7964601769911505, 23.083333333333332]
[2125.342000582769, 0.8306157810083096, 0.23270440251572327, 0.7672955974842768, 26.6875]



---------------------------------------------------------------------------------
> SPRINT LENGTH is considered as 30 days as given in the "ScrumBut" paper

> Time estimates are randomly assigned in the range(0,30) and are changed randomly based on team dynamism and culture.

> Velocity represents average time of completion for each sprint.

______________________________________________________________

Currently analyzing JMOO_V0 and hooking POM4 to JMOO. ETA: 2-3 days