PLAN C: Prune trees

1a) delete data from any leaf containing things from > 1 cluster.
Did this one . Check http://unbox.org/things/var/nave/lpj/out/10_June_2014/dats/pruned_tree.dat

for pruned tree and actual tree. 
To check if the code is working or not see: http://unbox.org/things/var/nave/lpj/out/10_June_2014/dats/short_example.dat

1b) descend the trees generated from CART looking for sub-trees that
whose items with cluster ID have HIGHER entropy than the parents, then
delete all items in those sub-trees-of-confusion

1c) for all sub-trees built by CART, compute the entropy of the leaf
items in that tree. sort those entropies to find "too much confusion"
e.g. half way down that list. delete all sub-trees with MORE than "too
much confusion"

then after step1, rebuild the trees using the reduced data sets.

Apparently scikit-learn dtrees do not maintain samples in their trees. meaning. at the leafs or at the nodes there is no sample data. It runs through the sample data and gathers values(results) but not store samples themselves. So at a leaf, I can only find stats of samples, like
[  7.   8.   3.   0.   0.   0.   0.   0.   3.   0.   5.   2.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]

indicating there are 7 samples of first cluster, 8 of second and so on.

So I cannot actually go back to dataset remove those rows and rebuild the tree.

Instead I can calculate entropies using above array at each leaf/node and prune ones using 1b and 1c.

PS: One more problem, those arrays of values are not maintained at nodes! there are only available at leaves, so I have to traverse down entire branch beneath a node and add those arrays together to get that above array at that node.

results Contrast set learner

55555 rounding
Before rounding:

 Techniques         -effort         -months        -defects          -risks    #
           T0 m              29              67              14               2    #
  T3 C25 N100 m               2              10               3               0    #
      T9:j/j_ m               0              35               0              21    #
-------------------------------------------------------------------------------------
           T0 q              26              16              20               6    #
  T3 C25 N100 q               3               0               5               1    #
      T9:j/j_ q               3              23               4              38    #
-------------------------------------------------------------------------------------
           T0 w             100             100             100              19    #
  T3 C25 N100 w              32              21              49               4    #
      T9:j/j_ w              32             100              56             100    #
-------------------------------------------------------------------------------------
            100         2687.38            41.4        17612.38             8.6    #
              0          117.21            1.93          381.83            0.05    #

After rounding:
Dataset rounded to integers:

 Techniques         -effort         -months        -defects          -risks    #
           T0 m              29              67              14               0    #
  T3 C25 N100 m               3              10              10               1    #
      T9:j/j_ m               0              35               0              22    #
-------------------------------------------------------------------------------------
           T0 q              28              17              24               2    #
  T3 C25 N100 q               3               0              13               2    #
      T9:j/j_ q               3              23               4              38    #
-------------------------------------------------------------------------------------
           T0 w             100              99             100              15    #
  T3 C25 N100 w              26              19              78               4    #
      T9:j/j_ w              32             100              59             100    #
-------------------------------------------------------------------------------------
            100          2695.0            41.4        16771.11             8.6    #
              0          117.21            1.85          381.83             0.0    # 

7777 Trees of all

http://unbox.org/things/var/nave/lpj/out/02_June_2014/dats/tree*

88888 Results for all models after rounding (makes sense):

flight:

     Techniques         -effort         -months        -defects          -risks    #
           T0 m              27              67              12               4    #
  T3 C25 N100 m               0              10               0               0    #
    T9:j/jall m               4              46               1              16    #
-------------------------------------------------------------------------------------
           T0 q              24              16              18              10    #
  T3 C25 N100 q               0               0               2               1    #
    T9:j/jall q              10              29               6              43    #
-------------------------------------------------------------------------------------
           T0 w             100             100             100              35    #
  T3 C25 N100 w              31              21              47               7    #
    T9:j/jall w              32              97              35             100    #
-------------------------------------------------------------------------------------
            100         2687.38           41.35        17612.38             4.8    #
              0          172.58            1.92          892.87            0.04    #

ground:

 
Techniques         -effort         -months        -defects          -risks    #
           T0 m              15              55               8              10    #
  T3 C25 N100 m               2              12               6               0    #
 T9:j/jground m               4              42               0               9    #
-------------------------------------------------------------------------------------
           T0 q              12              13              13              24    #
  T3 C25 N100 q               0               0               6               0    #
 T9:j/jground q              15              47               2              15    #
-------------------------------------------------------------------------------------
           T0 w              68              84             100              54    #
  T3 C25 N100 w              26              21              44               4    #
 T9:j/jground w             100             100              98             100    #
-------------------------------------------------------------------------------------
            100          2819.5            43.9        27175.92             4.8    #
              0          155.25            1.55          806.33            0.12    # 

osp:

Techniques         -effort         -months        -defects          -risks    #
           T0 m              30              54              38              37    #
  T3 C25 N100 m               0               8               2               0    #
    T9:j/josp m               4              47               0              11    #
-------------------------------------------------------------------------------------
           T0 q               9               5              21              29    #
  T3 C25 N100 q               0               0               4               0    #
    T9:j/josp q               4              30              10              30    #
-------------------------------------------------------------------------------------
           T0 w              54              67             100              64    #
  T3 C25 N100 w              13              16              55              10    #
    T9:j/josp w             100             100              41             100    #
-------------------------------------------------------------------------------------
            100          3063.6            41.7        14504.17             7.5    #
              0           89.69            1.47          722.41            0.26    #
 

osp2:

      Techniques         -effort         -months        -defects          -risks    #
           T0 m              49              67              11               5    #
  T3 C25 N100 m               1              12               1               0    #
   T9:j/josp2 m               2              44               0               9    #
-------------------------------------------------------------------------------------
           T0 q               9               5               2               7    #
  T3 C25 N100 q               0               0               4               2    #
   T9:j/josp2 q              16              21              18              28    #
-------------------------------------------------------------------------------------
           T0 w              90              82              34              15    #
  T3 C25 N100 w              44              23              93              17    #
   T9:j/josp2 w             100             100             100             100    #
-------------------------------------------------------------------------------------
            100          1198.0            33.4          7610.0             5.1    #
              0          112.45            1.57           637.4            0.21    #

all:

Techniques         -effort         -months        -defects          -risks    #
           T0 m              20              54              11              21    #
  T3 C25 N100 m               0              12               5               0    #
    T9:j/jall m               2              42               0              14    #
-------------------------------------------------------------------------------------
           T0 q              20              18              19              35    #
  T3 C25 N100 q               0               0               7               2    #
    T9:j/jall q               8              26               4              40    #
-------------------------------------------------------------------------------------
           T0 w             100             100             100             100    #
  T3 C25 N100 w              42              24              64              10    #
    T9:j/jall w              31              89              27              94    #
-------------------------------------------------------------------------------------
            100          2645.2           44.82        22300.72            5.08    #
              0          225.33            2.24         1028.96            0.12    #

99999 Cohen <=> Bootstrap

Bootstrap:

 Techniques         -effort         -months        -defects          -risks    #
           T0 m              27              67              12               4    #
  T3 C25 N100 m               0              10               0               0    #
    T9:j/jall m               4              46               1              16    #
-------------------------------------------------------------------------------------
           T0 q              24              16              18              10    #
  T3 C25 N100 q               0               0               2               1    #
    T9:j/jall q              10              29               6              43    #
-------------------------------------------------------------------------------------
           T0 w             100             100             100              35    #
  T3 C25 N100 w              31              21              47               7    #
    T9:j/jall w              32              97              35             100    #
-------------------------------------------------------------------------------------
            100         2687.38           41.35        17612.38             4.8    #
              0          172.58            1.92          892.87            0.04    #

Cohen:

  Techniques         -effort         -months        -defects          -risks    #
           T0 m              27              67              11               4    #
  T3 C25 N100 m               0              10               0               0    #
    T9:j/jall m               3              46               0              16    #
-------------------------------------------------------------------------------------
           T0 q              24              16              18              10    #
  T3 C25 N100 q               1               0               2               1    #
    T9:j/jall q               9              28               5              43    #
-------------------------------------------------------------------------------------
           T0 w             100             100             100              35    #
  T3 C25 N100 w              27              21              51               8    #
    T9:j/jall w              32              97              34             100    #
-------------------------------------------------------------------------------------
            100         2687.38           41.35        17612.38             4.8    #
              0          185.14            2.08          989.83            0.04    #
 

Runtimes:

Bootstrap:

real 1m51.292s
user 1m44.647s
sys 0m6.395s 

Cohen:

real 0m56.032s
user 0m52.087s
sys 0m3.737s
 

Summary

-555--Rounding dataset into integers makes more sense as it would be similar to the daataset we receive. Results indicate minor changes.

-777--Trees as indicated are very big. Working on axe to get it down.

-888--Results for other models are similar to flight. a little better.

-999--Swapping cohens to bootstrap gave little change to output. but runtime is reduced by half.

Results of contrast set learning techniques

Results of contrast set learning techniques 

What was done using different techniques:

1. Cluster jplflight(-->C1) with xy_proj.py -->C2

2. Build decision trees using xy_dt.py

3. Use diff.py to get decisions(contrast sets) to be made for worse cluster to be better cluster.

4. Using the contrast sets generate 500 samples with gen.py. (used xomo)-->C3

5. Compare initial clusters to newly generated data.

6. Represent results as in fig 9 of http://menzies.us/pdf/12gense.pdf .

Techniques:
T0: asIs
T2 =C1+C3 
T3 = C2+C3


Flight data

 Techniques         -effort         -months        -defects          -risks    #
           T0 m              43              74              14               9    #
           T2 m               0               3               0               1    #
           T3 m               0               4               0               0    #
           T0 q              32              17              21              26    #
           T2 q               0               0               1               2    #
           T3 q               0               0               1               2    #
           T0 w             100             100             100             100    #
           T2 w               2               7              25              15    #
           T3 w               2               7              21              13    #
            100         30166.1            88.6         27118.6             1.8    #
              0          9598.8            16.4          4340.4             0.2    #

Ground data

  Techniques         -effort         -months        -defects          -risks    #
           T0 m              43              74              15              10    #
           T2 m               0               3               0               1    #
           T3 m               0               3               0               0    #
           T0 q              32              17              21              27    #
           T2 q               0               0               1               3    #
           T3 q               0               0               1               3    #
           T0 w             100             100             100             100    #
           T2 w               1               7              18              17    #
           T3 w               1               6              16              15    #
            100         30166.1            88.6         27118.6             1.8    #
              0          9598.8            16.4          4340.4             0.2    #

Osp data

 Techniques         -effort         -months        -defects          -risks    #
           T0 m              43              74              12               0    #
           T2 m               0               4               0              12    #
           T3 m               0               4               0              12    #
           T0 q              32              18              19              19    #
           T2 q               0               0               1               9    #
           T3 q               0               0               0               8    #
           T0 w             100             100             100             100    #
           T2 w               1               6              24              33    #
           T3 w               1               6              20              33    #
            100         30166.1            88.6         27118.6             1.8    #
              0          9598.8            16.4          6021.0             0.2    #

Osp2 data

Techniques         -effort         -months        -defects          -risks    #
           T0 m              43              74              14               3    #
           T2 m               0               4               0               0    #
           T3 m               0               4               0               0    #
           T0 q              32              18              21              22    #
           T2 q               0               0               0               2    #
           T3 q               0               0               0               2    #
           T0 w             100             100             100             100    #
           T2 w               1               6              14              14    #
           T3 w               1               6              11              14    #
            100         30166.1            88.6         27118.6             1.8    #
              0          9598.8            16.4          6021.0             0.2    #