Thursday, September 25, 2014

New Results Format

10/30/14 - Rank Sums, NSGAII-style Selection

Random Forest

Table of Rank Sums Across All Data-sets
60  : Default Cur -> Cur
35  : Tuned Prev -> Cur
28  : Tuned Cur -> Cur
  0  : Default Prev -> Cur

Overall Rankings




Params (387 permutations)
('bootstrap', ['values', True])
('min_samples_leaf', ['values', 1])
('n_estimators', ['values', 8, 16, 32])
('min_samples_split', ['values', 2])
('criterion', ['values', 'gini'])
('max_features', ['values', 2, 4, 8, 16])
('max_depth', ['values', 2, 4, 6, 8, 10, 12, 14, 16, 18])

Bernoulli Bayes

Table of Rank Sums Across All Data-sets
29  : Tuned Cur -> Cur
28  : Tuned Prev -> Cur
21  : Default Cur -> Cur
  9  : Default Prev -> Cur

Overall Rankings




Params (50 permutations)
('binarize', ['values', 0.0, 0.2, 0.4, 0.6, 0.8])
('alpha', ['values', 0.0, 0.2, 0.4, 0.6, 0.8])
('fit_prior', ['values', True, False])

Logistic Regression

Table of Rank Sums Across All Data-sets
31  : Default Cur -> Cur
27  : Tuned Cur -> Cur
24  : Tuned Prev -> Cur
  5  : Default Prev -> Cur

Overall Rankings




Params (36 permutations)
('penalty', ['values', 'l1', 'l2'])
('C', ['values', 0.5, 1, 2])
('class_weight', ['values', None])
('intercept_scaling', ['values', 0.5, 1, 2])
('fit_intercept', ['values', True, False])


10/16/14 - Summary results, 3 frontiers, and F-measure rankings.

The rig used for these results is different (simplified) in the following ways:
-RF only
-One parameter (maxDepth) swept from 0 to 16
-All other parameters fixed
-Three ND frontiers are returned from tuning instead of one.
-Due to multiple frontiers, pD/pF AUC is out. F-measure is used as ranking field instead.



New Results Format

Link to the results: (I think landscape is easiest to see)
https://www.dropbox.com/sh/iowdac5ki9hyyw4/AAAzRAdOf0pI571udSY0qf-ya?dl=0

In this format, the top item is a comparison of the prev and current version dataset stats. These are the usual suspects plus "overlapping instances" where the software module name is the same in both versions and "identical instances" where the software module's metrics are unchanged from one version to the next.

The rest of the chart shows the results of parameter tuning on both the previous and current versions. There is a table for each learner which lists all of its explored parameter values and the frequency with which they were selected by the grid search in both the previous and current versions. For example, a parameter value of "False" appearing in 90% of the selected combinations in the previous version and 43% of the current version selections would be represented as "False: (90/43)".

It also shows the pD/pF performance of each learner's non-dominated turnings applied in-set and out of set. in this case we have four combinations:

  • tune on prev -> apply in-version
  • tune on prev -> apply out-of-version (current)
  • tune on current -> apply in-version
  • tune on current -> apply out-of-version (prev)
In this case, the major effect that we see is that the green sticks with the blue and the red sticks with the purple. This scenario arises when one dataset is more difficult to preform well on than the other. Beyond that, the performance in-version and out-of version seem pretty comparable. There are occasional exceptions, but not a real trend towards in-version or out-of-version doing better.

Tuesday, September 23, 2014

1000 bootstrap

XOMOflight

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomofl500 m                   14                   50                    8                    6    #
     CT0_xomofl500 m                   11                   44                    4                    7    #
     CT1_xomofl500 m                    7                   24                    4                    4    #
    NSGA xomofl500 m                    0                   21                    0                    0    #
-------------------------------------------------------------------------------------
Base Line xomofl500 q                   13                    8                   10                   14    #
     CT0_xomofl500 q                   12                    7                    6                   13    #
     CT1_xomofl500 q                    8                    0                    5                   11    #
    NSGA xomofl500 q                    1                    2                    0                   13    #
-------------------------------------------------------------------------------------
Base Line xomofl500 w                   50                   79                   38                   43    #
     CT0_xomofl500 w                  100                  100                  100                   60    #
     CT1_xomofl500 w                   49                   52                   47                   36    #
    NSGA xomofl500 w                    6                   55                    3                  100    #
-------------------------------------------------------------------------------------
                 100              6505.94                 56.0             72279.91                  4.0    #
                   0                 81.3                 5.79               259.29                  0.0    #


XOMOground

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomogr500 m                   11                   48                    5                   14    #
     CT0_xomogr500 m                    9                   43                    3                   14    #
     CT1_xomogr500 m                    5                   21                    2                    5    #
    NSGA xomogr500 m                    0                   23                    0                    0    #
-------------------------------------------------------------------------------------
Base Line xomogr500 q                   10                    8                    6                   20    #
     CT0_xomogr500 q                   10                    8                    5                   20    #
     CT1_xomogr500 q                    5                    0                    3                    4    #
    NSGA xomogr500 q                    1                    5                    0                    7    #
-------------------------------------------------------------------------------------
Base Line xomogr500 w                   39                   74                   24                   62    #
     CT0_xomogr500 w                  100                  100                  100                  100    #
     CT1_xomogr500 w                   34                   44                   24                   47    #
    NSGA xomogr500 w                   13                   69                    1                   92    #
-------------------------------------------------------------------------------------
                 100              5260.07                51.62            115072.54                 4.29    #
                   0                61.16                  4.5               174.68                 0.54    #


XOMO OSP

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomoos500 m                   29                   65                   17                   46    #
     CT0_xomoos500 m                   26                   63                   17                    8    #
     CT1_xomoos500 m                   12                   26                    7                   16    #
    NSGA xomoos500 m                    0                   28                    0                    0    #
-------------------------------------------------------------------------------------
Base Line xomoos500 q                    9                    6                    8                   33    #
     CT0_xomoos500 q                    8                    4                   11                   11    #
     CT1_xomoos500 q                    3                    0                    4                   13    #
    NSGA xomoos500 q                    1                    4                    0                   12    #
-------------------------------------------------------------------------------------
Base Line xomoos500 w                   54                   82                   37                  100    #
     CT0_xomoos500 w                  100                  100                  100                   32    #
     CT1_xomoos500 w                   31                   38                   26                   23    #
    NSGA xomoos500 w                    9                   77                    8                   93    #
-------------------------------------------------------------------------------------
                 100              3360.27                37.54             32578.89                 5.95    #
                   0                67.25                 1.64               191.75                 0.67    #


XOMO OSP2

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomoo2500 m                   57                   81                   35                    4    #
     CT0_xomoo2500 m                   27                   41                   17                    0    #
     CT1_xomoo2500 m                   40                   56                   34                    9    #
    NSGA xomoo2500 m                    0                   38                    0                   21    #
-------------------------------------------------------------------------------------
Base Line xomoo2500 q                   15                    5                   11                   11    #
     CT0_xomoo2500 q                    7                    0                    9                    7    #
     CT1_xomoo2500 q                   11                    2                   13                    9    #
    NSGA xomoo2500 q                    0                   27                    1                   32    #
-------------------------------------------------------------------------------------
Base Line xomoo2500 w                  100                  100                   75                   14    #
     CT0_xomoo2500 w                   74                   62                   93                    7    #
     CT1_xomoo2500 w                   94                   80                  100                   10    #
    NSGA xomoo2500 w                   19                   70                   19                  100    #
-------------------------------------------------------------------------------------
                 100              1185.47                29.91              3115.06                  7.0    #
                   0                 59.1                 2.31                165.2                  0.2    #


XOMO AL

         Techniques              -effort              -months             -defects               -risks    #
Base Line xomoal500 m                   10                   33                    7                   12    #
     CT0_xomoal500 m                    7                   28                    3                   12    #
     CT1_xomoal500 m                    3                    9                    5                    0    #
    NSGA xomoal500 m                    0                   10                    0                    5    #
-------------------------------------------------------------------------------------
Base Line xomoal500 q                   10                   10                   10                   16    #
     CT0_xomoal500 q                    8                    9                    5                   17    #
     CT1_xomoal500 q                    4                    0                    7                    3    #
    NSGA xomoal500 q                    0                    2                    0                   22    #
-------------------------------------------------------------------------------------
Base Line xomoal500 w                   41                   67                   45                   56    #
     CT0_xomoal500 w                  100                  100                  100                  100    #
     CT1_xomoal500 w                   31                   35                   84                   29    #
    NSGA xomoal500 w                    3                   34                    2                   53    #
-------------------------------------------------------------------------------------
                 100              7691.63                72.55              82741.7                10.49    #
                   0                47.89                 4.77               164.88                 0.74    #


POM3A
          Techniques                -cost          +completion                -idle    #
Base Line pom3A500 m                   42                   89                   29    #
      CT0_pom3A500 m                   35                   69                   46    #
      CT1_pom3A500 m                   19                   41                   14    #
     NSGA pom3A500 m                    4                   86                   28    #
-------------------------------------------------------------------------------------
Base Line pom3A500 q                   28                   13                   39    #
      CT0_pom3A500 q                    0                    0                    0    #
      CT1_pom3A500 q                   12                    5                   20    #
     NSGA pom3A500 q                    2                   19                   50    #
-------------------------------------------------------------------------------------
Base Line pom3A500 w                  100                  100                   76    #
      CT0_pom3A500 w                   81                   87                   49    #
      CT1_pom3A500 w                   49                   47                   43    #
     NSGA pom3A500 w                    9                   96                  100    #
-------------------------------------------------------------------------------------
                 100              1672.42                 1.04                  0.8    #
                   0                  0.0                  0.0                  0.0    #


POM3B
         Techniques                -cost          +completion                -idle    #
Base Line pom3B500 m                   34                   87                   35    #
      CT0_pom3B500 m                   39                   80                    6    #
      CT1_pom3B500 m                   32                   83                   34    #
     NSGA pom3B500 m                    6                   85                   20    #
-------------------------------------------------------------------------------------
Base Line pom3B500 q                   28                   13                   41    #
      CT0_pom3B500 q                    0                    0                    0    #
      CT1_pom3B500 q                   27                   12                   39    #
     NSGA pom3B500 q                    2                    9                   37    #
-------------------------------------------------------------------------------------
Base Line pom3B500 w                  100                  100                   77    #
      CT0_pom3B500 w                   50                   78                   27    #
      CT1_pom3B500 w                   97                   96                   75    #
     NSGA pom3B500 w                   14                   96                  100    #
-------------------------------------------------------------------------------------
                 100             22987.74                 1.04                  0.8    #
                   0               117.64                 0.01                 0.01    #




POM3C
         Techniques                -cost          +completion                -idle    #
Base Line pom3B500 m                   34                   87                   35    #
      CT0_pom3B500 m                   39                   80                    6    #
      CT1_pom3B500 m                   32                   83                   34    #
     NSGA pom3B500 m                    6                   85                   20    #
-------------------------------------------------------------------------------------
Base Line pom3B500 q                   28                   13                   41    #
      CT0_pom3B500 q                    0                    0                    0    #
      CT1_pom3B500 q                   27                   12                   39    #
     NSGA pom3B500 q                    2                    9                   37    #
-------------------------------------------------------------------------------------
Base Line pom3B500 w                  100                  100                   77    #
      CT0_pom3B500 w                   50                   78                   27    #
      CT1_pom3B500 w                   97                   96                   75    #
     NSGA pom3B500 w                   14                   96                  100    #
-------------------------------------------------------------------------------------
                 100             22987.74                 1.04                  0.8    #
                   0               117.64                 0.01                 0.01    #

250 bootstrap results

250


FLIGHT

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomofl500 m                   27                   67                   18                    6    #
     CT0_xomofl500 m                    2                    7                    1                    0    #
     CT1_xomofl500 m                   14                   37                    8                    4    #
    NSGA xomofl500 m                    0                   33                    0                    0    #
-------------------------------------------------------------------------------------
Base Line xomofl500 q                   26                   18                   25                   14    #
     CT0_xomofl500 q                    2                    0                    2                    2    #
     CT1_xomofl500 q                   15                    9                   11                   10    #
    NSGA xomofl500 q                    2                   12                    1                   13    #
-------------------------------------------------------------------------------------
Base Line xomofl500 w                   97                  100                   90                   43    #
     CT0_xomofl500 w                   24                   17                   30                   10    #
     CT1_xomofl500 w                  100                   70                  100                   37    #
    NSGA xomofl500 w                   11                   72                    7                  100    #
-------------------------------------------------------------------------------------
                 100              3408.55                45.35             30775.62                  4.0    #
                   0                 81.3                 1.68               259.29                  0.0    #

Ground:

          Techniques              -effort              -months             -defects               -risks    #
Base Line xomogr500 m                   28                   66                   20                   19    #
     CT0_xomogr500 m                    7                   16                    4                    0    #
     CT1_xomogr500 m                   13                   29                    7                    7    #
    NSGA xomogr500 m                    0                   34                    0                    4    #
-------------------------------------------------------------------------------------
Base Line xomogr500 q                   27                   15                   27                   25    #
     CT0_xomogr500 q                    7                    0                    6                    2    #
     CT1_xomogr500 q                   14                    3                    9                    6    #
    NSGA xomogr500 q                    1                   11                    1                   12    #
-------------------------------------------------------------------------------------
Base Line xomogr500 w                  100                  100                  100                   69    #
     CT0_xomogr500 w                   75                   41                   97                   31    #
     CT1_xomogr500 w                   89                   57                   93                   44    #
    NSGA xomogr500 w                   34                   94                    3                  100    #
-------------------------------------------------------------------------------------
                 100              2065.56                39.28             27430.38                  4.0    #
                   0                61.16                 2.97               174.68                 0.38    #


OSp


          Techniques              -effort              -months             -defects               -risks    #
Base Line xomoos500 m                   35                   65                   16                   46    #
     CT0_xomoos500 m                   33                   62                   14                   33    #
     CT1_xomoos500 m                   16                   27                    9                   20    #
    NSGA xomoos500 m                    0                   28                    0                    0    #
-------------------------------------------------------------------------------------
Base Line xomoos500 q                   11                    6                    8                   33    #
     CT0_xomoos500 q                   12                    6                   11                   42    #
     CT1_xomoos500 q                    4                    0                    4                   14    #
    NSGA xomoos500 q                    1                    4                    0                   12    #
-------------------------------------------------------------------------------------
Base Line xomoos500 w                   65                   82                   35                  100    #
     CT0_xomoos500 w                  100                  100                  100                   99    #
     CT1_xomoos500 w                   40                   39                   27                   26    #
    NSGA xomoos500 w                   11                   77                    7                   93    #
-------------------------------------------------------------------------------------
                 100              2798.94                37.57             34928.22                 5.95    #
                   0                67.25                 1.66               191.75                 0.67    #


osp2



          Techniques              -effort              -months             -defects               -risks    #
Base Line xomoo2500 m                   58                   82                   48                    7    #
     CT0_xomoo2500 m                    6                   11                    3                    0    #
     CT1_xomoo2500 m                   25                   35                   29                    5    #
    NSGA xomoo2500 m                    2                   41                    3                   23    #
-------------------------------------------------------------------------------------
Base Line xomoo2500 q                   16                   10                   17                   13    #
     CT0_xomoo2500 q                    0                    0                    0                    3    #
     CT1_xomoo2500 q                    6                    3                   12                    4    #
    NSGA xomoo2500 q                    2                   31                    3                   34    #
-------------------------------------------------------------------------------------
Base Line xomoo2500 w                  100                  100                  100                   16    #
     CT0_xomoo2500 w                   19                   17                   25                    3    #
     CT1_xomoo2500 w                   55                   46                   88                    5    #
    NSGA xomoo2500 w                   20                   72                   27                  100    #
-------------------------------------------------------------------------------------
                 100              1185.47                29.91               2385.2                  7.0    #
                   0                 39.1                 0.65               103.31                  0.0    #


ALL


          Techniques              -effort              -months             -defects               -risks    #
Base Line xomoal500 m                   24                   52                   11                   29    #
     CT0_xomoal500 m                    0                    3                    0                    0    #
     CT1_xomoal500 m                   11                   25                    6                   11    #
    NSGA xomoal500 m                    0                   21                    0                   17    #
-------------------------------------------------------------------------------------
Base Line xomoal500 q                   23                   21                   15                   35    #
     CT0_xomoal500 q                    1                    0                    1                    1    #
     CT1_xomoal500 q                   12                   10                    8                   19    #
    NSGA xomoal500 q                    0                   11                    1                   46    #
-------------------------------------------------------------------------------------
Base Line xomoal500 w                  100                  100                   69                  100    #
     CT0_xomoal500 w                   19                   11                   10                   12    #
     CT1_xomoal500 w                   97                   67                  100                   79    #
    NSGA xomoal500 w                    7                   54                    3                   96    #
-------------------------------------------------------------------------------------
                 100              3151.95                50.35             54140.25                 6.17    #
                   0                47.89                 1.07               164.88                 0.17    #


POM3A


          Techniques                -cost          +completion                -idle    #
Base Line pom3A500 m                   24                   89                   15    #
      CT0_pom3A500 m                   20                   85                   13    #
      CT1_pom3A500 m                    7                   42                    0    #
     NSGA pom3A500 m                    1                   85                   13    #
-------------------------------------------------------------------------------------
Base Line pom3A500 q                   15                    6                   27    #
      CT0_pom3A500 q                   19                    3                   25    #
      CT1_pom3A500 q                    4                    0                   12    #
     NSGA pom3A500 q                    0                   13                   40    #
-------------------------------------------------------------------------------------
Base Line pom3A500 w                   58                  100                   72    #
      CT0_pom3A500 w                  100                   96                   84    #
      CT1_pom3A500 w                   22                   48                   40    #
     NSGA pom3A500 w                    4                   96                  100    #
-------------------------------------------------------------------------------------
                 100              2873.72                 1.04                  0.8    #
                   0                 37.6                 0.07                 0.13    #


POM3B


          Techniques                -cost          +completion                -idle    #
Base Line pom3B500 m                   33                   87                   34    #
      CT0_pom3B500 m                    1                    9                    0    #
      CT1_pom3B500 m                   16                   50                   16    #
     NSGA pom3B500 m                    4                   85                   18    #
-------------------------------------------------------------------------------------
Base Line pom3B500 q                   26                   13                   39    #
      CT0_pom3B500 q                    1                    0                    1    #
      CT1_pom3B500 q                   15                    5                   26    #
     NSGA pom3B500 q                    0                    9                   35    #
-------------------------------------------------------------------------------------
Base Line pom3B500 w                  100                  100                   77    #
      CT0_pom3B500 w                   13                   10                    8    #
      CT1_pom3B500 w                   70                   56                   52    #
     NSGA pom3B500 w                   12                   96                  100    #
-------------------------------------------------------------------------------------
                 100             22987.74                 1.04                  0.8    #
                   0                494.8                 0.01                 0.03    #


POM3C


          Techniques                -cost          +completion                -idle    #
Base Line pom3C500 m                   33                   90                   32    #
      CT0_pom3C500 m                   30                   86                   32    #
      CT1_pom3C500 m                   19                   70                   28    #
     NSGA pom3C500 m                    8                   87                   37    #
-------------------------------------------------------------------------------------
Base Line pom3C500 q                   16                    1                   28    #
      CT0_pom3C500 q                   20                   11                   33    #
      CT1_pom3C500 q                    9                    3                   25    #
     NSGA pom3C500 q                    0                    0                    0    #
-------------------------------------------------------------------------------------
Base Line pom3C500 w                   74                  100                   88    #
      CT0_pom3C500 w                  100                   96                  100    #
      CT1_pom3C500 w                   55                   78                   80    #
     NSGA pom3C500 w                   16                   96                   83    #
-------------------------------------------------------------------------------------
                 100              2613.81                 1.04                  0.7    #
                   0                 47.7                  0.1                  0.1    #