ai @ wvu

Concept Lattice Generation

2012-01-24T05:49:00.000-08:00

Using a tool called Lattice Miner (written in Java), I was able to generate the following lattices.

NOTE: This lattice was generated using only PD, PF, and PREC as objectives (meaning 3 dimensions). This generated 25 rules. Using all 6 dimensions I have been using (number of bugs found, lines of code, bugs/loc), I was getting 70 rules, which made the lattice chart very large and unwieldy, and therefore not ideal for the presentation:

Limitations: it doesn't allow labelling of edges, only of points.
This tool doesn't have a summary tool (to average the PDs, PFs, etc) that I have found yet.
No command line interface that I have found yet.

Benefits: File structure makes it very easy to generate lattice grid files on the fly
Very quick
Visually appealing

Worth noting: Lattices higher on the tree (towards the top) tend to have higher PD, higher PF, lower precision
As you move down the tree (towards more complicated rules), you start to see lower PF and lower precision, but also lower PD

Repairing feature models

2012-01-24T05:38:00.000-08:00

From http://www.splot-research.org/

INPUT:

   <feature_model name="My feature model">     <-- feature model start tag and name attribute (mandatory)
  <meta>                                                                  <-- Optional
  <data name="description">Model description</data>                 <-- Optional
  <data name="creator">Model's creator</data>                       <-- Optional
  <data name="email">Model creator's email</data>                   <-- Optional
  <data name="date">Model creation date</data>                      <-- Optional
  <data name="department">Model creator's department</data>         <-- Optional
  <data name="organization">Model creator's organization</data>     <-- Optional
  <data name="address">Model creator's address</data>               <-- Optional
  <data name="phone">Model creator's phone</data>                   <-- Optional
  <data name="website">Model creator's website</data>               <-- Optional
  <data name="reference">Model's related publication</data>         <-- Optional
  </meta>
  <feature_tree>                <-- feature tree start tag
    :r root (root_id)                 <-- root feature named 'root' with unique ID 'root_id'         
      :o opt1 (id_opt1)               <-- an optional feature named opt1 with unique id id_opt1
      :o opt2 (id_opt2)               <-- an optional feature named opt2, child of opt1 with unique id id_opt2
      :m man1                         <-- an mandatory feature named man1 with unique id id_man1
        :g [1,*]                      <-- an inclusive-OR feature group with cardinality [1..*] ([1..3] also allowed)
          : a (id_a)                  <-- a grouped feature name a with ID id_a
          : b (id_b)                  <-- a grouped feature name b with ID id_b
            :o opt3 (id_opt3)         <-- an optional feature opt3 child of b with unique id id_opt3
          : c (id_c)                  <-- a grouped feature name c with ID id_c
        :g [1,1]                      <-- an exclusive-OR feature group with cardinality [1..1]
          : d (id_d)                  <-- a grouped feature name d with ID id_d
          : e (id_e)                  <-- a grouped feature name e with ID id_e
            :g [2,3]                      <-- a feature group with cardinality [2..3] children of feature e
              : f (id_f)                  <-- a grouped feature name f with ID id_f
              : g (id_g)                  <-- a grouped feature name g with ID id_g
              : h (id_h)                  <-- a grouped feature name h with ID id_h
  </feature_tree>               <-- feature tree end tag (mandatory)
  <constraints>                 <-- extra constraints start tag (mandatory)
    c1: ~id_a or id_opt2        <-- extra constraint named c1: id_a implies id_opt2 (must be a CNF clause)
    c2: ~id_c or ~id_e          <-- extra constraint named c2: id_c excludes id_e (must be a CNF clause)
  </constraints>                <-- extra constraint end tag (mandatory)
</feature_model>                <-- feature model end tag  (mandatory)

Important:

Tags are optional but encouraged
Feature tree must have at least one feature, i.e., the root node
Extra constraints are optional
Each extra constraint must be a CNF clause (any arity)

State of the art in checking these models:

A Performance Comparison of Contemporary Algorithmic Approaches for Automated Analysis Operations on Feature Models
Richard Pohl, Kim Lauenroth and Klaus Pohl. IEEE ASE 2011

Exponential runtimes

also; just says "no" and maybe offers one counter example

here, a linear-time greedy bayes search (think simpler than W2) generates this in 6 seconds in awk from 10,000 random models from a 300+ class system

Output:

                                                  MEDIAN      N    treatment
                                                  ------    ---    ---------
[                                ------ | ------   ] 80 | 10000    all rows
[                                     ----|  ---   ] 85 |  4945    _id_35 = 1
[                                       ---  |--   ] 90 |  2521    _id_32 = 1
[                                       ---  |--   ] 90 |  2213    _id_68 = 0
[                                       ---  |--   ] 90 |  1903    _id_63 = 0
[                                       ---  |--   ] 90 |  1427    _id_77 = 0
[                                       ---  |--   ] 90 |   711    _id_15 = 1
[                                       ---  | -   ] 90 |   355    _id_17 = 1
[                                       ---  | -   ] 90 |   170    _id_22 = 1
[                                       ---  | -   ] 90 |    86    _id_86 = 0
[                                       ---  |--   ] 90 |    47    _id_24 = 1
[                                       -----| ----] 90 |    20    _id_3 = 0

What's fun here is that it is telling us what to do and what not to do.

CHALK Weekly Progress

2012-01-24T04:41:00.000-08:00

Erin's weekly report

Idea's MRE 0, 25, 50, 75 and 100th Percentile Results

2012-01-24T03:33:00.000-08:00

Here are Idea's MRE results for 19 datasets since I'm missing sdr, I also added them to Ekrems MRE .CSV file results. But still working on ranking them against Ekrems results. Installed node.js and Node Packet Manager as well as coffeescript. Translated bootstrap.coffee to javascript but still haven't managed to understand it fully. Haven't been able to run bootstrap.coffee either. Looked over wilcoxonTest.awx, mwuTest.awk as well but no success.

Intrinsic Complexity of SEE

2012-01-23T21:34:00.000-08:00

How complex should a learner working on SEE data sets be? Our intuition is that there is a need for complex learners, if the data set has enough complexity to it, i.e. if the data set can be divided into complex and self-contained regions. On the other hand, if most of the data is noise and if the data sets can be summarized with only a small number of instances and features, then there is no need for complex learners. The updated version of the QUICK paper investigates this question. The new version with all the results/figures updated is here.

Preview of the Paper ["First" Draft]

2012-01-23T19:49:00.000-08:00

Conference: Games and Software Engineering 2012.

Location: Zurich Switzerland. (Part of ICSE 2012.)

Due Date: Feb. 17th

Paper Link

2012-01-17T07:05:00.000-08:00

What I read this week

CHALK proposal

Li H, Homer N. A survey of sequence alignment algorithms for next-generation sequencing. Brief Bioinform 2010;11:473-483.

- hash tables, spaced seed, q-gram filter, multiple seed hits, suffix/prefix tries, seed extension, suffix trie, FM-index, inexact matches, gapped alignment,

- Aligning sequence reads: gapped alignment, paired-end and mate-pair mapping, base quality, long-read aligner, capillary read aligner, SOLiD reads, bisulfite-treated reads, spliced reads, realignment

- Speed, memory considerations

Revital Eres, Gad M Landau, Laxmi Parida. Permutation pattern discovery in biosequences.Journal of computational biology a journal of computational molecular cell biology 2004 11 (6) p. 1050-1060

- sliding window technique for pattern matching with examples

Smith TF, Waterman MS. Indentification of common molecular subsequences. J Mol Biol 1981;147:195-7.

- Similarity (homology) measure

Smith TF, Waterman MS. Comparison of Biosequences. Advances in Applied Mathematics 1981;489:482-489.

- More detailed algorithm for the Smith-Waterman homology measure, comparison to Sellers and Needleman and Wunsch algorithms

MacIsaac KD, Fraenkel E (2006) Practical strategies for discovering regulatory DNA sequence motifs. PLoS Comput Biol 2(4): e36. DOI: 10.1371/journal.pcbi0020036

- DNA Encoding Schemes with examples

o consensus sequence of preferred nucleotides (ACGT)

o position weight matrix (PWM)

o example: seq to pwm 6 positions

- Clustering of DNA - techniques, dimensionality

o k-medroids

o SOM,

o hierarchical clustering to the motifs and combined clusters with a similarity exceeding 70% by computing a consensus sequence

- Distance / similarity measure for DNA sequences

o fraction of common bits as a similarity metric

o Pearson correlation coefficient between motif PWMs as the similarity measure

Next week:

I need to understand basics of DNA, Computational Chemistry, and Bioinformatics better. I have a few book chapters downloaded from the WVU libraries electronic collection.

Core Games and Casual Games

2012-01-17T05:37:00.000-08:00

No person is entirely a casual or core gamer, nor can you define them as “50% casual, 50% core.” What about games?

Some people have mixed opinions about the nature of the games they love to play. Survey on FlyFF: http://www.esurveyspro.com/Survey.aspx?id=92c00c57-ee20-4047-9ed4-100fae8b7644

From that, perhaps we can assign replayability-aspect importance to different core-casual scores.

http://www.grabilla.com/02111-22a4d3b4-88fc-496f-be33-a683f61d40a9.png

Frustration with concept Lattices

2012-01-17T04:02:00.000-08:00

I've been trying to implement this:
http://www.gbspublisher.com/ijtacs/1010.pdf

Unfortunately, I just haven't been able to so far. Hopefully have something by the meeting.

First week of research

2012-01-17T01:17:00.000-08:00

First week of research

View more presentations from Jarilyn Hernandez

Virtual Machine and You: A match made in coding heaven

2012-01-10T08:55:00.001-08:00

Here's the link to set up Ubuntu Virtual Machine, you crazy coders you:

http://menzies.us/cs472/

From here, go to Projects->Homework 1

E-mail me if you have questions.

NSGA-II

2012-01-09T13:34:00.000-08:00

Original takes about 900 ms.

Clustered takes about 1100 ms.

Measurements of Stability

2011-12-16T07:48:00.000-08:00

unsupervised learning results

2011-12-05T20:33:00.001-08:00

nasa93

> ,kloc,0
> ,acap,0.22927988981346761332
> ,data,0.2334438373917348819
> ,time,0.25423355037952316549
> ,plex,0.26866055225810964169
> ,stor,0.27441589382858594393
> ,pmat,0.3093114132807298633
> ,cplx,0.34470954768369121979
> ,apex,0.35198812495229314656
> ,pcap,0.36553252022821636213
> ,rely,0.40329996377433946497
> ,ltex,0.46603801062070232542
> ,pvol,0.56369464132541236001
> ,sced,0.58780717952192207409
> ,tool,0.82277351898917538975
> ,docu,1
> ,flex,1
> ,pcon,1
> ,prec,1
> ,resl,1
> ,ruse,1
> ,site,1
> ,team,1

coc81

> ,tool,0
> ,acap,1
> ,aexp,1
> ,cplx,1
> ,data,1
> ,dev_mode,1
> ,lexp,1
> ,loc,1
> ,modp,1
> ,pcap,1
> ,project_id,1
> ,rely,1
> ,sced,1
> ,stor,1
> ,time,1
> ,turn,1
> ,vexp,1
> ,virt,1

desharnais

>> ,YearEnd,0
>> ,Adjustment,1
>> ,Entities,1
>> ,Langage,1
>> ,Length,1
>> ,ManagerExp,1
>> ,PointsAjust,1
>> ,PointsNonAdjust,1
>> ,Project,1
>> ,TeamExp,1
>> ,Transactions,1

finnish

>> ,prod,0
>> ,at,1
>> ,co,1
>> ,FP,1
>> ,hw,1

nasa93n

>> ,prec,0
>> ,flex,0.11243402160650950439
>> ,docu,0.33052121547798030132
>> ,resl,0.40212091296737206836
>> ,apex,0.46229399349528710328
>> ,data,0.46229399349528710328
>> ,acap,0.61482323785435888386
>> ,pvol,0.61482323785435888386
>> ,sced,0.61482323785435888386
>> ,stor,0.61482323785435888386
>> ,cplx,0.70398054754829086921
>> ,kloc,0.78682242656022605143
>> ,ltex,0.78682242656022605143
>> ,pcap,0.78682242656022605143
>> ,pmat,0.78682242656022605143
>> ,rely,0.78682242656022605143
>> ,ruse,0.78682242656022605143
>> ,team,0.78682242656022605143
>> ,time,0.78682242656022605143
>> ,tool,0.78682242656022605143
>> ,plex,0.87700247550119436735
>> ,site,0.87700247550119436735
>> ,pcon,1

china

>> ,Duration,0
>> ,Resource,0.18076822524622718213
>> ,Enquiry,0.70229300937753547096
>> ,Added,0.70418156239280693676
>> ,NPDR_AFP,0.71748509491228684709
>> ,AFP,0.84023872986505276916
>> ,N_effort,0.84023872986505276916
>> ,Interface,0.85325277904975838084
>> ,File,0.9226111617867434056
>> ,Input,0.94562919153491631352
>> ,Output,0.96948001254205484756
>> ,Changed,0.98391349972332420304
>> ,Deleted,1
>> ,Dev.Type,1
>> ,ID,1
>> ,NPDU_UFP,1
>> ,PDR_AFP,1
>> ,PDR_UFP,1

miyazaki94

>> ,KLOC,0
>> ,FORM,0.40638957366031353002
>> ,ESCRN,0.43471405099857385324
>> ,EFILE,0.63742030743007560556
>> ,FILE,0.74270091796517645477
>> ,EFORM,0.89605538051121036425
>> ,SCRN,1

MRE results

Just listening to the best X percent of attributes, sorted by entropy.

#data sets        cutoff a  ignore           0   25   50    75     100
----------------  ------ -  -------------   ---  ---  ----  ----   ----
data/nasa93.arff, 0.25,  5, 1, , , 10, 0, |, 0,  0.6, 0.81, 0.91,  9.99
data/nasa93.arff, 0.33,  7, 1, , , 10, 0, |, 0, 0.51, 0.77, 0.89,  7.33
data/nasa93.arff,  0.5, 11, 1, , , 10, 0, |, 0, 0.35, 0.69, 0.85, 33.08
data/nasa93.arff, 0.66, 15, 1, , , 10, 0, |, 0, 0.37, 0.61, 0.84,     7
data/nasa93.arff, 0.75, 23, 1, , , 10, 0, |, 0,  0.4, 0.63,  0.9,  12.5
data/nasa93.arff,    1, 23, 1, , , 10, 0, |, 0, 0.42, 0.63, 0.83,  7.33

data/desharnais.arff, 0.001,  1, 1, , , 10, 0, |, 0, 0.34, 0.46, 0.65, 5.92
data/desharnais.arff,  0.25, 11, 1, , , 10, 0, |, 0, 0.22, 0.37, 0.56, 3.87
data/desharnais.arff,  0.33, 11, 1, , , 10, 0, |, 0, 0.25, 0.37, 0.56, 3.87
data/desharnais.arff,   0.5, 11, 1, , , 10, 0, |, 0, 0.25, 0.37, 0.56, 3.87
data/desharnais.arff,  0.66, 11, 1, , , 10, 0, |, 0, 0.25, 0.37, 0.56, 3.87
data/desharnais.arff,  0.75, 10, 1, , , 10, 0, |, 0, 0.27, 0.48, 0.62, 4.18
data/desharnais.arff,     1, 11, 1, , , 10, 0, |, 0, 0.25, 0.37, 0.56, 3.87

data/finnish.arff, 0.001, 1, 1, , , 10, 0, |, 0.02, 0.48, 0.71, 0.81, 5.23
data/finnish.arff,  0.25, 1, 1, , , 10, 0, |, 0.05, 0.59, 0.79, 0.85, 4.77
data/finnish.arff,  0.33, 1, 1, , , 10, 0, |, 0.01, 0.58, 0.72, 0.84,  4.9
data/finnish.arff,   0.5, 5, 1, , , 10, 0, |, 0.09, 0.61, 0.76, 0.85,  9.5
data/finnish.arff,  0.66, 5, 1, , , 10, 0, |, 0.09, 0.61, 0.76, 0.85,  9.5
data/finnish.arff,  0.75, 5, 1, , , 10, 0, |, 0.09, 0.61, 0.76, 0.85,  9.5
data/finnish.arff,     1, 5, 1, , , 10, 0, |, 0.09, 0.61, 0.76, 0.85,  9.5

data/coc81.arff, 0.001,   , 1, , , 10, 0, |, 0.07, 0.63, 0.82, 0.98, 15.92
data/coc81.arff,  0.25, 18, 1, , , 10, 0, |, 0.04, 0.76, 0.93,    1, 15.25
data/coc81.arff,  0.33, 18, 1, , , 10, 0, |,    0,  0.6, 0.77, 0.93, 18.58
data/coc81.arff,   0.5, 18, 1, , , 10, 0, |, 0.04, 0.63, 0.86, 0.98,  30.5
data/coc81.arff,  0.66, 18, 1, , , 10, 0, |, 0.04, 0.62, 0.89, 0.99, 27.75
data/coc81.arff,  0.75, 18, 1, , , 10, 0, |,    0, 0.49, 0.85, 0.98, 15.92
data/coc81.arff,     1, 18, 1, , , 10, 0, |,    0, 0.63, 0.84, 0.98, 17.47

data/coc81.arff, 0.001,  1, 1, , , 10, 0, |, 0.04, 0.69,  0.9, 0.98, 14.57
data/coc81.arff,  0.25, 18, 1, , , 10, 0, |,    0, 0.47, 0.82, 0.96, 27.75
data/coc81.arff,  0.33, 18, 1, , , 10, 0, |, 0.01, 0.66, 0.86, 0.98, 15.92
data/coc81.arff,   0.5, 18, 1, , , 10, 0, |, 0.04, 0.58, 0.86, 0.98, 16.96
data/coc81.arff,  0.66, 18, 1, , , 10, 0, |, 0.05, 0.62, 0.94, 1.45, 25.86
data/coc81.arff,  0.75, 18, 1, , , 10, 0, |,    0, 0.62, 0.91, 0.96, 15.92
data/coc81.arff,     1, 18, 1, , , 10, 0, |, 0.06, 0.69, 0.88, 0.99, 15.92

data/miyazaki94.arff, 0.001, 1, 1, , , 10, 0, |,    0, 0.24, 0.45,  0.7,  6.02
data/miyazaki94.arff,  0.25, 1, 1, , , 10, 0, |,    0,  0.2, 0.46, 0.76, 14.25
data/miyazaki94.arff,  0.33, 2, 1, , , 10, 0, |, 0.03, 0.39, 0.61, 0.72, 14.25
data/miyazaki94.arff,   0.5, 3, 1, , , 10, 0, |, 0.01, 0.24, 0.43, 0.63,  4.36
data/miyazaki94.arff,  0.66, 4, 1, , , 10, 0, |,    0,  0.4, 0.61, 0.79,  3.79
data/miyazaki94.arff,  0.75, 5, 1, , , 10, 0, |, 0.02, 0.31, 0.45, 0.72,  3.57
data/miyazaki94.arff,     1, 7, 1, , , 10, 0, |,    0,  0.2, 0.63, 0.78,  5.35

data/china.arff, 0.001,  1, 1, , , 10, 0, |, 0,  0.5, 0.76, 0.94, 93.48
data/china.arff,  0.25,  5, 1, , , 10, 0, |, 0, 0.14, 0.27, 0.45, 14.67
data/china.arff,  0.33,  5, 1, , , 10, 0, |, 0, 0.44, 0.66, 0.83, 46.84
data/china.arff,   0.5,  9, 1, , , 10, 0, |, 0, 0.21, 0.37, 0.58,  8.46
data/china.arff,  0.66, 13, 1, , , 10, 0, |, 0, 0.24, 0.38, 0.59, 13.63
data/china.arff,  0.75, 18, 1, , , 10, 0, |, 0, 0.38, 0.58, 0.76, 12.08
data/china.arff,     1, 18, 1, , , 10, 0, |, 0, 0.37, 0.57, 0.75, 19.35

2011-12-05T13:48:00.000-08:00

Comparison of main results from surveys on "all board games together" versus just a single board game: The Settlers of Catan. Social and Impact seem most similar.

Tuning a learner

2011-11-29T06:29:00.001-08:00

See http://unbox.org/stuff/var/timm/11/idea/data/ideasMAR_fss.txt for what happens when I run and http://unbox.org/stuff/var/timm/11/idea/html/idea.html to generate http://unbox.org/stuff/var/timm/11/idea/data/ideasMAR.csv, then summarize with http://unbox.org/stuff/var/timm/11/idea/ideas.sh through feature selection

Active Learning for Defect Data

2011-11-28T20:36:00.001-08:00

Here is a nice report about the active learning experiments.

Replayability Surveys

2011-11-14T19:17:00.001-08:00

Survey results for a few types of games:

MMO RPG Game (FlyFF):

Board games:

2011-11-14T13:59:00.001-08:00

1. Bias Variance Analysis
a) Anova -> Bias is 8/20, Variance is 14/20
b) Scott&Anova -> Bias is 8/20, Variance is 15/20
c) Anova plus Cohen's correction: Bias is 8/20
d) Kruskal-Wallis not so good: Bias is 4/20 and variance is 10/20

e) Sorted bias values of all the data sets plotted are here.
The above plots showed us that there are extreme values due to new experimental conditions.
f) The Friedman-Nemenyi plots are here for separate data sets.
g) The same test as before for the entire 20 data sets together are here.
h) Re-generate boxplots with the new experimental conditions.

I applied a correction on the analysis, because...
Here is the paper with new boxplots..
Here is the Friedman Nimenyi plots on separate data sets, bias is the same for 15 out of 20 data sets..
Below is the result of Friedman Nimenyi over all data sets, i.e. if we see it as 20 repetitions.

2. Papers
a) HK paper is revised and is ready to be seen here.
b) Active learning paper's major revision is submitted to TSE.
c) Camera ready versions of ensemble and kernel papers are submitted to TSE and ESEM.

3. The break-point experiments

Clustering Accelerates NSGA-II

2011-11-14T12:25:00.000-08:00

Using a natural clustering method, and replacing poor solutions with solutions bred within "interesting" clusters, appears to accelerate how quickly NSGA-II converges upon the Pareto frontier with little impact on the diversity of the solutions or the speed of the algorithm.

Clusters are divided by heuristically and recursively dividing space along a natural axis for the individuals. The FastMap heuristic is used to pick individuals that are very distant from each other by their independent attributes. All the other individuals are placed in the coordinate space by the magnitude of their orthogonal projections onto the line between the two extreme individuals. A division point is chosen along this axis that minimizes the weighted variance of the individuals' ranks. The splitting continues on each half of the split down to "sqrt(individuals)".

An "interesting" cluster is one that has at least one first-rank individual within it and a low ratio of first-rank individuals to total individuals in the cluster. A "boring" cluster is either completely saturated with first-rank individuals or lower-ranked individuals.

Raw data from hall et al

2011-11-01T06:31:00.000-07:00

https://bugcatcher.stca.herts.ac.uk/slr2011/

A Systematic Review of Fault Prediction Performance in Software Engineering

http://www.computer.org/portal/web/csdl/abs/trans/ts/5555/01/tts555501toc.htm

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6035727&tag=1

stats methids

2011-11-01T06:26:00.000-07:00

divide numbers such that the entropy of the treatments they select is minimal: http://unbox.org/stuff/var/timm/11/id/html/entropy.htmldivide performancescores such that the variance t is minimal: http://unbox.org/stuff/var/timm/11/id/html/variance.html

given N  arrays of performance results p .... of treatments 1,2,3...
with means  m[1],m[2], etc
with stand devs of s[1], s[2], etc then ...

for i in N
   for j in N
        boring[i,j] = (abs(m[i] - m[i])/s <= 0.3) #cohen

samples=1000
N*samples timesRepeat: {
   i = any index 1..N
   j= any index 1..N
  if (boring[i,j] continue
   x = the i-th member of p[i]
   y = the j-th member of p[j]
   diff = x - y
   better = if(errorMeasures) diff<0 else diff>0
   win[i] += better
}
for(i in win)
   win[i] = win[i]/(N*samples) * 100 # so now its percents

rank treatments by their position in win (higher rank = more wins)

Reviews and unsupervised feature-selectors

2011-10-10T22:55:00.000-07:00

a) Papers: In case you want to take a look at some of the recent reviews we have submitted, here are two papers: 1) kernel density estimation and 2) ensemble methods.

Another one with 40% of the reviews done is about active learning (here).

b) QUICK+ (under implementation): An unsupervised instance and feature selector, which works in two steps:

Step1: Run QUICK on the dataset and find how many instances (q) you need.
Step2:

For q instances, transpose the dataset and find the popularity of features in a space defined by instances.
Throw away features from least popular to most popular until there is a statistically significant deterioration in performance.
Stop when performance drops significantly and put the last-thrown feature back.

NSGA-II with Clustering Results

2011-10-10T16:50:00.000-07:00

Clustering appears to be capable of helping NSGA-II converge faster, but it does not appear to do so reliably. For the two easier problems (Schaffer and Fonseca), the clustering version converges in significantly fewer generations than the baseline version. For the harder problems (Kursawe and Tanaka), it appears to have had little impact.

The clustering appears to have no impact on the variance in convergence.

The following are graphs of the convergences of the two NSGA-II versions on the four problems. The left column shows the median performance, and the right shows the 25th and 75th percentile performances.

The following are graphs of the running times for solving the previous problems. The clustering version is linearly slower than the baseline version.

There are some odd blips in some of the running times that seem to be noise. I tested the Tanaka problem with a population size of 2000 for 2000 generations, and I got a very linear result:

The only thing that can determine whether clustering truly helps NSGA-II perform better is a distribution metric. If clustering helps distribute solutions across the Pareto frontier in fewer generations than the baseline, it may be a viable technique even when it provides no benefit for the convergence.

The Continuing Adventures of NSGA-II

2011-10-04T06:10:00.000-07:00

I have implemented clustering inside the official NSGA-II implementation in C. It seems to converge faster and with more spread than the baseline:

The trick is processing these results to determine whether the apparent convergence is real or not. The only way I know to test the convergence and distribution is to use the Upsilon (average distance to Pareto frontier) and Delta (distances to neighbors weighted by average distances) measures Deb used, but these require knowing the mathematical definition of the Pareto frontier to calculate equally distributed points across it.

To approximate this, I tried writing an incremental search through the problem space using kd-clustering to refine the search space. To my surprise, it did not seem to work well:

So, I've tried switching to use calculus to determine these points by hand. This is not going well, either:

Schaffer Pareto Frontier: f2 = (sqrt(f1) - 2)^ 2
Area under Frontier: Integral fx dx = 1/2 x^2 - 8/3 x sqrt(x) + 4x + C
Calculating x's that generate precise areas... ?

Crowd pruning

2011-10-04T05:36:00.000-07:00

Tried 2 methods:

1) Based on the ratio of (max allowed / current rule count), I randomly decide whether to include each point.

2) I sort by the first dimension, then add in in order every nth item to get to 100. (If n is not an integer, it selects every item that makes the "step" go over the next integer value.

Using algorithm one, I saw a fair amount of performance loss, as gaps began to appear in the frontier.

However, when I used the second algorithm, I saw mostly the same level of performance as the data where I didn't crowd prune the rules, and generated the results is minutes rather than a few hours.

For comparison, here's the fonseca curve at generation 9 without rule pruning.

Still reviewing..

2011-10-04T01:10:00.000-07:00

Finished the reviews of the Comba paper minor revision for TSE. The paper is here. I still need a proof-read.

Submitted version of the kernel paper to EMSE is here.

Active learning paper major revisions for TSE are in progress..

Discussion on the TSE paper asserting that OLS is the best.

Experiments: 1) unsupervised feature selectors, 2) different ranking methods.

Current Progress of Lisp NSGA-II

2011-09-27T07:06:00.000-07:00

This was the first comparison between the C version and my version. I would have made a new graph, but I somehow lost the C frontier data. I saw that something was very wrong, so I have spent a lot of time trying to fix it. This is as close as I've gotten thus far:

The results are closer, but something is still missing.

How to find data for effort estimation

2011-09-27T06:25:00.001-07:00

How to Find Relevant Data for Effort Estimation

View more presentations from Tim Menzies

See also http://menzies.us/tmp/IST_Invited_2011_v6.pdf

Which-2 Function Optimizer Early results

2011-09-27T05:25:00.000-07:00

Primarily, we are going to look at the Fonseca data set.

The final curve I was able to generate after 10 generations of running which looks like this:

http://unbox.org/stuff/var/will/frontierfinder/ml/generations/fonseca9.png

It is worth noting that the frontier does not approach f1(x)=0 or f2(x) = 0. Also, on the bottom (leg) of the curve, we see less consistency. This did not seem to improve with future generations.

The overall progression of the generations is here:

http://unbox.org/stuff/var/will/frontierfinder/ml/generations/fonseca1.png

After generation 5, very little seems to change other than a slight change between generations 6 to 9:

http://unbox.org/stuff/var/will/frontierfinder/ml/generations/fonseca.png

Numeric results have not yet been calculated.

Paper Reviews

2011-09-26T22:50:00.000-07:00

Kernel paper is good to go with 10 pages of replies. The latest pdf is here.

TSE paper (active learning) major review is under construction here.

CITRE Data Analysis

2011-09-13T05:08:00.000-07:00

Not shockingly, the field that is an order of magnitude larger than the others most affects the data

By comparison, when looking at the other fields, few of the percentages were substantially different from 20% (if, within a class, each of the possible values had a 20% distribution, it would mean true random distribution)

Data (Big file)

Insofar as progress into Which and NSGA-2, I've made progress, though due to illness, not as much as I would like. Hopefully to complete both (or at least 1) by next week with some results.

First Stab at NSGA-2 Optimization

2011-09-12T22:59:00.000-07:00

I am working on a modified NSGA-2 to converge on the Pareto frontier in fewer generations than the standard version. My idea is to use clustering to determine the region of independent variable space that is the most promising of those tried in a given generation. By knowing the best spatial region, it should be possible to generate solutions in that region to gain more information than could be gathered breeding initial poor solutions.

My first attempt took a rather simple approach. I made it so that the fast non-dominated sort (FNDS) chose some percentage of the parents for the next generation, and the rest were generated within the most promising space as determined by the above sort. This space was determined by clustering solutions by reducing the standard deviations of the independent variables down to a minimum cluster size of 20, then selecting the cluster that had the lowest mean frontier rank.

This attempt seems to have some promise. Here is a graph showing the baseline performance of the standard NSGA-2 on the Fonseca study:

This is a graph of the two objective scores for all members of the first-ranked frontier for each generation. By generation 20 (far right), NSGA-2 has found a very nice distribution of answers across the Pareto frontier. The first generation that looks close enough to me is generation 14. Note how the first few generations are rather noisy; it isn't until generation 8 that the solutions starts to mostly lie on the Pareto frontier.

As a counterpoint, here is a like graph of my modified NSGA-2. For this run, FNDS selects 80% of the parents, and the remaining 20% are generated from the best cluster:

First, note how the generations converge toward the frontier more rapidly than the baseline. Fewer of the solutions for the initial generations are far away from the Pareto frontier, and by generation 6, the Pareto frontier is outlined almost as strongly as it is in generation 8 from the baseline.

There seem to be diminishing returns from this approach, however. After the baseline NSGA-2 has "discovered" the Pareto frontier, it smoothly converges onto it and distributes solutions across it rather nicely. My modified NSGA-2 seems to be injecting noise that keeps it from converging smoothly across the solution space. This is probably because NSGA-2 is designed to distribute solutions across the Frontier, whereas my current clustering approach searches only one part of it. Furthermore, no adjustments are currently made to account for the algorithm getting closer to the answer, so good solutions are getting replaced by guesses made by the clusterer.

For the next step, I will explore ways to make the clusterer explore all the promising spaces as opposed to just the best one, and also to dynamically adjust the ratio of FSDN-selected parents to cluster-selected parents based upon the proximity to the Pareto frontier.

Wrapping up after the quals

2011-09-12T21:31:00.000-07:00

1. ESEM 2011 Talk: Version1 of our presentation is here.
2. The papers waiting for revision are:

Active Learning Paper at TSE (major changes (for 2 goods, 1 excellent) due 5th December 2011)
Kernel Paper at ESEM (minor changes due 29th September 2011)
Ranking Stability Paper at ASE (major-awaiting the return from TSE about ensemble paper)
Bias-Variance Paper (new statistical test)
Hong-Kong paper (waiting for submission after the reviews of Dr. Keung & Dr. Menzies)

3. Design of experiments for unsupervised feature selection. My action-plan is:

Find a list of 10+ unsupervised feature selectors
See this work as a follow-up or complement of the active learning paper, i.e.
Use the same datasets as the active learning paper
Run only feature selectors and see which datasets can give the same performance with less features. This will show if it works at all.
If prior succeeds, then run active learner (with k=1) followed by successful feature selectors. Then change this order and get another run. Compare performance to using the whole dataset and see if feature selection and active learning can complement one another.

4. Design of experiment for the ET test. My action plan for the paper is:

Reading the TSE paper for better understanding
Formulating the ET test in a formal manner
Replicating the ASE paper to confirm/refute the ranking changes
If the previous one succeeds, then building ensembles to see if the ET test outcomes are confirmed by successful multi-methods (???)

Game AI vs Traditional AI

2011-09-05T18:28:00.000-07:00

Very recent topic about Traditional AI and Game AI, and what is best for entertainment in games.

http://www.ai-blog.net/archives/000183.html

http://altdevblogaday.com/2011/07/11/students-game-ai-vs-traditional-ai/

We are still very far behind in terms of "good AI"; Game developers tend to avoid Traditional AI

"Game Developers don't get paid to be clever" - most of the stuff in games today are just tricks, smokes and mirrors, designed to increase believability and make for better games.

The important part is to make a fun and compelling game. Traditional AI takes lots of time and memory, and it doesn't even get used much.

Dungeon Game

2011-09-05T18:22:00.001-07:00

Dungeon Game is updated with a simple competitive game play. You need to find the key and then take it to the exit (faster than the Agent.)

We want the Agent to play like a human, so that the game is fair, and as fun as it can be.

The Algorithm powering the Agent is based simply on how I think humans explore. However, data is forthcoming on how humans really explore.

Screenshot: The Agent wins.

Results on CITRE from last semester

2011-08-30T06:01:00.000-07:00

5 "bands" further examination

ReviewDocs2 = 250
ReviewDocs2 = 275
ReviewDocs2 = 300
ReviewDocs2 = 325
ReviewDocs2 = 350

Density Distribution Map

2011-08-30T05:10:00.000-07:00

Here is the density distribution map that I came up with for the China Data set. The larger the density, the darker the shade of gray. The pure white spots are where there are no clusters present.

Which2 Multidimensional optimizer

2011-08-29T16:17:00.000-07:00

Immediate results using the given multi-dimensional functions are poor, but promising

Fonseca data

All of our rules with 2-bin discretization using Which were in the tiny green square. However, the goal with fonseca is to minimize, so being in the top right corner is very bad. By comparison, with 8-bin, are rules were mostly in the top right blue square, but we had one rule with coordinates f1=0.2497 f2=0.9575.

In Kursawe

Our rules were all in the mass in the center left when I chose maximize to optimize. With 8-bin, the rules were spread out with than with 2-bin.

This is because 8-bin allows more detail than 2-bin. However, I posit once I am able to recurse this process, applying the constraints of the rules, that 2-bin will be better overall.

Further exploring these rules (by applying the rules as new constraints on the randomized input vectors on the data database) will involve a massive recoding. However, having done some manual constraints using the generated rules, the results improve in Round 2 (treating this as round 1). However, you cannot simply pick one rule to explore. Basically, your unconstrained start point is the head of an infinite tree, the branches from each node are the rules generated by each run of which using that node and all ancestor nodes to that node's rules as constraints on the input data. The rules can then be mapped to coordinates in the space of (f1, f2, ...fn). Ideally, these rules will approach the Pareto frontier.

Which is running through the data very quickly, but until I have further results which will take a massive reworking of code, can't say anything definitive about it's long term usefulness just yet.

Privacy Algorithms

2011-08-23T07:44:00.000-07:00

4 Privacy algorithms tested against 4 learners (random forests, naive bayes, k-nearest neighbour and logistic regression.

privacy.pdf

what i did on my holidays

2011-08-23T05:54:00.001-07:00

think before you report

2011-08-23T05:53:00.000-07:00

fayola eg1

fayola eg2

Fastgrid

2011-08-23T05:42:00.000-07:00

This summer I worked on creating a stronger version of idea in lisp. The first code was fastmap which was the 2D point generator. Then instead of storing the data in a tree structure, they were stored in a grid structure. The size and number of quadrantas was determined by the square-root of the amount of the data divided by two. When looking to cluster the quadrants, the gridclus function could take a look at all directions surrounding it to find its' closest neighbors.
Then the neighbors of the neighbors were searched for acceptance rate of 0.5 also. Gaps was then coded to compare clusters and locate the closet neighbor feared by the cluster being looked at. Finally, Keys was created to look for the best treatment. It used (b/B)^2/((b/B) +(r/R)) to determine the best rule overall. b is the frequency the rules appears in the 20% best, while r is the frequency that the rule appears in the 80% rest. While all of these functions were coded up individually, they are not yielding the correct results.

As seen from the image, the grid still contains too many clusters. When taking a closer look, it can be seen tath several of tehse clusters should be mereged t into one. The gridclus error is the start of the problems with the different functions interacting.

This example is from the velocity 1.6 data.

There are 30 clusters with the most in one cluster equally 4 quadrants.
An example of the printout when keys is run on this data follows:

KEYS
(CLUSTER AB)
(ENVIES AL)
(TREATMENT #((0.0 $MOA)))
KEYS
(CLUSTER AL)
(ENVIES AZ)
(TREATMENT #((0.0 $NOC)))
KEYS
(CLUSTER AG)
(ENVIES AB)
(TREATMENT #((0.0 $LCOM)))

Lua Games

2011-08-22T12:02:00.000-07:00

Over the summer, I worked on and developed three games in Lua. The game concepts come out of the book; Land of Lisp.

Attack of the Robots

- Avoid robots, and as they chase you down, get them to run into each other

Grand Theft Wumpus

- The Wumpus is hiding in Congestion City, but where?

- Use clues to track down his blood trail, and avoid Glow-worm gangs and Police

- Fire your one and only shot at the Wumpus if you think you've got his location tracked down

Evolution

- Watch as animals reproduce while eating and expanding across the game world

- Notice how most animals will remain in the lush jungle; very few wander out across the steppes.

Graphics Engine chosen for development was Lua LOVE: http://love2d.org/

Dungeon Gen & Explorer

2011-08-22T11:53:00.000-07:00

Dungeon built based off an algorithm by Jamis Buck:http://blog.kromatyk.fr/wp-content/random-dungeon-design.pdf.

The agent doing the exploring is what is interesting in the Dungeon Project - I developed an algorithm on my own in which the agent discovers and uses pathfinding to track down "darkways" - points of interest which the agent wants to go explore. The algorithm marks down all darkways, and chooses the closest one to go explore.

This is a human-way of exploring an unknown dungeon, and it is human because the agent has been restricted in what it knows about the dungeon. For instance, the agent only knows what has been revealed into vision, and it's pathfinding includes only visible regions.

Left: Regions the agent has explored. Right: The entire dungeon; explored.

Scale Up Stability Multiple Datasets

2011-06-08T06:58:00.001-07:00

Scale Up Study - so far I've only gone to 2433 datums but I'm planning to combine all of the versions we have with all other sets, which will be ~7400 datums and also to adopt your new report format for these results/new ones. [COMBO] [JEDIT] [XERCES] [LUCENE] [VELOCITY].

Attempted for a neat result, color each dataset and graph them. They're all over the place. Perhaps if I report the majority dataset in each cluster there could be something there.

Here are more stability results for 6 datasets: [HERE]

Active Learning Paper

2011-04-19T06:22:00.000-07:00

Current version of the paper is here.

Contrast Sets

2011-04-18T23:49:00.000-07:00

I've got which2n ported from Will1 for a baseline result. It's spitting out results like THIS.

Also, I've figured out how to embed an on_release event to the matplotlib canvas inside of my wxpython rig so that we can record clicks and display contrasts for selected regions. There will probably be 1,000,000,000 variations of the contrasts that we show (or that the user would like to see. THIS against ALL, THIS against THOSE, THIS against THAT...)

I'm having some issues with which2 and very small sections of instances so the display is waiting on that.

Planner Demo & Lua Search Algorithms

2011-04-18T13:54:00.001-07:00

Download demo from here: https://sites.google.com/site/tanzastic/research/planner.love

Need to download LOVE from http://love2d.org, (and install; simple, takes 30 seconds) then you can just double-click to launch planner.love

- Simple demo in lua, using various search methods.

- Map and sprites based on a popular game (that I'm active with.)

Noise Reduction in Defect Datasets Using Cliff

2011-04-12T05:53:00.000-07:00

No subsampling applied.

http://www.unbox.org/wisp/var/kel/cliff-effort/results6/gnuplot/40/true-chart.pdf
http://www.unbox.org/wisp/var/kel/cliff-effort/results6/gnuplot/40/false-chart.pdf

Subsampling applied (all (N) defective class and N non-defective.
http://www.unbox.org/wisp/var/kel/cliff-effort/SubSamResults3/gnuplot/40/true-chart.pdf
http://www.unbox.org/wisp/var/kel/cliff-effort/SubSamResults3/gnuplot/40/false-chart.pdf

Tasks this week:
Grammatech report (by Thursday morning at the very latest)
IDEA rig for summer.

Active 1NN vs. Passive&Random 1NN, CART

2011-04-11T21:44:00.000-07:00

The plots for 24 datasets are here.

An executive summary of all these plots is here.

Histograms and EMD

2011-04-05T06:17:00.001-07:00

Taken from Wikipedia, the source of all knowledge and truth:

"The earth mover's distance (EMD) is a measure of the distance between two probability distributions over a region D. Informally, if the distributions are interpreted as two different ways of piling up a certain amount of dirt over the region D, the EMD is the minimum cost of turning one pile into the other; where the cost is assumed to be amount of dirt moved times the distance by which it is moved."

Largest Earth Mover in the world -- German made. 311 feet tall, 705 feet long, 45,000 tons, can move 76, 455 cubic meters each day.

Weka style attribute Histograms (10 bins).

I can compute the EMD of two numeric columns in JM1 (5440 samples) in 0.0035 seconds.

Cliff and Application to Defect Data

2011-04-05T04:55:00.000-07:00

Charts for TRUE (defects) class
Charts for FALSE (no defects) class

Sampling methods and Bias-Variance Trade-off

2011-04-05T00:25:00.000-07:00

Final version of the bias-variance paper is here.

Active Learning via kNN-Based Popularity

2011-03-15T06:16:00.000-07:00

The initial results for the active learning are here.

2011-03-15T06:07:00.000-07:00

Defect EK Matrix.

http://www.unbox.org/wisp/var/butcher/fromtimm/EK.pdf

Some terminology

2011-03-08T01:09:00.000-08:00

Something that caused me a lot of time to correct..
When speaking of ABE, instance selection and instance retrieval correspond to entirely different things. Instance selection = filtering, whereas retrieval = analogy selection. Corrected ESEM paper is here.

If only I had perfect experts!

2011-03-08T01:01:00.000-08:00

Here is a short report about the results of active learning experiments in effort estimation.
For full plots, refer here.
The moral of the story is:
1) When using 1NN as estimation method, (more or less) half of the instances are not closest neighbor to any other instances.
2) When project instances are labeled by an expert based on their popularity, MdMRE values very easily drop below regular 1NN (with LOO).
3) ASSUMPTION/CHEAT is that the expert perfectly predicts the project effort, which is (probably) never the case.

Even More on Business Intelligence

2011-03-07T23:50:00.000-08:00

New runtime chart can be found at this location:

http://unbox.org/wisp/var/butcher/runtime/idea_runtime_training.xlsx

---

Empirical (not fitted) Weibull CDF of the effort distribution w/ you are here in red:

-----

Takes on the 3 shot cycle:

More on Business Intelligence and Other Things

2011-02-28T22:04:00.000-08:00

Quadrant Neighborhood

Mann-Whitney reports the following regarding the distributions in the chart below.

Rank Dataset

0 albrecht

0 china

0 coc81

1 cocomo_sdr

1 desharnais_1_1

1 finnish

1 kemerer

1 maxwell

1 nasa93

Are there favourite subsets?

2011-02-28T20:13:00.001-08:00

Noise injection experiments showed that it is difficult to introduce noise to effort datasets. Then this raised the question: Are certain subsets of the datasets more preferable than the others? To answer that we can look at neighbor-ordering matrices. Two forms of neighbor-ordering are:
* Absolute : here.
* Percentage : here.

Business Intelligence & Quadrant Differences Graphed

2011-02-15T06:27:00.001-08:00

All of the above can be found HERE.

Results from Noise Generation in Effort Sets

2011-02-15T06:03:00.000-08:00

Ran kNN and kNN with Cliff instance selection for k = 1,3,5.

Noise Injection method:

for 10%, 20%,...,80% of train data:
Replace with a randomly selected effort score is not equal to the original value.
Perform similar transform with the top 3 attributes as selected by BORE also ensuring replaced value is not equal to the original value.

Results are MDMRE from 5x5 cross-validation.

albrecht
china
cocomo
desharnais
finnish
isbsg
maxwell
miyazaki94
nasa93

Results from Noise Reduction in Effort Sets

2011-02-08T06:33:00.000-08:00

http://www.unbox.org/wisp/var/kel/cliff-effort/results/graph/

Six sets:
albrecht
china
desharnais
maxwell
miyazaki94
nasa93

Business Intelligence

2011-02-08T06:08:00.000-08:00

Initial effort results and some images HERE and my first take on visual model is HERE.

ESEM Paper Initial Version

2011-02-08T01:44:00.000-08:00

A very basic initial version is here..

Initial BAMBOO Results

2011-02-07T20:39:00.000-08:00

http://unbox.org/wisp/var/bill/comba/reports/february/feb8.pdf

Initial results from running BAMBOO are provided in the above file.

Results from Active Learning Projects This Week

2011-02-01T06:32:00.000-08:00

Comparison between NB, NB+CLiff, and 1NN+Cliff:
http://www.unbox.org/wisp/var/kel/doc/ActiveLearning/CliffResults/1-31/

Comparison between NB treatments:
http://www.unbox.org/wisp/var/kel/doc/ActiveLearning/CliffResults/1-31-2/

Secret mixture?

2011-02-01T01:08:00.000-08:00

Is there a secret percentage mixture when selecting instances from your own company and from a cross-company? Find out here.

Open Issues in Empirical SE

2011-01-28T17:57:00.000-08:00

Q1: Generality of the Microsoft conclusions?

See "Myths in Software Engineering: From the Other Side" by Nachiappan Nagappan, TAPS'10

A list of 10 conclusions made from Msoft research
The validity of these, for sites outside of Msoft, is an open issue.

Q2: How to support the following kinds of business intelligence?

Click to enlarge (from http://research.microsoft.com/apps/pubs/?id=144543):

Other approaches to business intelligence:

Watch my recent talk: (250MB) : http://menzies.us/empiricalSEversion2.wmv
Read my recent talk (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx

Or see below

Talk at Microsoft Research, Jan 27, 2011

2011-01-28T02:20:00.000-08:00

Talk at Microsoft Research, Jan 27, 2011

Slides (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx

Download (250MB) : http://menzies.us/empiricalSEversion2.wmv

The rapid pace of software development innovation challenges empirical software research to keep up, if it is to deliver actionable and useful results to practitioners. The empirical software engineering research field has not always been able to deliver this. Recently, it has become increasingly apparent that rigorous data collection and analysis can be so expensive and time-consuming that empirical software engineering studies, which seek to understand the costs and benefits of software development solutions in practice, greatly lag the pace of innovation in the field. In too many cases, a trusted body of empirical results can only be built up after the innovative solutions that they are studying are already well on their way to obsolescence or standard practice. However, we argue that recent advances put a sustainable and increased research pace within our reach. A suitably scaled-up and nimble empirical research approach must be based upon the “crowd-sourcing” of tough empirical problems.

empirical software engineering, v2.0

View more presentations from Tim Menzies.

Will McBurney

2011-01-25T16:17:00.000-08:00

Currently a graduate student in Computer Science at West Virginia University, Will McBurney is a programmer, musician, and blogger on the side. Will McBurney began programming in his Senior year of High School in Charleston, WV, his hometown. Originally attending WVU as an undergraduate aspiring to become a Mechanical Engineer, Will switched majors midway through his freshmen year to Computer Science.

It is not secret Will is a major nerd. Will skipped his Senior Prom to go to National Science Bowl in 2006 and has read more Star Wars books than most people have read at all. On the inter-tubes, Will uses the psuedonym "Death by Smiley", a name that was made up spur of the moment while playing Halo in the dorms at WVU.

-Will McBurney

Active Learning

2011-01-25T06:29:00.000-08:00

Kel w/ Cliff:

http://www.unbox.org/wisp/var/kel/doc/cliff/results.txt

Andrew w/ Heuristic Randomness:

http://afrolegs.com/kc1_eraSize150.png

http://unborx.org/wisp/var/butcher/active/

Rank Deltas for COMBA

2011-01-25T05:00:00.000-08:00

http://unbox.org/wisp/var/bill/comba/reports/rankdeltas.png

Which features define CC?

2011-01-25T00:12:00.000-08:00

Division of available datasets into subsets according to different criteria (language, hardware, resource etc.) may define closeness of a cross-company setting. The results and subsets are here.

Bias and Varince of Testing Strategies on Effort Datasets

2011-01-25T00:11:00.000-08:00

A short report is here.

Another look at Fayola's data

2011-01-24T20:54:00.000-08:00

http://unbox.org/wisp/var/11/pretty/forensics.pdf

map-reduce & Hadoop

2011-01-24T20:46:00.000-08:00

Map-Reduce

MapReduce is a patented^[1] software framework introduced by Google to support distributed computing on large data sets on clusters of computers.^[2]

MapReduce libraries have been written in C++, C#, Erlang, Java, Ocaml, Python, Ruby, F#, R and other programming languages.

http://en.wikipedia.org/wiki/MapReduce

Appache HadoopOpen-source system layer for map-reduce. Implements (e.g.) a distributed file system

Material for interactive new media design

2011-01-24T20:42:00.000-08:00

http://forgottenlegacywwi.org/exhibit.html

popularity is (almost) a perfect predictor for defects

2011-01-24T20:40:00.000-08:00

http://unbox.org/wisp/var/11/popular/popular-v6.pdf

One surprising observation from the Helix study [50] was that(a) most classes are not popular and (b) patterns of the most pop-ular classes emerged very early in the lifetime of an open-sourceproject. Specifically, popularity and age of a class maintain a pos-itive monotonic relationship throughout the lifetime of a system[50], that is, as a system matures, popular classes tend to becomeeven more popular.

This contradicts our reading of Fowler [20] andBeck’s work [6] that object-oriented quality assurance requires in-volved constant refactoring (e.g., Beck’s TDD loop of write tests,run test, refactor). We cannot see evidence of constant, widespreadrefactoring in open-source projects studied by Helix [50].

The implications of this lack-of-refactoring for the theory of object-oriented development is discussed elsewhere [50]. This paperfocuses of class popularity since (a) it is a stable concept for thelifetime of an object-oriented system and (b) popularity can lead todefects via:

Defect injection:As developers work with the popular classes, they make the occasional mistake. Some of these mistakesresult in code defects. Since developers work on popular classes more than other classes, then most developer defects accumulate in the popular classes.
Defect discovery: Since developers work mostly on popular classes, they are most likely to uncover those classes’ defects.

We demonstrate that in 33 open-source Java projects popularity-based defect predictors work within 4% of a theoreti-cal upper bound on predictor performance (this is the basis for ourclaim that such predictors are “nearly perfect”).

empirical SE v2.0

2011-01-24T20:34:00.001-08:00

timm, trying to map out a research agenda for the next decade

empirical software engineering, v2.0

View more presentations from Tim Menzies.

What makes a game fun

2011-01-18T20:26:00.001-08:00

Hello, hi,

My name is Joe and I'm a new PhD student at WVU, got my MS in CS just last year, and working towards the big goal with a vague idea and a genuine interest for AI in games. I'm hard of hearing, and generally have a tough time understanding people in a group setting, but usually if there's only one person speaking -towards- me, with a good voice, then it's fine. (Sorry if I seem confused sometimes, usually trying to figure out what you just said.)

Last semester I worked on the idea of "What makes a game Fun?" and found myself digging through a lot of unofficial resources towards concepts in a game that heighten the level of enjoyment had by players. I wrote a survey-type paper and outlined what I thought were the core values of "Fun in video games." Those four were Gameplay, Originality, Replayability and Story (GORS). Replayability is key; any time a player picks up a controller, there is a drawing factor that keeps them attracted to the game and continuously replaying. Once that allure is lost, replayability is lost, and the game no longer has potential to provide fun (at least for a while, but there's always a chance the allure returns at some later point - yet another key point to replayabilty.)

Anyway, second half of semester I put the four dimensions of fun together in a game which the user could play and "set" the levels of each dimension. i.e., the user would set originality to be of "low" setting, making item names and enemies sound very generic, as opposed to original and unique. Called Cre-Ativity, the game plays like Zelda, in a top-down fashion developed with OpenGL in C++. Players can use one of four weapons; acquire armor, gain experience, become stronger and bash enemies while exploring in aims of completing a general story-driven quest.

At this point, the semester was over and one of the key questions that remained was; "How could you develop story content according to user-input?" More on that topic to come.

Screenshots:
https://sites.google.com/site/tanzastic/research/s01.jpg
https://sites.google.com/site/tanzastic/research/s02.jpg
https://sites.google.com/site/tanzastic/research/s03.jpg
https://sites.google.com/site/tanzastic/research/s04.jpg
https://sites.google.com/site/tanzastic/research/s05.jpg

Cross vs. Within Company Experiments

2011-01-18T06:34:00.001-08:00

Tentative results are here.

Selected cross-company datasets are given below..

finnishAppArea = {'finnishAppArea2','finnishAppArea7','finnishAppArea10'};

finnishAppType = {'finnishAppType1','finnishAppType2345'};

kemererHardware = {'kemererHardware1','kemererHardware23456'};

maxwellAppType = {'maxwellAppType1','maxwellAppType2','maxwellAppType3'};

maxwellHardware = {'maxwellHardware2','maxwellHardware3','maxwellHardware5'};

maxwellSource = {'maxwellSource1','maxwellSource2'};

chinaResource = {'chinaResource1','chinaResource2','chinaResource3','chinaResource4'}

Questions:

Is 1*LOO ok or do I need 20*LOO?

Is using only static-k better?

Testing using only the X dimension in IDEA

2011-01-18T06:31:00.001-08:00

http://www.unbox.org/wisp/var/butcher/dragons/x_validity_test.pdf

New results after discretizing with Fayyad Irani

2011-01-18T06:15:00.000-08:00

http://unbox.org/wisp/var/fayola/forensics/results/

This week...

2011-01-18T05:26:00.000-08:00

Tim has written two subjects

Learned about map-reduce and clojure and wondered about applying it to WHICH.

Coded up idea, knn, cross-val, and run lots of experiments

Worked with physics on an NSF proposal

Planned next week's talk at microsoft, redmond

Worked with Martin Shepperd on a journal special issue.

Worked with Ayse Bener on a journal special issue

And written a research paper

Over to you....

Fayyad Irani Discretization for Row Pruning Using Variance

2011-01-11T06:43:00.000-08:00

New results for reducing variance with W: here

Latex classes for building resumes

2010-12-10T13:18:00.000-08:00

http://rpi.edu/dept/arc/training/latex/resumes/

includes some nice sample resumes including http://rpi.edu/dept/arc/training/latex/resumes/res9b.pdf

New papers

2010-11-09T10:55:00.000-08:00

http://www.mendeley.com/library/

searched based SE

defect prediction

cascade generalization

world cloude

2010-11-09T10:52:00.000-08:00

of papers in my foser track.

combined cloud in the midle

individual papers all around

http://menzies.us/tmp/Track5uberwordle.pdf

MoE Experiments

2010-11-09T08:16:00.001-08:00

On the x-axis algorithms are ordered w.r.t. loss values over all err. measures and all datasets. Lines show the max. ordering change w.r.t. win and loss values over any of the error measures. The individual rankings are in this sheet.

3Way-LOO Rank Changes + 3Bands Appearances

2010-10-26T10:47:00.000-07:00

Above is the plot of 3-Way vs. LOO
The appearance of methods in 3bands are here.

More About Stability...

2010-10-22T03:28:00.000-07:00

Rank distributions and frequency tables
Win Percentages across methods and datasets as per error measure

IDEA - Stained Glass Visualization

2010-10-12T10:27:00.000-07:00

Defect Sets:

Boxes with color indicate a cluster. Clusters are self-tested and ranked based on harmonic mean for the TRUE defect class. Green indicates the cluster performed in the top 25%. Yellow indicates the middle 50%, and red indicates the bottom 25%.

KC1
2,109 examples
Run time: 12.5 seconds

JM1
10,880 examples
Run time: 2 minutes

Effort sets:

Effort set performance is based on self-test MDMRE. Clusters are colored in the same manner as above.

China
499 examples
2.6 seconds

Datasets, charts for more datasets, and time results are available in WISP.

Stability and Bias-Variance of LOO vs. 3Way

2010-10-12T02:18:00.000-07:00

Stability across datasets: For each evaluation method, over 19 datasets, see how many times solos appear in top 16. The numbers and plots for all methods are here.

Bias & Variance of LOO and 3-Way: Let L be the squared loss function, f be a predictor and y be the predicted value for a particular x. Also let y* be the optimal prediction (actual response for x), and ym be the main prediction. Under squared loss function, main function becomes mean of all predictions: ym=mean(all y's). Then the following definitions follow:

Bias(x) = L(y*, ym)
Var(x) = ED(L(ym, y)) where D is the occurrence of x in all training sets.

Above definitions are valid for single instances, however they can be average over all instances:

Biasavg = Ex(B(x))
Varavg = Ex(Var(x))

When we follow these definitions we get bias(x-axis) vs. variance(y-axis) figures here.

Do Reductions with Total Defects Select Smaller Data?

2010-10-05T11:40:00.000-07:00

Does W perform better with total defects than defects/kloc by simply picking small projects?

Mann Whitney says no

Stability of Combination Methods

2010-10-05T10:55:00.000-07:00

http://unbox.org/wisp/var/10/comba/only-plots-to-discuss.pdf

IDEA - Progress

2010-10-05T08:49:00.001-07:00

List completed so far:

Flipping the X-axis so that the dense area of points are at the West pole (essential for logging the x axis).
Logging of X and Y Coordinates
Equal width and equal frequency grid creation for m x n divisions.
Separating data point into Quadrants based on the axes. (Not recursive yet)
Coloring the plot where data exists.
Next step will be to color in interesting* neighboring quadrants.

Quadrants w/points, N=10

Quadrants w/points, N=4

Images below do not demonstrate the quadrant highlighting demonstrated above.

jm1 defect dataset

jm1 defect dataset with log(x)

kc1 defect dataset

kc1 defect dataset with log(x)

Linear v. Dynamic Search in Molecular Structures

2010-10-05T08:46:00.000-07:00

In the field of cheminformatics, a common task is navigating through a database of molecules that could have thousands of entries.

One task is how to store the molecules themselves. An approach in cheminformatics is a "SMILES" string. A smiles string stores the molecules involved and special characters which indicate structural properties of the molecule. Previous tests in the Lewis Research Group have involved RDX adsorption. (RDX is an explosive compound) The Lewis Research Group uses a default file format of .xyz, but an open file conversion system exists called OpenBabel. The SMILES string for RDX is provided below, along with a picture for comparison.

RDX Molecule
O=N(=O)N1CN(N(=O)=O)CN(N(=O)=O)C1

SMILES strings can be represent the same structure in different ways, so for databases a special form of SMILES known as Canonical SMILES form is used to prevent duplicate entries. The advantage of SMILES strings is that when they break in to comparable substructures.

There are clever indexing schemes for quickly searching structural properties of molecules. One is substructure keys, in which binary flags about structural properties are stored for each molecule. Another is using a hash table encoding that serves as a proximity filter.

There is also the area of molecular similarity and molecular diversity. This field seeks to find similar molecules by noting differences in derived attributes. For example, it is computationally easy to compute the molecular weight of a molecule. This attribute, and others, can be used as a search therm to find similar molecules or correlation with more complex attributes such as molecular adsorption in repeating lattice structures.
Other features can also be noted and collected, providing a linear database for searching.

Overall, the question is whether to use structural similarity analysis (dynamic) or molecular diversity measures (linear) for the system.
The answer, at the current moment, appears to be using both.
We are currently not sure what variables correlate with performance in the desired chemical property of adsorption in a lattice structure, so having more features provides a greater possibility for correlation and more accurate estimates. The linear search could also be used to select candidates for dynamic search, using a hybridized preprocessing approach.

The second is that the system has the advantage of being domain specific and thus having access to domain specific algorithms and methods. The database in question will have many properties which need to be handled in relation to their environment. For example, it is likely that there will be missing or blank features for entries in the database. The algorithm might have access to be able to call a function to compute missing entries, which is a highly domain specific solution not appropriate to general algorithms.