Comparison between NB, NB+CLiff, and 1NN+Cliff:
http://www.unbox.org/wisp/var/kel/doc/ActiveLearning/CliffResults/1-31/
Comparison between NB treatments:
http://www.unbox.org/wisp/var/kel/doc/ActiveLearning/CliffResults/1-31-2/
Tuesday, February 1, 2011
Secret mixture?
Is there a secret percentage mixture when selecting instances from your own company and from a cross-company? Find out here.
Friday, January 28, 2011
Open Issues in Empirical SE
Q1: Generality of the Microsoft conclusions?
Q2: How to support the following kinds of business intelligence?
Other approaches to business intelligence:
- See "Myths in Software Engineering: From the Other Side" by Nachiappan Nagappan, TAPS'10
- A list of 10 conclusions made from Msoft research
- The validity of these, for sites outside of Msoft, is an open issue.
Q2: How to support the following kinds of business intelligence?
- Click to enlarge (from http://research.microsoft.com/apps/pubs/?id=144543):
Other approaches to business intelligence:
- Watch my recent talk: (250MB) : http://menzies.us/empiricalSEversion2.wmv
- Read my recent talk (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx
- Or see below
Talk at Microsoft Research, Jan 27, 2011
Talk at Microsoft Research, Jan 27, 2011
Slides (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx
Download (250MB) : http://menzies.us/empiricalSEversion2.wmv
Slides (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx
Download (250MB) : http://menzies.us/empiricalSEversion2.wmv
The rapid pace of software development innovation challenges empirical software research to keep up, if it is to deliver actionable and useful results to practitioners. The empirical software engineering research field has not always been able to deliver this. Recently, it has become increasingly apparent that rigorous data collection and analysis can be so expensive and time-consuming that empirical software engineering studies, which seek to understand the costs and benefits of software development solutions in practice, greatly lag the pace of innovation in the field. In too many cases, a trusted body of empirical results can only be built up after the innovative solutions that they are studying are already well on their way to obsolescence or standard practice. However, we argue that recent advances put a sustainable and increased research pace within our reach. A suitably scaled-up and nimble empirical research approach must be based upon the “crowd-sourcing” of tough empirical problems.
Tuesday, January 25, 2011
Will McBurney
Currently a graduate student in Computer Science at West Virginia University, Will McBurney is a programmer, musician, and blogger on the side. Will McBurney began programming in his Senior year of High School in Charleston, WV, his hometown. Originally attending WVU as an undergraduate aspiring to become a Mechanical Engineer, Will switched majors midway through his freshmen year to Computer Science.It is not secret Will is a major nerd. Will skipped his Senior Prom to go to National Science Bowl in 2006 and has read more Star Wars books than most people have read at all. On the inter-tubes, Will uses the psuedonym "Death by Smiley", a name that was made up spur of the moment while playing Halo in the dorms at WVU.
-Will McBurney
Active Learning
Which features define CC?
Division of available datasets into subsets according to different criteria (language, hardware, resource etc.) may define closeness of a cross-company setting. The results and subsets are here.
Monday, January 24, 2011
map-reduce & Hadoop
popularity is (almost) a perfect predictor for defects
One surprising observation from the Helix study [50] was that(a) most classes are not popular and (b) patterns of the most pop-ular classes emerged very early in the lifetime of an open-sourceproject. Specifically, popularity and age of a class maintain a pos-itive monotonic relationship throughout the lifetime of a system[50], that is, as a system matures, popular classes tend to becomeeven more popular.
This contradicts our reading of Fowler [20] andBeck’s work [6] that object-oriented quality assurance requires in-volved constant refactoring (e.g., Beck’s TDD loop of write tests,run test, refactor). We cannot see evidence of constant, widespreadrefactoring in open-source projects studied by Helix [50].
The implications of this lack-of-refactoring for the theory of object-oriented development is discussed elsewhere [50]. This paperfocuses of class popularity since (a) it is a stable concept for thelifetime of an object-oriented system and (b) popularity can lead todefects via:
- Defect injection:As developers work with the popular classes, they make the occasional mistake. Some of these mistakesresult in code defects. Since developers work on popular classes more than other classes, then most developer defects accumulate in the popular classes.
- Defect discovery: Since developers work mostly on popular classes, they are most likely to uncover those classes’ defects.
We demonstrate that in 33 open-source Java projects popularity-based defect predictors work within 4% of a theoreti-cal upper bound on predictor performance (this is the basis for ourclaim that such predictors are “nearly perfect”).
empirical SE v2.0
timm, trying to map out a research agenda for the next decade
empirical software engineering, v2.0
View more presentations from Tim Menzies.
Tuesday, January 18, 2011
What makes a game fun
Hello, hi,
My name is Joe and I'm a new PhD student at WVU, got my MS in CS just last year, and working towards the big goal with a vague idea and a genuine interest for AI in games. I'm hard of hearing, and generally have a tough time understanding people in a group setting, but usually if there's only one person speaking -towards- me, with a good voice, then it's fine. (Sorry if I seem confused sometimes, usually trying to figure out what you just said.)
Last semester I worked on the idea of "What makes a game Fun?" and found myself digging through a lot of unofficial resources towards concepts in a game that heighten the level of enjoyment had by players. I wrote a survey-type paper and outlined what I thought were the core values of "Fun in video games." Those four were Gameplay, Originality, Replayability and Story (GORS). Replayability is key; any time a player picks up a controller, there is a drawing factor that keeps them attracted to the game and continuously replaying. Once that allure is lost, replayability is lost, and the game no longer has potential to provide fun (at least for a while, but there's always a chance the allure returns at some later point - yet another key point to replayabilty.)
Anyway, second half of semester I put the four dimensions of fun together in a game which the user could play and "set" the levels of each dimension. i.e., the user would set originality to be of "low" setting, making item names and enemies sound very generic, as opposed to original and unique. Called Cre-Ativity, the game plays like Zelda, in a top-down fashion developed with OpenGL in C++. Players can use one of four weapons; acquire armor, gain experience, become stronger and bash enemies while exploring in aims of completing a general story-driven quest.
At this point, the semester was over and one of the key questions that remained was; "How could you develop story content according to user-input?" More on that topic to come.
Screenshots:
https://sites.google.com/site/tanzastic/research/s01.jpg
https://sites.google.com/site/tanzastic/research/s02.jpg
https://sites.google.com/site/tanzastic/research/s03.jpg
https://sites.google.com/site/tanzastic/research/s04.jpg
https://sites.google.com/site/tanzastic/research/s05.jpg
My name is Joe and I'm a new PhD student at WVU, got my MS in CS just last year, and working towards the big goal with a vague idea and a genuine interest for AI in games. I'm hard of hearing, and generally have a tough time understanding people in a group setting, but usually if there's only one person speaking -towards- me, with a good voice, then it's fine. (Sorry if I seem confused sometimes, usually trying to figure out what you just said.)
Last semester I worked on the idea of "What makes a game Fun?" and found myself digging through a lot of unofficial resources towards concepts in a game that heighten the level of enjoyment had by players. I wrote a survey-type paper and outlined what I thought were the core values of "Fun in video games." Those four were Gameplay, Originality, Replayability and Story (GORS). Replayability is key; any time a player picks up a controller, there is a drawing factor that keeps them attracted to the game and continuously replaying. Once that allure is lost, replayability is lost, and the game no longer has potential to provide fun (at least for a while, but there's always a chance the allure returns at some later point - yet another key point to replayabilty.)
Anyway, second half of semester I put the four dimensions of fun together in a game which the user could play and "set" the levels of each dimension. i.e., the user would set originality to be of "low" setting, making item names and enemies sound very generic, as opposed to original and unique. Called Cre-Ativity, the game plays like Zelda, in a top-down fashion developed with OpenGL in C++. Players can use one of four weapons; acquire armor, gain experience, become stronger and bash enemies while exploring in aims of completing a general story-driven quest.At this point, the semester was over and one of the key questions that remained was; "How could you develop story content according to user-input?" More on that topic to come.
Screenshots:
https://sites.google.com/site/tanzastic/research/s01.jpg
https://sites.google.com/site/tanzastic/research/s02.jpg
https://sites.google.com/site/tanzastic/research/s03.jpg
https://sites.google.com/site/tanzastic/research/s04.jpg
https://sites.google.com/site/tanzastic/research/s05.jpg
Cross vs. Within Company Experiments
Tentative results are here.
Selected cross-company datasets are given below..
finnishAppArea = {'finnishAppArea2','finnishAppArea7','finnishAppArea10'};
finnishAppType = {'finnishAppType1','finnishAppType2345'};
kemererHardware = {'kemererHardware1','kemererHardware23456'};
maxwellAppType = {'maxwellAppType1','maxwellAppType2','maxwellAppType3'};
maxwellHardware = {'maxwellHardware2','maxwellHardware3','maxwellHardware5'};
maxwellSource = {'maxwellSource1','maxwellSource2'};
chinaResource = {'chinaResource1','chinaResource2','chinaResource3','chinaResource4'}Questions:
Is 1*LOO ok or do I need 20*LOO?
Is using only static-k better?
This week...
Tim has written two subjects
Learned about map-reduce and clojure and wondered about applying it to WHICH.
Coded up idea, knn, cross-val, and run lots of experiments
Worked with physics on an NSF proposal
Planned next week's talk at microsoft, redmond
Worked with Martin Shepperd on a journal special issue.
Worked with Ayse Bener on a journal special issue
And written a research paper
Learned about map-reduce and clojure and wondered about applying it to WHICH.
Coded up idea, knn, cross-val, and run lots of experiments
Worked with physics on an NSF proposal
Planned next week's talk at microsoft, redmond
Worked with Martin Shepperd on a journal special issue.
Worked with Ayse Bener on a journal special issue
And written a research paper
Over to you....
Tuesday, January 11, 2011
Fayyad Irani Discretization for Row Pruning Using Variance
New results for reducing variance with W: here
Friday, December 10, 2010
Latex classes for building resumes
http://rpi.edu/dept/arc/training/latex/resumes/
includes some nice sample resumes including http://rpi.edu/dept/arc/training/latex/resumes/res9b.pdf
includes some nice sample resumes including http://rpi.edu/dept/arc/training/latex/resumes/res9b.pdf
Tuesday, November 9, 2010
world cloude
of papers in my foser track.
combined cloud in the midle
individual papers all around
http://menzies.us/tmp/Track5uberwordle.pdf
combined cloud in the midle
individual papers all around
http://menzies.us/tmp/Track5uberwordle.pdf
MoE Experiments
On the x-axis algorithms are ordered w.r.t. loss values over all err. measures and all datasets. Lines show the max. ordering change w.r.t. win and loss values over any of the error measures. The individual rankings are in this sheet.
Tuesday, October 26, 2010
Friday, October 22, 2010
Tuesday, October 12, 2010
IDEA - Stained Glass Visualization
Defect Sets:
Boxes with color indicate a cluster. Clusters are self-tested and ranked based on harmonic mean for the TRUE defect class. Green indicates the cluster performed in the top 25%. Yellow indicates the middle 50%, and red indicates the bottom 25%.
KC1
2,109 examples
Run time: 12.5 seconds

JM1
10,880 examples
Run time: 2 minutes

Effort sets:
Effort set performance is based on self-test MDMRE. Clusters are colored in the same manner as above.
China
499 examples
2.6 seconds

Datasets, charts for more datasets, and time results are available in WISP.
Boxes with color indicate a cluster. Clusters are self-tested and ranked based on harmonic mean for the TRUE defect class. Green indicates the cluster performed in the top 25%. Yellow indicates the middle 50%, and red indicates the bottom 25%.
KC1
2,109 examples
Run time: 12.5 seconds

JM1
10,880 examples
Run time: 2 minutes

Effort sets:
Effort set performance is based on self-test MDMRE. Clusters are colored in the same manner as above.
China
499 examples
2.6 seconds

Datasets, charts for more datasets, and time results are available in WISP.
Stability and Bias-Variance of LOO vs. 3Way
Stability across datasets: For each evaluation method, over 19 datasets, see how many times solos appear in top 16. The numbers and plots for all methods are here.
Bias & Variance of LOO and 3-Way: Let L be the squared loss function, f be a predictor and y be the predicted value for a particular x. Also let y* be the optimal prediction (actual response for x), and ym be the main prediction. Under squared loss function, main function becomes mean of all predictions: ym=mean(all y's). Then the following definitions follow:
Bias & Variance of LOO and 3-Way: Let L be the squared loss function, f be a predictor and y be the predicted value for a particular x. Also let y* be the optimal prediction (actual response for x), and ym be the main prediction. Under squared loss function, main function becomes mean of all predictions: ym=mean(all y's). Then the following definitions follow:
- Bias(x) = L(y*, ym)
- Var(x) = ED(L(ym, y)) where D is the occurrence of x in all training sets.
- Biasavg = Ex(B(x))
- Varavg = Ex(Var(x))
Tuesday, October 5, 2010
IDEA - Progress
List completed so far:
- Flipping the X-axis so that the dense area of points are at the West pole (essential for logging the x axis).
- Logging of X and Y Coordinates
- Equal width and equal frequency grid creation for m x n divisions.
- Separating data point into Quadrants based on the axes. (Not recursive yet)
- Coloring the plot where data exists.
- Next step will be to color in interesting* neighboring quadrants.
Linear v. Dynamic Search in Molecular Structures
In the field of cheminformatics, a common task is navigating through a database of molecules that could have thousands of entries.
One task is how to store the molecules themselves. An approach in cheminformatics is a "SMILES" string. A smiles string stores the molecules involved and special characters which indicate structural properties of the molecule. Previous tests in the Lewis Research Group have involved RDX adsorption. (RDX is an explosive compound) The Lewis Research Group uses a default file format of .xyz, but an open file conversion system exists called OpenBabel. The SMILES string for RDX is provided below, along with a picture for comparison.
One task is how to store the molecules themselves. An approach in cheminformatics is a "SMILES" string. A smiles string stores the molecules involved and special characters which indicate structural properties of the molecule. Previous tests in the Lewis Research Group have involved RDX adsorption. (RDX is an explosive compound) The Lewis Research Group uses a default file format of .xyz, but an open file conversion system exists called OpenBabel. The SMILES string for RDX is provided below, along with a picture for comparison.

RDX Molecule
O=N(=O)N1CN(N(=O)=O)CN(N(=O)=O)C1
O=N(=O)N1CN(N(=O)=O)CN(N(=O)=O)C1
SMILES strings can be represent the same structure in different ways, so for databases a special form of SMILES known as Canonical SMILES form is used to prevent duplicate entries. The advantage of SMILES strings is that when they break in to comparable substructures.
There are clever indexing schemes for quickly searching structural properties of molecules. One is substructure keys, in which binary flags about structural properties are stored for each molecule. Another is using a hash table encoding that serves as a proximity filter.
There is also the area of molecular similarity and molecular diversity. This field seeks to find similar molecules by noting differences in derived attributes. For example, it is computationally easy to compute the molecular weight of a molecule. This attribute, and others, can be used as a search therm to find similar molecules or correlation with more complex attributes such as molecular adsorption in repeating lattice structures.
Other features can also be noted and collected, providing a linear database for searching.
Overall, the question is whether to use structural similarity analysis (dynamic) or molecular diversity measures (linear) for the system.
The answer, at the current moment, appears to be using both.
We are currently not sure what variables correlate with performance in the desired chemical property of adsorption in a lattice structure, so having more features provides a greater possibility for correlation and more accurate estimates. The linear search could also be used to select candidates for dynamic search, using a hybridized preprocessing approach.
The second is that the system has the advantage of being domain specific and thus having access to domain specific algorithms and methods. The database in question will have many properties which need to be handled in relation to their environment. For example, it is likely that there will be missing or blank features for entries in the database. The algorithm might have access to be able to call a function to compute missing entries, which is a highly domain specific solution not appropriate to general algorithms.
There are clever indexing schemes for quickly searching structural properties of molecules. One is substructure keys, in which binary flags about structural properties are stored for each molecule. Another is using a hash table encoding that serves as a proximity filter.
There is also the area of molecular similarity and molecular diversity. This field seeks to find similar molecules by noting differences in derived attributes. For example, it is computationally easy to compute the molecular weight of a molecule. This attribute, and others, can be used as a search therm to find similar molecules or correlation with more complex attributes such as molecular adsorption in repeating lattice structures.
Other features can also be noted and collected, providing a linear database for searching.
Overall, the question is whether to use structural similarity analysis (dynamic) or molecular diversity measures (linear) for the system.
The answer, at the current moment, appears to be using both.
We are currently not sure what variables correlate with performance in the desired chemical property of adsorption in a lattice structure, so having more features provides a greater possibility for correlation and more accurate estimates. The linear search could also be used to select candidates for dynamic search, using a hybridized preprocessing approach.
The second is that the system has the advantage of being domain specific and thus having access to domain specific algorithms and methods. The database in question will have many properties which need to be handled in relation to their environment. For example, it is likely that there will be missing or blank features for entries in the database. The algorithm might have access to be able to call a function to compute missing entries, which is a highly domain specific solution not appropriate to general algorithms.
Tuesday, September 28, 2010
Visualizing Datasets with Compass - Progress
Work is partially finished. Need to add different colors for TRUE/FALSE and gradiant for MRE.
Tuesday, September 21, 2010
Does Active Learning Work?
Here's work from yesterday. Unfortunately, I encountered several walls during my look at NASA93 before I begin working on all of the other datasets.
The first section of the document describes the variables and our ranking function for selecting devious children pairs.
The second section details everything I worked on yesterday and the pitfalls encountered.
Status of Compass Comparison Defect Prediction Experiments
I'm currently working my way through NASA and Softlab datasets shown in the table testing two Compass variations with different stopping rules to K-means with k = 1,2,4,8,16 and bisecting K-means with k = 4,6 and 8. I'm using leave-one-out and 66%/33% train-test slices repeated 20 times for cross-val.

The NASA datasets are still running, but ugly, non-formatted Softlab results are available in: http://www.unbox.org/wisp/var/kel/CompassDefectResults/

The NASA datasets are still running, but ugly, non-formatted Softlab results are available in: http://www.unbox.org/wisp/var/kel/CompassDefectResults/
Thursday, September 2, 2010
Flight of the MILL
Four brave researchers take time from their busy schedules to embark on a journey via helicopter that will bring them closer together than ever before. Their adventure can be now be shared thanks to the power of a Blackberry and YouTube.
A special thanks to Jacky Keung for the helicopter. Dr. Menzies will be lucky if he gets it back.
Tuesday, August 31, 2010
Problem with Stopping Rule in Compass Defect Prediction
I've found that the stopping rule for walking through the Compass tree stops too early. Compass currently stops when the variance of the node is less than the weighted variance of the node's children.
Example:
http://github.com/abutcher/compass/raw/master/doc/dot/defect/pruned/jm1.dot.png
In the defect prediction data sets, this condition occurs at a high level (usually 1 or 2). The result is that the cluster for majority voting is often most of the data set rather than similar instances.(if (< (node-variance c-node)(weighted-variance c-node))
(defun weighted-variance (c-node)(if (and (null (node-right c-node)) (null (node-left c-node)))(node-variance c-node)(if (or (null (node-right c-node)) (null (node-left c-node)))(if (null (node-right c-node))(node-variance (node-left c-node))(node-variance (node-right c-node)))(/ (+ (* (node-variance (node-right c-node))(length (node-contents (node-right c-node))))(* (node-variance (node-left c-node))(length (node-contents (node-left c-node)))))(+ (length (node-contents (node-right c-node)))(length (node-contents (node-left c-node))))))))
http://github.com/abutcher/compass/raw/master/trunk/src/lisp/variance.lisp
Example:
http://github.com/abutcher/compass/raw/master/doc/dot/defect/pruned/jm1.dot.png
Results from Splitting Data // Oracle
Results can be found HERE.
The System
1. Randomize 20x
2. Divide data into two halves.
a. Separate first half into eras of 10 instances.
b. Use the second half to build an oracle compass tree.
3. Using the eras from (2a), build an initial tree with the first several eras.
4. Incrementally insert the instances from eras (2a) into the compass tree formed in (3).
a. After each era find the most interesting instances by finding high variance children pairs and removing the center of their union.
b. Classify instances in the naughty list using the oracle from (2b).
c. Insert the naughty list back into the incremental tree formed in (3), (4) by using their classification information from (4b).
d. To keep the tree growing, re-compass high areas of variance (leaves in the top X% of variance among leaves).
6. Compare with a standard tree.
The System
1. Randomize 20x
2. Divide data into two halves.
a. Separate first half into eras of 10 instances.
b. Use the second half to build an oracle compass tree.
3. Using the eras from (2a), build an initial tree with the first several eras.
4. Incrementally insert the instances from eras (2a) into the compass tree formed in (3).
a. After each era find the most interesting instances by finding high variance children pairs and removing the center of their union.
b. Classify instances in the naughty list using the oracle from (2b).
c. Insert the naughty list back into the incremental tree formed in (3), (4) by using their classification information from (4b).
d. To keep the tree growing, re-compass high areas of variance (leaves in the top X% of variance among leaves).
6. Compare with a standard tree.
Wednesday, August 25, 2010
Compass demos
Andrew Butcher writes:
Once you have source from http://unbox.org/wisp/var/butcher/compass/tags/compass-0.0.1
cd compass-0.0.1/src
make demo1
etc...
They will all pipe to less, except demo10 which outputs maxwell.dot.png.
Once you have source from http://unbox.org/wisp/var/butcher/compass/tags/compass-0.0.1
cd compass-0.0.1/src
make demo1
etc...
They will all pipe to less, except demo10 which outputs maxwell.dot.png.
Tuesday, August 24, 2010
Pred Comparisons, Datasets and Bolstering
Pred(25) and loss values as well as comparison of algorithms&datasets of 3 points over all datasets are here.
New 30+ datasets for checking is here.
Bolstering: Is it inverse of boosting?
Boosting is supervised - Neither TEAK nor Compass is supervised
Boosting is a set of weak learners - TEAK and Compass are like cascading of one algorithm
Boosting weights the instances in a way (sending the misclassifieds to next weak learner) - TEAK and Compass also weights instances (0-1 weighting, keep or eliminate for the next round)
Boosting similar algorithms that are sometimes confused are called "leveraging algorithms" (need to read more on this)
New 30+ datasets for checking is here.
Bolstering: Is it inverse of boosting?
Boosting is supervised - Neither TEAK nor Compass is supervised
Boosting is a set of weak learners - TEAK and Compass are like cascading of one algorithm
Boosting weights the instances in a way (sending the misclassifieds to next weak learner) - TEAK and Compass also weights instances (0-1 weighting, keep or eliminate for the next round)
Boosting similar algorithms that are sometimes confused are called "leveraging algorithms" (need to read more on this)
Sunday, August 22, 2010
Agenda, August 24
Stuff in blue shows addits after posting0.
Stuff in green shows addits after posting1.
ALL:
Stuff in green shows addits after posting1.
ALL:
- need to name our desk. landscape paper, name in 30 point, photo.
- we now have 2 sets of ugrads students working the
FAYOLA:
- range experiment, and on other data sets.
- need that graphic max wanted showing the complexity of our data
- thesis outline (on paper).
ADAM1:
- can you attend? want to talk abut china and java W
ADAM2:
- all the latexing of the e,d,m and edm results done.
- ditto with the discretization experiments. outline of thesis.
- thesis outline (on paper)
- co-ordination with adam re java W
- need a revision of the paper that shows a good example of instability and
how it is less with W2
- receipt from japan
EKREM:
- when you used "log" pre-preprocessor, did you unlog afterwards before (say) MRE?
- check jacky's TSE corrections
- the abstract only mentions MRE. add others? or delete reference to MRE?
- check that our quotes from the paper in the review section are still accurate
- FIG5: when you checked (median(a) < median(b)), did you reverse that for pred?
- needs a bridge at end of 3.2. As can be seen in this example, it is not necessarily true that moving to a smaller set of neighbors decreases variance. As shown below, it can improve prediction accuracy if ABE takes this matter into account.
- bring (small) copies of ASE poster
- revist all old results. which need to be done with strong data sets?
- need your full passport name
- can we do ensembles just as a simple TEAK extension? just by randomly selecting from the current sub-tree 90%, 10 times?
ANDREW:
- seeking results on
- splitting data in half
- building a predictor (?teac) from one half
- walking the other one half in in (say) 10 eras
- after clustering the other half from era[i], find the most informative regions to query and send those queries to the predictor
- extend the era[i] cluster tree with the new results found during (c)
- test the new tree on era[i+i]
- increment "i" and goto (c)
KEL:
- photos, videos of the helicopter on the blog
- Effort Estimation Experiments with Compass and K-means
- Code is ready to go (Thanks in very large part to Andrew's previous work)
- Want to discuss experiments about effort estimation and defect predictions.
- Working on Matlab versions of TEAK and Compass to work with Ekrem's Matlab rig later.
- we now have 2 sets of ugrads students working the
FAYOLA:
- range experiment, and on other data sets.
- need that graphic max wanted showing the complexity of our data
- thesis outline (on paper).
ADAM1:
- can you attend? want to talk abut china and java W
ADAM2:
- all the latexing of the e,d,m and edm results done.
- ditto with the discretization experiments. outline of thesis.
- thesis outline (on paper)
- co-ordination with adam re java W
- need a revision of the paper that shows a good example of instability and
how it is less with W2
- receipt from japan
EKREM:
- when you used "log" pre-preprocessor, did you unlog afterwards before (say) MRE?
- check jacky's TSE corrections
- the abstract only mentions MRE. add others? or delete reference to MRE?
- check that our quotes from the paper in the review section are still accurate
- FIG5: when you checked (median(a) < median(b)), did you reverse that for pred?
- needs a bridge at end of 3.2. As can be seen in this example, it is not necessarily true that moving to a smaller set of neighbors decreases variance. As shown below, it can improve prediction accuracy if ABE takes this matter into account.
- bring (small) copies of ASE poster
- revist all old results. which need to be done with strong data sets?
- need your full passport name
- can we do ensembles just as a simple TEAK extension? just by randomly selecting from the current sub-tree 90%, 10 times?
- (the following will take some time)
- redo the icse paper's results with
- compass
- (sum, pred25, mre)
- win%,loss%, (win-loss)%
- leave-1-one, cross-val
- china broken up. jacky says it naturally divides into a few large chunks. chinaA, chinaB, etc.
- report the number of rank positions things change.
- i.e. what is expected wriggle in an algorithm's rankings
- a simple MSR paper http://2011.msrconf.org/important-dates
ANDREW:
- seeking results on
- splitting data in half
- building a predictor (?teac) from one half
- walking the other one half in in (say) 10 eras
- after clustering the other half from era[i], find the most informative regions to query and send those queries to the predictor
- extend the era[i] cluster tree with the new results found during (c)
- test the new tree on era[i+i]
- increment "i" and goto (c)
KEL:
- photos, videos of the helicopter on the blog
- Effort Estimation Experiments with Compass and K-means
- Code is ready to go (Thanks in very large part to Andrew's previous work)
- Want to discuss experiments about effort estimation and defect predictions.
- Working on Matlab versions of TEAK and Compass to work with Ekrem's Matlab rig later.
Tuesday, August 17, 2010
Defect Sets and GUI's
Defect Sets
Very large (19k pixels wide) Defect set Compass trees can be found --> HERE.
GUI
Working on a way to display variance in the Compass Explorer... either going with labels or colors --> HERE.
Changes of Note:
In-line pruning, along with square root min cluster size.
Very large (19k pixels wide) Defect set Compass trees can be found --> HERE.
GUI
Working on a way to display variance in the Compass Explorer... either going with labels or colors --> HERE.
Changes of Note:
In-line pruning, along with square root min cluster size.
Monday, August 16, 2010
Agenda, August 17
TIM
ADAM1
- news: the great MSR vs PROMISE marriage
- wboy tv coverage
- request: all on the blog
- need a new time for meetings
- GUI for HALE
ADAM1
- feel free not to attend but have you got a start date from ye yang?
- Please get with Adam2 regarding "W"
- Reaching data mining. got a copy of chapter 4 of witten?
- TSE rewrite (should have gone to Jacky by now)
- Next TSE article
- PRED(25) results for X=12, X=1, X=-10
- What is we build COMBA by just fliiping the pre-preprocessor?
- What is we build COMBA by just using 10 90% samples and log(K=1)?
- Boolstering vs boosting (just an idea)
- atteding ase. need to write a poster. see http://ai-at-wvu.blogspot.com/2010/08/how-to-write-posters.html
- new data sets to PROMISE
- travel to ase. you got visas to belgium
- seeking results on
- (a) splitting data in half
- (b) building a predictor (?teac) from one half
- (c) walking the other one half in in (say) 10 eras
- (c) after clustering the other half from era[i], find the most informative regions to query and send those queries to the predictor
- (d) extend the era[i] cluster tree with the new results found during (c)
- (e) test the new tree on era[i+i]
- (f) increment "i" and goto (c)
- note that by end august we will need at least one run of IDEA on a large defect data set (e.g. PC1)
- note by mid-oct we need some (possibly gawd awful) prototype to describe to GT
- what can we brainstorm as tasks for IDEA?
- Progress towards masters thesis?
- Got time to do experiments on IDEA (formally known as "compass")
- IDEA and TEAK in ekrem's matlab rig?
- Try defect prediction in IDEA?
- Try effort estimation with K-means, IDEA?
- word version
- need an example of error reduction in a medical domain. UCI?
- Outline for thesis. Checked in with Cox?
- Teaching LISP: do you have a copy of graham?
- decision regarding phd direction
- Outline for thesis. have you checked in with cox? update to locallessons.pdf table of results for coc/nasa 20 trials updated. right now, each cell presents one number. i want three written down as a,b,c where "a" is for just reducing effort, "b" is for just reducing detects and "c" is the current ones reducing effort,defects, time
- also, for that week, i need the discretization study repeated. some succinct representation showing that if we discretize to small N we do as well as larger N.
- then which of the following do you want to do?
- port it to your favorite language. python?
- build a web front end where it is easy to alter the goal function and the query. in python? in awk? check it: http://awk.info/?tools/server
- start writing your thesis
- do it all!
- get with adam1 regarding understanding W and a gui for china
- still need to sort out the payment receipts. got a publication date from jsea?
- what is your timetable for the nsf stuff
Sunday, August 15, 2010
How to write posters
- For a sample poster, see http://ourmine.googlecode.com/svn/trunk/share/pdf/poster.pdf
- You can write this using
- Latex (use "svn export http://unbox.org/wisp/var/09/pom2/doc/ase09/poster/pom2poster/ poster" then run "make")
- Powerpoint : layup one page in 5pt text. Won't look as good as the Latex, but some will find it easier.
- Open Office, etc (see the Powerpoint advice)
- Basement of library can print these; you can print non-gloss ($6) or full gloss ($12).
- For test prints, A3 sheets are readable, and cheap to produce
Subscribe to:
Posts (Atom)














