Tukutuku paper

The latest version of the paper, re-worded, re-positioned according to transfer learning viewpoint is here.

Ekrem Kocaguneli

Ekrem started his CS career at Bogazici University, where he received his BSc and MS degrees. Currently he is a PhD candidate in LCSEE department at West Virginia University. His main research interests are using machine learning methods to find solutions to difficult software estimation problems, analysis of various types of software data and investigating for stable conclusions in empirical software engineering. He currently maintains a personal web site (www.kocaguneli.com), where he posts his ideas about software engineering problems, announces new papers accepted and so on. 
A list of his publications are:

• Journal
• J. Keung, E. Kocaguneli, T. Menzies, “Finding Conclusion Stability for Selecting the Best Effort Predictor in Software Effort Estimation”, Automated Software Engineering, to appear in 2012.
• E. Kocaguneli, T. Menzies, A. Bener, J. Keung, “Exploiting the Essential Assumptions of Analogy-based Effort Estimation”, IEEE Transactions on Software Engineering, vol. 38, no.2, pp. 425-438, March-April 2012.
• E. Kocaguneli, T. Menzies, J. Keung, “On the Value of Ensemble Effort Estimation”, IEEE Transactions on Software Engineering, pre-prints, 2011.
• E. Kocaguneli, T. Menzies, J. Keung, “Kernel Methods for Software Effort Estimation”, Empirical Software Engineering Journal, pre-prints, 2011.
• A. Brady, T. Menzies, E. Kocaguneli, “What Is Enough Quality for Data Repositories?”, Software Quality Professional, 2011.
• Conference
• E. Kocaguneli, T. Menzies, “How to Find Relevant Data for Effort Estimation”, International Symposium on Empirical Software Engineering and Measurement (ESEM) 2011
• E. Kocaguneli, B.Caglayan, A. Tosun, A. Bener, “Experiences on Developer Participation and Effort Estimation”, EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA) 2011 
• E. Kocaguneli, G. Gay, Y. Yang, T. Menzies, “When to Use Data from Other Projects for Effort Estimation”, International Conference on Automated Software Engineering (ASE) 2010
• E. Kocaguneli, A. Tosun, A. Bener, “AI-Based Models for Software Effort Estimation”, 36th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA) 2010
• E. Kocaguneli, Y. Kultur, A. Bener, “Combining Multiple Learners Induced on Multiple Datasets for Software Effort Prediction”, Proceedings of International Symposium on Software Reliability Engineering (ISSRE) 2009.
• E. Kocaguneli, A. Tosun, A. Bener, B. Turhan, B. Caglayan, “Prest: An Intelligent Software Metrics Extraction, Analysis and Defect Prediction Tool”, Proceedings of International Conference on Software Engineering and Knowledge Engineering (SEKE) 2009.
• Ayse Tosun, Ayse Bener, Ekrem Kocaguneli, “BITS: Issue Tracking and Project Management Tool in Healthcare Software Development”, Proceedings of International Conference on Software Engineering and Knowledge Engineering (SEKE) 2009.
• A. Bakir, E. Kocaguneli, A. Tosun, A.Bener and B. Turhan “Xiruxe: An Intelligent Fault Tracking Tool”, Proceedings of International Conference on Artificial Intelligence and Pattern Recognition (AIPR) 2009.
• Y. Kultur, E. Kocaguneli, A. Bener, “Discovering Cost Related Features: Focus on Classification Models”, Poster Paper in Empirical Software Engineering and Measurement (ESEM) 2009.
• E. Kocaguneli, A. Tosun, B. Caglayan, A. Bener, T. Aytac, B. Turhan, “Bulutlarda Akilli Bir Yazilim Olcumleme, Hata Analiz ve Tahmin Araci: Prest”, (in Turkish), 2nd National Symposium on Software Engineering (UYMS) 2009
• Y. Kultur, E. Kocaguneli, A. Bener, “Domain Specific Phase by Phase Effort Estimation in Software Projects”, Proceedings of International Symposium on Computer and Information Sciences (ISCIS) 2009.

Tukutuku paper

The first set of corrections are here.
The second set of corrections are here.
The final version of the paper after these corrections are here.

Active Learning on Defect Paper

I ran all the data sets through weka's Bayes and Random Forest learners. The g-values produced by weka are very close to the g-values given by my implementation, but I still kept the g-values of weka. The figures are placed together in an under-construction version of the paper.

What does zero test tell us about the SEE world?

Here is a detailed report using the Blom correction.

Current Status of Papers

In the Pipeline

• Active learning paper
• Bias variance paper
• LOC comparison paper

In reply process

• Theoretical Maximum Prediction Accuracy paper

To be written (with experiments ready)

• Active learning for defect
• Tukutuku paper

LOC-compensation Paper

The latest version of the paper is here. There is still a lot of polishing to do, but the general frame of the paper tells our story (particularly the abstract).

Grammatech Report

I added in active learning literature review of 6 pages.
2 more pages of motivation regarding why our active learning solutions matter.
The other details of the progress (also 2 questions) is here.
The report's latest version is here.

Possibility of Effort Estimates without Software Size

The detailed results are:

• Only instance selection: 
• Quick vs. 1NN: 2 losses out of 13 data sets
• Quick vs. CART: 4 losses out of 13 data sets
• Only feature selection:
• Quick vs. 1NN: 4 losses
• Quick vs. CART: 6 losses
• Both feature and instance selection:
• Quick vs. 1NN: 3 losses
• Quick vs. CART: 5 losses

The paper (under progress) is here.

Taking Size Out of Data Sets

For the time being I have cocomo, nasa, sdr and desharnais data sets.
The comparison of with-size vs. without-size results for the top13 solo-methods are here.

ASE paper

The last version of the paper with new results and text for the sanity check section (see pages 15 to 18) is here.

We can submit(?) within this week (after a proof-read and a little tweaking).

Final Report

A report with a total of 27+4(for references) pages on our side is here.

How would data cluster 90 solo-methods

This ad-hoc procedure uses 2 statistical test

• Wilcoxon Rank Sum Test: To compare 2 arrays of MRE's
• Kruskal-Wallis: To compare n arrays of MRE's

The procedure for each data set can be summarized as follows:

1. Sort the solo-methods by their MdMRE's
2. Start with j'th group (initially j=1) and place the i'th solo-method into this group
3. Compare MRE's of i'th and i+1'th solo-method w.r.t. Wilcoxon
4. Compare MRE's of all solo-methods in j'th group and MRE of i+1'th w.r.t. Kruskal-Wallis
5. If step3 and step4 both agree that i+1'th method is statistically the same, then place it into j'th group
6. Else (if one of step3 or step4 says i+1'th is statistically different), then increment j by 1, i.e. form a new group.
7. i = i+1;
8. Go to step2

The results of the above procedure are incorporated into the jiggle paper on p16-17.
Concise summary of results:

• Some dataset are awfully flat without any ability to form a second group: telecom and kemerer.
• The datasets that show a good distribution of the solo-methods to their groups is around half the data sets: cocomo81e, nasa93_center_5, desharnais, nasa_93_center_2, sdr, finnish, cocomo81, miyazaki94, nasa93
• The occurrence of the top13 solo-methods within these groups still proves our point, i.e. they are mostly on the top group. However, for certain datasets (desharnaisL3, cocomo81o) the generality does not hold, and the top13 solo-methods occur in lower-performance groups.
• The above result shows that, the generality that we have found from the aggregate of 7 error measures and 20 data sets via win-tie-loss values, still holds when we look at the specifics (just using the MRE's w.r.t. another procedure based on multiple statistical tests). However, we still cannot say that there will be no exceptions due to data sets like desharnaisL3 and cocomo81o.

This week

We are done with the active learning paper (waiting for reviews). Submission this week.

Read all the available information on Grammatech and placed them here.

• Question: I have only 3 reports... More stuff to read for me to catch up?

Started writing the report. The draft is here.

• Question: How to manage the citations? Mendeley is pretty easy to use (see the draft).

Active Learning Paper and Tukutuku

Active learning paper is here.

The tukutuku stuff (ongoing) is here.

Intrinsic Complexity of SEE

How complex should a learner working on SEE data sets be? Our intuition is that there is a need for complex learners, if the data set has enough complexity to it, i.e. if the data set can be divided into complex and self-contained regions. On the other hand, if most of the data is noise and if the data sets can be summarized with only a small number of instances and features, then there is no need for complex learners. The updated version of the QUICK paper investigates this question. The new version with all the results/figures updated is here.

Active Learning for Defect Data

Here is a nice report about the active learning experiments.

1. Bias Variance Analysis
    a) Anova -> Bias is 8/20, Variance is 14/20
    b) Scott&Anova -> Bias is 8/20, Variance is 15/20
    c) Anova plus Cohen's correction: Bias is 8/20
    d) Kruskal-Wallis not so good: Bias is 4/20 and variance is 10/20
 
    e) Sorted bias values of all the data sets plotted are here.
           The above plots showed us that there are extreme values due to new experimental conditions.
    f) The Friedman-Nemenyi plots are here for separate data sets.
    g) The same test as before for the entire 20 data sets together are here.
    h) Re-generate boxplots with the new experimental conditions.

I applied a correction on the analysis, because...
Here is the paper with new boxplots..
Here is the Friedman Nimenyi plots on separate data sets, bias is the same for 15 out of 20 data sets..
Below is the result of Friedman Nimenyi over all data sets, i.e. if we see it as 20 repetitions.

2. Papers
    a) HK paper is revised and is ready to be seen here.
    b) Active learning paper's major revision is submitted to TSE.
    c) Camera ready versions of ensemble and kernel papers are submitted to TSE and ESEM.

3. The break-point experiments

Reviews and unsupervised feature-selectors

a) Papers: In case you want to take a look at some of the recent reviews we have submitted, here are two papers: 1) kernel density estimation and 2) ensemble methods.

Another one with 40% of the reviews done is about active learning (here).

b) QUICK+ (under implementation): An unsupervised instance and feature selector, which works in two steps:

• Step1: Run QUICK on the dataset and find how many instances (q) you need.
• Step2: 
• For q instances, transpose the dataset and find the popularity of features in a space defined by instances. 
• Throw away features from least popular to most popular until there is a statistically significant deterioration in performance.
• Stop when performance drops significantly and put the last-thrown feature back.

Still reviewing..

Finished the reviews of the Comba paper minor revision for TSE. The paper is here. I still need a proof-read.

Submitted version of the kernel paper to EMSE is here.

Active learning paper major revisions for TSE are in progress..

Discussion on the TSE paper asserting that OLS is the best.

Experiments: 1) unsupervised feature selectors, 2) different ranking methods.