Open Issues in Empirical SE

Q1: Generality of the Microsoft conclusions?

• See "Myths in Software Engineering: From the Other Side" by Nachiappan Nagappan, TAPS'10
• A list of 10 conclusions made from Msoft research
• The validity of these, for sites outside of Msoft, is an open issue. 

Q2: How to support the following kinds of business intelligence?

• Click to enlarge (from http://research.microsoft.com/apps/pubs/?id=144543):

See also: http://www.slideshare.net/tom.zimmermann/analytics-for-software-development

t

Other approaches to business intelligence:

• Watch my recent talk:  (250MB) : http://menzies.us/empiricalSEversion2.wmv
• Read my recent talk  (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx
• Or see below

Talk at Microsoft Research, Jan 27, 2011

Talk at Microsoft Research, Jan 27, 2011

Slides (7MB) : http://unbox.org/wisp/var/timm/10/swinburne/bi-v5.pptx

Download (250MB) : http://menzies.us/empiricalSEversion2.wmv

The rapid pace of software development innovation challenges empirical software research to keep up, if it is to deliver actionable and useful results to practitioners. The empirical software engineering research field has not always been able to deliver this. Recently, it has become increasingly apparent that rigorous data collection and analysis can be so expensive and time-consuming that empirical software engineering studies, which seek to understand the costs and benefits of software development solutions in practice, greatly lag the pace of innovation in the field. In too many cases, a trusted body of empirical results can only be built up after the innovative solutions that they are studying are already well on their way to obsolescence or standard practice. However, we argue that recent advances put a sustainable and increased research pace within our reach. A suitably scaled-up and nimble empirical research approach must be based upon the “crowd-sourcing” of tough empirical problems. 

empirical software engineering, v2.0

View more presentations from Tim Menzies.

Will McBurney

Currently a graduate student in Computer Science at West Virginia University, Will McBurney is a programmer, musician, and blogger on the side. Will McBurney began programming in his Senior year of High School in Charleston, WV, his hometown. Originally attending WVU as an undergraduate aspiring to become a Mechanical Engineer, Will switched majors midway through his freshmen year to Computer Science.

It is not secret Will is a major nerd. Will skipped his Senior Prom to go to National Science Bowl in 2006 and has read more Star Wars books than most people have read at all. On the inter-tubes, Will uses the psuedonym "Death by Smiley", a name that was made up spur of the moment while playing Halo in the dorms at WVU.

-Will McBurney

Active Learning

Kel w/ Cliff:

http://www.unbox.org/wisp/var/kel/doc/cliff/results.txt

Andrew w/ Heuristic Randomness:

http://afrolegs.com/kc1_eraSize150.png

http://unborx.org/wisp/var/butcher/active/

Rank Deltas for COMBA

http://unbox.org/wisp/var/bill/comba/reports/rankdeltas.png

Which features define CC?

Division of available datasets into subsets according to different criteria (language, hardware, resource etc.) may define closeness of a cross-company setting. The results and subsets are here.

Bias and Varince of Testing Strategies on Effort Datasets

A short report is here.

Another look at Fayola's data

http://unbox.org/wisp/var/11/pretty/forensics.pdf

map-reduce & Hadoop

Map-Reduce

MapReduce is a patented^[1] software framework introduced by Google to support distributed computing on large data sets on clusters of computers.^[2]

MapReduce libraries have been written in C++, C#, Erlang, Java, Ocaml, Python, Ruby, F#, R and other programming languages.

http://en.wikipedia.org/wiki/MapReduce

Appache HadoopOpen-source system layer for map-reduce. Implements (e.g.) a distributed file system

Material for interactive new media design

http://forgottenlegacywwi.org/exhibit.html

popularity is (almost) a perfect predictor for defects

http://unbox.org/wisp/var/11/popular/popular-v6.pdf

One surprising observation from the Helix study [50] was that(a) most classes are not popular and (b) patterns of the most pop-ular classes emerged very early in the lifetime of an open-sourceproject. Specifically, popularity and age of a class maintain a pos-itive monotonic relationship throughout the lifetime of a system[50], that is, as a system matures, popular classes tend to becomeeven more popular. 

This contradicts our reading of Fowler [20] andBeck’s work [6] that object-oriented quality assurance requires in-volved constant refactoring (e.g., Beck’s TDD loop of write tests,run test, refactor). We cannot see evidence of constant, widespreadrefactoring in open-source projects studied by Helix [50].

The implications of this lack-of-refactoring for the theory of object-oriented development is discussed elsewhere [50]. This paperfocuses of class popularity since (a) it is a stable concept for thelifetime of an object-oriented system and (b) popularity can lead todefects via:

• Defect injection:As developers work with the popular classes, they make the occasional mistake. Some of these mistakesresult in code defects. Since developers work on popular classes more than other classes, then most developer defects accumulate in the popular classes.
• Defect discovery: Since developers work mostly on popular classes, they are most likely to uncover those classes’ defects.

We demonstrate that in 33 open-source Java projects popularity-based defect predictors work within 4% of a theoreti-cal upper bound on predictor performance (this is the basis for ourclaim that such predictors are “nearly perfect”).  

empirical SE v2.0

timm, trying to map out a research agenda for the next decade

empirical software engineering, v2.0

View more presentations from Tim Menzies.

More on story content generation

http://grandtextauto.org/2008/02/19/ep-52-a-tale-spin-story/

What makes a game fun

Hello, hi,

My name is Joe and I'm a new PhD student at WVU, got my MS in CS just last year, and working towards the big goal with a vague idea and a genuine interest for AI in games. I'm hard of hearing, and generally have a tough time understanding people in a group setting, but usually if there's only one person speaking -towards- me, with a good voice, then it's fine. (Sorry if I seem confused sometimes, usually trying to figure out what you just said.)

Last semester I worked on the idea of "What makes a game Fun?" and found myself digging through a lot of unofficial resources towards concepts in a game that heighten the level of enjoyment had by players. I wrote a survey-type paper and outlined what I thought were the core values of "Fun in video games." Those four were Gameplay, Originality, Replayability and Story (GORS). Replayability is key; any time a player picks up a controller, there is a drawing factor that keeps them attracted to the game and continuously replaying. Once that allure is lost, replayability is lost, and the game no longer has potential to provide fun (at least for a while, but there's always a chance the allure returns at some later point - yet another key point to replayabilty.)



Anyway, second half of semester I put the four dimensions of fun together in a game which the user could play and "set" the levels of each dimension. i.e., the user would set originality to be of "low" setting, making item names and enemies sound very generic, as opposed to original and unique. Called Cre-Ativity, the game plays like Zelda, in a top-down fashion developed with OpenGL in C++. Players can use one of four weapons; acquire armor, gain experience, become stronger and bash enemies while exploring in aims of completing a general story-driven quest.

At this point, the semester was over and one of the key questions that remained was; "How could you develop story content according to user-input?" More on that topic to come.


Screenshots:
https://sites.google.com/site/tanzastic/research/s01.jpg
https://sites.google.com/site/tanzastic/research/s02.jpg
https://sites.google.com/site/tanzastic/research/s03.jpg
https://sites.google.com/site/tanzastic/research/s04.jpg
https://sites.google.com/site/tanzastic/research/s05.jpg

Cross vs. Within Company Experiments

Tentative results are here.

Selected cross-company datasets are given below..

finnishAppArea = {'finnishAppArea2','finnishAppArea7','finnishAppArea10'};

finnishAppType = {'finnishAppType1','finnishAppType2345'};

kemererHardware = {'kemererHardware1','kemererHardware23456'};

maxwellAppType = {'maxwellAppType1','maxwellAppType2','maxwellAppType3'};

maxwellHardware = {'maxwellHardware2','maxwellHardware3','maxwellHardware5'};

maxwellSource = {'maxwellSource1','maxwellSource2'};

chinaResource = {'chinaResource1','chinaResource2','chinaResource3','chinaResource4'}

Questions:

Is 1*LOO ok or do I need 20*LOO?

Is using only static-k better? 

Testing using only the X dimension in IDEA

http://www.unbox.org/wisp/var/butcher/dragons/x_validity_test.pdf

New results after discretizing with Fayyad Irani

http://unbox.org/wisp/var/fayola/forensics/results/

This week...

Tim has written two subjects

Learned about map-reduce and clojure and wondered about applying it to WHICH.

Coded up idea, knn, cross-val, and run lots of experiments

Worked with physics on an NSF proposal

Planned next week's talk at microsoft, redmond

Worked with Martin Shepperd on a journal special issue.

Worked with Ayse Bener on a journal special issue

And written a research paper

Over to you....

Fayyad Irani Discretization for Row Pruning Using Variance

New results for reducing variance with W: here