Tuesday, November 19, 2013

Version Tracking Visualization

Results 1/21/14


Results of A/B/C/D prediction: dismal


Results 2:



Back to the CSV: class names are listed

['ant-1.3.csv', 'ant-1.4.csv', 'ant-1.5.csv', 'ant-1.6.csv', 'ant-1.7.csv']
Type A: 4%    B: 11%    C: 12%    D: 71%    NoMatch: 0%
Type A: 3%    B: 17%    C: 8%    D: 63%    NoMatch: 5%
Type A: 5%    B: 5%    C: 18%    D: 69%    NoMatch: 0%
Type A: 17%    B: 8%    C: 15%    D: 58%    NoMatch: 0%

['camel-1.0.csv', 'camel-1.2.csv', 'camel-1.4.csv', 'camel-1.6.csv']
Type A: 3%    B: 0%    C: 22%    D: 51%    NoMatch: 22%
Type A: 15%    B: 18%    C: 3%    D: 55%    NoMatch: 6%
Type A: 9%    B: 7%    C: 10%    D: 71%    NoMatch: 1%

['ivy-1.1.csv', 'ivy-1.4.csv', 'ivy-2.0.csv']
Type A: 7%    B: 47%    C: 2%    D: 40%    NoMatch: 1%
Type A: 0%    B: 0%    C: 0%    D: 0%    NoMatch: 100%

['jedit-3.2.csv', 'jedit-4.0.csv', 'jedit-4.1.csv', 'jedit-4.2.csv', 'jedit-4.3.csv']
Type A: 17%    B: 15%    C: 5%    D: 58%    NoMatch: 2%
Type A: 16%    B: 7%    C: 9%    D: 62%    NoMatch: 4%
Type A: 9%    B: 15%    C: 3%    D: 64%    NoMatch: 6%
Type A: 0%    B: 11%    C: 0%    D: 47%    NoMatch: 38%

['log4j-1.0.csv', 'log4j-1.1.csv', 'log4j-1.2.csv']
Type A: 16%    B: 6%    C: 8%    D: 41%    NoMatch: 27%
Type A: 30%    B: 1%    C: 56%    D: 5%    NoMatch: 5%

['lucene-2.0.csv', 'lucene-2.2.csv', 'lucene-2.4.csv']
Type A: 33%    B: 12%    C: 24%    D: 28%    NoMatch: 1%
Type A: 42%    B: 15%    C: 21%    D: 15%    NoMatch: 4%

['synapse-1.0.csv', 'synapse-1.1.csv', 'synapse-1.2.csv']
Type A: 5%    B: 4%    C: 22%    D: 63%    NoMatch: 3%
Type A: 13%    B: 12%    C: 19%    D: 53%    NoMatch: 1%

['velocity-1.4.csv', 'velocity-1.5.csv', 'velocity-1.6.csv']
Type A: 40%    B: 34%    C: 2%    D: 2%    NoMatch: 20%
Type A: 26%    B: 37%    C: 3%    D: 29%    NoMatch: 2%

['xalan-2.4.csv', 'xalan-2.5.csv', 'xalan-2.6.csv', 'xalan-2.7.csv']
Type A: 9%    B: 4%    C: 36%    D: 44%    NoMatch: 4%
Type A: 27%    B: 20%    C: 15%    D: 31%    NoMatch: 4%
Type A: 44%    B: 0%    C: 51%    D: 1%    NoMatch: 2%

['xerces-1.2.csv', 'xerces-1.3.csv', 'xerces-1.4.csv']
Type A: 3%    B: 11%    C: 10%    D: 72%    NoMatch: 1%
Type A: 7%    B: 0%    C: 38%    D: 25%    NoMatch: 27%

Idea: New dataset consisting of:

  • All attributes of N
  • All attributes of N+1
  • The delta between N and N+1
  • Class of defect change

Result1


  • Preliminary feature selection with info gain selecting top 50%
  • Normalized and discredited with Fayyed-Irani
  • PCA via FastMap
  • Grid clustering
  • Centroids plotted along with version n+1 nearest neighbor lines. (Not terribly useful)
  • Do I smell transforms of best fit around the corner?











Results0

k-means 5 to cluster each data-set within itself
Eigenvalues used to determine select features with most influance
Actual selected columns are plotted, not synthesized dimensions
     -- significant correlations could be reported as synonmyms
rules for connecting the dots?







Monday, November 11, 2013

Tree query languages for MOEA

Method

  1. Cluster the data
  2. Find deltas of interest between the clusters
    • Score each cluster
      • Let each row have objective scores, normalized 0..1, min..max
      • Let the score of a row by the sum of the normalized scores
      • Let the score of a cluster be the mean of the score of its rows
      • Technically, this is almost the cdom predicate used in IBEA
    • For each cluster C1
      • Find its nearest neighbor C2 with a better score 
      • Assert one  (leave,goto)  tuple for  (C1,C2)
  3. Build and prune a decision tree on the clusters
    • Label each instance with the cluster it belongs to
    • Build a decision tree on the labelled data set.
    • Find the clusters that are only weakly recognized by the decision tree learner
      • e.g. use a three-way cross val and prune anything with F < 0.5
    • Remove the weakly recognized clusters
  4. For each (C1,C2) tuple where both are not weakly recognized, 
    • Query the tree to find the delta
Observation: the trees are so small that this can be done manually.

Example

Nasa93 clustered into 2D (one color per cluster)

Cluster details 
  • All the following values are normalized 0..1, min..max
  • Defects and months are connected, but not always
  • Effort is not what separates the projects- its more about defects and calender time to develop
  • Clearly, cluster 2 is a bad place and 10 and 13 look nicest.


A decision tree learned from the data labelled with each cluster (ignoring the objectives) generated:

 acap = h
|   apex = h
|   |   pmat = h
|   |   |   plex = h: _2 (6.0/1.0)
|   |   |   plex = n: _4 (3.0)
|   |   pmat = l
|   |   |   cplx = vh: _3 (2.0)
|   |   |   cplx = h
|   |   |   |   time = vh: _3 (3.0)
|   |   |   |   time = n: _6 (4.0/1.0)
|   |   |   cplx = n: _5 (2.0)
|   |   pmat = n: _6 (4.0/1.0)
|   apex = n
|   |   data = h: _6 (2.0/1.0)
|   |   data = n: _4 (3.0/1.0)
|   |   data = l: _13 (1.0)
|   apex = vh
|   |   pcap = h: _10 (3.0)
|   |   pcap = vh: _7 (2.0/1.0)
acap = n
|   sced = n
|   |   stor = xh: _7 (1.0)
|   |   stor = n
|   |   |   cplx = h
|   |   |   |   pcap = h: _10 (3.0/1.0)
|   |   |   |   pcap = n: _13 (3.0)
|   |   |   cplx = n: _7 (3.0/1.0)
|   |   stor = vh: _11 (3.0)
|   |   stor = h: _11 (2.0)
|   sced = l
|   |   $kloc <= 16.3: _9 (5.0)
|   |   $kloc > 16.3: _8 (6.0)
acap = vh: _12 (7.0/1.0)

A 3-way cross-val yielded following confusion matrix. 
  • The underlined and bold entries are the correctly classified rows.
  • The red entries are errors.
  •  Note the poor performance for recognizing clusters 4,5,6,7,10,13

 a b c d e f g h i j k l   <-- as="" classified="" font="">
 5 0 0 0 0 0 0 0 0 0 0 0 | a = _2
 0 4 1 1 0 0 0 0 0 0 0 0 | b = _3
 1 0 2 0 1 0 0 0 1 0 0 0 | c = _4
 0 1 1 0 3 0 0 0 0 0 0 0 | d = _5
 0 1 2 3 0 0 0 0 1 0 0 1 | e = _6
 0 0 0 0 0 0 0 0 1 2 1 1 | f = _7
 0 0 0 0 0 0 6 0 0 0 0 0 | g = _8
 0 0 0 0 0 0 1 4 0 0 0 0 | h = _9
 0 0 1 0 0 0 0 0 3 0 0 2 | i = _10
 0 0 0 0 0 1 0 0 0 4 0 0 | j = _11
 0 0 0 0 0 1 0 0 0 0 6 0 | k = _12
 0 0 1 0 0 0 0 0 1 1 0 2 | l = _13


The above confusion matrix is mapped into the "f" measures of the following table. 
  • The "goto" column marks the deltas of interest.
  •  Low "f" values are marked in gray. 
  • Any "goto" that comes or goes into gray is marked with gray.

cluster n effort defects months f     goto
2 5 43% 25% 72% 91% 3
3 6 5% 32% 42% 67% 6
4 5 6% 17% 37% 31% 8
5 5 7% 29% 43% 0% 6
6 8 6% 24% 40% 0% 10
7 5 7% 16% 36% 0% 13
8 6 2% 6% 17% 92% 9
9 5 0% 1% 3% 89%
10 6 2% 9% 22% 46% 12
11 5 7% 18% 31% 67% 13
12 7 2% 7% 18% 86%
13 5 7% 15% 26% 36%
total: 68




 
If we prune the above tree of any branch that leads only to gray classes, we get, as promised above, a very small tree.

acap = h
|   apex = h
|   |   pmat = h
|   |   |   plex = h: _2 (6.0/1.0)
|   |   pmat = l
|   |   |   cplx = vh: _3 (2.0)
|   |   |   cplx = h
|   |   |   |   time = vh: _3 (3.0)
acap = n
|   sced = n
|   |   stor = vh: _11 (3.0)
|   |   stor = h: _11 (2.0)
|   sced = l
|   |   $kloc <= 16.3: _9 (5.0)
|   |   $kloc > 16.3: _8 (6.0)
acap = vh: _12 (7.0/1.0)

Summary

  • The definite statements that clearly make changes in SE data are very succinct.
    • But they might not cover everything.
Question: what would you baseline this against? I.e. how would you certify this as a good/crappy idea?

Monday, October 28, 2013

Short aptamer presentation

Basic Presentation for DM and Aptamers
for NanoSAFE fellows on Oct 31,2012
Target: General audience

Monday, October 21, 2013

DataSet ID Comparison

All comments based on KNN performance

http://unbox.org/things/var/fayola/docs/newcliff-v6.pdf
fig. 5

Data Set---ID----- Comment
labor-neg :: 0.68     Noticeable Downtrend
glass        :: 0.75     Down Trend with some improvement at 20%
iris           :: 1.11     Flat
hepatitis   :: 2.45     Noticeable Downtrend on ACC, Prec Not Convinced PD, PF
ecoli        :: 3.27      Flat (acts like Software sets for pd, prec, pf)
bcancer   :: 0.00      No Results (unsure of source)
heartc     :: 0.00       No Results(restricted)
lymph      :: 0.00      No Results (restricted)
vote        :: 0.00       No Results

Note: I am interested to see what the UCI Data Sets look like with Error Bars to see if they look similar in any way to the Software Sets

http://unbox.org/things/var/brian/2013/projects/data-quality/mucker2/output/10-7-2013%20plots/
----------------
xerces1.4:: 0.01    Flat/Not Convinced
poi-3.0   :: 1.45     Flat/Not Convinced
ivy-1.1   :: 1.74     Flat/Not Convinced
synap1.2:: 1.89     Flat/Not Convinced
xalan2.6 :: 1.90     Flat/Not Convinced
jedit-4    :: 2.03     Flat/Not Convinced
ant-1.7   :: 2.08     Flat/Not Convinced
log4j-1.1:: 2.67     Flat/Not Convinced
lucene2.4:: 2.94    Flat/Not Convinced
veloci1.6:: 3.00     Flat/Not Convinced


TODO:
*Create Random DataSets as per "Reflections on the NASA MDP data sets" - D.Gray
*Get My (Vasil's) rig running with the UCI data sets to see if we get similar results?      
*Find sources for [bcancer, lymph, vote]                                                                    
*Get to the bottom of why the noise is not a problem, if ID does not explain it.
*????            

Tuesday, October 8, 2013

UPC Find

Update 5/01/14

Nutritional Facts information: we've got it for about 30% of the items.

                                  Unique UPCs | Total Items
Items matching Walmart.com      :  5436 34.3% |  5904 35.0%
Items with Walmart nutrition    :  4514 28.5% |  4940 29.0%
                     'Sodium'   :  4540 28.7% |  4972 30.0% 
         'Total Carbohydrate'   :  4540 28.7% |  4973 30.0% 
                    'Protein'   :  4514 28.5% |  4940 29.8% 
                  'Total Fat'   :  4512 28.5% |  4949 29.9% 
                     'Sugars'   :  3924 24.8% |  4251 25.7% 
              'Saturated Fat'   :  3778 23.8% |  4228 25.5% 
                'Cholesterol'   :  3744 23.6% |  4175 25.2% 
                  'Trans Fat'   :  3446 21.7% |  3849 23.2% 
              'Dietary Fiber'   :  3296 20.8% |  3503 21.1% 
                  'Potassium'   :  1462 9.2% |  1506 9.1% 
                   'Calories'   :   614 3.9% |   729 4.4% 
                    'Dietary'   :   360 2.3% |   567 3.4% 
        'Monounsaturated Fat'   :   274 1.7% |   333 2.0% 
                           ''   :   250 1.6% |   248 1.5% 
                  'Saturated'   :    64 0.4% |    69 0.4% 
                      'Trans'   :    64 0.4% |    54 0.3% 
           'Sugars Less Than'   :    12 0.1% |     9 0.1% 
                      'Total'   :     8 0.1% |     4 0.0% 
    'Dietary Fiber Less Than'   :     6 0.0% |     4 0.0% 
    'Monounsaturated Fat 1.5'   :     2 0.0% |     1 0.0% 
            'Dietary Fiber 2'   :     2 0.0% |     7 0.0% 
Items matching upcdatabase.com  :  4344 27.4% |  5125 30.0%
Items matching local UPC list   :  2594 16.4% |  3046 18.0%

Items matching ONLY Walmart.com :  3352 21.2%  |  3606 21.0%
Items matching ONLY upcdatabase :  3352 21.2%  |  3606 21.0%
Items matching ONLY local list  :  3352 21.2%  |  3606 21.0%
Items matching all three sources:  1474 9.3%  |  1717 10.0%
Items matching any source       :  7886 49.8%  |  8891 53.0%

Update 3/27/14

Criteria for match success: must match at ONE of the following:
-Walmart.com result
-UPC database result + NDB match
-Food database result + NDB match


Graphs (some of these are repeats from lost blog posts):


Tabular Results:
top 1000 items



Update--Web Sources for Barcode Lookup:





Summary:

  • Of the 50 Items scanned from my fiance's kitchen, a match was found for 29 items (58%).  
  • If beverages are removed from the list, the match rate for this set of scans jumps to 28/40 (70%). 
  • An interesting observation: in this set of scans, ALL store-brand products failed to match.
  • Partial matching of UPCs yielded poor results
  • For this set, 81% of the items which failed to match had a brand code which matched to something in the database. Perhaps brand could be used in some circumstances to make an educated guess about the properties of the item if a match cannot be made.
    • positive example: Florida's Natural (low brand variance)
    • negative example: Sam's Choice (high brand variance)


UPCA == TMMMMMPPPPPX
where T is type (0 for US UPC)
           M is manufacturer code
           P is product code
           X is check digit

1) UPCA => Jif Peanut Butter
2) UPCA => Nutella Hazelnut Spread with Skim Milk & Cocoa
3) UPCA => No Match 078742095233
4) UPCA => No Match 011110833303
5) UPCA => No Match 041498127824
6) UPCA => No Match 044000031138
7) UPCA => Bush's Best Baked Beans
8) UPCA => Kraft Dinners Easy Mac
9) UPCA => No Match 038000844966
10) UPCA => Campbell's Pasta
11) UPCA => Betty Crocker Instant Potatoes
12) UPCA => Campbell's R&W Condensed Soup
13) UPCA => No Match 085000016176
14) UPCA => No Match 085000019894
15) UPCA => No Match 081172780006
16) UPCA => Smart Balance Cooking Spray
17) UPCA => No Match 031200002945
18) UPCA => No Match 078742351896
19) UPCA => Kraft Philadelphia Cream Cheese Spread
20) UPCA => No Match 016300151304
21) UPCA => No Match 070847000037
22) UPCA => Coca-Cola Cola
23) UPCA => No Match 087692591009
24) UPCB => UPCA 8857378 => 088573000078
24) UPCA => No Match 088573000078
25) UPCA => Sweet Baby Ray's Barbecue Sauce
26) UPCB => UPCA 1364008 => 013000006408
26) UPCA => Heinz Ketchup
27) UPCA => French's Classic Yellow Mustard
28) UPCA => Hellmann's Mayonnaise
29) UPCA => DiGiorno Pizza & Breadsticks
30) UPCA => Birds Eye Steamfresh Corn
31) UPCA => No Match 011110673565
32) UPCA => Lance Toastchee
33) UPCA => No Match 078742434377
34) UPCA => McCormick Grill Mates Seasoning
35) UPCA => Sun Chips Flavored Multigrain Snack
36) UPCA => Doritos Tortilla Chips
37) UPCA => No Match 050000497256
38) UPCA => Smucker's Preserves
39) UPCA => No Match 011110786715
40) UPCA => Tostitos Dip
41) UPCA => Prego Italian Sauce
42) UPCA => Quaker Oatmeal Instant Oatmeal
43) UPCA => Swiss Miss Hot Cocoa Mix
44) UPCA => Swanson RTS Broth
45) UPCA => No Match 078742030104
46) UPCA => No Match 011110492630
47) UPCA => Betty Crocker Loaded Mashed
48) UPCA => No Match 072736014880
49) UPCA => Duncan Hines Cake Mix
50) UPCA => Knorr Side Dishes Fiesta Sides
match: 29/50 58.0%


Items not found:
Note: one-off, two-off are the number of database items which differ by one or two characters

     UPCA    | FoundBrand | one-off | two-off | type (* indicates store-brand)
078742095233 |     T      |    0    |    2    |    trail mix*
011110833303 |     T      |    0    |    0    |    trail mix*
041498127824 |     T      |    0    |    0    |    baking soda*
044000031138 |     T      |    0    |    9    |    ritz snack packs
038000844966 |     T      |    0    |    0    |    pringles
085000016176 |     F      |    0    |    2    |    gin
085000019894 |     F      |    0    |    0    |    wine
081172780006 |     T      |    0    |    0    |    gummy candy
031200002945 |     T      |    0    |    18  |    crasins
078742351896 |     T      |    0    |    2    |    skim milk*
016300151304 |     T      |    0    |    4    |    orange juice
070847000037 |     T      |    0    |    2    |    energy drink
087692591009 |     F      |    0    |    0    |    beer
088573000078 |     F      |    0    |    0    |    beer
011110673565 |     T      |    0    |    0    |    jello*
078742434377 |     T      |    0    |    1    |    paprika*
050000497256 |     T      |    0    |    0    |    coffee mate
011110786715 |     T      |    0    |    0    |    applesauce*
078742030104 |     T      |    0    |    1    |    chicken breast*
011110492630 |     T      |    0    |    2    |    cola*
072736014880 |     T      |    0    |    0    |    vinaigrette

List of Items Scanned:
1) Gif reduced fat PB
2) Nutella
3) tropical Trail Mix
4) traditional trail mix
5) baker's corner baking soda
6) ritz fresh snacks
7) busch's baked beans
8) kraft Easy mac
9) pringles original
10) Spaghetto0s meatballs
11) betty crocker sour cream and chives
12) campbell's chicken noodle
13) New Amsterdam Gin
14) Barefood Red moscato
15) PVZ gummies
16) smart balance cooking spray
17) crasins
18) great value skim milk
19) phillidelpha cream cheese
20) florida's natural
21) monster absolutely zero
22) coke 2 litre
23) sam adams latitude 48
24) schiner bock
25) sweet baby ray's
26) heinz
27) french's
28) hellman's
29) pizza and breadsticks
30) bird's eye corn
31) kroger strawberry jello
32) lance toast chee
33) great value paprika
34) grill mates mesquite spice
35) sun chips
36) dorritos
37) coffee mate french vanilla
38) smuckers strawberry preserves
39) kroger applesauce
40) tostitos creamy spanish
41) prego meat
42) quaker fruit n cream
43) swiss miss
44) swanson chicken broth
45) great value chicken breast
46) big k vanilla cola
47) betty crocker loaded mash
48) vinaigrette
49) duncan hines spice cake
50) knorr taco rice

Tuesday, October 1, 2013

WMC3CDA Results

Shown here are results for GALE/NSGA-II/SPEA2 for a single run each on the CDA simulation for the WMC3 framework  (i.e. WMC3CDA).  

Note that each single run evaluates every member of every population for comparison purposes, but with GALE, not every evaluation is necessary for optimization.  Thus, each algorithm takes ~ about 4 hours to do a single run.

The results below are for a single run of the algorithm.  A more complete analysis would require ~ 10 runs at the very least, or 20 to be minimally satisfying to put up a solid statistical comparison.

Results:
Dot Plots: 1
http://i.imgur.com/jyfLyDH.png

Line Plots:
GALE is on top.   Bottom plots have messed up plot marker-choices, so read carefully.
http://i.imgur.com/LxkL6eh.png


Comments:
Although GALE reaches some lows not seen by the other algorithms, GALE has much larger variance, which could theoretically be fixed by removing random jiggles.


Note that the adjustments in making GALE deterministic (removing all randoms) are not used in these results.


Whats Next:
Need more Decision Variables!
 - Different landing routes
 - Radar Tower parameters?

Prune some objectives away?  Focus on time lost instead of number of delayed/interrupted tasks.

NASA furloughs - no access to their servers for now.  Need to get stuff working on my machines.

Erin's stuff

Poster for NSF EPSCoR Conference

Abstract for Poster

Paper, needs new title, abstract

Tuesday, September 24, 2013

Bounce at "1 minus (east-west)"?

WMC3 - CDA

Recap Problem Statement: We're trying to use GALE to make safety-critical decisions in short time.  We're using the CDA model from WMC3, built by Georgia Tech scholars, to learn what organizational structures within the cockpit (in terms of how the pilots should handle their taskload) best optimize objectives such as minimizing time of delay/interruptions in getting these tasks completed.  This is a cognitive science domain.

Buggy dataset: http://i.imgur.com/GCZFxie.png.  Takes about 12-15 minutes to run 100 times.

Comparing to results (example: http://i.imgur.com/znv2xzv.png) from http://cci.drexel.edu/NextGenAA/pdf/jcedm-2013-Pritchett-measuring%20fa.pdf.  This is our main reference paper for sanity checks on the model.

Few obvious correlations.  Sanity check.

Task: Figure out the bugs.  Then we can take off.

GALE

What I did:

Took away all non-determinism (random injections).

Only randomness that exists is with regeneration of individuals - but this isn't really non-deterministic (probability bin distributions).

Results: No change.

Self-reflection: we're mutating some distance, usually going too far.  Then we try to come back towards it, and we go too far.  Repeat, bouncing back and forth across where we want to go.

Problem: How far do we really want to bounce to get to exactly where we want to go?

Solution: Our projection axis is typically 1.0 units long.  In the optimal ideal world, east = 1.0, and west = 0.0  (or vice versa).  Thus absolute difference is 1.0 - 0.0 = 1.0.  Taking 1 minus this, this tells us how far we need to bounce.   "1 - (east - west)".

Observations: In early generations, this bounce distance is on the order of 0.30.  In later generations, it's very close to 0.0.  In addition, once we get close to 0.0, we STAY there.

Hypotheses: In problems that are hard to optimize, we don't get close to 0.0.  Bounce distance tells us on its own how good we're doing.

New Results:

Some standard Models (Osyczka2, Vienett2, Schaffer): http://i.imgur.com/MUUC2pD.png

POM3 Models: http://i.imgur.com/aSK1gWj.png

GALE POM3B Decision Chart: http://i.imgur.com/zoL8ZiW.png

Observations: Less variance in the dots.  Clearer image than before  (active v6 link: http://unbox.org/things/var/joe/active/active-v6.pdf)

Tuesday, September 3, 2013

Noise Injection Results

BK(1) = Linear NN K=1

http://unbox.org/things/var/brian/2013/projects/data-quality/mucker2/output/9-2-2013%20plots/

Since we are still getting flat results, perhaps we should bump up the amount of noise from 0.1 - 0.5 instead of 0 - 0.4? Or even back to the 0, 0.1, 0.25, 0.5, 1 results?

Intrinsic Dimensionality Baseline

New graph: The effect of (dataset length / critical length) on results. This was done at Q=0.7 and Rmax=len(R)/16. Shown here with 2, 3, 4, and 5 dim random data.


Some more progress on the paper: latest pdf

Consolidated some of my code... relevant files: 


Results from revised SE effort data:


















Simple stuff first. here's some terrible results from high-dimension random sets:

I had some problems with finding a good slope value for small datasets. The solution I found was the median of two-point slopes to a point. This seems to yield good results in places where the finite slope failed (see below).

Unique IDs crater ID. See Miyazaki94 and my modified Miyazaki2:

Things like dates, test fields which represent comments or notes also do nasty things. The effects of this are still visible in some of the effort datasets. I'll need to re-format the by hand to make sure it's well-conditioned if more accurate results are desired.


While staring at these graphs, I realized:

  • We're seeing a spectrum of ID from extremely local to extremely global. 
  • The local data isn't particularly useful because you're getting an average value of all localities rather than information about a specific locality.
  • The global data isn't particularly useful because as r -> max; D -> 0 (a single point)

But what if you were to ignore the summation across all points and focus instead on distances from a single point (say a cluster centroid):

  • At the low end, the dimensionality of the cluster would be represented
  • As r reaches the edge of the cluster, if there are other nearby clusters, the changes in D will likely be small. If there are no other clusters nearby, D will approach zero until other clusters are reached
  • This could potentially be used to select a scale for visualizations or cross-project learning. (If you're going to use a 2-d map, find a subset of points with an ID close to 2.0 to maximize the amount of information conveyed.)
  • This might not work, but seems intuitive. I'll do a few conceptual tests for next week.

I also made some progress on certifying results. Smith88Intrinsic derives this relation for the minimum number of test points to show dimensionality M at range R=rmax/rmin with quality Q:
At first, the results were wild. Nmin went through the roof. By restricting R (lowering the maximum value of r which is being used to determine the ID), the ID becomes more representative of the local area and Nmin becomes reduced. (you're not making a statistical claim about as large of a sphere of influence) I set this up as a after-the fact certification envelope check on the figures. After limiting rmax to the 6.25th ptile of all distances and setting Q to 50%, the results look good (N/Nmin > 1) on most things except the effort sets that I need to re-format. If a tighter certification is needed, additional options such as tuning the parameters on a dataset-by-dataset basis is possible.




Results from Brian's Data:
note: the Lymphography dataset is not present because is not available for public download due to private data.

ecoli.csv :: 3.27
glass.csv :: 0.75
hepatitis.csv :: 2.45
iris.csv :: 1.11
labor-negotiations.csv :: 0.68



Intrinsic Dimensionality Results:

ant-1.3.csv :: 3.40
ant-1.4.csv :: 3.61
ant-1.5.csv :: 2.26
ant-1.6.csv :: 2.69
ant-1.7.csv :: 2.08
ivy-1.1.csv :: 1.74
ivy-1.4.csv :: 2.67
ivy-2.0.csv :: 2.21
jedit-3.2.csv :: 2.04
jedit-4.0.csv :: 1.72
jedit-4.1.csv :: 2.03
jedit-4.2.csv :: 2.22
jedit-4.3.csv :: 2.01
log4j-1.0.csv :: 1.95
log4j-1.1.csv :: 2.67
bad file: ././arff/.svn
ant-1.7.arff :: 2.08
camel-1.0.arff :: 2.41
camel-1.2.arff :: 1.98
camel-1.4.arff :: 2.17
camel-1.6.arff :: 2.16
ivy-1.1.arff :: 1.74
ivy-1.4.arff :: 2.67
ivy-2.0.arff :: 2.21
ivy-2.0.csv :: 2.21
jedit-3.2.arff :: 2.04
jedit-4.0.arff :: 1.72
jedit-4.1.arff :: 2.03
jedit-4.2.arff :: 2.22
jedit-4.3.arff :: 2.01
kc2.arff :: 0.01
log4j-1.0.arff :: 1.95
log4j-1.1.arff :: 2.67
log4j-1.2.arff :: 2.12
lucene-2.0.arff :: 2.14
lucene-2.2.arff :: 2.69
lucene-2.4.arff :: 2.94
mc2.arff :: 1.13
pbeans2.arff :: 1.82
poi-2.0.arff :: 1.75
poi-2.5.arff :: 1.13
poi-3.0.arff :: 1.45
prop-6.arff :: 1.83
redaktor.arff :: 2.02
serapion.arff :: 0.32
skarbonka.arff :: 0.66
synapse-1.2.arff :: 1.89
tomcat.arff :: 2.76
tomcattomcat.arff :: 2.76
bad file: ././arff/too_big
velocity-1.5.arff :: 2.12
velocity-1.6.arff :: 3.00
xalan-2.6.arff :: 1.90
xalan-2.7.arff :: 1.96
xerces-1.2.arff :: 2.18
xerces-1.3.arff :: 0.00
xerces-1.4.arff :: 0.01
zuzel.arff :: 1.54


Space Dim | Mean Dimensionality | SD
    5     |         4.60        |  0.10
   10     |         8.37        |  0.20
   15     |         11.34        |  0.13
   20     |         14.64        |  0.26
   25     |         16.99        |  0.26
   30     |         19.30        |  1.08
   35     |         22.32        |  0.58
   40     |         23.71        |  0.72



Axes only
Axial Planes only










80% Sparse 3D Dataset

Numeric 3D Dataset

Numberic 10D Dataset

Numberic 100D Dataset




UPDATE 2: PITS vs Randomized:
Original and Word-Randomized Sets of Various Length

Original Data, Word-Randomized with Original Distribution, Word-Randomized with Uniform Distribution
Original Data, Randomized with Original Wordcount Distribution, Randomized with Flat Wordcount Distribution 

Original, Randomized with Original Wordcount dist, Randomized with Gamma Wordcount dist
Original and Randomized with Gamma dist Wordcounts at various Alpha

Original and Randomized with Gamma dist Wordcounts at various Beta



UPDATE: PITS Data:


Cube of uniform distributuion (3-D):

Line of uniform distributuion (1-D):

Spiral of uniform distributuion (1-D):


Plane of uniform distributuion (2-D):

Roling Plane of uniform distribution (2-D):

Line with noise in 1-D, 2-D (line is of length 10, noise is Gaussian SD=0.1):