56 comments

[ 3.4 ms ] story [ 114 ms ] thread
SPSS -> Cluster Analysis
(comment deleted)
Excel, compare standard deviation + means in a slightly hand-wavey way. The mean is around the same for each interestingly, however the standard deviation varies in such a way as to imply which belongs to which.

I am quite rusty on this stuff I must say!

Weird, just looking at the raw numbers I would have expected A to be the weights and B to be the temperature averages because A has a bigger range.

I guess one would correlate each row to the A and B one and look which correlate the most?

Standard dev seamed to be enough. I sorted and graphed the data to to be sure.
My thought was to match the bell curves, but this is nearly the same thing now that I think about it.
Temps have standard deviation around 10ish, weights around 5ish.
I would plot some histograms and do it by eye.
Several people have said they would, or have, ploted histograms - but how?

* By hand?

* With Excel?

* With R?

* With processing?

I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.

(comment deleted)
use R

#copy data into a txt file without the first two lines from the xls

par(mfrow = c(3,2))

dat <- read.table("data.txt", sep="\t", header = TRUE)

for(let in c("A","B","C","D","E","F"))

plot(density(dat[,let]), main = let)

easy...(why doesnt HN recognise newlines?)

why doesnt HN recognise newlines?

It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this:

    par(mfrow = c(3,2))
    dat <- read.table("data.txt", sep="\t", header = TRUE)
    for(let in c("A","B","C","D","E","F"))
    plot(density(dat[,let]), main = let)
Thanks Dove. I probably should read the posting FAQ or something.
Not your fault, actually. That particular quirk may be written down somewhere, but I've not found it in my three years here.

I offer you knowledge obtained by sheer trial and error. :)

Bash

for i in {1..6}; do COL='$'$i; awk -F, "{delta = $COL - avg; avg += delta / NR; mean2 += delta * ($COL - avg); } END { print sqrt(mean2 / NR); }" list.csv; done

[troll]

Troll... or hero?
haha thanks, but definitely trolling. This is perhaps the most inelegant solution in the world. But it does the job in this case. Output is 10.6014, 4.44403, 5.71353, 9.46718, 10.6471, 5.12933, for columns A to F, respectively
I would look at the mean and variance in each column and cluster by that. My intuition is that the variance in Temperature measured in Fahrenheit will be greater then that of weight in Kg.

Checking it seems that this is borne out. So is the data real? What is being assessed here?

Interesting to see that people prefer std dev to variance for this problem.

I can just scan the numbers and see the patterns. Column A has much more extreme ranges, lots of 40s and 70s, whereas B is much narrower.

So D and E are temperature, and C and F are weights.

An interesting problem. My first guess is that teenagers' weights distribution is two-peaked (boys & girls) but this assumes that the teenagers are around the same age and that the peaks are far enough to be detectable in the samples. I can't be arsed to check this hypothesis though :-)

Anyway, a serious approach to this problem would require comparing against solid real-life data. Off the bat assumptions about distributions of real life data are often very wrong.

I used my app https://app.prettygraph.com to quickly look at the histograms. B and F look almost identical with a very normal distribution. The others look more like twin-peaks.

Edit: If you want to try, copy the data and click the "Paste data from Excel link" on the left under the Data Tab, paste and load. Then choose histogram under the Graph tab. It plots the X var (ignores Y). Sorry it's a bit shabby, I haven't worked on it in a while.

Nice app! Computing the histogram hung on the Apple stock data for me.
Thanks! It gets hung sometimes, but it should work if you select any field other than the Date as the X variable. Refresh and try again if you like.
as others have said, first thing to do is plot the data. but if you want a statistical test (with significances, so that you have some idea of how confident you can be in your decision) then you could use the two-sample ks test. it's likely more useful that a simple t-test here because it's sensitive to shape.

but to be honest the first thing i would do is google for suitable approaches...

Does the thread starter wonder about what software tool (i.e. language) people would use, or does he wonder about what method people would use? As far as language goes, the problem requires mathematics on a level so simple that practically any programming language is more than enough sufficient - the histographical method I'd use for solving this problem would be doable in BASIC on a VIC-20 or a TI-81 pocket calculator.
I calculated the mode in excel, the weight columns have one while the farenheit columns dont.

EDIT: I realize that this was a somewhat curious result.

I would use one of these two tools:

Orange http://orange.biolab.si/ "Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. Packed with features for data analytics."

Weka http://www.cs.waikato.ac.nz/ml/weka/ "Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes."

I used a spreadsheet to calculate the standard deviations. A,D and E have twice the deviation of B,C and F. (10 vs 5)

The means are all the same.

I would use my right hand to slap Alison and Charlie and get them to do their work again... this time forcing them to label their samples.

I would then take them into my office and hold an inquest on how two completely different data-sets got mixed up in the first place.

At which point I would probably find out Alison and Charlie have been knocking boots on company time.. Alison and Charlie would then be fired as this is against company policy.

Due to the difficulty in finding new jobs Alison and Charlie's fledgling romance would end... leaving Alison unexpectedly pregnant and Charlie with an 18 year bill for child support.

The morale of this story? Label your fking work.

Note: I was unable to solve this problem as my math skills are poor.

You're on the management fast track aren't you ?
I probably would be if I spent less time writing fictional stories to accompany math problems. ^^
And ask Charlie why he's storing weights to 13 decimal places.
I'm sure you can easily identify which set is which using statistical methods. It's probably as simple as finding out which type of distribution each set resembles. Then again, my undergrad statistics is very rusty.