Excel, compare standard deviation + means in a slightly hand-wavey way. The mean is around the same for each interestingly, however the standard deviation varies in such a way as to imply which belongs to which.
Several people have said they would, or have, ploted histograms - but how?
* By hand?
* With Excel?
* With R?
* With processing?
I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.
It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this:
par(mfrow = c(3,2))
dat <- read.table("data.txt", sep="\t", header = TRUE)
for(let in c("A","B","C","D","E","F"))
plot(density(dat[,let]), main = let)
haha thanks, but definitely trolling. This is perhaps the most inelegant solution in the world. But it does the job in this case. Output is 10.6014, 4.44403, 5.71353, 9.46718, 10.6471, 5.12933, for columns A to F, respectively
I would look at the mean and variance in each column and cluster by that. My intuition is that the variance in Temperature measured in Fahrenheit will be greater then that of weight in Kg.
Checking it seems that this is borne out. So is the data real? What is being assessed here?
Interesting to see that people prefer std dev to variance for this problem.
An interesting problem. My first guess is that teenagers' weights distribution is two-peaked (boys & girls) but this assumes that the teenagers are around the same age and that the peaks are far enough to be detectable in the samples. I can't be arsed to check this hypothesis though :-)
Anyway, a serious approach to this problem would require comparing against solid real-life data. Off the bat assumptions about distributions of real life data are often very wrong.
I used my app https://app.prettygraph.com to quickly look at the histograms. B and F look almost identical with a very normal distribution. The others look more like twin-peaks.
Edit: If you want to try, copy the data and click the "Paste data from Excel link" on the left under the Data Tab, paste and load. Then choose histogram under the Graph tab. It plots the X var (ignores Y). Sorry it's a bit shabby, I haven't worked on it in a while.
as others have said, first thing to do is plot the data. but if you want a statistical test (with significances, so that you have some idea of how confident you can be in your decision) then you could use the two-sample ks test. it's likely more useful that a simple t-test here because it's sensitive to shape.
but to be honest the first thing i would do is google for suitable approaches...
Does the thread starter wonder about what software tool (i.e. language) people would use, or does he wonder about what method people would use? As far as language goes, the problem requires mathematics on a level so simple that practically any programming language is more than enough sufficient - the histographical method I'd use for solving this problem would be doable in BASIC on a VIC-20 or a TI-81 pocket calculator.
Orange http://orange.biolab.si/
"Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. Packed with features for data analytics."
Weka http://www.cs.waikato.ac.nz/ml/weka/
"Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes."
I would use my right hand to slap Alison and Charlie and get them to do their work again... this time forcing them to label their samples.
I would then take them into my office and hold an inquest on how two completely different data-sets got mixed up in the first place.
At which point I would probably find out Alison and Charlie have been knocking boots on company time.. Alison and Charlie would then be fired as this is against company policy.
Due to the difficulty in finding new jobs Alison and Charlie's fledgling romance would end... leaving Alison unexpectedly pregnant and Charlie with an 18 year bill for child support.
The morale of this story? Label your fking work.
Note: I was unable to solve this problem as my math skills are poor.
I'm sure you can easily identify which set is which using statistical methods. It's probably as simple as finding out which type of distribution each set resembles. Then again, my undergrad statistics is very rusty.
56 comments
[ 3.4 ms ] story [ 114 ms ] threadI am quite rusty on this stuff I must say!
I guess one would correlate each row to the A and B one and look which correlate the most?
* By hand?
* With Excel?
* With R?
* With processing?
I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.
#copy data into a txt file without the first two lines from the xls
par(mfrow = c(3,2))
dat <- read.table("data.txt", sep="\t", header = TRUE)
for(let in c("A","B","C","D","E","F"))
plot(density(dat[,let]), main = let)
easy...(why doesnt HN recognise newlines?)
It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this:
I offer you knowledge obtained by sheer trial and error. :)
for i in {1..6}; do COL='$'$i; awk -F, "{delta = $COL - avg; avg += delta / NR; mean2 += delta * ($COL - avg); } END { print sqrt(mean2 / NR); }" list.csv; done
[troll]
Checking it seems that this is borne out. So is the data real? What is being assessed here?
Interesting to see that people prefer std dev to variance for this problem.
So D and E are temperature, and C and F are weights.
Anyway, a serious approach to this problem would require comparing against solid real-life data. Off the bat assumptions about distributions of real life data are often very wrong.
Edit: If you want to try, copy the data and click the "Paste data from Excel link" on the left under the Data Tab, paste and load. Then choose histogram under the Graph tab. It plots the X var (ignores Y). Sorry it's a bit shabby, I haven't worked on it in a while.
but to be honest the first thing i would do is google for suitable approaches...
EDIT: I realize that this was a somewhat curious result.
Orange http://orange.biolab.si/ "Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. Packed with features for data analytics."
Weka http://www.cs.waikato.ac.nz/ml/weka/ "Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes."
The means are all the same.
I would then take them into my office and hold an inquest on how two completely different data-sets got mixed up in the first place.
At which point I would probably find out Alison and Charlie have been knocking boots on company time.. Alison and Charlie would then be fired as this is against company policy.
Due to the difficulty in finding new jobs Alison and Charlie's fledgling romance would end... leaving Alison unexpectedly pregnant and Charlie with an 18 year bill for child support.
The morale of this story? Label your fking work.
Note: I was unable to solve this problem as my math skills are poor.