Some problems with the use of data mining in social sciences

Old science?

Anderson 2008 believes that the impact of huge databases has changed the methodology of Science:

"The scientific method is built around testable hypotheses. These models, for the most part, are systems visualized in the minds of scientists. The models are then tested, and experiments confirm or falsify theoretical models of how the world works. This is the way science has worked for hundreds of years.

Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two. Once you have a model, you can connect the data sets with confidence. Data without a model is just noise.

But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. ....

There is now a better way. Petabytes allow us to say: "Correlation is enough." We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

The best practical example of this is the shotgun gene sequencing by J. Craig Venter. Enabled by high-speed sequencers and supercomputers that statistically analyze the data they produce, Venter went from sequencing individual organisms to sequencing entire ecosystems. In 2003, he started sequencing much of the ocean, retracing the voyage of Captain Cook. And in 2005 he started sequencing the air. In the process, he discovered thousands of previously unknown species of bacteria and other life-forms.

If the words "discover a new species" call to mind Darwin and drawings of finches, you may be stuck in the old way of doing science. Venter can tell you almost nothing about the species he found. He doesn't know what they look like, how they live, or much of anything else about their morphology. He doesn't even have their entire genome. All he has is a statistical blip — a unique sequence that, being unlike any other sequence in the database, must represent a new species.

This sequence may correlate with other sequences that resemble those of species we do know more about. In that case, Venter can make some guesses about the animals — that they convert sunlight into energy in a particular way, or that they descended from a common ancestor. But besides that, he has no better model of this species than Google has of your MySpace page. It's just data. By analyzing it with Google-quality computing resources, though, Venter has advanced biology more than anyone else of his generation."

I have issues with this approach. The first is that it is naive to believe that pure induction is possible.  An old, probably apocryphal story is told of an amateur scientist in the early days of the Royal Society who became enamoured of the Baconian inductive method of Science. So for thirty years he observed everything, wrote up his diaries and presented them, a mass of unorganised data, to the RS where they still lie, in the archives, unanalysed. But the 'scientist' was fooling himself. Before you collect data you must decide what to measure, before you observe you must select what you observe. Thus your beliefs (which are essentially unformed, unacknowledged theories) influence and pre-judge the facts from which you then construct your theories. There is likely to be a little confirmation bias in this loop, particularly so if the data is about people as in the social sciences.

My second concern is with the danger of false positives.  (Rajaraman 2008) suggests that "adding more, independent data usually beats out designing ever-better algorithms to analyze an existing data set".  Perhaps they key word here is 'independent' because if you look for patterns you are going to find them. Statisticians use significance limits, often 1%,  to determine whether a pattern occurs by pure random chance. But if you have a large data set of say 100 variables and you correlate each with each of the others you will trawl 10,000 patterns. At a 1% level you would expect to find 100 correlations by chance! How will you tell which are true correlations and which are false positives?

Thirdly, there is a concern about the objectivity of the data. Genome sequences are relatively easy to observe although there is always the possibility of contamination. But in the social sciences it is far more difficult for the observer NOT to 'contaminate' the observation. For example, if a subject is aware that they are being observed they may behave differently, often to conform to what they believe the observer expects of them. An example of this is social desirability bias. This can be exacerbated in action research in education when the observing experimenter may also be the teacher seeking to achieve better grades for the pupils who are at the same time the subjects of the experiment. Ethical considerations suggest that you can't not get involved. "On one hand, institutions might be vulnerable to charges of “profiling” students when they draw conclusions from student data; on the other, they could be seen as irresponsible if they don’t take action when data suggest a student is having difficulty." Educause 2010. But Goodhart's Law, quoted by Snowdon (2011) as  "the minute a measure becomes a target it ceases to be a measure" (although more accurately it is 'once a measure becomes a target it loses its value as a measure') suggests that the taking of action significantly undermines the research. 

This feedback effect has huge implications. Traditionally social sciences have used statistics based on the Gaussian bell-curve. There is significant research (Taleb 2007; Ball 2004; Buchanan 2000) to suggest that Mandelbrotian power-law statistics may be more appropriate. This is because there is often feedback between observer and observed. Feedback (also found in earthquake modelling and avalanches of grains in sand-piles) changes the maths. 

I still agree that the data mining has massive potential but its application to social sciences raises significant concerns which need to be addressed.

References


Anderson C 2008 The end of theory: the data deluge makes the scientific method obsolete Wired Magazine 23rd June 2008 available at http://www.wired.com/science/discoveries/magazine/16-07/pb_theory accessed 17th January 2011


Ball P 2004 Critical Mass: how one things leads to another Heinemann London


Buchanan M 2000 Ubiquity Weidenfield & Nicolson, London


Educause 2010 7 things you should know about analytics available at http://net.educause.edu/ir/library/pdf/ELI7059.pdf accessed 17th January 2011


Rajaraman A 2008 More data usually beats better algorithms Blog post 24th March 2008 in Datawocky available at http://anand.typepad.com/datawocky/2008/03/more-data-usual.html accessed 17th January 2011


Snowdon D 2011 A grain of sand: innovation diffusion blog posted on 11th January 2011 in Cognitive Edge available at http://www.cognitive-edge.com/blogs/dave/2011/01/a_grain_of_sand_innovation_dif.php accessed 17th January 2011


Taleb NN 2007 The Black Swan: the impact of the highly improbable Random House, New York

Data Mining: a very simple start for a beginner (me)

Data mining is "the process of extracting patterns from data" (wikipedia). A data miner changes data into information. Educational data mining is a process of extracting data about learners and using that data to teach better.

What sort of things can you do?

You can use data to develop categories, clusters and classifications. 

In k-Nearest Neighbour (k-NN) classification a data point is classified by majority vote of its nearest neighbours. If k=1 the green circle will be classed with the red triangles, because its nearest neighbour is a red triangle. If k=3 it will again be red triangles because the majority of the (k=)3 nearest neighbours are triangles. If k=5 it will be classified as a blue square because 3 of the (k=)5 nearest neighbours are blue squares. Clearly the choice of k is critical. An alternate method is to weight the classification by the distance to each of the nearest neighbours.

You can try to discover behaviours which occur together. For example, in the sentence: "This is the life!", there are 2xe, 1xf, 2xh etc. If we only count where there are 2 or more occurrences, the "frequent 1 sequences" are: 2xe, 2xh, 3xi, 2xs, 2xt. If we seek the 2-sequences (only for these) we have: e_, e!, hi, he, is, is, s_, s_, th, th. Using a frequency threshold of 2 again, we are left with is, s_, and th as our frequent 2 sequences. Moving to 3 sequences (again only using those we have identified as frequent 2 sequences) and another threshold of 2 we have only 1 frequent 3 sequence: is_. Moving to 4 sequences we find is_i and is_t. Neither of these pass the threshold so the algorithm stops. What have we learnt? The 1 sequences could tell us something about the commonest letters in English and the 2 sequences tell us that is and th are frequent combinations and that s often happens at the end of words. The 3 sequences tell us that is often happens at the end of words. 

So what?

We now have a predictive framework: if you get an i expect an s (and then a space), if you get an s expect a space, if you get a t expect an h.

We could use this process to create 'recommendations' a la Amazon: if you enjoyed doing those sums you might like to try these. Or diagnoses, enabling us to identify the appropriate intervention for the measured behaviour.

References

Baker, S.J.D. & Yacef, K. (2009) The State of Educational Data Mining in 2009: A Review and Future Visions: http://www.educationaldatamining.org/JEDM/images/articles/vol1/issue1/JEDMVol1Issue1_BakerYacef.pdf accessed 10th January 2011

International Working Group on Educational Data Mining available at http://educationaldatamining.org/ accessed 10th January 2011

Wikipedia Data Mining available at http://en.wikipedia.org/wiki/Data_mining accessed 10th January 2011

Learning Analytics

"Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs." (Siemens 2010)


Baer 2011 points out that online courses produce a vast amount of data. This can be used to monitor the progress of students on a day to day basis; this in turn can be used to tailor further teaching, for example by putting remedial measures in place.


This is all good formative assessment stuff (also called AfL, assessment for learning or ASL, assessment to support learning). 






But there are three buts to which Baer alludes:

• You have to know what to assess;
• You have to be able to assess it accurately;
• You have to know what to do in response to any particular measurement.

What to assess

Normally we assess through tests and relating the scores to the standards but Wiley 2011 points out that Learning Analytics can measure other things (eg 'how long did the student spend reading the book?') although he realises that it can be difficult to see the correlations between this data and the final achievements.

How to respond

We might use Learning Analytics to create a sort of profile of each learner which could then be used (like Amazon) to provide personalised recommendations of resources and techniques that have created success for similar types of learners (Duval 2011)

How does all this match what Nicol and McFarlane-Dick (2006) describe as "good feedback practice"? Can any automated system feedback as well as an experienced assessor?

N&M-D make it clear that feedback should be prompt so that it can actually influence learning rather than occurring after learning. Clearly the day-by-day assessment envisaged by Baer will do this. But other key elements of good feedback include:

• Telling teachers how to shape their teaching:
• On a meta level, Learning Analytics may "help us to realise how much of what we do is not very effective" and so enable teaching pedagogy to evolve (Duval 2011).
• Making clear what good performance is
• Helping learners reflect
• Helping learners to take action to correct deficiencies
• Encouraging T-S and S-S dialogue

It is clear that these last four could be integrated into the feedback from a Learning Analytics system; it is also clear that they should be used to shape the design of such feedback.

References 



Baer J 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=50 accessed 8th January 2011

Duval E 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=54 accessed 8th January 2011

Nicol D & Macfarlane-Dick D, 2006 Formative assessment and self-regulated learning: A model and seven principles of good feedback practice Studies in higher Education 31(2):199-218

Siemens G 2010 Learning Analytics and Knowledge available at https://tekri.athabascau.ca/analytics/ accessed 8th January 2011

Wiley D 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=46 accessed 8th January 2011