Teaching Assistance in schools

The Deployment and Impact of Support Staff (DISS) Project suggests that the way we use TAs may need to change.

The key differences are that “teachers generally ‘open up’ the pupils, whereas TAs ‘close down’ the talk linguistically and cognitively”:

• Teachers encourage students to think for themselves,
• Teachers explain concepts; TAs’ explanations sometimes inaccurate
• TAs use closed questions to support pupils to complete tasks but with minimal exploration of concepts
• Teachers use prompts and questions to encourage thinking; TAs more frequently supplied answers and completed work for pupils.

It would appear that TAs are creating a dependency culture and closing down opportunities for a pupil to think for themselves.

This meant that “the more TA support pupils received, the less academic progress made.” There was a significantly negative effect on progress in every year studied in English, in every year but year 10 in Maths, and in every year but years 1, 3, 7 and 10 in Science. No year group and no subject made significantly positive progress.

The only objective positive findings were that TA support improves pupils’ Positive Approaches to Learning but only in year 9.

Notes

‘TA’s include Teaching Assistants, Higher Level Teaching Assistants, Classroom Assistants, Learning Support Assistants, LSAs for SEN, Nursery Nurses and Therapists

References

 Blatchford P, Bassett P, Brown P, Martin C, Russell A, Webster R, Babayigit S, Haywood N, Koutsoubou M, & Rubie-Davies C 2010 The Deployment and Impact of Support Staff (DISS) Project available at www.schoolsupportstaff.net

Some problems with the use of data mining in social sciences

Old science?

Anderson 2008 believes that the impact of huge databases has changed the methodology of Science:

"The scientific method is built around testable hypotheses. These models, for the most part, are systems visualized in the minds of scientists. The models are then tested, and experiments confirm or falsify theoretical models of how the world works. This is the way science has worked for hundreds of years.

Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two. Once you have a model, you can connect the data sets with confidence. Data without a model is just noise.

But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. ....

There is now a better way. Petabytes allow us to say: "Correlation is enough." We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

The best practical example of this is the shotgun gene sequencing by J. Craig Venter. Enabled by high-speed sequencers and supercomputers that statistically analyze the data they produce, Venter went from sequencing individual organisms to sequencing entire ecosystems. In 2003, he started sequencing much of the ocean, retracing the voyage of Captain Cook. And in 2005 he started sequencing the air. In the process, he discovered thousands of previously unknown species of bacteria and other life-forms.

If the words "discover a new species" call to mind Darwin and drawings of finches, you may be stuck in the old way of doing science. Venter can tell you almost nothing about the species he found. He doesn't know what they look like, how they live, or much of anything else about their morphology. He doesn't even have their entire genome. All he has is a statistical blip — a unique sequence that, being unlike any other sequence in the database, must represent a new species.

This sequence may correlate with other sequences that resemble those of species we do know more about. In that case, Venter can make some guesses about the animals — that they convert sunlight into energy in a particular way, or that they descended from a common ancestor. But besides that, he has no better model of this species than Google has of your MySpace page. It's just data. By analyzing it with Google-quality computing resources, though, Venter has advanced biology more than anyone else of his generation."

I have issues with this approach. The first is that it is naive to believe that pure induction is possible.  An old, probably apocryphal story is told of an amateur scientist in the early days of the Royal Society who became enamoured of the Baconian inductive method of Science. So for thirty years he observed everything, wrote up his diaries and presented them, a mass of unorganised data, to the RS where they still lie, in the archives, unanalysed. But the 'scientist' was fooling himself. Before you collect data you must decide what to measure, before you observe you must select what you observe. Thus your beliefs (which are essentially unformed, unacknowledged theories) influence and pre-judge the facts from which you then construct your theories. There is likely to be a little confirmation bias in this loop, particularly so if the data is about people as in the social sciences.

My second concern is with the danger of false positives.  (Rajaraman 2008) suggests that "adding more, independent data usually beats out designing ever-better algorithms to analyze an existing data set".  Perhaps they key word here is 'independent' because if you look for patterns you are going to find them. Statisticians use significance limits, often 1%,  to determine whether a pattern occurs by pure random chance. But if you have a large data set of say 100 variables and you correlate each with each of the others you will trawl 10,000 patterns. At a 1% level you would expect to find 100 correlations by chance! How will you tell which are true correlations and which are false positives?

Thirdly, there is a concern about the objectivity of the data. Genome sequences are relatively easy to observe although there is always the possibility of contamination. But in the social sciences it is far more difficult for the observer NOT to 'contaminate' the observation. For example, if a subject is aware that they are being observed they may behave differently, often to conform to what they believe the observer expects of them. An example of this is social desirability bias. This can be exacerbated in action research in education when the observing experimenter may also be the teacher seeking to achieve better grades for the pupils who are at the same time the subjects of the experiment. Ethical considerations suggest that you can't not get involved. "On one hand, institutions might be vulnerable to charges of “profiling” students when they draw conclusions from student data; on the other, they could be seen as irresponsible if they don’t take action when data suggest a student is having difficulty." Educause 2010. But Goodhart's Law, quoted by Snowdon (2011) as  "the minute a measure becomes a target it ceases to be a measure" (although more accurately it is 'once a measure becomes a target it loses its value as a measure') suggests that the taking of action significantly undermines the research. 

This feedback effect has huge implications. Traditionally social sciences have used statistics based on the Gaussian bell-curve. There is significant research (Taleb 2007; Ball 2004; Buchanan 2000) to suggest that Mandelbrotian power-law statistics may be more appropriate. This is because there is often feedback between observer and observed. Feedback (also found in earthquake modelling and avalanches of grains in sand-piles) changes the maths. 

I still agree that the data mining has massive potential but its application to social sciences raises significant concerns which need to be addressed.

References


Anderson C 2008 The end of theory: the data deluge makes the scientific method obsolete Wired Magazine 23rd June 2008 available at http://www.wired.com/science/discoveries/magazine/16-07/pb_theory accessed 17th January 2011


Ball P 2004 Critical Mass: how one things leads to another Heinemann London


Buchanan M 2000 Ubiquity Weidenfield & Nicolson, London


Educause 2010 7 things you should know about analytics available at http://net.educause.edu/ir/library/pdf/ELI7059.pdf accessed 17th January 2011


Rajaraman A 2008 More data usually beats better algorithms Blog post 24th March 2008 in Datawocky available at http://anand.typepad.com/datawocky/2008/03/more-data-usual.html accessed 17th January 2011


Snowdon D 2011 A grain of sand: innovation diffusion blog posted on 11th January 2011 in Cognitive Edge available at http://www.cognitive-edge.com/blogs/dave/2011/01/a_grain_of_sand_innovation_dif.php accessed 17th January 2011


Taleb NN 2007 The Black Swan: the impact of the highly improbable Random House, New York

Connectivism

For me reflective learning is usually a solitary act.

Siemens 2005 believes that the rapid pace of change with the total amount of human knowledge doubling every 18 months means that we are in a new learning 'age' which requires a new pedagogy. 


As usual we have to cycle back to theories of knowledge. I do get a little frustrated with the perpetual debates about whether reality is external or internal and whether knowledge is knowable. These seem to be epistemological equivalents of counting the number of angels who can dance on the head of a pin. The universe will not cease with my death. I learn by interacting with reality. It may be that I construct my understanding through this interaction but my belief in the realness of reality is so powerful that if I step of a cliff I will fall.


So I am quite happy to believe that reality is out there while at the same time agreeing with Siemens 2005 that "learners are not empty vessels to be filled with knowledge. Instead, learners are actively attempting to create meaning."


But then Siemens suggests that learning is a social act, not an individual act. "Connectivism presents a model of learning that acknowledges the tectonic shifts in society where learning is no longer an internal, individualistic activity." Whilst many would agree that the process of learning often happens in social groups, he seems to take a more extreme position. "We can no longer personally experience and acquire learning that we need to act. We derive our competence from forming connections." 


Can this be right? Sanger (2010) seems to suggest the opposite: that although one can acquire information in groups and discuss information in  groups, in the end one turns that information into knowledge by encoding it within the neurons of one's individual brain.


Of course, the construction of understanding, the making of meaning, is down to the creation of connections. I understand interatomic collisions using a billiard ball metaphor; I have connected these two realms. But the connections are between ideas rather than between people. It may be that I have used other people to learn, to suggest the connections, but my knowledge is not inside their heads, it is inside mine.


"Learning may reside in non-human appliances" says Siemens and he is thinking of databases etc. And yet he defines learning as "actionable knowledge". Well I can't take action over knowledge that lies in some dusty tome on a library shelf unless I have actually accessed that knowledge, and learnt it, so that it resides inside my head. Otherwise I can learn simply by bookmarking all the articles I one day hope to read; I can learn just by buying books.

"The pipe is more important than the content within the pipe" states Siemens which suggests that the skull is more important than the mind. I think I understand what he means but it seems to me that Thornburg's separation of the learning around the campfire and at the watering hole from learning in the cave is still a vital distinction.

We might access ideas in connected groups but in the end the internalisation of that learning is a solitary process.

References


Sanger, L. (2010). Individual Knowledge in the Internet Age. Educause Review, March/April 2010. pp14-24) http://net.educause.edu/ir/library/pdf/ERM1020.pdf


Siemens G 2005 Connectivism: A learning theory for the digital age International Journal of Instructional Technology and Distance learning 2:1 avaliable at http://www.itdl.org/journal/jan_05/article01.htm accessed 19th January 2011

Thornburg D 2004 Campfires in Cyberspace: Primordial metaphors for learning in the 21st Century International Journal of Instructional Technology and Distance Learning 1:10 available at http://itdl.org/journal/oct_04/invited01.htm

Data Mining: a very simple start for a beginner (me)

Data mining is "the process of extracting patterns from data" (wikipedia). A data miner changes data into information. Educational data mining is a process of extracting data about learners and using that data to teach better.

What sort of things can you do?

You can use data to develop categories, clusters and classifications. 

In k-Nearest Neighbour (k-NN) classification a data point is classified by majority vote of its nearest neighbours. If k=1 the green circle will be classed with the red triangles, because its nearest neighbour is a red triangle. If k=3 it will again be red triangles because the majority of the (k=)3 nearest neighbours are triangles. If k=5 it will be classified as a blue square because 3 of the (k=)5 nearest neighbours are blue squares. Clearly the choice of k is critical. An alternate method is to weight the classification by the distance to each of the nearest neighbours.

You can try to discover behaviours which occur together. For example, in the sentence: "This is the life!", there are 2xe, 1xf, 2xh etc. If we only count where there are 2 or more occurrences, the "frequent 1 sequences" are: 2xe, 2xh, 3xi, 2xs, 2xt. If we seek the 2-sequences (only for these) we have: e_, e!, hi, he, is, is, s_, s_, th, th. Using a frequency threshold of 2 again, we are left with is, s_, and th as our frequent 2 sequences. Moving to 3 sequences (again only using those we have identified as frequent 2 sequences) and another threshold of 2 we have only 1 frequent 3 sequence: is_. Moving to 4 sequences we find is_i and is_t. Neither of these pass the threshold so the algorithm stops. What have we learnt? The 1 sequences could tell us something about the commonest letters in English and the 2 sequences tell us that is and th are frequent combinations and that s often happens at the end of words. The 3 sequences tell us that is often happens at the end of words. 

So what?

We now have a predictive framework: if you get an i expect an s (and then a space), if you get an s expect a space, if you get a t expect an h.

We could use this process to create 'recommendations' a la Amazon: if you enjoyed doing those sums you might like to try these. Or diagnoses, enabling us to identify the appropriate intervention for the measured behaviour.

References

Baker, S.J.D. & Yacef, K. (2009) The State of Educational Data Mining in 2009: A Review and Future Visions: http://www.educationaldatamining.org/JEDM/images/articles/vol1/issue1/JEDMVol1Issue1_BakerYacef.pdf accessed 10th January 2011

International Working Group on Educational Data Mining available at http://educationaldatamining.org/ accessed 10th January 2011

Wikipedia Data Mining available at http://en.wikipedia.org/wiki/Data_mining accessed 10th January 2011

Learning Analytics

"Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs." (Siemens 2010)


Baer 2011 points out that online courses produce a vast amount of data. This can be used to monitor the progress of students on a day to day basis; this in turn can be used to tailor further teaching, for example by putting remedial measures in place.


This is all good formative assessment stuff (also called AfL, assessment for learning or ASL, assessment to support learning). 






But there are three buts to which Baer alludes:

• You have to know what to assess;
• You have to be able to assess it accurately;
• You have to know what to do in response to any particular measurement.

What to assess

Normally we assess through tests and relating the scores to the standards but Wiley 2011 points out that Learning Analytics can measure other things (eg 'how long did the student spend reading the book?') although he realises that it can be difficult to see the correlations between this data and the final achievements.

How to respond

We might use Learning Analytics to create a sort of profile of each learner which could then be used (like Amazon) to provide personalised recommendations of resources and techniques that have created success for similar types of learners (Duval 2011)

How does all this match what Nicol and McFarlane-Dick (2006) describe as "good feedback practice"? Can any automated system feedback as well as an experienced assessor?

N&M-D make it clear that feedback should be prompt so that it can actually influence learning rather than occurring after learning. Clearly the day-by-day assessment envisaged by Baer will do this. But other key elements of good feedback include:

• Telling teachers how to shape their teaching:
• On a meta level, Learning Analytics may "help us to realise how much of what we do is not very effective" and so enable teaching pedagogy to evolve (Duval 2011).
• Making clear what good performance is
• Helping learners reflect
• Helping learners to take action to correct deficiencies
• Encouraging T-S and S-S dialogue

It is clear that these last four could be integrated into the feedback from a Learning Analytics system; it is also clear that they should be used to shape the design of such feedback.

References 



Baer J 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=50 accessed 8th January 2011

Duval E 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=54 accessed 8th January 2011

Nicol D & Macfarlane-Dick D, 2006 Formative assessment and self-regulated learning: A model and seven principles of good feedback practice Studies in higher Education 31(2):199-218

Siemens G 2010 Learning Analytics and Knowledge available at https://tekri.athabascau.ca/analytics/ accessed 8th January 2011

Wiley D 2011 video interview with George Siemens available at http://www.learninganalytics.net/?page_id=46 accessed 8th January 2011