Category Archives: Assessment

Markopalypse Now

AHT VAL: And once you’ve finished marking your students’ books and they have responded IN DETAIL to your DETAILED comments, you must take them in again and mark them a second time using a different coloured pen!

AHT HARVEY: A page that’s marked in only one colour is a useless page! 

NQT BENJAMIN: Erm, if you say so. But why?

AHT VAL: It’s basic Ofsted-readiness, Benjamin. Without a clearly colour-coded dialogue between teacher and student, how can we prove that the student has made progress as a result of teacher feedback?

NQT BENJAMIN: But I’ve only got this red biro…

AHT HARVEY GRINS UNPLEASANTLY AND OPENS A CABINET FULL OF PENS OF MANY COLOURS.

AHT HARVEY: In this school we wage a constant battle against teacher sloth and indifference!

(With apologies to The League Of Gentlemen)

I have been a teacher for more than 26 years and I tell you this: I have never marked as much or as often as I am now. We are in the throes of a Marking Apocalypse — a Markopalypse,  if you will.

And why am I doing this? Have I had a Damascene-road conversion to the joy of rigorous triple marking?

No. I do it because I have to. I do it because of my school’s marking policy. More to the point, I do it because my school expends a great deal of time and energy checking that their staff is following the policy. And my school is not unique in this. 

Actually, to be fair, I think my current school has the most nearly-sensible policy of the three schools I have worked in most recently, but it is still an onerous burden even for an experienced teacher who can take a number of time-saving short cuts in terms of lesson planning and preparation.

Many schools now include so-called “deep marking” or “triple marking” in their lists of “non-negotiables”, but there are at least two things that I think all teachers should know about these policies.

1. “We have to do deep/triple marking because of Ofsted”

No, actually you don’t. In 2016, Sean Harford (Ofsted National Director, Education) wrote:

 [I]nspectors should not report on marking practice, or make judgements on it, other than whether it follows the school’s assessment policy. Inspectors will also not seek to attribute the degree of progress that pupils have made to marking that they might consider to be either effective or ineffective. Finally, inspectors will not make recommendations for improvement that involve marking, other than when the school’s marking/assessment policy is not being followed by a substantial proportion of teachers; this will then be an issue for the leadership and management to resolve.


2. “Students benefit from regular feedback”

Why yes, of course they do. But “feedback” does not necessarily equate to marking.

Hattie and Timperley write:

[F]eedback is conceptualized as information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding. A teacher or parent can provide corrective information, a peer can provide an alternative strategy, a book can provide information to clarify ideas, a parent can provide encouragement, and a learner can look up the answer to evaluate the correctness of a response. Feedback thus is a “consequence” of performance.

So a textbook, mark scheme or model answer can provide feedback. It does not have to be a paragraph written by the teacher and individualised for each student.

Daisy Christodoulo makes what I think is a telling point about the “typical” feedback paragraphs encouraged by many school policies:

[T]eachers end up writing out whole paragraphs at the end of a pupils’ piece of work: ‘Well done: you’ve displayed an emerging knowledge of the past, but in order to improve, you need to develop your knowledge of the past.’ These kind of comments are not very useful as feedback because whilst they may be accurate, they are not helpful. How is a pupil supposed to respond to such feedback? As Dylan Wiliam says, feedback like this is like telling an unsuccessful comedian that they need to be funnier.

Advertisements

5 Comments

Filed under Assessment, Education, Humour, Uncategorized

Look at the pretty pictures…

Uniformity of practice seldom continues long without good reason.

So opined the estimable Dr Johnson in 1775. In other words, if a thing is done in a certain way, and continues to be done in that same way for a number of years by many different people, then it is a pretty safe bet that there is a good reason for doing the thing that way. And this is true even when that reason is not immediately apparent.

For the choice of this situation there must have been some general reason, which the change of manners has left in obscurity.

— Samuel Johnson, A Journey To The Western Islands of Scotland (1775).

Consider the following examples of “uniformity of practice”:

wp-1487108711715.png

wp-1487109018433.pngexam1

They are fairly bog-standard GCSE examination questions from the last two years from three different exam boards. But compare and contrast with an O-level Physics paper from 1966:

exam2.jpg

exam3.jpg

 

The “uniformity of practice” that leaps out at me is that the more modern papers, as a rule, have many more illustrations than the older paper. Partly, of course, this is to do with technology. It would have been (presumably) vastly more expensive to include illustrations in the 1966 paper.

Even if we assume that the difficulty level of the questions in the modern and older papers are equivalent (and therein lies a really complex argument which I’m not going to get into), there is a vast difference in the norms of presentation. For example, the modern papers seems to eschew large blocks of dense, descriptive text; this extends to presenting the contextual information in the ultrasound question as a labelled diagram.

Now I’m not saying that this is automatically a good or a bad thing, but there does seem to be a notable “uniformity of practice” in the modern papers.

Now what could the “general reason” for this choice?

Rather than leave the “change of manners” responsible for the choice “in obscurity”, I will hazard a guess: the examiners know or suspect that many of their candidates will struggle with reading technical prose at GCSE level, and wish to provide visual cues in order for students to play “guess the context” games.

Now I’m not assigning blame or opprobrium on to the examiners here. If I was asked to design an exam paper for a wide range of abilities I might very well come up with a similar format myself.

But does it matter? Are we testing Physics or reading comprehension here?

My point would be that there can be an elegance and beauty in even the most arid scientific prose. At its best, scientific prose communicates complex ideas simply, accurately and concisely. It may seem sparse and dry at first glance, but that is only because it is designed to be efficient — irrelevancies have been ruthlessly excised. Specialised technical terms are used liberally, of course, but this is only because they serve to simplify rather than complicate the means of expression. 

Sometimes, “everyday language” serves to make communication less direct by reason of vagueness, ambivalence or circumlocution. You might care to read (say) one of Ernest Rutherford’s papers to see what I mean by good scientific prose.

The O-level paper provides, I think, a “beginner’s guide” to the world of scientific, technical prose. Whereas a modern question on falling objects might tack on the sentence “You may ignore the effects of air resistance” as an afterthought or caveat, the O-level paper uses the more concise phrase “a body falling freely” which includes that very concept.

To sum up, my concern is that in seeking to make things easier, we have actually ended up making things harder, and robbing students of an opportunity to experience clear, concise scientific communication.

22 Comments

Filed under Assessment, Education, Physics

The Gamesters of Sparta

Sir. It must be considered, that a man who only does what every one of the society to which he belongs would do, is not a dishonest man. In the republick of Sparta, it was agreed, that stealing was not dishonourable, if not discovered.

— Samuel Johnson

At a recent event, the speaker asked us to consider a hypothetical conundrum: what if one GCSE Triple Science student was strong in (say) Chemistry and Biology, but significantly weaker in GCSE Physics? 

What course of action would you recommend? Extra support in Physics, was the consensus reply. 

Actually, said the speaker, the smart “Progress 8 Maximisation Strategy” would be to:

  1. Tell the student to focus her efforts entirely on Biology and Chemistry and completely ignore Physics. . .
  2. . . . but keep her entered for GCSE Physics anyway, and make sure that she goes into the exam hall and writes her name on the Physics papers, even if she does nothing else.

That way, she has ostensibly followed a full and balanced curriculum. She has, after all, been entered for all three Science subjects.  And, since Progress 8 counts only the two highest Science grades (or so I’m told), the student’s contribution to the school’s league table position would be also be secure.

H’mm. Dishonest? No. In the school’s best interests? Definitely. In the student’s best interests? Erm . . . on balance, no.

Sadly, as the character Joseph Sisko (ably played by Brock Peters) once observed on Star Trek: Deep Space Nine: “There isn’t a test that’s been created that a smart man can’t find his way around!” And that includes Progress 8 . . .

Sir, I do not call a gamester a dishonest man; but I call him an unsocial man, an unprofitable man. Gaming is a mode of transferring property without producing any intermediate good. 

— Samuel Johnson

3 Comments

Filed under Assessment, Education, Society

They Wouldn’t Let It Lie: The Twelve Physics Pracs of Gove (Part 3)

image

One time a whole lot of the animals made up their minds they would go shares in building a house. There was old Brer Bear and Brer Fox and Brer Wolf and Brer Raccoon and Brer Possum — everyone right down to old Brer Mink. There was a whole crowd of them, and they set to work in building a house in less than no time.

Brer Rabbit was there too, of course, but he said it made his head swim to climb up the scaffold and build like the others, and he said too that he always got sun-stroke if he worked in the sun — but he got a measure, and he stuck a pencil behind his ear, and he went round measuring and marking, measuring and marking. He always looked so busy that all the other creatures said to each other that Brer Rabbit was doing a mighty lot of work.

And folk going along the road said that Brer Rabbit was doing more work than anyone. But really Brer Rabbit wasn’t doing anything much, and he might just as well have been lying by himself in the shade, fast asleep!

Brer Rabbit Gets A House by Enid Blyton

Of all the things that could be dumped overboard during a radical curriculum overhaul, the dreadful, unholy mess known variously as “controlled assessment” or “coursework” or “practical assessment” (but whose names are actually legion) would certainly get my vote.

So, I was actually faintly encouraged by the reformed A-levels insistence that students have to DO twelve “required” practicals, and that all schools have to provide is evidence that their students have DONE those practicals to allow a “practical endorsement” to be ticked on exam certificates. Assessment of students’ practical skills would be in the final examinations.

In my naivety, I thought that a set of properly written laboratory notebooks would be sufficient evidence for the practical endorsement to be awarded. I actually enjoyed explaining the protocols for keeping a lab book to our AS Physics group; for example, the idea that it should be a contemporaneous working document, replete with mistakes and crossings out — proper science in the raw, so to speak, warts and all. And not a suspiciously pristine, antiseptic and bowdlerised “neat” copy. And I thought our students responded gamely to the challenge, even down to worrying whether the pen with erasable ink counted as an ‘indelible pen” or not.

But, goddammit, the latest email from our exam board shows that the JCQ wouldn’t let it lie, they wouldn’t let it lie.

Now, we have to minutely “track” (dread word!) our students’ practical skills, verily even unto recording onto the Holy Spreadsheet if we have indubitable observational evidence of each student reading an Instruction sheet or not.

Oh deary deary me. It calls to mind Wilfred Owen’s memorable lines about being fit to “bear Field-Marshal God’s inspection”.

But it won’t be Field-Marshal God inspecting us. Instead, it will be a vast floppy-eared army of snaggletoothed practical-assessor-Brer Rabbits, hopping all over the land, measuring and marking, marking and measuring…

I give up: this seems to me like defeat, a return to the discredited and unlamented paradigm of controlled assessment. This is defeat, a totally avoidable defeat that has been snatched from the ravening jaws of victory…

The Twelve Physics Prac of Gove Part 1
The Twelve Physics Pracs of Gove Part 2
Bring Back POAE!

8 Comments

Filed under Assessment, Science

Educational Defeat Devices

image

The Volkswagen Emissions Test Defeat Device needs no introduction:

Full details of how [the defeat device] worked are sketchy, although the EPA has said that the engines had computer software that could sense test scenarios by monitoring speed, engine operation, air pressure and even the position of the steering wheel.
When the cars were operating under controlled laboratory conditions – which typically involve putting them on a stationary test rig – the device appears to have put the vehicle into a sort of safety mode in which the engine ran below normal power and performance. Once on the road, the engines switched out of this test mode.
The result? The engines emitted nitrogen oxide pollutants up to 40 times above what is allowed in the US.
BBC News 4/11/15

This perceptive post from cavmaths shows , I think, the danger of relying on widely used educational “best practice” short cuts. They can actually be deleterious to student understanding. In short, many of them are simply “educational defeat devices”, clever tricks designed to give a false impression of student performance under artificial test conditions, cheats that fall apart when tested in the real world.

1 Comment

Filed under Assessment, PIXL, Society

PIXL: panacea or poison?

“How the understanding is best conducted to the knowledge of science, by what steps it is to be led forwards in its pursuit, how it is to be cured of its defects, and habituated to new studies, has been the inquiry of many acute and learned men, whose observations I shall not either adopt or censure”.
–Samuel Johnson, The Rambler, April 1750

A colleague described a recent visit to a highly successful science department that has drunk mighty deep of the PIXL well. I shall summarise some of her observations and comments below. My reactions varied from intrigued to puzzled to horrified, but in keeping with the Johnson quote above, I shall endeavour to urge neither adoption nor censure — at least until I have thought about it some more.

Item the first: textbooks are forbidden. Students are taught from in-house PowerPoints and worksheets which are made available online for individual study by students. My colleague reported that she visited several classes in the same year group, and all the teachers were teaching the same topic with the same PowerPoint — and were often on exactly the same slide at the same time! Reportedly, this system was set up because science leaders were not satisfied with the quality of lessons being planned by individual teachers. For myself, I couldn’t help but be reminded of the Gaullist education minister who claimed to know which page of which textbook children throughout France would be studying on that very day . . .

Item the second: science leaders have exhaustively analysed the GCSE exam board specification to produce the materials mentioned above. Every learning point is translated into “student friendly” language and covered in detail. My information is that a typical starter activity might be for students to copy down a summary of important information from a PowerPoint, before practising application using worksheets and past paper questions. These are often peer marked. Since planning and resource making have been centralised, the workload of the classroom teacher appeared to be more manageable than in many schools.

Item the third: students are regularly tested. Test papers are gone over with a fine tooth comb by the science team and areas of weakness identified. These are addressed in large, multiclass study skills sessions led by the head of science in the assembly hall, teaching from the front (brave woman!) using an old fashioned OHP and transparencies! (Sigh! Now that takes me back: I can almost smell the banda machine solvent as we speak.) Students are sat at exam desks for the session, and the hall is supervised by teaching staff and SLT (including the headteacher on the day my colleague visited). This is followed by a “walk and talk” mock (i.e. the answer is modelled by the Head of Science on her trusty OHP), followed by individual exam practice under exam conditions.

And so we come to the question: shall we adopt or censure these observations?

The truth is: I am not sure.

On the one hand, I can see how this might be a rapid and effective way to improve results, especially in a school with an inexperienced science team. And the part of me that actually likes writing schemes of work and resources would relish the challenge of developing such a scheme. And I’m told that percentage science pass rates improved significantly from the low teens to the high eighties . . . over the course of a single year! And you can’t really argue with such success, can you? (Actually, yes you can — see this post on the Halo Effect) Also as Lt. Worf of the starship Enterprise once observed: “If winning isn’t important, why keep score?”

And yet . .

Part of me rebels at such regimentation. Is this an example of the “mcdonaldisation” of education, the continuing process of deskilling the classroom practitioner? I genuinely hate to say this, but given this model maybe Sir Ken Robinson has a point; although this particular iteration seems to owe more to Taylorism rather that the nineteenth century workhouse.

Use another teacher’s PowerPoint? Ugh! I’d rather be forced to use his toothbrush . . .

And, while I grant that many examination questions are indeed fit for purpose and thoughtfully designed to expose misunderstandings and misconceptions, I cannot help the feeling that our examination system has become an overly-powerful tail wagging an emaciated dog.

Is learning truly synonymous with exam success? Have we become so enamoured of the assessment of learning rather than learning itself that we, like the Scarecrow in The Wizard of Oz, would not consider ourselves truly learnèd unless we hold a diploma saying that we are?

image

Why, anybody can have a brain. That's a very mediocre commodity . . . great thinkers . . . think deep thoughts and with no more brains than you have. But they have one thing you haven't got: a diploma.

I shall leave the final word to my friend Sam Johnson:

“The great differences that disturb the peace of mankind are not about ends, but means. We have all the same general desires, but how those desires shall be accomplished will for ever be disputed.”
The Idler, December 1758

12 Comments

Filed under Assessment, Education, PIXL, Science, Society

Never Mind The Data, Feel The Noise (or, seek the signal, young Jedi)

Everyone in education loves data.

image

This is the only time it is correct to use the word "Data" in the singular...

Or at least claims to. One sometimes wonders what would happen to the UK education system if a computer virus disabled every Excel spreadsheet overnight — h’mmm, perhaps someone should get in touch with those nice hacker people at Anonymous . . .
image

However, I digress. I wanted to share a recent epiphany that I’d had about data, particularly educational data. Perhaps it’s not much of an epiphany, but I’ve started so I’ll finish.

It came when I was listening to an interview on the evergreen The Jodcast (a podcast produced by the Jodrell Bank Radio Observatory). Dr Alan Duffy was talking about some of the new technologies that need to be invented in order to run the new massive Square Kilometre Array radio telescope (due to begin observing in 2018):

And then we have to deal with some of the data rates . . . essentially we recreate all of the information that exists on the internet today, and we do that every year without fail, it just keeps pouring off the instrument. And what you’re looking for is the proverbial needle in the haystack . . . how do you pick out the signal that you’re interested in from that amount of data?
The Jodcast, October 2014, 18:00 – 21:00 min approximately [emphasis added]

The realisation that hit me was: it isn’t the data that should be centre stage — it’s the signal that’s contained within that data. And that signal can be as hard to find as the proverbial needle in a haystack, even without data volumes that are multiples of the 2014 Internet.

A simple example from the history of science: Edwin Hubble’s famous graph from 1929 that was one of the first pieces of evidence that we exist in an expanding universe. The data are the difficult and painstaking measurements made by Hubble and his colleague Vesto Slipher that are plotted as small circles on the graph.

image

The signal is the line of best fit that makes sense of the data by suggesting a possible relationship between the variables. Now, as you can see, not all the points lie on, or even close, to the line of best fit. This is because of noise — random fluctuations that affect any measurement process. Because Hubble and Slipher were pushing the envelope of available technology at the time, their measurements were unavoidably ‘noisy’, but they were still able to extract a signal, and that signal has been both confirmed and honed over the years.

In my experience, when the dread phrase “let’s look at the data” is uttered in education, the “search for a signal” barely extends beyond simplistic numerical comparisons: increase=doubleplus good, decrease=doubledoubleplus ungood.

The way we use currently use data in schools reminds me of SF author William Gibson’s coining of the term cyberspace (way back in the pre-internet 1980s) as the
 

consensual hallucination experienced daily by billions of legimate operators . . . a graphic representation of data abstracted from the banks of every computer in the human system
— William Gibson, Neuromancer (1984)

In my opinion, almost the whole statistical shebang associated with UK education, from the precipitous data-mountains of the likes of RAISEOnline (TM) to the humblest tracking spreadsheet for a department of one, is actually nothing more than a ‘consensual hallucination’.

The numbers, levels and grades mean something because we say they mean something. And sometimes, it is true, they can tell a story.

Let’s say a student has variable test scores in one subject over a few months: does this tell us something about the child’s actual learning, or about possible inconsistencies in the department’s assessment regime, or about the child’s teachers?

My point is that WE DON’T KNOW without cross referencing other sources of information and using — wait for it — professional judgement.

I believe that the search for a signal should be central to any examination of data, and that this is best done with a human brain through the lens of professional experience. And, given the inevitability of noise and uncertainty in any measurement process, with a generous number of grains of statistical salt.

12 Comments

Filed under Assessment, Education, Levels, Physics, Science, Society