Making it real?

Notes on the sample materials for the

KS3 English tests in 2003

A level playing field

National tests in which the sole outcomes are overall levels are simply rough sorting procedures with no particular pedagogic functions. The aims are to place pupils on national scales of achievement in defined areas of the curriculum, ‘subjects’. It is not the purpose of these notes to discuss the particular changes to the KS3 English tests in 2003, but simply to review the sample test materials in these terms, the key questions being those of validity and reliability:-

• What do the tests purport to assess?
• What do they actually test?
• Is either of these worth testing?
• Are the tests reliable: do they properly and consistently discriminate between the target levels 4-7 in terms of valid aims, objectives and content?

A set of tests with overlapping objectives is projected for this overall sorting procedure:-

- with sets of assessment focuses (AFs) variously ascribed. In the Shakespeare paper the ‘Writing’ tasks have only a tangential connection with the plays studied.

With some fine scoring of particular points in the tasks set, this seems on the face of it a somewhat elaborate procedure for the simple reporting in overall levels required - with reliability at risk. Unless marks are awarded across the whole range of broad mark bands there can be a cumulative regression of marks to the mean in the aggregations of question, paper and subject marks resulting in a consequent loss of overall discrimination and the maximum numbers of pupils at risk at the level boundaries, especially in the mid ranges in over-peaked curves of distribution. (The ideal ‘bell curve’ of distribution would be a horizontal straight line, with pupils evenly spread across the whole aggregated mark ranges and so relatively few at risk at the level boundaries.)

‘Making it Real’:-

- and apparently linked thematically:-

- except that only one question (14) appears to make any connections at all, and that for but two of the 32 marks, and for but two of the three passages. This proves spurious in any case:-

• one purpose of each text
• one word or phrase to describe the language used in each text

- in which no link is actually made between the two texts. So, this apparently attractive contextualisation of the three passages is entirely spurious as far as the set tasks are concerned; they are each considered quite independently of each other. Where is the prompt to consider the texts in relation to each other and write on some aspect of the outlined linking theme that might have been looked for in a concluding, ‘synoptic’ question? But this is ‘Reading’, and the assessment focuses forbid:-

So this final question has a degree of triviality at the highest level (AF6: ‘writers’ purposes and viewpoints, and the effect of the text’) matched only by that of the first question: ‘one purpose…’, ‘one word or phrase to describe the language used’ - for each text and for but one mark each? The mark scheme lists contrasting one-word ‘correct answers’, as if anything half relevant will do (but ‘Do not accept descriptive’) and no further reflection is required. In these pre-tested sample KS3 tests it would have been helpful to know how successfully such questions discriminated the four levels of achievement for the tests, but no such information is provided in the sample materials.

The first question is one of three – for a total of four marks - aimed at the lowest level of achievement for the tests (AF2: information retrieval):

- as plainly stated in the onset of the first and third sentences:-

How productive a question was this: did significant numbers at the lowest level in the pre-tests really manage to misread the time as the ‘one pm’, as suggested by the mark scheme (not that ‘already one in the morning’ indicates the ‘precise time’)? Unless significant numbers at particular levels fail questions we are not testing anything.

We leap levels for the next question (AF4), if still for but one mark:-

- with a number of stylistic features to chose from, for the markers that is, not for the pupils, who have to predict as best they may what kinds of things are looked for:

But there is a mismatch here in any case, since, while the tension is built up, at text level, in the narrative context of the whole paragraph - with the indications of stress in the preceding sentences (‘dreary’, ‘labours’, ‘anxiety’, ‘agony’, ‘lifeless thing’…) - all the features noted in the mark scheme are internal to this final sentence, and so at sentence level.

But we are now confronted by one of the fundamental issues in this kind of testing: the assessment focuses used by the setters and markers are not made available to the pupils. If such questions are to be asked – if – then it was some virtue at least of multiple-choice testing to make the marking criteria more explicit in the sets of possible answers in the question formats, and so to concentrate setters’ minds on what was actually being asked. Question 3, similarly, also on AF4, would have been better set in a multiple-choice form, but at least it looks at inter-sentence features, if only for one mark in picking contrasts and one for a comment on their effectiveness. And that is a further problem with a handful of one-mark questions; they either question trivial points or trivialise more significant points.

The fourth question on this first passage does ask for some discussion – twenty lines allowed in the answer booklet, for five marks - on the writer’s ‘use of language’, in which pupils are asked to ‘comment on how the writer’:

• sets the scene
• describes the creature
• shows Dr Frankenstein’s reactions

This is under AF5, which includes ‘grammatical and literary features at word and sentence level’, though this is not made explicit in the question nor covered in the mark scheme by anything more than ‘Some relevant words and phrases, suggested by the prompts [in the question], are clearly identified’.

It is a similar pattern for the four questions on the second passage, with a minor piece of commentary set similarly as the fourth question (8), but under AF6 – as for the final question (14) on the third passage (above), except that no writing is required at all on the third passage. ‘Writing’ is set as a separate paper.

This comprises one piece of writing, contextualised in terms of audience and purpose in a given scenario:

A planning page is provided:

• words and phrase to begin and end your speech
• the advantages of teenagers having part-time jobs
• the disadvantages of teenagers having part-time jobs

But how long a speech? The only guidance given is the timing (45 minutes, including 15 minutes’ planning time) and the total marks (30). And a written speech - as an appropriate model for either speaking or writing in Year 9? We might ask pupils to write notes for a speech – assessing the ensuing speech, not the notes – or to write a piece (in so many words) for, say, a newsletter, but this bogus task is neither one nor the other, and no supported personal opinion is sought. The guidance on the three strands of the assessment given to the markers:

- again withheld from the pupils – gives the greatest weight to those surface features in writing of least significance in a script for speaking, supposing this were ‘making it real’.

But Writing is also the objective of the first task (Section A) in the Shakespeare paper – for over half the marks (20):

- not that the pupils are to know this – and the Twelfth Night Writing task is similarly misconceived:

For this half the marks are for ‘Sentence structure, punctuation and text organisation’ (6 marks) and ‘Spelling’ (4 marks), again, largely those surface features of writing of least significance in a script for talking.

There are further levels of unreality in the Twelfth Night task:

‘In charge of…’ - is that even remotely likely? There are real issues for 14-year olds to address about allowing ‘chips in the canteen’, and about the roles of ‘singing/music’ and ‘games/sports’ in the curriculum, but the other options put up for banning as ‘fun or enjoyment’ – ‘jokes’ and ‘smiling’ - are simply as silly as the task set, for which the play studied is the barest excuse.

It is ‘smiling’ that gives the game away in this bogus scenario. Malvolio is certainly fooled into attempts at smiling himself, but it is not actually something he is shown to have condemned in others; this is merely a specious nod to the play that manages to get the reference wrong into the bargain! Are the pupils supposed to refer to Malvolio in their remarks? No reference to the play is required; the only tangential feature is that humour is required in the role for the higher marks under ‘Composition and effect’ (10 marks), but how are they to know?

The scenario for the Macbeth Writing task is also bogus: supposedly a request to ‘The Publicity Department’ for ‘a publicity booklet for teachers’ for ‘a video of Macbeth in modern dress’? In the absence of any other information offered about the video the pupils are asked to outline:

• what’s on the video;
• why it will appeal to young people;
• what the educational benefits are.

- as if all this were evident . Apart from the passing reference to ‘Macbeth’s delusions’ Example 2, awarded full marks in the sample mark scheme, shows no knowledge of the play whatsoever; this is ‘Writing’, not ‘Shakespeare’.

But what writing? A leaflet might be a single sheet, possibly folded; a ‘booklet’ presupposes several pages, a length way beyond the scope of a 30 minutes’ task. No target length is indicated, though there would be a tight word limit in a real publicity brief, with constraints on the writing and format imposed by graphics (for a video publicity leaflet!), layout and house style. In short, a more realistic task for group work using ICT – after watching such a video – but quite unrealistic as a short hand-written piece in an individual test task when none of this applies. All the pupils are asked to do is to ‘Write the text’, but that begs all the questions. The mark scheme does make some concessions to the use of an informal style – though, again, how are the pupils to know? - but it is prose paragraphs that seem to be required.

The context provided for the Henry V Writing scenario also undermines the task set: ‘The library is producing a collection of writing called "People We Admire"… Write your contribution…’ No length noted, but in half an hour the pupils are expected to:

• choose a famous person or someone you know;
• review what this person has achieved;
• explain what qualities make this person so special

- all without the access to library resources that would be the focus of such an invitation. And the tangential link to the play only confuses the issue:

What about heroes who oppose strong leaders (the ‘William Tell’ factor), for instance? The play is irrelevant to the task set. And that is the nagging question about these ‘Writing’ tasks set in the ‘Shakespeare’ paper: what have they got to do with the study of the plays?

That is down to the ‘Reading’ tasks – with more time advised (45 minutes, allowing 15 minutes for reading the extracts - if not suggested?) - but for fewer marks (18). The sample tasks are set on:

• the language
of the text (Henry V)
• the text in performance
(Macbeth)
• the motivation and behaviour of the characters
(Twelfth Night)

- in conventional enough extract questions, though with some unreality again in the ‘performance’ scenario set for the Macbeth scenes: