Patoss-BCS Readability Project

A successful and innovative Patoss project

by Kath Morris
Former Bulletin Editor and Member of the Patoss Readability Project Team

What is readability? What are the factors which decide whether an examination candidate can access questions easily if s/he has reading difficulties? How can exam boards make questions readable without compromising the integrity of the exam by making it 'easier' for some candidates than others? These are questions which must be considered by exam boards and their scriptwriters now that equality of access to fair assessment is required by the Disability Discrimination Act (DDA) 2004. This Act extended to 'organisations that confer, renew or extend a professional or trade qualification' the requirements of Part 2 of the Disability Discrimination Act (DDA 1995). From 2004, it has been unlawful for qualifications bodies to discriminate against a disabled person, and these bodies will 'have a duty to make a "reasonable adjustment" if a provision, criterion or practice, or a physical feature of premises, places a disabled person at a substantial disadvantage in getting or retaining a qualification'.

The European Computer Driving Licence(ECDL) is an internationally recognised qualification which enables people to demonstrate their competence in computer skills. It is the fastest growing IT user qualification in over 125 countries. 1.5 million people in the UK have chosen ECDL as the way to learn all important IT skills. Candidates can start working for ECDL without prior knowledge of computer skills and the age range of candidates, working in schools, colleges and other Approved Centres is from 8 to 80+. Because of the wide range of candidates the ECDL attracts, access to examinations is a particularly important issue for the British Computer Society (BCS) which is licensed to manage the European Computer Driving Licence (ECDL) qualification in the UK.

PATOSS was approached by BCS to advise on revising question papers from their suite of ECDL examinations. The aim was to make them 'dyslexia-friendly' in the sense that they covered exactly the same knowledge and skills base as before, but were more accessible for those who found reading difficult. The papers in question were made up of paper-based exams, with some multiple choice and some context-based questions, which required candidates to draw information from computer files.

The ECDL syllabus is designed to cover the key concepts of computing, its practical applications and their use in the workplace and society. It is broken down into seven modules, each of which must be passed before an ECDL certificate is awarded.

"1. Basic concepts of IT
2. Using the computer and managing files
3. Word processing
4. Spreadsheets
5. Database
6. Presentation
7. Information and Communication"

The PATOSS project team consisted of Lynn Greenwold (project co-ordinator), Sarah Fitzalan Howard, Katherine Kindersley and Kath Morris. All members of the team were experienced learning support tutors, not IT specialists, but with a good working knowledge of the applications covered in the seven modules.

Setting up the project
Our first task was to decide:

  • what were the constituents of good readability in both multiple choice and context-based questions
  • how these could best be measured.

We found that although there was a little published research into the readability of school texts, there was very little information or research available concerning the readability of examination questions, particularly in relation to multiple choice questions. We also found there was little published research into the difficulties dyslexic candidates experienced in accessing such questions. The team therefore relied heavily on their own experience in supporting dyslexic candidates, and on feedback from the member of the team who was herself dyslexic. We should dearly have loved to test out our ideas on a 'guinea pig' group of dyslexic students across the age range and incorporate feedback from them into our recommendations, but this was not possible as we were dealing with live questions.

Usefulness of Readability tests
One of the first issues we looked at was the usefulness of readability tests. We considered using the Fog Index but decided that as the Flesch Kincaid was accessible to most computer users on Word, this would be more useful - especially as we were thinking of our findings being passed on to the BCS Qualifications team of exam script writers.
We used the Flesch Reading Ease formula and found it helpful for longer passages of text. (It is easily accessed on up to-date versions of word through the Tools Index.) However, the findings are not valid for passages of fewer than 100 words. For shorter passages which comprised most of the ones we dealt with, it provided only a very limited prediction of readability. We relied much more on our own judgements about readability, which we developed in the reviewing process, to provide a list of standards for readability, based on the broad issues of presentation, syntax and vocabulary. For each of these issues we devised a number of criteria some of which were applicable to multiple choice questions, some to non multiple choice questions, some to both.

Subject vocabulary
Subject vocabulary was also an issue in an area of study where so much vocabulary is used which is specific to IT and to individual computer applications. This vocabulary could not be discarded or replaced, but our project highlighted the importance of training candidates to understand any particular subject vocabulary they need. This applies to all students, but especially to those with dyslexia or other language difficulties, perhaps especially in IT. The other type of vocabulary which all exam candidates need to be trained to recognise is question language (for example 'Which of the following is...?').

Devising a protocol
Having considered some of these issues, the PATOSS team devised a protocol to analyse the exemplar questions from each module which we had been asked to make more readable, then to present them in a new dyslexia-friendly format. It was important that the questions covered exactly the same knowledge and skills base which the exam was originally designed to test. The aim was simply to make the questions easier to read and comprehend.

Completing the project
Once this preliminary work was done, the actual project was really underway and we worked through the summer to have it completed in order to offer our revised questions together with a final report at the end of August 2006. During the working period of the project close contact was maintained with BCS Qualifications, who kept a watching brief on our activities. The project team also met twice to discuss progress and kept regularly in touch by email.

Outcomes
As the project neared completion, as well as writing the report, extensive final revision of all our work was needed to make sure that it was consistent. However, our report was submitted in time, together with the revised questions.

We were rewarded when the BCS Qualifications informed us that they accepted our report and would be using the revised dyslexia-friendly paper from 2006/7. Initially these papers would be presented as an access option for centres to offer to those candidates identified as having difficulties with reading and comprehension. During 2007, BCS Qualifications is looking to incorporate the principles of readability which we proposed into all new papers when they devise a new bank of questions. PATOSS is then likely to have an advisory role.

It is an indication of the success of the project that BCS Qualifications have awarded PATOSS their Quality Award as a Manual Test Provider to recognise the high quality of the work that the organisation has produced. We are now entitled, as an organisation, to use a special logo to this effect on our official documents and correspondence.

Below we outline some of the issues which arose from the project which might be of interest to PATOSS members:

  • The importance of readability as an issue in examinations. If papers were more readable there could be less need for readers and extra time.
  • The constituents of readability which are embedded in layout and presentation, syntax and vocabulary.
  • The difficulties posed by subject vocabulary.
  • The demands on working memory posed by multiple choice questions
  • The problems posed for candidates with visual tracking difficulties in dealing with separate question and answer papers (usually for multiple choice questions) where answers have to be entered into a complex grid with a different format to the question paper.
  • The importance of plain, clear English for all candidates.
  • The need for research into the readability of exam papers, particularly in relation to candidates with reading and comprehension difficulties.
  • The fact that such difficulties might arise from dyslexia or other learning difficulties, but also from having English as a Second Language.

This article first appeared in the Patoss Bulletin, November 2006

Click here to download the full report
Patoss-BCS Readability Project Report

Back to all sheets