Originally published in Communication News 5/1 1997 as a review of Aspects of Literacy: Assessed Skills Levels Australia 1996, W. McLennan, Australian Statistician, Australian Bureau of Statistics 1997.
The issues raised are perennial, and the methods used in national literacy studies remain unchanged.
This review is a companion to Literacy: an ecological view.

LADY BRACKNELL: The whole theory of modern education is radically unsound. Fortunately in England, at any rate, education produces no effect whatsoever. If it did, it would prove a serious danger to the upper classes, and probably lead to acts of violence in Grosvenor Square. Oscar Wilde’s The Importance of being Earnest, first performed at St James’s Theatre 14 February, 1895.

The above quotation is, among other things, prophetic. In 1968, at the height of the Vietnam war protests, there was a violent demonstration in Grosvenor Square outside the US Embassy.

A political environment

The recent political debates surrounding literacy statistics confirm, yet again, that literacy is a potent political issue. The timely publication of the ABS study of adult literacy has added an authoritative source of data to the debate, though, regrettably, the data is potentially misleading. In its customary professional manner, the ABS has conducted a methodologically rigorous survey resulting in high quality data. But there are assumptions built into this study about the nature of literacy—what it is, how it might be measured, and how we might improve it—which cannot be taken at face value. The opening sentence of the introduction firmly locates literacy within a particular socio-economic view of the world.

The increasing complexity of our society and the need for a more flexible and highly educated workforce mean that individuals need to be able to read, comprehend and apply information of varying difficulty from a range of different sources to function effectively at work and in everyday life. (p. viii)

In this view of the world, literacy is firmly linked to employability and wealth creation, and the ‘problem’ of inadequate literacy is a problem for individuals. It follows from this view that if there is a deficit in individuals’ literacy, then it must be remedied by changing the individuals with the deficit. In the current debate, this view often manifests itself by a focus on teachers. Teachers, so the argument runs, are responsible for helping individuals become literate. Therefore, if our society has a literacy deficit, it must be the teachers’ fault. We pay teachers to give individuals literacy skills so that the individuals can make their economic contribution to society. Poor literacy means poor value for money. In other words, if the ‘evidence’ shows that the products of our educational system have a literacy deficit—and by implication cannot work as a result—then the teachers as the manufacturers of this product are to blame not only for the literacy ‘problem’ but also for the resultant unemployment ‘problem’: a neat and superficially plausible causal chain leading from poor teaching to unemployment, with the teachers as the villains. But such simple stories rarely enable us to deal with complex issues, and what such a story tells us is really about the current political environment and the limited nature of political debates in our society. On the issue of literacy, it tells us very little.

A different story

It is possible to tell a different type of story about literacy if one begins with an environmental metaphor rather than a physical science metaphor. Rather than a causal chain of events—from individuals, through teachers, to the unemployed—it is possible to think of literacy as a series of interdependencies, much in the way that an ecologist would look at the interdependence of organisms within an ecosystem. Within the literacy environment, there is minimally an interdependency between writers and readers, between those who create texts and those who use them; writers exist for readers and vice versa. In such a view the health of literacy in our society would be measurable by looking closely at the health of the relationship between readers and writers rather than at the skills of readers as an isolated abstract entity. Using evidence of authors’ capacity to create texts in combination with evidence of readers’ capacity to use texts, we can tell a very different story about literacy levels. Few people who have commented on the ABS data have mentioned the documents which the ABS used as test material in its study. This in itself is a symptom of seeing literacy as a reader problem, rather than a problem of interdependence between readers and authors. Appendix A of the ABS study gives an account of how the literacy tests were developed in the United States and shows the documents used in the testing. By examining this test material, it is possible to make some observations about the capacity of our society to generate understandable documents. One of CRIA’s methodologies for examining documents is to apply a suite of benchmark criteria that compare the content and structure of the text with known features of good document design. Applied to the test material used in the ABS study, not one of these documents would satisfy these benchmark criteria. I will illustrate this point with one example.

Label design: is the designer illiterate?

One of the ‘simple’ documents used in the ABS literacy test is a Canadian over-the-counter medicine label (see Figure 1).

Figure 1. Canadian label used in ABS study


The clumsy language and layout requirements of regulation have been at work in Canada in much the same way as they work in Australia. Indeed, it is not altogether unlike an Australian label (Figure 2) that we used in a study which explored some of the same issues as those explored by the ABS, although from a different point of view (Rogers et al 1995).

[Ed’s note: the regulations for medicine label design in Australia were changed in 2004 as a result of the Rogers et al study.]

Figure 2. Australian label used in our study

Our study, like the ABS’s, was to find out what tasks readers could perform correctly with the label. But, unlike the ABS, we were also interested to see whether redesigning the label as a result of diagnostic testing and modification (Figure 3) would lead to any improvements in the number of tasks that readers could perform correctly.

Figure 3. Improved Australian label used in our study

cri_prinsol_online

The results (Table 1) show that in most instances the redesigned label out-performed the original. Whose literacy then do these tests measure: that of the label readers or that of the label authors? Clearly, there is a component of the literacy ‘problem’ in our society which cannot be blamed on the readers or those who teach them.

Table 1 Percentage of people giving correct answers to questions

Question old label new label
What can this medicine be used for? 18% 79%:
Who should not use this medicine? 14% 48%
How much should you use? 60% 71%
How often should you use it? 78% 77%
When should you stop using this medicine? 6% 20%
How are you supposed to use it? 27% 40%
Is there anything you should not do while taking this medicine? 44% 74%
How should you keep this medicine? 7% 54%
When should you dispose of this medicine? 50% 76%
What are the active ingredients? 70% 88%
Average 37% 63%

In any economic analysis of the costs and benefits of different strategies for improving literacy, we might ask whether it is more cost effective to improve educational outcomes for the large number of readers in our society, or whether it might be more cost effective to improve the quality of outcomes with the relatively small number of authors.

I suspect that we might achieve a significant and cost effective improvement in literacy if we devoted attention to those who create poorly designed documents, rather than those who have to read them. But within an environmental view it would be wrong, and indeed incorrect, to point the finger at authors instead of readers. Authors too are constrained by the systemic features of the environment in which they function. Nothing so clearly illustrates this than medicine label designs. The reason why the labels are poorly designed is the many regulations surrounding labelling in both Australia and Canada. Neither provide an environment in which good design can flourish.

Blaming the author in this case is as pointless as blaming the reader or those who teach us to read. In the ABS study, poor document design has not been eliminated to get closer to readers’ actual abilities. Instead, poor design of both text and layout are all too evident, making understanding difficult, if not impossible.

These poor design features are referred to in the ABS study as ‘distractors’: that is, they distract readers from the task they have been set in the test. In other words, the test asks the reader to compensate for poor design; and the measure of one’s literacy is presumably the measure of one’s ability to compensate.

The best that can be said from the ABS data from this point of view is that we Australians are as good (or bad) as Canadians at dealing with, compensating for, and putting up with poorly designed documents.

This leads us to a final observation and the most serious question about the ABS study. The reason why the ABS used the USA methodology and material was to maintain comparability between Australia and other countries. In certain areas of data collection, such a practice is clearly desirable. But when data from such internationalised studies are used to diagnose and then offer remedies for local problems, we must seriously question whether the literacy environment in Australia is comparable to that of other countries.

I think there are good cultural reasons for suggesting that our environment is sufficiently different. For example, in the case of the medicine label, it is clear that the Canadian regulations lead to a different ‘dialect’ of illiteracy—a different set of distractors—than the Australian regulations. We have to put up with a different type of distractor in our daily use of medicines. I suspect this could have some effect on the results. We try to perform specific tasks in accordance with our specific needs, not in accordance with abstract definitions of intellectual tasks. If the ABS had drawn on local material such as Tattslotto or TAB, it would probably have found a higher level of literacy and numeracy against the same type of abstracted intellectual task.

The ABS is funded by Australia to undertake collections that are meaningful and useful to Australians. But this study has paid more attention to potentially misleading international comparisons and notions of literacy than it has to the specific Australian environmental factors that have to be taken into account in the ongoing literacy debate.

References

McLennan W 1997 Aspects of Literacy: Assessed Skills Levels Australia 1996. Canberra: Australian Bureau of Statistics

Rogers D, Shulman A, Sless D, & Beach R 1995 Designing Better Medicine Labels: report to PHARM. Canberra: Communication Research Institute of Australia