Assessment: reports from the ATN Conference
Assessment Practice: a manifesto for change
ATN Assessment Conference Keynote; Chris Rust
Rust reviews the literature and popular inquiry to establish that assessment is a major focus for concern. In particular, he points to Boud’s words: “students … cannot (by definition if they wish to graduate) escape the effects of poor assessment…. We must confront the ways in which assessment is undermining learning.” (Boud, 1995).
Rust then outlines the process by which the Manifesto, for which he played a leading role, was developed in 2007. A key aspect of the manifesto is the establishment of standards: standards provide guidance to students, enable them to monitor, and through evaluation ensure reliability and relative judgments to be made about students completing courses at university.
What is the manifesto? Its tenets are all about what is wrong and why it needs to change.
Why is change needed? The system is broken, assessment can even lead students to become less interested in deep learning; our practices – for example grading students across %ages (61% or 62%), or adding up score from different assignments, or working from different scales conventionally; is irrelevant and non-supportable. Rust emphasises that the institutional rules – length of assignments; numbers of assessment tasks and so on that are meant to produce consistency – are actually false. Points to deeply held disciplinary differences – for example the difficulty of getting a First in history compared to mathematics.
Rust also critiques the emphasis on ‘transparency’ and explicitness: not that being explicit is bad, it is just that it doesn’t achieve all that we claim it might. The explicitness of criteria and process and so on simply refines the levels of debate about what the words mean. There are other steps to be taken beyond just detailing in words the ways we might judge good work. Notes in particular the work of Royce Sadler (1987, 2008) on this point and concludes with Polanyi’s comment “we can know more than we can tell”. To some extent back this up by arguing for a community of practice model: students need to learn about the culture and conventions of the field in which they are working so as to interpret the criteria and processes of assessment; thus Rust asserts the importance of feedback as dialogue. Ultimately, (Astin 1997) student achievement is predicted by the degree of staff-student and student-student interaction. Gibbs (2007) – high levels of student involvement in a department generate excellence in research and teaching (rankings of departments).
Time, meaning duration, is important: lengthy involvement (assessment, feedback and re-assessment) includes “rehearsal” across the length of a degree, not just within a unit (Knight and Yorke, 2004). Cites examples of quality research – higher-grading areas, even when they covered very different subject matter – were not marking higher; they were more likely to be holistic programs in which there was a clear coherence between year 1 and year 3, and staff worked together on the assessment patterns. Such programs are more likely to create a community within which apprenticeships can occur over a long period of time; they are also likely to enable staff to inculcate new academics into the culture of assessment practice. One reason for the value of such apprenticeships in assessment is that being able to self- and peer-assess is a key graduate outcome.
A critical commentary
One of the slightly troubling aspects of the current debates on, and calls for change in, assessment is that they cite literature from at least two decades. Undoubtedly there has been a lot of conservatism over this time, but to be fair, there has also been significant change. So, is it legitimate to use ‘calls for action’ from the 1980s and 1990s to found action in the 2010 period onwards? It is also clear that there is significant overgeneralisation: comments about examinations, for example, imply that everywhere, everywhen, there is an examination. In fact, significant areas of the academy – particularly in the creative and humanities disciplines – are not especially interested in exams, do not use them, and have already got a strong tradition of authentic, deep-learning, and student-centred assessment. What place is there for these academics in all of the heat and light about our failings?Rust’s work puts forward an argument for the invalidity of many policies which are used to guide assessment; yet he does not go further to consider the reasons for these policies – the attempt to standardise and control the diverse and complex disciplinary array which makes up a university. Thus, to some extent, Rust’s arguments for the need for greater standardisation, and control and overarching unification of assessment is challenged by the fact that the problems he describes have in fact been caused by this kind of movement in higher education in the first place. That said, Rust usefully points out that – in some cases – people do not distinguish between one discipline and another (for example, in employment of graduates generally); thus different between disciplines have a real impact on the general conditions of higher education as a social institution: universities, while traditionally an aggregration of disciplines and professions engaged in parallel play, have become – or are becoming – singular entities whose brand, products (graduates) and value proposition constrain and corral disciplinary diversity to the central corporate mission.
It is also important, when thinking about the current practical debates and operational plans for assessment improvement, to emphasise the standards and quality assurance agenda. Rust contrasts US and British systems: no attempt in the US to presume that all graduates from all universities can be provided with equality such that one can compare graduates across degrees, disciplines and institutions. Such a move would be madness from a US perspective. Rust neatly constrasts that view with the British government’s enthusiasm for standards, based in large measure on the massive public investment in universities. Perhaps more attention should be paid, therefore, to how assessment is not about learning but about the management of the complex political economy of public expenditure in nations that have responded to the triumph of the market by attempting to explain and justify public expenditure to avoid public political concern about taxation.