I have written many times about Assessment for Learning and the self assessment or evaluation of learning. Assessment for Learning is the idea of formative assessment to support the learning process, rather than most of our present assessment systems which are designed to support comparisons or as a screening mechanism for entry into higher education or education and training or into employment.
And self evaluation – it is what it says. The idea that learners are able to evaluate or assess tehir own learning, often with a surprising degree of insight and accuracy. Of course when they do this they own the assessment – it ceases ot be something that is done ot them but is part of their own reflective learning process.
But, say teachers, this is hard to do. Learners will not know how to do it. they will over-rate their own abilities.
So practical examples are always welcome and I was lucky enough to see today the self evaluation of one of my friend’s children in a school in Bremen (reproduced above).
The process went something like this. Last week the students – aged 8 – were asked to fill in their own assessments in the left had column. Then the sheets were passed to their two form teachers who also filled in the assessment in the right hand column. And then today there were individual meetings between teachers and the students to discuss the results. (It is interesting to note that like in previous expercises of this sort that I have seen, teachers tended to rate students slightly higher than the students themselves).
Seems pretty cool to me (even if a little overly emphasising behaviour and conformity) and much, much more useful than the UK Standard Assessment tests (SATs).

Last week I took part in a fairly impassioned Flash meeting debate about the use of the European Qualification Framework for the training of teachers and trainers. Opinions varied greatly between those who saw the EQF as a useful tool for promoting new qualifications for teachers and trainers to those who saw it as a barrier in this field.

Firstly, for non European readers, it may be useful to recapitulate what the EF is all about. The European Qualifications Framework (EQF), says the European Commission,  “acts as a translation device to make national qualifications more readable across Europe, promoting workers’ and learners’ mobility between countries and facilitating their lifelong learning.” The primary users of the EQF are seen as being bodies in charge of national and/or sectoral qualification
systems and frameworks. The idea is that once they have related their respective systems to the EQF, the EQF will help individuals, employers and education and training providers compare individual qualifications from different countries and education and training systems.

To achieve this the European Commission has designed a framework of eight different levels. Each of the 8 levels is defined by a set of descriptors indicating the learning outcomes expressed as knowledge, skills and outcomes relevant to qualifications at that level.

The problem is that it doesn’t work. For the moment I will ignore the more epistemological issues related to the definition of knowledge kills and competences. The biggest problem for me relates to levels. The Framework has mixed a series of different indicators to derive the levels. Some of the indicators are based on academic attainment. For instance the descriptor for knowledge at Level 8 states (can) “demonstrate substantial authority, innovation, autonomy, scholarly
and professional integrity and sustained commitment to the development of new ideas or processes at the forefront of work or study contexts including research.” Some are based on levels of responsibility and autonomy in work roles.  Level 4 talks of the ability to “supervise the routine work of others, taking some responsibility for the evaluation and improvement of work or study activities.” Others are based the complexity of the work being undertaken. Level four skills says: :advanced skills, demonstrating mastery and innovation, required to solve complex and unpredictable
problems in a specialised field of work or study.” Yet others are based on quite abstract ideas of knowledge. Level six knowledge comprises of “advanced knowledge of a field of work or study, involving a critical understanding of theories and principles.”

One of the biggest problems is the framework attempts to bring together applied knowledge and skills within a work process, work roles as expressed by responsibility and knowledge as expressed through academic achievement. And of course it is impossible to equate these, still less to derive a hierarchical table of progression and value. It is easy to pick holes: why for instance is “exercise management and supervision in contexts of work or study activities where there is unpredictable change” level 5, whilst having a “critical awareness of knowledge issues in a field and at the interface between different fields.”

There would appear to be a serie sof unspoken and implicit value judgements related both to the value of academic versus vocational and applied learning and to different roles within the workplace. There also seems to be an attempt to deal with work organisation with teamwork being written in as a high level function. Of course it maybe, but then again in particualr contexts teamwortk may not be so important. In some jobs, the ability for autonomous work may be important, in others not so. And how can we translate between such abilities, competences or whatever they are called and qualifications.

I do not think it is possible to design such a frameworks, nor do I think that levels are a useful concept, especially given the hierarchical structures of this and other similar frameworks. Why should one particular competence or skill be valued over another. Even more important is the idea of hierarchical progression. Lest this be thought to be merely an academic question, the UK government has already withdrawn funding support for those wishing to progress from one qualification to another at the same EQF related level.

one of teh aims of the Frameowrk is to promote Lifelong Learning. But an individual may not wish ot advance their learning in the social forms envisaged by the Framework. There is an assumption for instance that a teacher or trainer will progress towards being a manager, as represented in the higer levels within the EQF. But many teachers and trainers that I have talked to actually want to improve their practice as tecahers or trainers. Or they many wish to move ‘sideways’ – to learn more about working with particular groups or to undertake work as a counsellor for instance. The implicit progression routes inherent in the Frameowkr do not necessarily represent the way we work and learn. far better nore me is the idea of Learning Pathways. Learning pathways can represent a progression in our learning based on the context of our life and our work and based on individual interest and motivation. Our Learning Pathway may at times go upwards, downwards or sideways in the table of skills, competnces and knowledge as representd in the Framework. Such an idea of Learning Pathways is contiguous with the idea of Perosnal Learning Environments and of teh idea of more self directed learning. Of course it is useful to have a framework to assist in selecting progression routes and for counselling, guidance and support. But lets abolish the taxonomy of levels and start representing learning opportunties as what they are, rather than a somewhat oddly derived taxonomy trying to make things fit neatly which do not and embodying implict social values.

Last weeks Emerging Mondays seminar was on the topic of MOOCs and Open Course Models. The speaker was Dave Cormier from the University of Prince Edward Island.

Dave spoke about his experiences, so far, of the CCK MOOC on Connectivism and Connected Knowledge, the technological platforms being used to support participants, the tensions that exist within the course design and the peer support models that are being embraced.  Dave’s introduction led to a wide ranging discussion including the nature and furture of courses and communities, issues of scale, how to support learners, open accreditation and the future of open education – and …Humpty Dumpty and Alice in Wonderland!

If you missed the session – or would like to hear it again – we are providing you with three different versions. You can watch a replay of the event in Elluminate. This provides you with access to the sidebar chat discussion as well as to the audio.

Or – if you are short of time you can listen to an MP3 podcast of Dave’s introduction.

Or you can listen to the full session inline or on your MP3 player.

This is the link to the Elluminate version.

Podcast: Play in new window | Download

Subscribe: RSS

Podcast (podcast1): Play in new window | Download

Subscribe: RSS

Another post on the IPTS seminar on Learning 2.0 in Seville. This workshop was interesting becuase it brought togther researchers and practitioners from all over Europe. And, somewhat to my surpise, there was a fair degree of consensus. We agreed social software provided many opportunties for creating, raher than passively consuming learning. We agreed that learnng opportunities were being developed outside the classroom. We even agreed that the locus of control was switching from institutions to the learners and that this might well be a good thing. We agreed we were moving towards individual learning pathways and that learners needed to be supported to finding their pathways.

We agreed that the context of learning was important. Mobile learning would become increasingly important with the development of context sensitive devices. (Also see Serge Ravet’s post on User Generated Content or User Generated Contexts).

But there were also limits to the consensus. Whilst there appeared agreement on new roles for teachers, no-one was sure what that role was?

Much of the discussion centred on the scaffolding of learning. How much support did leaners need and how much of that support would come from teachers?

Neither were participants agreed on the future role of institutions. More critically, was Learning 2.0 something which happened outside the school, and had only a limited impact on institutional practice, or did it pose a fundamental challenge for the future of schooling?

There was even greater disagreement over curriculum. Should there be a curriculum for basic skill and knowledge that everyone should learn? Did learners need a basic grounding in their subject before theyc oudl develop their own learning pathways? Who should define such a curriculum? What was the role of ‘experts’ and who were they anyway?

And perhaps the greatest disagreement was over assessment and accreditation. Many of us felt that we needed to move towards community based formative assessment. Employers, we said, would be more interested in what people were able to do than formal certicates. Others, pointing to occupations such as doctors and plumbers felt there should be some form of standards against which people should be assessed and accredited.

A final comment on the form of the project. Although the work is about Learning 2.0 the present form of the work is decidedly Research 1.0. This research is important enough that it needs to be opened out to the community. It seems a wiki is being d veloped and when it is up I will blog here about it. In the meantime here are some of the photos of the flip charts used for brainstorming around different issues at the workshop. I will pass on any comments on this post to the project organisers.

Can we develop an Open Accreditation system.  What would we be looking for. In this post Jenny Hughes looks at criteria for a robust and effective cccreditation system.

An accreditation system depends on the robustness of the assessment system on which it is based.

Imagine you were in a shop that sold accreditation / assessment systems ‘off-the-peg” – what would criteria would you use if you went in to buy one?

Reliability
Reliability is a measure of consistency. A robust assessment system should be reliable; that is, it should be based on an assesssment process that yields the same results irrespective of who is conducting it or the environmental conditions under which it is taking place. Intra-tester reliability simply means that if the same asessor is assessing performance his or her judgement should be consistent and not influenced by, for example, another learner they might have just assessed or whether they feel unwell or just in a bad mood! Inter-tester reliability means that if two different assessors were given exactly the same questions, data collection tools, output data and so on, their conclusions should also be the same. Extra-tester reliability means that the assessor’s conclusions should not be influenced by extraneous circumstances, which should have no bearing on the assessment object.

Validity
Validity is a measure of ‘appropriateness’ or ‘fitness for purpose’. There are three sorts of validity. Face validity implies a match between what is being assessed or tested and how that is being done. For example, if you are assessing how well someone can bake a cake or drive a car then you would probably want them to actually do it rather than write an essay about it! Content validity means that what you are testing is actually relevant, meaningful and appropriate and there is a match between what the learner is setting out to do and what is being assessed. If an assessment system has predictive validity it means that the results are still likely to hold true even under conditions that are different from the test conditions. For example, performance assessment of airline pilots who are trained to cope with emergency situations on a simulator must be very high on predictive validity.

Replicability
Ideally an assessment should be carried out and documented in a way which is transparent and which allows the assessment to be replicated by others to achieve the same outcomes. Some ‘subjectivist’ approaches to assessment would disagree, however.

Transferability
Although each assessment should be designed around a particular piece of learning, a good assessment  system is one which could be adapted for similar  situations or could be extended easily to new activities. That is, if your situation evolves and changes over a period of time in response to need, it would be useful if you didn’t have to rethink your entire assessment system. Transferability is about the shelf-life of the assessment and also about maximising its usefulness

Credibility
People actually have to believe in yourassessment! It needs to be authentic, honest, transparent and ethical. If you have even one group of stakeholders questioning the rigour of the assessment process or doubting the results or challenging the validity of the conclusions, the assessment loses credibility and is not worth doing.

Practicality
This means simply that however sophisticated and technically sound the assessment is, if it takes too much of people’s time or costs too much or is cumbersome to use or the products are inappropriate then it is not a good assessment system !

Comparability
Although an assessment system should be customised to meet the needs of particular learning events, a good assessment system should also take into account the wider assessment ‘environment’ in which the learning is located. For example, if you are working in an environment where assessment is normally carried out by particular people (e.g teachers, lecturers) in a particular institution (e.g school or university) where ‘criteria reference assessment is the norm, then if you undertake a radically different type of assessment you may find that your audience will be less receptive and your results less acceptable. Similarly, if the learning that is being assessed is part of a wider system and everyone else is using a different system then this could mean that your input is ignored simply because it is too difficult to integrate.

Also, if you are trying to compare performance from one year to the next or compare learning outcomes with other people, then this needs to be taken into account.

Last week I write that with the emergence of open learning in the form of freely available courses, seminars and events on the internet, and with the increasing availability of open educational resources, open accreditation is the final frontier.

This week I will be writing a series of blog pots on accreditation. But first up an excellent quote from J,M Coetzee in his novel ‘Diary of a Bad Year’. Just substitute ‘open online networks for ‘peoples homes’

“In the days when Poland was under communist rule, there were dissidents who conducted night classes in their homes, running seminars on writers and philosophers excluded from the official canon (for example, Plato). No money changed hands though there may have been other forms of payment. if the spirit of the university is to survive, something along these lines may have to come into being in countries where tertiary eduction has been wholly subordinated to business principles. In other words the real university may have to move into peoples homes and grant degrees for which the sole backing will be the names of the scholars who sign the certificates.”

Continuing the open learning debate….I greatly like this diagramme by George Siemens. I think there is much of merit here. Very happpy to see acknowledgemnt of the importance of self publishing (as opposed to academic reporsitories). However there are a few things missed out.

Firstly if we take congniscence of Jenny Hughes’ defintion of learning as ‘to find and follow a track’ as counterposed to curriculum  from the latin ‘currere’, which means to run or race and ‘curriculum’ as race or racecourse, then instotutions and teacher have an important role in assisting learners in developing their own learning pathways.

A second important  role is that of assessment. But to understand this we need to decouple assessment from acceditation. If we design assessment as a learning process and move from assessemnt of learning to asssessement for learning this could become an integral part of the process of finding and developing learning pathways. This is not so utopian. Serendipitously. The Times newpaper today published an article about innovative assessment in UK universities. The struggle, though, as with self and peer assessment is in assessment having to match accreditation procedures. Without this link, we could open up all jinds of new forms of assessment.

A final point on accreditation. Many learners do not want or require accreditation. Indeed it is the formal accreditation procedures which deters them for signing up for a learning programme. And as Antnio Fini, talking about the home made certificate he got from the OpenEd2007 course, says: “all my connections, blog posts, comments, collective works, presentations, articles related to that experience, are still out there as tangible proofs of this learning. So I could equally put the OpenEd course in my CV and could ask to my supervisor to evaluate all that activity for credit in my PhD, also without that piece of paper!”

Why not put the learners in charge of accreditation. Lets leave it to them to decide how they wish to show what and how they have learned – albeit with support. I once co-ran a course with Jenny Hughes where we offered the particpants their certificate at the start of the courese. They refused! But it did raise the issue of why they were doing the course and how they valued learning. And that is an issue we need to bring to the fore.

Sort of like that saying. I don’t have much time ot post here – am constantly traveling – but will make a few quick commnets from the road (as an aside – countries like Estonia put other countries to shame when it comes to intrenet access – free fast wireless access is available almost everywhere).

Maybe it is juts the people i am talking too, buut there seems to be growing appreciation of the importance of informal learning and learning acquired in the workplace. At the same time I am slightly concerned that this appreciation for workplace and informal learning is being counterposed to formal training and qualifications. In this respect I think people are mixing up the schooling system and formal learning. Yes – I completely agree that our formal schooling system is out of date, frequently ineffective and promotes formal accreditation at the expense of learning. Putting it simply there are better ways to learn – and the money spent of formal schooling could be much more effectively deployed elsewhere.

But this is not to say there is no place for formal training and learning and for qualifications. Qualifications can play an important regulatory role – both for quality and in terms of preventing employer exploitation. Moreover such qualifications can prove aspirational – especially for young people. Yes – there are many issues around curriculum (I will return to this issue in a further post). Formal learning and training can provide a structure for learning. And formal learning and qualifications are not in opposition to informal learning – the two can go together.

I think there are problems in a fast changing economy as employment and work tasks and roles are fast changing. There is no guarantee that training for one particular occupation will guarantee employment in ten years time. Yet, all the empirical surveys we have carried out show that those who have undergone a formal training programme – regardless of subject – are more likely to participate in on-going learning in the future. Thus, even though the link between qualifications and employment may be weakening (especially in liberal market economies such as the UK) there remains a macro economic benefit to the provision of formal learning opportunities.

This story from thestar.com in Canada shows just why we need to tackle issues over assessment (thanks to Cristina Costa for forwarding). Ryerson University is threatening to expel a student for administering an on-line study group.

“First-year student Chris Avenir is fighting charges of academic misconduct for helping run an online chemistry study group via Facebook last term, where 146 classmates swapped tips on homework questions that counted for 10 per cent of their mark.

The computer engineering student has been charged with one count of academic misconduct for helping run the group – called Dungeons/Mastering Chemistry Solutions after the popular Ryerson basement study room engineering students dub The Dungeon – and another 146 counts, one for each classmate who used the site.

Avenir, 18, faces an expulsion hearing Tuesday before the engineering faculty appeals committee. If he loses that appeal, he can take his case to the university’s senate.

The incident has sent shock waves through student ranks, says Kim Neale, 26, the student union’s advocacy co-ordinator, who will represent Avenir at the hearing.

“All these students are scared s—less now about using Facebook to talk about schoolwork, when actually it’s no different than any study group working together on homework in a library,” said Neale.

“That’s the worst part; it’s creating this culture of fear, where if I post a question about physics homework on my friend’s wall (a Facebook bulletin board) and ask if anyone has any ideas how to approach this – and my prof sees this, am I cheating?” said Neale, who has used Facebook study groups herself.”

Ir seems fairly ridiculous that we spend so much time and trouble working on the use of new technologies for collaborative learning – and even more so for encouraging autonomous learning, only for academic institutions to take actions like this. Furthermore, every survey of employers suggests that the ability to work as part of a team is both one of the most sought after competencies and one which they feel is not being taught by the education system.

If the university is really worried about the integrity of its assessment system there are many things they could do. At a simple level why not set group assignments. Peer group assessment can provide for individual assessment within a group. Or, they could take a little more time and trouble and develop more authentic assessment to test each learner’s ability to apply knowledge developed through group work.

As Chris Avenir says ” if this kind of help is cheating, then so is tutoring and all the mentoring programs the university runs and the discussions we do in tutorials.”

Agggh – this is just what I feared. For some time I have been convinced we are at a turning point in our development of education and training systems and provision. I won’t go into all the reasons here – suffice to say that I believe education is intrinsically toed up in societal development – including economic development – and that our present education systems are based on the needs and forms of the first industrial revolutions and have failed to change to reflect the profound changes in society resulting from the digital revolution.

That systems will change and change dramatically is without doubt in my opinion. The direction of such change is less clear. I believe there is a big danger that systems will become privatized with state provision becoming a second class option. Furthermore, education is deeply tied up with societal values. Companies will not usually reflect those values in their totality.

I have no problem with MacDonald’s offering training programmes. Far from it. I do have two issues. One – why the hype? Why is the Prime Minister announcing new apprenticeship programmes by a few major companies. The rub is in the detail. MacDonald’s and other organisations will now be able to accredit their own programmes. They will become private examination boards. It could be argued that a number of the examination boards in the UK are effectively private organisations. But their purpose is to accredit learning. MacDonald’s primary purpose is to make p-rofit for shareholders by selling hamburgers.

Perhaps I am being paranoid, but I fear the reason that prime minister brown has announced these new programme sis that this represent another large shift towards privatising education and training in the UK.

From the Guardian: “The prime minister has defended the accreditation of in-company qualifications after it was announced that staff at McDonald’s could gain the equivalent of an A-level in burger bar management.

The fast food giant, Network Rail and the airline Flybe are the first three companies to win government approval to become an exam board.

The Qualifications and Curriculum Authority has approved a pilot “basic shift manager” course, which will train staff in everything they need to run a McDonald’s outlet, from marketing to human resources and customer service skills.

The budget airline Flybe will start piloting their “airline trainer programme” in the summer, which will cover everything from engineering to cabin crew training.

Much of the course will fit with the QCA’s Qualifications and Credit Framework, which allows credits for units of work to build up to full qualifications over time.

The company hopes to award qualifications equal to good GCSEs and up to university degrees.

Network Rail is piloting an initial qualification in track engineering and hopes to issue qualifications equivalent to GCSEs, but with some units at postgraduate level that could contribute to a master’s qualification.

Speaking on GMTV, Brown said: “You have got to do a pretty intensive course to get that qualification. It’s not that standards are going to fall. It’s going to be a tough course. Once you’ve got that qualification you can go anywhere.”