Assessment isn’t really my thing. That doesn’t mean I do not see it as important. I am interested in learning. Assessment for learning should help teachers and learners alike in developing their learning. But all too often assessment has little to do with learning. Indeed assessment has emerged as a barrier to the development of effective teaching and learning strategies especially collaborative learning using web 2.0 and social software tools.

This presentation by Luis Tinoca follows the present trend of adding 2.0 on the end of everything but is a useful exploration of how we can use technologies to support assessment for learning

The mailing for the seminar says” “The objective of Making Assessment Count is primarily to help students engage more closely with the assessment process, either at the stage where they are addressing an assignment or at the stage when they receive feedback on a completed assignment. In addition an underlying theme of MAC is to use technology to help connect student reflections on their assessment with their tutors. To facilitate the reflection aspect of MAC a web based tool called e-Reflect is often used. This tool enables the authoring of self-review questionnaires by tutors for students. On completion of an e-Reflect questionnaire a report is generated for the student containing responses that are linked to the options the student selected on the questionnaire.”

You can find out more ans sign up for the seminar at  http://jiscmac.eventbrite.co.uk/

The competition was aimed at 8-14 year olds and asked them to write a story using any combination of digital media

“The theme for your story should be on a political event that has happened – or is currently happening – in Wales.
We are not just interested in the facts but on your opinions and impressions. For example, how do you feel about the event you are describing? Who do you agree with and why? What have been the consequences of the event you have chosen?”
Suddenly life got a lot better! The black and white world of education that I seem to have lived in for the last few weeks was in brilliant technicolour. The stories were variously funny, poignant, angry, persuasive and insightful. All of them were well researched, referenced, technically at a level that would put many class teachers to shame and above all, they entertained me and taught me a whole lot I didn’t know. Surely the definition of a good learning experience!

(And by the time I had settled down with a glass of wine and a cigarette, the learning environment seemed pretty good as well).

The thing that cheered me up the most was that these kids had opinions – well argued, well expressed and authentic. I was pretty rubbish at history (Was? ‘Am’ actually! More maths and physics, me…) but short of those exam questions which always started “Compare and contrast….” or “What arguments would you use to support …something ” I don’t ever remember being allowed to have a ‘real’ opinion on anything historical, still less encouraged to express them if I did. Especially not in primary school – I think I was doing post-grad before I earned that privilege.

Which brings me on to my main point! There is a great public panic at the moment about Wales’s performance in the Programme for International Student Assessment (PISA) because they are two beans behind somewhere or other, half a Brownie point below an average or a nanopoint lower than last time. Puhlease!!

I am not being dismissive from a point of total ignorance here – some years ago I worked on the PISA statistics and the methodology for several months; I even remember doing a keynote presentation at European Conference for Education Research on PISA . Nor am I suggesting that standards do not matter. What I am saying is that the ‘Ain’t it awful’ media frenzy generated by the Smartie counting exercise that is PISA – and the politicians’ heavy-handed response – does a huge disservice to this generation of feisty, articulate and confident kids. And to the amazing generation of teachers that scaffold their learning.

Working in Pontydysgu, being a teacher trainer and a very active school governor means that I spend a lot of time in classrooms and my contention is that 99% of teachers are doing a fantastic job under pretty rubbish conditions. (Did I say this in a previous post? Yes? Well I don’t care – it needs to be shouted from the roof tops).

So what am I going to do about it? Firstly, I am tempted to rewrite the newspaper headlines showing that Welsh education is improving and is better than ‘average’. A claim I could easily back-up by a different manipulation of the PISA figures. Secondly, I could point out that the PISA survey takes place every four years but that changes at the lower age ranges – such as the introduction of the new 3-7 yr old Foundation Phase in Wales (which is awesome) will not impact on PISA results for another nine years so knee-jerk changes to ‘fix’ things seem a bit premature. Thirdly, I could argue that putting so much store on paper-based testing in Reading, Maths and Science as the measure of success of ‘a broad and balanced curriculum’ and ‘pupil-centred, experiential learning’ is a bit of an oxymoron. Fourthly, I could remind our government that Wales led the way on getting rid of SATs and league tables on the very valid grounds that comparisons are unfair because they are not comparing like with like. They funded research which showed standardised testing to be unhelpful, demotivating and did nothing to improve performance. So on a local and national level they don’t work – do they suddenly work on an international one? Or maybe I should become a politician and take on the establishment in the debating chamber – but Hey! I’ve just found there’s a whole new generation of politically astute, sussed and sorted 10year olds who are going to do that much better than I could. Fifteen years from now, it’s going to be move over Minister! Leighton Andrews – ‘your’ education system has much to be proud of.

P.S. I might put some of the entries on the Pontydysgu website over the next few weeks so that you can see for yourself. Any teacher interested in getting their kids to write and publish political stories too, have a look at the Learning About Politics website and get back to us.

The Open Badges framework, say the project developers, is designed to allow any learner to collect badges from multiple sites, tied to a single identity, and then share them out across various sites — from their personal blog or web site to social networking profiles. The infrastructure needs to be open to allow anyone to issue badges, and for each learner to carry the badges with them across the web and other contexts.

Now some of the issues. I am still concerned of attempts to establish taxonomies, be it those of hierarchy in terms of award structures or those of different forms of ability / competence / skill (pick your own terminology). Such undertakings have bedeviled attempts to introduce new forms of recognition and I worry that those coming more from the educational technology world may not realise the pitfalls of taxonomies and levels.

Secondly is the issue of credibility. There is a two fold danger here. One is that the badges will only be adopted for achievements in areas / subjects / domains presently outside ‘official’ accreditation schemes and thus will be marginalised. There is also a danger that in the desire to gain recognition, badges will be effectively benchmarked against present accreditation programmes (e.g. university modules / degrees) and thus become subject to all the existing restrictions of such accreditation.

And thirdly, as the project roils towards a full release, there may be pressures for restricting badge issuers to existing accreditation bodies, and concentrating on the technological infrastructure, rather than rethinking practices in assessment.

Lets look at some of the characteristics of any assessment system:

In terms of the use of technology for teaching and learning we saw limited technical innovation. OK, the UK saw an increasing trend towards providing Virtual Learning environments (mainly Moodle) in primary schools. Applications like Google docs and Dropbox allowed enhanced facilities for collaborative work and file sharing. However neither of these was designed specifically for educational use. Indeed the main technical trend may have been on the one hand the increased use of social software and cloud computing apps for learning and on the other hand a movement away from free social software towards various premium business models. Of course mobile devices are fast evolving and are making an increasing impact on teaching and learning.

But probably the main innovation was in terms of pedagogy and in wider approaches to ideas around learning. and here the major development is around open learning. Of course we do not have a precise or agreed definition of what open education or open learning means. But the movement around Open Educational Resources appears to be becoming a part of the mainstream development in the provision of resources for tecahing and learning, despite significant barriers still to be overcome.  And there is increasing open and free tecahing provision be it through online ‘buddy’ systems, say for language learning, various free courses available through online VLEs and the proliferation of programmes offered as Massive Open Online Courses (MOOCs) using a variety of both educational and social software. Whilst we are still struggling to develop new financial models for such programmes, perhaps the major barrier is recognition. This issue can be viewed at three different levels.

1. The first level is a more societal issue of how we recognise learning (or attainment). at the moment this tends to be through the possession of accreditation or certification from accredited institutions. Recognition takes the form of entry into a profession or job, promotion to a higher level or increased pay.
2. The second level is that of accreditation. Who should be able to provide such accreditation and perhaps more importantly what should it be for (this raises the question of curriculum).
3. The third is the issue of assessment. Although traditional forms of individual assessment can be seen as holding back more innovative and group based forms of teaching and learning there are signs of movement in this direction – see, for example the Jisc Effective Assessment in a Digital Age, featured as his post of the year by Stephen Downes.

These issues can be overcome and I think there are significant moves towards recognising broader forms of learning in different contexts. In this respect, the development of Personal Learning Environments and Personal Learning Networks are an important step forward in allowing access to both technology and sources of learning to those not enrolled in an institution.

However, such ‘progress’ is not without contradiction. One of the main gains of social democratic and workers movements over the last century has been to win free access to education and training for all based on nee4d rather than class or income. OK, there are provisos. Such gains were for those in rich industrialised countries – in many areas of the world children still have no access to secondary education – let alone university. Even in those rich countries, there are still big differences in terms of opportunities based on class. And it should not be forgotten that whilst workers movements have fought for free and universal access to education, it has been the needs of industry and the economic systems which have tended to prevail in extending access (and particularly in moulding the forms of provision (witness the widely different forms of the education systems in northern Europe).

Now those gains are under attack. With pressures on econo0mies due of the collapse of the world banking system, governments are trying to roll back on the provision of free education. In countries like the UK, the government is to privatise education – both through developing a market driven system and through transferring the cost of education from the state to the individual or family.

Students have led an impressive (and largely unexpected) fightback in the UK and the outcome of this struggle is by no means clear. Inevitably they have begun to reflect on the relation between their learning and the activities they are undertaking in fighting the increases in fees and cutbacks in finances, thus raising the issue of the wider societal purposes and forms of education.

And that also poses issues for those of us who have viewed the adoption of technology for learning as an opportunity for innovation and change in pedagogy and for extending learning (through Open Education) to those outside schools and universities. How can we defend traditional access to institutional learning, whilst at the same time attacking it for its intrinsic limitations.

At their best, both the movements around Open Education and the student movement against cuts have begun to pose wider issues of pedagogy and the purpose and form of education as will as the issues of how we recognise learning. One of the most encouraging developments in the student movement in the UK has been the appropriation of both online and physical spaces to discuss these wider issues (interestingly in opposition to the police who have in contrast attempted to close access to spaces and movement through he so-called kettling tactic).

I wonder now, if it is possibel to bring together the two different movements to develop new visions of education together with a manifesto or rather manifestos for aschieveing such visions.

But coming back to today’s article, reporting on a three day international conference on plagiarism at Northumbria University, the Guardian reports that “The conference will also hear that the problem of plagiarism at university could be reduced if students used “digital storytelling” – creating packages of images and voiceovers – rather than essays to explain their learning from an imagined personal perspective.

Phil Davies, senior lecturer at Glamorgan university’s computing school, said he had been using the technique for two years and had not seen any evidence of cheating. “Students find it really hard but it’s very rewarding, because they’re not copying and writing an essay, they have to think about it and bring their research into a personal presentation.”

Another approach is to focus on authentic assessment – or rather assessment of authentic learning tasks. In this case students are encouraged to use the internet for research but have to reflect on and re-purpose materials for reporting on their own individual research.

In both cases this goes beyond dealing plagiarism – it is good practice in teaching and learning. And I wonder if that might be a better starting point for the efforts of researchers, developers and teachers.

Several years ago, e-portfolios were the vogue in e-learning research and development circles. Yet today little is heard of them. Why? This is not an unimportant question. One of the failures of the e-learnng community is our tendency to move from one fad to the next, without ever properly examining what worked, what did not, and the reasons for it.

First of all it is important to note that  there was never a single understanding or approach to the development and purpose of an e-Portfolio. This can largely due be ascribed to different didactic and pedagogic approaches to e-Portfolio development and use. Some time ago I wrote that “it is possible to distinguish between three broad approaches: the use of e-Portfolios as an assessment tool, the use of e-Portfolios as a tool for professional or career development planning (CDP), and a wider understanding of e-Portfolios as a tool for active learning.”

In a paper presented at the e-Portfolio conference in Cambridge in 2005 (Attwell, 2005), I attempted to distinguish between the different process in e-Portfolio development and then examined the issue of ownership for each of these processes.

The diagramme reveals not only ownership issues, but possibly contradictory purposes for an e-Portfolio. Is an e-Portfolio intended as a space for learners to record all their learning – that which takes place in the home or in the workplace as well as in a course environment or is it a place or responding to prescribed outcomes for a course or learning programme? How much should a e-Portfolio be considered a tool for assessment and how much for reflection on learning? Can tone environment encompass all of these functions?

These are essentially pedagogic issues. But, as always, they are reflected in e-learning technologies and applications. I worked for a whole on a project aiming to ‘repurpose the OSPI e-portfolio (later merged into Sakai) for use in adult education in the UK. It was almost impossible. The pedagogic use of the e-Portfolio, essentially o report against course outcomes – was hard coded into the software.

Lets look at another, and contrasting, e-Portfolio application, ELGG. Although now used as a social networking platform, in its original incarnation ELGG stared out as a social e-portfolio, originating in research undertaken by Dave Tosh on an e-portfolio project. ELGG essentially provided for students to blog within a social network with fine grained and easy to use access controls. All well and good: students were not restricted to course outcomes in their learning focus. But when it came to report on learning as part of any assessment process, ELGG could do little. There was an attempt to develop a ‘reporting’ plug in tool but that offered little more than the ability to favourite selected posts and accumulate them in one view.

Mahara is another popular open source ePortfolio tool. I have not actively played with Maraha for two years. Although still built around a blogging platform, Mahara incorporated a series of reporting tools, to allow students to present achievements. But it also was predicated on a (university) course and subject structure.

Early thinking around e-Portfolios failed to take into account the importance of feedback – or rather saw feedback as predominately as coming from teachers. The advent of social networking applications showed the power of the internet for what are now being called personal Learning networks, in other words to develop personal networks to share learning and share feedback. An application which merely allowed e-learners to develop their own records of learning, even if they could generate presentations, was clearly not enough.

But even if e-portfolios could be developed with social networking functionality, the tendency for institutionally based learning to regard the class group as the natural network, limited their use in practice. Furthermore the tendency, at least in the school sector, of limited network access in the mistaken name of e-safety once more limited the wider development of ‘social e-Portfolios.”

But perhaps the biggest problem has been around the issue of reflection. Champions have lauded e-portfolios as a natural tools to facilitate reflection on learning. Helen Barrett (2004) says an “electronic portfolio is a reflective tool that demonstrates growth over time.” Yet  are e-Portfolios effective in promoting reflection? And is it possible to introduce a reflective tool in an educations system that values the passing of exams through individual assessment over all else? Merely providing spaces for learners to record their learning, albeit in a discursive style does not automatically guarantee reflection. It may be that reflection involves discourse and tools for recording outcomes offer little in this regard.

I have been working for the last three years on developing a reflective e-Portfolio for a careers service based din the UK. The idea is to provide students an opportunity to research different career options and reflect on their preferences, desired choices and outcomes.

We looked very hard at existing opens source e-portfolios as the basis for the project, nut could not find any that met our needs. We eventually decided to develop an e-Portfolio based on WordPress – which we named Freefolio.

At a technical level Freefolio was part hack and part the development of a plug in. Technical developments included:

• The ability to aggregate summaries of entries on a group basis
• The ability add custom profiles to see profiles of peers
• Enhanced group management
• The ability to add blog entries based on predefined xml templates
• More fine grained access controls
• An enhanced workspace view

Much of this has been overtaken by subsequent releases of WordPress multi user and more recently Buddypress. But at the time Freefolio was good. However it did  not work in practice. Why? There were two reasons I think. Firstly, the e-Portfolio was only being used for careers lessons in school and that forms too little a part of the curriculum to build a critical mass of familiarity with users. And secondly, it was just too complex for many users. The split between the front end and the back end of WordPress confused users. The pedagogic purpose, as opposed to the functional use was too far apart. Why press on something called ‘new post’ to write about your career choices.

And, despite our attempts to allow users to select different templates, we had constant feedback that there was not enough ease of customisation in the appearance of the e-Portfolio.

In phase two of the project we developed a completely different approach. Rather than produce an overarching e-portfolip, we have developed a series of careers ‘games; to be accessed through the Careers company web site. Each of the six or so games, or mini applications we have developed so far encourages users to reflect on different aspects of their careers choices. Users are encouraged to rate different careers and to return later to review their choices. The site is yet to be rolled out but initial evaluations are promising.

I think there are lessons to be learnt from this. Small applications that encourage users to think are far better than comprehensive e-portfolios applications which try to do everything.

Interestingly, this view seems to have concur with that of CETIS. Simon Grant points out: “The concept of the personal learning environment could helpfully be more related to the e-portfolio (e-p), as both can help informal learning of skills, competence, etc., whether these abilities are formally defined or not.”

I would agree: I have previously seen both as related on a continuum, with differing foci but similar underpinning ideas. However I have always tended to view Personal Learning Environments as a pedagogic capproach, rather than an application. Despite this, there have been attempts to ‘build a PLE’. In that respect (and in relation to rethinking e-Portfolios) Scott Wilson’s views are interesting. Simon Grant says: “As Scott Wilson pointed out, it may be that the PLE concept overreached itself. Even to conceive of “a” system that supports personal learning in general is hazardous, as it invites people to design a “big” system in their own mind. Inevitably, such a “big” system is impractical, and the work on PLEs that was done between, say, 2000 and 2005 has now been taken forward in different ways — Scott’s work on widgets is a good example of enabling tools with a more limited scope, but which can be joined together as needed.”

Simon Grant goes on to say the ““thin portfolio” concept (borrowing from the prior “personal information aggregation and distribution service” concept) represents the idea that you don’t need that portfolio information in one server; but that it is very helpful to have one place where one can access all “your” information, and set permissions for others to view it. This concept is only beginning to be implemented.”

This is similar to the Mash Up Personal Learning Environment, being promoted in a number of European projects. Indeed a forthcoming paper by Fridolin Wild reports on research looking at the value of light weight widgets for promoting reflection that can be embedded in existing e-learning programmes. This is an interesting idea in suggesting that tools for developing an e-Portfolio )or for that matter, a PLE can be embedded in learning activities. This approach does not need to be restricted to formal school or university based learning courses. Widgets could easily be embedded in work based software (and work flow software) and our initial investigations of Work Oriented Personal Learning Environments (WOMBLES) has shown the potential of mobile devices for capturing informal and work based learning.

Of course, one of the big developments in software since the early e-Portfolio days has been the rise of web 2.0, social software and more recently cloud computing. There seems little point in us spending time and effort developing applications for students to share powerpoint presentations when we already have the admirable slideshare application. And for bookmarks, little can compete with Diigo. Most of these applications allow embedding so all work can be displayed in one place. Of course there is an issue as to the longevity of data on such sites (but then, we have the same issue with institutional e-Portfolios and I would always recommend that students retain a local copy of their work). Of course, not all students are confident in the use of such tools: a series of recent studies have blown apart the Digital Native (see for example Hargittai, E. (2010). Digital Na(t)ives? Variation in Internet Skills and Uses among Members of the “Net Generation”. Sociological Inquiry. 80(1):92-113).  And some commercial services may be more suitable than other for developing an e-Portfolio: Facebook has in my view limitations! But, somewhat ironically, cloud computing may be moving us nearer to Helen Barrett’s idea of an e-Portfolio. John Morrison recently gave a presentation (downloadable here) based on his study of ‘what aspects of identity as learners and understandings of ways to learn are shown by students who have been through a program using course-based networked learning?’ In discussing technology he looked at University as opposed to personally acquired, standalone as opposed to networked and Explored as opposed to ongoing use.

He found that students:

Did not rush to use new technology

Used face-to-face rather than technology, particularly in early brainstorming phases of a project

Tried out software and rejected that which was not meeting a need

Used a piece of software until another emerged which was better

Restrained the amount of software they used regularly to relatively few programs

Certain technologies were ignored and don’t appear to have been tried out by the students

Students used a piece of software until another emerged which was better  which John equates with change. Students restrained the amount of software they used regularly to relatively few programs  which he equates with conservatism

Whilst students were previously heavy users of Facebook, they were now abandoning it. And whilst there was little previous use of Google docs, his latest survey suggested that this cloud application was now being heavily used. This is important in that one of the more strange aspects of previous e0Portolio development has been the requirement for most students to upload attached files, produced in an off line work processor, to the e-Portfolio and present as a file attachment. But if students (no doubt partly driven by costs savings) are using online software for their written work, this may make it much easier to develop online e-portfolios.

John concluded that :this cohort lived through substantial technological change. They simplified and rationalized their learning tools. They rejected what was not functional, university technology and some self-acquired tools. They operate from an Acquisition model of learning.” He concluded that “Students can pick up and understand new ways to learn from networks. BUT… they generally don’t. They pick up what is intended.” (It is also well worth reading the discussion board around John’s presentation – - although you will need to be logged in to the Elesig Ning  site).

So – the e-Portfolio may have a new life. But what particularly interests me us the interplay between pedagogic ideas and applications and software opportunities and developments in providing that new potential life. And of course, we still have to solve that issue of control and ownership. And as John says, students pick up what is intended. If we continue to adhere to an acquisition model of learning, it will be hard to persuade students to develop reflective e-Portfolios. We should continue to rethink e-Portfolios through a widget based approach. But we have also to continue to rethink our models of education and learning.

“Danish ‘A’ level students are likely to be able to use the Internet in their written exams if a test run later this year proves successful.

The Ministry of Education says that pupils already use the Internet for tests.

“It’s a good way to get hold of historical facts or an article that can be useful, for example, in a written social sciences exam,” Ministry Education Consultant Søren Vagner tells MetroXpress.

Digital hand-in

In order to prevent students from cheating by downloading translation programmes or communicating using chats, the idea is that papers should be handed in digitally and that there should be random checks on sites that students visit during an exam”

So early in the morning (at least for me) I got in and skyped into the BBC Cardiff newsroom. I was on the programme to defend the use of computers, Chris Woodhead, the ex Chief Inspector of Schools, was the opponent. And we had five minutes of knock around fun. The BBC preceded the item with three or four vox pops with ‘A’ Level school students from Monmouth in East Wales, who rather predictably said what a bad idea it was as it would penalise those who had worked hard to remember all the facts.

I said I thought on the whole it was a good idea becuas eit would allow students to use teh technolgie savaible in the ral worlls to show their creativity and ability to develop ideas and knowledge, Chris siad it was a bad thing because they would waste tiem surfing and it would prevent them showing their creativity and grasp on knowledge and ideas. and thatw a sit.

In reality, I think the discussion is a much deeper one over the nature and purpose of assessment. The ‘A’ level exam in the UK is essentially used as a filter mechanism, to select students for university. As such their is little authenticity. Students are inevitably taught for the exam. I saw some research a time ago suggesting that ‘A” levels are a poor predicator for later success in university but cannot find a reference ot that at the moment. The problem is that the examinations do not really test the students learned, but their ability to apply what they have learnt to a particular series of formalised tests. neither do the exams serve to help the students in their learning, Other than, I suppose, motivating them to learn a lot of facts in the run up to the exam. I fear that little of what we call revision for exams actually involves reflection on learning. And if the use of computers were to herald a move away from learning facts, to reflecting on meanings, then it could only be a good thing. But at then end of the day, I can’t get excited – and certainly couldn’t so early in the morning. The big issue for me is how to use technology to support learning. And that is another thing.

I have written many times about Assessment for Learning and the self assessment or evaluation of learning. Assessment for Learning is the idea of formative assessment to support the learning process, rather than most of our present assessment systems which are designed to support comparisons or as a screening mechanism for entry into higher education or education and training or into employment.
And self evaluation – it is what it says. The idea that learners are able to evaluate or assess tehir own learning, often with a surprising degree of insight and accuracy. Of course when they do this they own the assessment – it ceases ot be something that is done ot them but is part of their own reflective learning process.
But, say teachers, this is hard to do. Learners will not know how to do it. they will over-rate their own abilities.
So practical examples are always welcome and I was lucky enough to see today the self evaluation of one of my friend’s children in a school in Bremen (reproduced above).
The process went something like this. Last week the students – aged 8 – were asked to fill in their own assessments in the left had column. Then the sheets were passed to their two form teachers who also filled in the assessment in the right hand column. And then today there were individual meetings between teachers and the students to discuss the results. (It is interesting to note that like in previous expercises of this sort that I have seen, teachers tended to rate students slightly higher than the students themselves).
Seems pretty cool to me (even if a little overly emphasising behaviour and conformity) and much, much more useful than the UK Standard Assessment tests (SATs).

Firstly, for non European readers, it may be useful to recapitulate what the EF is all about. The European Qualifications Framework (EQF), says the European Commission,  “acts as a translation device to make national qualifications more readable across Europe, promoting workers’ and learners’ mobility between countries and facilitating their lifelong learning.” The primary users of the EQF are seen as being bodies in charge of national and/or sectoral qualification
systems and frameworks. The idea is that once they have related their respective systems to the EQF, the EQF will help individuals, employers and education and training providers compare individual qualifications from different countries and education and training systems.

To achieve this the European Commission has designed a framework of eight different levels. Each of the 8 levels is defined by a set of descriptors indicating the learning outcomes expressed as knowledge, skills and outcomes relevant to qualifications at that level.

The problem is that it doesn’t work. For the moment I will ignore the more epistemological issues related to the definition of knowledge kills and competences. The biggest problem for me relates to levels. The Framework has mixed a series of different indicators to derive the levels. Some of the indicators are based on academic attainment. For instance the descriptor for knowledge at Level 8 states (can) “demonstrate substantial authority, innovation, autonomy, scholarly
and professional integrity and sustained commitment to the development of new ideas or processes at the forefront of work or study contexts including research.” Some are based on levels of responsibility and autonomy in work roles.  Level 4 talks of the ability to “supervise the routine work of others, taking some responsibility for the evaluation and improvement of work or study activities.” Others are based the complexity of the work being undertaken. Level four skills says: :advanced skills, demonstrating mastery and innovation, required to solve complex and unpredictable
problems in a specialised field of work or study.” Yet others are based on quite abstract ideas of knowledge. Level six knowledge comprises of “advanced knowledge of a field of work or study, involving a critical understanding of theories and principles.”

One of the biggest problems is the framework attempts to bring together applied knowledge and skills within a work process, work roles as expressed by responsibility and knowledge as expressed through academic achievement. And of course it is impossible to equate these, still less to derive a hierarchical table of progression and value. It is easy to pick holes: why for instance is “exercise management and supervision in contexts of work or study activities where there is unpredictable change” level 5, whilst having a “critical awareness of knowledge issues in a field and at the interface between different fields.”

There would appear to be a serie sof unspoken and implicit value judgements related both to the value of academic versus vocational and applied learning and to different roles within the workplace. There also seems to be an attempt to deal with work organisation with teamwork being written in as a high level function. Of course it maybe, but then again in particualr contexts teamwortk may not be so important. In some jobs, the ability for autonomous work may be important, in others not so. And how can we translate between such abilities, competences or whatever they are called and qualifications.

I do not think it is possible to design such a frameworks, nor do I think that levels are a useful concept, especially given the hierarchical structures of this and other similar frameworks. Why should one particular competence or skill be valued over another. Even more important is the idea of hierarchical progression. Lest this be thought to be merely an academic question, the UK government has already withdrawn funding support for those wishing to progress from one qualification to another at the same EQF related level.

one of teh aims of the Frameowrk is to promote Lifelong Learning. But an individual may not wish ot advance their learning in the social forms envisaged by the Framework. There is an assumption for instance that a teacher or trainer will progress towards being a manager, as represented in the higer levels within the EQF. But many teachers and trainers that I have talked to actually want to improve their practice as tecahers or trainers. Or they many wish to move ‘sideways’ – to learn more about working with particular groups or to undertake work as a counsellor for instance. The implicit progression routes inherent in the Frameowkr do not necessarily represent the way we work and learn. far better nore me is the idea of Learning Pathways. Learning pathways can represent a progression in our learning based on the context of our life and our work and based on individual interest and motivation. Our Learning Pathway may at times go upwards, downwards or sideways in the table of skills, competnces and knowledge as representd in the Framework. Such an idea of Learning Pathways is contiguous with the idea of Perosnal Learning Environments and of teh idea of more self directed learning. Of course it is useful to have a framework to assist in selecting progression routes and for counselling, guidance and support. But lets abolish the taxonomy of levels and start representing learning opportunties as what they are, rather than a somewhat oddly derived taxonomy trying to make things fit neatly which do not and embodying implict social values.

Dave spoke about his experiences, so far, of the CCK MOOC on Connectivism and Connected Knowledge, the technological platforms being used to support participants, the tensions that exist within the course design and the peer support models that are being embraced.  Dave’s introduction led to a wide ranging discussion including the nature and furture of courses and communities, issues of scale, how to support learners, open accreditation and the future of open education – and …Humpty Dumpty and Alice in Wonderland!

If you missed the session – or would like to hear it again – we are providing you with three different versions. You can watch a replay of the event in Elluminate. This provides you with access to the sidebar chat discussion as well as to the audio.

Or – if you are short of time you can listen to an MP3 podcast of Dave’s introduction.

Or you can listen to the full session inline or on your MP3 player.

This is the link to the Elluminate version.

We agreed that the context of learning was important. Mobile learning would become increasingly important with the development of context sensitive devices. (Also see Serge Ravet’s post on User Generated Content or User Generated Contexts).

But there were also limits to the consensus. Whilst there appeared agreement on new roles for teachers, no-one was sure what that role was?

Much of the discussion centred on the scaffolding of learning. How much support did leaners need and how much of that support would come from teachers?

Neither were participants agreed on the future role of institutions. More critically, was Learning 2.0 something which happened outside the school, and had only a limited impact on institutional practice, or did it pose a fundamental challenge for the future of schooling?

There was even greater disagreement over curriculum. Should there be a curriculum for basic skill and knowledge that everyone should learn? Did learners need a basic grounding in their subject before theyc oudl develop their own learning pathways? Who should define such a curriculum? What was the role of ‘experts’ and who were they anyway?

And perhaps the greatest disagreement was over assessment and accreditation. Many of us felt that we needed to move towards community based formative assessment. Employers, we said, would be more interested in what people were able to do than formal certicates. Others, pointing to occupations such as doctors and plumbers felt there should be some form of standards against which people should be assessed and accredited.

A final comment on the form of the project. Although the work is about Learning 2.0 the present form of the work is decidedly Research 1.0. This research is important enough that it needs to be opened out to the community. It seems a wiki is being d veloped and when it is up I will blog here about it. In the meantime here are some of the photos of the flip charts used for brainstorming around different issues at the workshop. I will pass on any comments on this post to the project organisers.

An accreditation system depends on the robustness of the assessment system on which it is based.

Imagine you were in a shop that sold accreditation / assessment systems ‘off-the-peg” – what would criteria would you use if you went in to buy one?

Reliability
Reliability is a measure of consistency. A robust assessment system should be reliable; that is, it should be based on an assesssment process that yields the same results irrespective of who is conducting it or the environmental conditions under which it is taking place. Intra-tester reliability simply means that if the same asessor is assessing performance his or her judgement should be consistent and not influenced by, for example, another learner they might have just assessed or whether they feel unwell or just in a bad mood! Inter-tester reliability means that if two different assessors were given exactly the same questions, data collection tools, output data and so on, their conclusions should also be the same. Extra-tester reliability means that the assessor’s conclusions should not be influenced by extraneous circumstances, which should have no bearing on the assessment object.

Validity
Validity is a measure of ‘appropriateness’ or ‘fitness for purpose’. There are three sorts of validity. Face validity implies a match between what is being assessed or tested and how that is being done. For example, if you are assessing how well someone can bake a cake or drive a car then you would probably want them to actually do it rather than write an essay about it! Content validity means that what you are testing is actually relevant, meaningful and appropriate and there is a match between what the learner is setting out to do and what is being assessed. If an assessment system has predictive validity it means that the results are still likely to hold true even under conditions that are different from the test conditions. For example, performance assessment of airline pilots who are trained to cope with emergency situations on a simulator must be very high on predictive validity.

Replicability
Ideally an assessment should be carried out and documented in a way which is transparent and which allows the assessment to be replicated by others to achieve the same outcomes. Some ‘subjectivist’ approaches to assessment would disagree, however.

Transferability
Although each assessment should be designed around a particular piece of learning, a good assessment  system is one which could be adapted for similar  situations or could be extended easily to new activities. That is, if your situation evolves and changes over a period of time in response to need, it would be useful if you didn’t have to rethink your entire assessment system. Transferability is about the shelf-life of the assessment and also about maximising its usefulness

Credibility
People actually have to believe in yourassessment! It needs to be authentic, honest, transparent and ethical. If you have even one group of stakeholders questioning the rigour of the assessment process or doubting the results or challenging the validity of the conclusions, the assessment loses credibility and is not worth doing.

Practicality
This means simply that however sophisticated and technically sound the assessment is, if it takes too much of people’s time or costs too much or is cumbersome to use or the products are inappropriate then it is not a good assessment system !

Comparability
Although an assessment system should be customised to meet the needs of particular learning events, a good assessment system should also take into account the wider assessment ‘environment’ in which the learning is located. For example, if you are working in an environment where assessment is normally carried out by particular people (e.g teachers, lecturers) in a particular institution (e.g school or university) where ‘criteria reference assessment is the norm, then if you undertake a radically different type of assessment you may find that your audience will be less receptive and your results less acceptable. Similarly, if the learning that is being assessed is part of a wider system and everyone else is using a different system then this could mean that your input is ignored simply because it is too difficult to integrate.

Also, if you are trying to compare performance from one year to the next or compare learning outcomes with other people, then this needs to be taken into account.

This week I will be writing a series of blog pots on accreditation. But first up an excellent quote from J,M Coetzee in his novel ‘Diary of a Bad Year’. Just substitute ‘open online networks for ‘peoples homes’

“In the days when Poland was under communist rule, there were dissidents who conducted night classes in their homes, running seminars on writers and philosophers excluded from the official canon (for example, Plato). No money changed hands though there may have been other forms of payment. if the spirit of the university is to survive, something along these lines may have to come into being in countries where tertiary eduction has been wholly subordinated to business principles. In other words the real university may have to move into peoples homes and grant degrees for which the sole backing will be the names of the scholars who sign the certificates.”

Firstly if we take congniscence of Jenny Hughes’ defintion of learning as ‘to find and follow a track’ as counterposed to curriculum  from the latin ‘currere’, which means to run or race and ‘curriculum’ as race or racecourse, then instotutions and teacher have an important role in assisting learners in developing their own learning pathways.

A second important  role is that of assessment. But to understand this we need to decouple assessment from acceditation. If we design assessment as a learning process and move from assessemnt of learning to asssessement for learning this could become an integral part of the process of finding and developing learning pathways. This is not so utopian. Serendipitously. The Times newpaper today published an article about innovative assessment in UK universities. The struggle, though, as with self and peer assessment is in assessment having to match accreditation procedures. Without this link, we could open up all jinds of new forms of assessment.

A final point on accreditation. Many learners do not want or require accreditation. Indeed it is the formal accreditation procedures which deters them for signing up for a learning programme. And as Antnio Fini, talking about the home made certificate he got from the OpenEd2007 course, says: “all my connections, blog posts, comments, collective works, presentations, articles related to that experience, are still out there as tangible proofs of this learning. So I could equally put the OpenEd course in my CV and could ask to my supervisor to evaluate all that activity for credit in my PhD, also without that piece of paper!”

Why not put the learners in charge of accreditation. Lets leave it to them to decide how they wish to show what and how they have learned – albeit with support. I once co-ran a course with Jenny Hughes where we offered the particpants their certificate at the start of the courese. They refused! But it did raise the issue of why they were doing the course and how they valued learning. And that is an issue we need to bring to the fore.

Maybe it is juts the people i am talking too, buut there seems to be growing appreciation of the importance of informal learning and learning acquired in the workplace. At the same time I am slightly concerned that this appreciation for workplace and informal learning is being counterposed to formal training and qualifications. In this respect I think people are mixing up the schooling system and formal learning. Yes – I completely agree that our formal schooling system is out of date, frequently ineffective and promotes formal accreditation at the expense of learning. Putting it simply there are better ways to learn – and the money spent of formal schooling could be much more effectively deployed elsewhere.

But this is not to say there is no place for formal training and learning and for qualifications. Qualifications can play an important regulatory role – both for quality and in terms of preventing employer exploitation. Moreover such qualifications can prove aspirational – especially for young people. Yes – there are many issues around curriculum (I will return to this issue in a further post). Formal learning and training can provide a structure for learning. And formal learning and qualifications are not in opposition to informal learning – the two can go together.

I think there are problems in a fast changing economy as employment and work tasks and roles are fast changing. There is no guarantee that training for one particular occupation will guarantee employment in ten years time. Yet, all the empirical surveys we have carried out show that those who have undergone a formal training programme – regardless of subject – are more likely to participate in on-going learning in the future. Thus, even though the link between qualifications and employment may be weakening (especially in liberal market economies such as the UK) there remains a macro economic benefit to the provision of formal learning opportunities.

“First-year student Chris Avenir is fighting charges of academic misconduct for helping run an online chemistry study group via Facebook last term, where 146 classmates swapped tips on homework questions that counted for 10 per cent of their mark.

The computer engineering student has been charged with one count of academic misconduct for helping run the group – called Dungeons/Mastering Chemistry Solutions after the popular Ryerson basement study room engineering students dub The Dungeon – and another 146 counts, one for each classmate who used the site.

Avenir, 18, faces an expulsion hearing Tuesday before the engineering faculty appeals committee. If he loses that appeal, he can take his case to the university’s senate.

The incident has sent shock waves through student ranks, says Kim Neale, 26, the student union’s advocacy co-ordinator, who will represent Avenir at the hearing.

“All these students are scared s—less now about using Facebook to talk about schoolwork, when actually it’s no different than any study group working together on homework in a library,” said Neale.

“That’s the worst part; it’s creating this culture of fear, where if I post a question about physics homework on my friend’s wall (a Facebook bulletin board) and ask if anyone has any ideas how to approach this – and my prof sees this, am I cheating?” said Neale, who has used Facebook study groups herself.”

Ir seems fairly ridiculous that we spend so much time and trouble working on the use of new technologies for collaborative learning – and even more so for encouraging autonomous learning, only for academic institutions to take actions like this. Furthermore, every survey of employers suggests that the ability to work as part of a team is both one of the most sought after competencies and one which they feel is not being taught by the education system.

If the university is really worried about the integrity of its assessment system there are many things they could do. At a simple level why not set group assignments. Peer group assessment can provide for individual assessment within a group. Or, they could take a little more time and trouble and develop more authentic assessment to test each learner’s ability to apply knowledge developed through group work.

As Chris Avenir says ” if this kind of help is cheating, then so is tutoring and all the mentoring programs the university runs and the discussions we do in tutorials.”

That systems will change and change dramatically is without doubt in my opinion. The direction of such change is less clear. I believe there is a big danger that systems will become privatized with state provision becoming a second class option. Furthermore, education is deeply tied up with societal values. Companies will not usually reflect those values in their totality.

I have no problem with MacDonald’s offering training programmes. Far from it. I do have two issues. One – why the hype? Why is the Prime Minister announcing new apprenticeship programmes by a few major companies. The rub is in the detail. MacDonald’s and other organisations will now be able to accredit their own programmes. They will become private examination boards. It could be argued that a number of the examination boards in the UK are effectively private organisations. But their purpose is to accredit learning. MacDonald’s primary purpose is to make p-rofit for shareholders by selling hamburgers.

Perhaps I am being paranoid, but I fear the reason that prime minister brown has announced these new programme sis that this represent another large shift towards privatising education and training in the UK.

From the Guardian: “The prime minister has defended the accreditation of in-company qualifications after it was announced that staff at McDonald’s could gain the equivalent of an A-level in burger bar management.

The fast food giant, Network Rail and the airline Flybe are the first three companies to win government approval to become an exam board.

The Qualifications and Curriculum Authority has approved a pilot “basic shift manager” course, which will train staff in everything they need to run a McDonald’s outlet, from marketing to human resources and customer service skills.

The budget airline Flybe will start piloting their “airline trainer programme” in the summer, which will cover everything from engineering to cabin crew training.

Much of the course will fit with the QCA’s Qualifications and Credit Framework, which allows credits for units of work to build up to full qualifications over time.

The company hopes to award qualifications equal to good GCSEs and up to university degrees.

Network Rail is piloting an initial qualification in track engineering and hopes to issue qualifications equivalent to GCSEs, but with some units at postgraduate level that could contribute to a master’s qualification.

Speaking on GMTV, Brown said: “You have got to do a pretty intensive course to get that qualification. It’s not that standards are going to fall. It’s going to be a tough course. Once you’ve got that qualification you can go anywhere.”

Why is the Scottish Chartered Teacher scheme so interesting.

Firstly it recognises that teachers can develop their career without leaving the classroom. In other words it values the activities of being a teacher, rather than in most systems where career advancement is based on becoming a manager.

Secondly it introduces a framework for Continuing Professional Development, based on professional values and personal commitments.

Perhaps most important is the the Chartered Teacher qualification is based largely on reflective learning and self evaluation.

The Standard has four key components:

a. professional values and personal commitments;
b. professional knowledge and understanding;
c. professional and personal attributes;
d. professional action.

The basic assumption is that the Chartered Teacher is characterised by four central professional values and personal commitments:

a. effectiveness in promoting learning in the classroom;
b. critical self-evaluation and development;
c. collaboration and influence;
d. educational and social values.

You can find out more about the Chartered Teacher Scheme on the Scottish Government’s Standard for Chartered Teachers web site.

DRIVE: “First, drive. I define drive as self-motivation — people who will walk right through brick walls, on their own power, without having to be asked, to achieve whatever goal is in front of them. People with drive push and push and push and push and push until they succeed.”

CURIOSITY: “Second criterion: curiosity. Curiosity is a proxy for, do you love what you do? Anyone who loves what they do is inherently intensely curious about their field, their profession, their craft. They read about it, study it, talk to other people about it… immerse themselves in it, continuously. And work like hell to stay current in it. Not because they have to. But because they love to.

ETHICS: “Third and final criterion: ethics. Ethics are hard to test for. But watch for any whiff of less than stellar ethics in any candidate’s background or references. And avoid, avoid, avoid. Unethical people are unethical by nature, and the odds of a metaphorical jailhouse conversion are quite low.”

I think this is interesting and would agree with much of it but it raises some questions. Firstly, I would add a fourth category:

“COLLABORATION. The fourth criteria is collaboration. The ability to work with others is a critical source of learning. Even more so the ability to collaborate is central to developing and sharing knowledge. Collaboration leads to informal learning, innovation and productivity. Collaboration includes listening and valuing other peoples opinions as much as putting forward one’s own.”

The problem is that even if new pedagogic approaches involve curiosity and collaboration for learning, when we seek to assess and certificate competences, these are not qualities we value.

Is it possible to develop new forms of assessment that value drive, curiosity, ethics and collaboration? Is it even desirable that we seek to measure such things? What is the relation between our measurement of general educational learning or vocational skills and knowledge and what might be called the soft skills highlighted by Marc Andreessen.

Technorati Tags: competence development, assessment