Archive for the ‘Communities of Practice’ Category

The Learning Layers project scales up informal learning at the workplace

November 12th, 2012 by Graham Attwell

Last week Dirk, Jenny and I attended the first meeting of the European Commission IST programme Learning layers project. You are going to be hearing much more about the project on this blog, but here is a quick summary in the project’s first press release.

“With significant co-funding from the European Community, the Learning Layers project has recently started to research the role of information technologies in informal learning at the workplace. “It has been known for a long time that most learning that happens at the workplace is informal in nature”, says Tobias Ley, Professor at Tallinn University and scientific coordinator of the project, “but if we look at the learning technologies that are employed today, then most follow a very traditional model that mirrors course-based classroom learning.”

At the same time, mobile and social information and communication technologies have entered all parts of our lives. Nowadays we use them naturally to keep in contact with our friends, to seek information, to buy things and to work. There is now a realization that these emerging technologies are a key enabler to refocus efforts on informal learning, but  few companies have taken these technologies up in a systematic way to include them into their learning strategy.

So why is this? “One of the reasons is that, although informal interactions, like asking your colleague across the room for help, are very effective, they don’t scale very well beyond the immediate context”, says Tobias Ley. While the help is effective, not many others can benefit from it. Could our personal technologies provide a key for scaling up these interactions? This is what the project has set out to discover, and a number of technological solutions will be developed that should help to make this endeavor a reality. First of all, technologies need to be where informal learning really takes place. “In many prior projects, we have concentrated on people sitting at their computer desktops. In contrast, the Learning Layers Project will look at workplaces that are inherently mobile,” says Stefanie Lindstaedt, Professor at Graz University of Technology in Austria, who will be one of the main project partners for developing technologies for these settings.

The European Commission has specifically asked for proposals that benefit Small and Medium Sized Enterprises in sectors that have been less inclined in the past to take up technologies for learning. The Learning Layers project has therefore selected two very challenging application areas, the building and construction industry and the healthcare sector. Informal learning has traditionally played an important role in these sectors, but both have been hesitant to embrace learning technologies for different reasons. “With people using their personal devices, we now see a great opportunity in the building industry to connect our course offering much more directly with what people are experiencing at the workplace”, says Melanie Campbell who is representing a regional training center for the building and construction industry in Northern Germany. “Doctors work in interprofessional teams and have often learned from colleagues, technology could provide us with a way of sharing, enhancing and recording this informal learning”, says Tamsin Treasure-Jones from the Leeds Institute of Medical Education at the University of Leeds.

“Understanding the current situation in these two sectors, how they work, learn and share their knowledge, is a key priority in the project”, says Ronald Maier, Professor at the University of Innsbruck, “we will therefore conduct in-depth studies and use our findings to help improve the way people work and learn.” Based on these insights, the project will design interaction technologies to ensure that they are firmly embedded in the daily work practices, and to create new knowledge, learning and work practices that fit to the existing ones.

Apart from direct person to person interactions, such as asking colleagues, the Learning Layers project will also look at how learning materials are created informally and then increasingly shared, improved and used on a wider scale. And because people’s experiences with physical objects, like machines or materials, is crucial in the two sectors, the project will also consider how these experiences can be better exploited for learning purposes. For example, it is nowadays quite easy to make short video sequences and share them to explain how to use a certain tool.

“In order to scale up the use of these technologies, we really need to show take up in large user groups”, says Graham Attwell, Director of Pontydysgu and responsible for the project’s outreach strategy. One of the key measures the project will take is to make use of existing regional economic clusters in which enterprises are already collaborating on a whole range of activities. The project has proposed a roll-out strategy through which technologies would become part of the regional innovation and learning system, thereby reaching out to 1000 end users within the lifetime of the project.

The Learning Layers project started on 01 November 2012 with a total budget of over 12 million Euros over a four-year lifetime. Seventeen partners from Austria, Estonia, Finland, Germany, Norway, Spain, and the UK have teamed up for what will be an intense research collaboration. The project is jointly coordinated by CIMNE, a research institute in Barcelona/Spain and Tallinn University in Estonia. More information is available at http://www.learning-layers.eu.”

Disruptive Education

October 29th, 2012 by Graham Attwell

Last Friday, Fred Garnett and I made presentations to the weekly virtual Teaching and Learning Conversations (TLC) organised by Cristina Costa and Chrissie Nerantzi from Salford University. The title of the conversation, which took place on the Blackboard Collaborate platform, was disruptive education.

Fred lives in London and I was also in London for meetings, so we decided to meet up at the Westminster Hub (more on that later this week). And it was great fun! Fred and me both shared our presentations and so it evolved into a genuine conversation. I don’t know about the others, but i learned a lot (including that there is nothing like face to face proximity for a real conversation. We both agreed that globalisation is probably more disruptive to educatio0n at the moment than the introduction of new technologies, which are only an enabling factor.

I will post my slides tomorrow (and a link to the recording which seems to be broken at the moment). Here are Fred’s slides – slightly changed after the session. I especially like his distinction between disruption applied to education, which he says needs

  • new distance learning resources
  • new business models
  • globalisation
  • competition
  • capitalism
  • You!

and disruption applied to learning, which needs:

  • critical pedagogies
  • new collaborations
  • human-scale
  • Per to peer
  • social
  • Us!

Technology Enhanced Learning, Dialogicality and Practice

August 21st, 2012 by Graham Attwell

I like writing position papers! This is the second submission for the  Alpine Redezvous  workshop on the topic of TEL, the Crisis and the Response. This position paper is co-written with Dirk Stieglitz and Ilona Buchem.

We are aware of the increasing concerns about the commodification, monetarisation and privatization of education and academic labour. We also acknowledge the concern that the current mode of [neo-liberal] late-capitalism relies on “the continuous extension and validation of the infrastructure and the optimistic discourses of the new information technologies” (Hoofd, 2010)

However, rather than focus on concerns about the role of technology in the organisation and control of the educational infrastructure, in this position paper we which to examine the potential – and potential contradictions – of technology for learning. This in turn, leads to a focus on pedagogy, defined here as the theory and practice of teaching and learning. Technology is not pedagogically neutral – all technology enhances or hinders particular approaches to learning.

It is not hard to criticize the uses of educational technology in institutions. In its earlier phases technology was used to manage learners rather than facilitate learning. In its latter phase technology is being deployed both to commodify and monetarize knowledge (and the academic labour which produces such knowledge) and at the same time to sell education as just another consumer product (hence the present hype around so called learning analytics). Almost inevitably, attempts to develop an alternative ecology or milieu and an alternative pedagogy – such as MOOCs – are being absorbed by the dominant culture. Interestingly, in this regard, we can perceive the contradiction between an understanding of academic staff who wish to open up new horizons for learning to students with the concerns of the students who wish only to receive the necessary knowledge to achieve the credentials for which they believe they have paid. This in turn reinforces push technologies to support funnel delivery of learning objects to receivers (clients or customers). And despite the hype about the uses of technology by digital residents, repeated surveys have shown very limited use of social technologies by students to create, rather than consume digital artefacts and knowledge.

However, there is an alternative perspective. The almost indecent rush to commodify academic knowledge through the use of technology[1] may, to some extent, be driven by a realization that knowledge has escaped from the walled garden of the academy.

We would argue that the education systems grew in response to the needs of industrial capitalisms (in this respect it is informative to note that many Victorian schools in the UK were deigned to look like factories and were organised on a factory model). Despite the efforts of communities and organisations such as the Miners Hall, the Workers Educational Association and the Mechanics Institutes (and similar bodies and movements in other countries than the UK), access to education – and knowledge – was largely a monopoly of the education system, which in turn was ideologically driven by the needs of capitalist enterprises.

Despite the efforts of institutions and others – including publishers – to maintain control of knowledge, the internet allows an abundance of access to knowledge and learning, especially through informal and self managed learning. In a study we undertook of the use of information and communication for learning in Small and Medium Enterprises in six countries in Europe, in 106 case studies we found only one instance of the use of ICT for formal learning. Yet we found numerous uses of technology for informal learning (although often the users did not recognize this as learning themselves). We found:

–       the web was the platform for learning

–       in most cases the managers did not know such learning was happening

–       there was more likelihood of learning taking place where people had more control of work processes

–       learning was sometimes driven by just in time needs stemming from the work but was often driven by learners’ interests

–       learners had little interest in formal accreditation or credentials and no interest in assessment

Such learning often took place through contacting friends or through participating in informal, online communities of practice. Support for learning was through peers or those who Vygotsky called a More Knowledgeable Other and learning was largely self-directed.

Learning was heavily contextual, depending on both the subject and level of learning, the nature of the problem or the culture of the community.

Through a combination of the physical workplace and subject based culture and the culture of the online interactions, users were making new meanings for their own practice. This chimes with Bakhtin’s reasoning that others or other meanings are required for any cultural category to generate meaning and reveal its depths.

“Contextual meaning is potentially infinite, but it can only be actualized when accompanied by another (other’s) meaning, if only by a question in the inner speech of the one who understands. Each time it must be accompanied by another contextual meaning in order to reveal new aspects of its own infinite nature (just as the word reveals its meanings only in context). (Bakhtin, 1986, pp. 145–146).”

Akkerman and Bakker suggest that boundary crossing and the understanding of learning as a process that involves multiple perspectives and multiple parties is “different from most theories on learning that, first, often focus on a vertical process of progression in knowledge or capabilities (of an individual, group, or organization) within a specific domain and, second, often do not address aspects of heterogeneity or multiplicity within this learning process.”

Akkerman and Bakker advance “four dialogical learning mechanisms of boundaries:

  1. identification, which is about coming to know what the diverse practices are about in relation to one another;
  2. coordination, which is about creating cooperative and routinised exchanges between practices;
  3. reflection, which is about expanding one’s perspectives on the practices; and,
  4. transformation, which is about collaboration and co-development of (new) practices.”

The interesting point here is the relation to practices, and to dialogical learning processes, as opposed to the reified and top down nature of knowledge acquisition through institutional online learning and traditional TEL.

We suggest that if the TEL community is to contribute towards a response to the crisis, that response requires a move from a focus on formal knowledge transmission through educational technology controlled by institutions, to a perspective of supporting community knowledge acquisition and self directed learning focused on practice.  It equally requires a change in developmental approaches with technology co-developed with the communities of practice. Interestingly, it could be argued that such a change, although explicitly opposed to the use of TEL to commodify formal education, would provide a better social and economic use of technology in existing economies.

References

Akkerman, S. F., & Bakker, A. (2011). Boundary crossing and boundary objects. Review of Educational Research, 81, 132-169, http://rer.sagepub.com/cgi/reprint/0034654311404435v1?ijkey=4LKMx60v0wQzc&keytype=ref&siteid=sprer

Bakhtin, M. (1986). From notes made in 1970-71 (V. McGee, Trans.). In C. Emerson, & M. Holquist (Eds.), Speech genres & other late essays (pp. 132–158). Austin: University of Texas Press.

Hoofd, I. (2010), The accelerated university: Activist- academic alliances and the simulation of thought, in ephemera 2010 www.ephemeraweb.org volume 10(1): 7-24



[1] See for instance http://chronicle.com/blogs/wiredcampus/coursera-hits-1-million-students-with-udacity-close-behind/38801 although it is notable that this trend differs in different countries and economies

Boundary Crossing and Learning

August 13th, 2012 by Graham Attwell

I have been long interested in the idea of boundary objects especially in relation to the use of technology for learning in the workplace. In general I think one of the issues with Technology Enhanced Learning is that we have tended to ignore the importance of physical objects in learning and practice.

Following the presentation by Alan Brown and myself on Technologically Enhanced Boundary Objects (for use in careers guidance) at the final Mature-IP review meeting, Uwe Riss kindly referred us to two papers:

This is Not a Boundary Object: Reflections on the Origin of a Concept by Susan Leigh Star

and

Boundary Crossing and Boundary Objects by Sanne F. Akkerman and Arthur Bakker.

Sadly neither is available for open access (I have university access but I find it very frustrating when there is no open access to important work).

I haven’t read Star’s paper yet, but found the paper by Akkerman and Bakker very useful. They define boundaries as “sociocultural differences that give rise to discontinuities in interaction and action.” They have undertaken an extensive literature review of the use of the idea of boundary crossing in education. In particular I think that Baktin’s idea of ‘dialogicality’ helps explain how learning takes place with multiple sources of ideas and knowledge (which some are referring to as ‘abundance’, through the internet as well as through structured, course based learning.

Bakhtin’s basic line of reasoning was that others or other meanings are required for any cultural category to generate meaning and reveal its depths:

Contextual meaning is potentially infinite, but it can only be actualized when accompanied by another (other’s) meaning, if only by a question in the inner speech of the one who understands. Each time it must be accompanied by another contextual meaning in order to reveal new aspects of its own infinite nature (just as the word reveals its meanings only in context). (Bakhtin, 1986, pp. 145–146)

This Bakhtinian notion of dialogicality comes to the fore in the various claims on the value of boundaries and boundary crossing for learning: learning as a process that involves multiple perspectives and multiple parties. Such an understanding is different from most theories on learning that, first, often focus on a vertical process of progression in knowledge or capabilities (of an individual, group, or organization) within a specific domain and, second, often do not address aspects of heterogeneity or multiplicity within this learning process.

In the second part of their research Akkerman and Bakker look at the “four dialogical learning mechanisms of boundaries”:

  1. identification, which is about coming to know what the diverse practices are about in relation to one another;
  2. coordination, which is about creating cooperative and routinized exchanges between practices;
  3. reflection, which is about expanding one’s perspectives on the practices; and,
  4. transformation, which is about collaboration and codevelopment of (new) practices.

Work based Personal Learning Environments

July 10th, 2012 by Graham Attwell
View more presentations from GrahamAttwell
The Personal Learning Envrinments conference is a flipped conference. Traditional paper sessions are frowned on and presenters are invited to put forward their idea of how to engage participants in teh different sessions. Sessionc hairs are asked to negotiate with presenters. Thus it was that this afternoon I received an email from Linda Castenda who is chairing the session with my paper about Developing work Based Personal Learning Environments.
“Dear authors”, she said, “As you may already now, I’ll be the chair of your session in the PLE Conference in Aveiro. …
We will be together in the A3 session, on the Thursday 11th at 011 O’clock in the room Number 2.

I’m really happy of sharing with you this moment and I hope to have the possibility of trying a different kind of session that help us to find new ways for enriching or work and for find new ways of collaborating together.

I’ve thought on the possibility of organizing a different session, I’m absolutely open to all of your suggestions, but I would like to propose you the following organizing, if you like it:

11:10 to 11:15 Presentation and explanation of the session dynamic  (me).
For it I would love to have a picture of you for including on the presentation.

from 11:15 to 11:45 an “Speed dating” session:
Each paper will be localized in three different “spaces” in the same classroom. The audience will be divided into three groups that will be passing by each paper and  each one of you will have to explain your work in 9 minutes to three different groups of people. After 9 minutes I’ll ring an alarm to change the group.
I know you have to repeat the explanation 3 times, but It would help them to be more close to you and to be more “active” during the explanations. You could use for your presentation a slideshow (in your computer or tablet, a pamphlet, a paper, a trip tic, or whatever you want… BUT taking into account that you are presenting for few people only.
Attendees will have some papers to include questions and comments of the presentations that we will recover after the round of presenting.
If you don’t mind (if you do, please, let it me know)  I will love to record each presentation in video for uploading to the web after the session.

11:50 to 12:05 Panel for answering:
We will try to make a panel with you three and try to answer to questions. I will have also some question for you, only in case you have not any Q from the public

12:05 to 12:30 Conclusions and PLN
After the questions I will ask you for doing a deliverable by paper WITH the attendees… maybe you can think in something to do around your paper (questions, short activity, or whatever) , or maybe we can do something generic… what about a kind of map of relationships between the research presented by you and the attendees research area (including contact details)?

I’m sure it seems a bit complicated, but it would be very active and challenging in order to take advantage of the papers and the groups that are going to be there…”

I am very happy with the idea. But then the problem – I had brought no slides for this session. So in record time I have hacked together a quick presentation and thanks to the speed of teh network here in Aveiro, have got it up on slideshare already.

Open and Linked Data and Mediation

April 13th, 2012 by Graham Attwell

There has been an explosion of interest in Open Data and the potential for linking data to produce new social apps. Yet despite all this attentions, and the growing access to data in some countries such as the UK, the development of new apps has been less than impressive.

Rather than full apps, probably the main use has been the development of interactive visualizations allowing users to explore different data sets and quick visualisations of different data sets. The Guardian newspaper data blog has led the way in the UK and in particular has shown the value of open journalism such as in this discussion on how they got the colours of the maps right.

But the development of more advanced apps has been slower. Probably the biggest take off has been around transport allowing real time timetable tracking etc. But even here the problem of the social purpose and use of data apps is an issue. take this compelling app from the German newspaper Suddeutsche . Its hows graphic representations of train journeys in Germany, providing information on each train’s itinerary and the details of any delay. There is also an interactive timeline, allowing you to watch previous days’ travel play out. Its fun. But I can’t really see that it is much use! Or take this app – available in various forms – using crowd sourced data to find the nearest post box in the UK. Do we really need it? Why not just ask somebody / anybody?

In education there are a number of apps for finding schools etc. But there is little use of open and linked data for learning.

We have been working with a number of organisations to produce open and linked data apps for use in careers guidance. There are now three iterations of what we variously call a TEBO (Technologically Enhanced Boundary Object) or Careers dashboard.

The first was a quick demonstrator which we built to see how it might work. The second works through an API to the Careers Wales beta web site. And the third – more technically advanced – iteration is a database and API developed for UKCES which is not publicly available at present.

One of issues being raised in this work is mediation. In general government / agencies seem to regard data as just standing on its own. Within the TEBO concept we always stressed the need for social mediation and had ideas for a number of ways in which this might happen using social software e.g Question and Answer applications.

In fact mediation takes place at a series of levels – including the selection of data originally collected, and the way data is selected for use and display within an application. Different people will need different apps for interrogating the same data. For instance our Careers Dashboard may have potential interest and use for:

  • Young people thinking about career choices;
  •  Young people applying to further or higher education, seeking an apprenticeship or employment;
  • Adults who are newly unemployed;
  •  Long term unemployed adults;
  •  Adults considering re-entering education and training (e.g. women returners);
  • Adults thinking about a change in career direction (e.g. mid-career changers);
  • Parents and carers supporting young people wishing to enter further education, vocational training or employment
  • Career professionals – careers teachers, careers advisers and subject teachers; and
  • Various others (e.g. educational planners and policymakers, professionals preparing funding applications, researchers).

However, mediation seems to be commonly understood as intervention and then posed as a dichotomy between non intervention or intervention or to put it another way – let end users access to data or only let professionals access to data. This seems to me a misunderstanding of both the potentials and limitations of the data but of the potentially rich ways in which mediation happens and the ways in which technologically can be used in such processes.

It would be interesting to look at mediation within physical communities and through extended web and social media based communities. It would also be interesting to link mediation to the potential quality of careers interventions (i.e. after mediation takes place.)

More to follow…..

 

 

 

 

Evaluation 2.0: How do we progress it?

October 11th, 2011 by Jenny Hughes

Have been in Brussels for the last two days – speaking at 9th European Week of Regions and Cities organized by DG Regio and also taking the opportunity to join other sessions. My topic was Evaluation 2.0. Very encouraged by the positive feedback I’ve been getting all day both face-to-face and through twitter. I thought people would be generally resistant to the idea as it was fairly hard-hitting (and in fairness, some were horrified!) but far more have been interested and very positive, including quite a lot of Commission staff. However, the question now being asked by a number of them of them is “How do we progress this?” – meaning, specifically, in the context of the evaluation of Regional Policy and DG Regio intervention.

Evaluation 2.0 in Regional Policy evaluation
I don’t have any answers to this – in some ways, that’s not for me to decide! I have mostly used Evaluation 2.0 stuff in the evaluation of education projects not regional policy. And my recent experience of the Cohesion Fund, ERDF, IPA or any of the structural funds is minimal. However, the ideas are generic and if people think that there are some they could work with, that’s fine!

That said, here are some suggestions for moving things forward – some of them are mine, most have been mooted by various people who have come to talk to me today (and bought me lots of coffee!)

Suggestions for taking it forward

  • Set up a twitter hashtag #evaluation2.0. Well that’s easy but I don’t know how much traffic there would be as yet!
  • Set up a webpage providing information and discussion around Evaluation 2.0. More difficult – who does that and who keeps it updated? Maybe, instead, it is worth feeding in to the Evalsed site that DG Regio maintain, which currently provides information and support for their evaluators. I gather it is under the process of review – a good opportunity to make it more interactive, to make more use of multimedia and with space for users to create content as well as DG Regio!
  • Form a small working group or interest group – this could be formal or informal, stand alone or tied to their existing evaluation network. Either way, it needs to be open and accessible to people who are interested in developing new ideas and trying some stuff out rather than a representative ‘committee’.
  • Alternatively, set up an expert group to move some ideas forward.
  • Or how about a Diigo group?
  • Undertake some small-scale trials with specific tools – to see whether the ideas do cross over from the areas I work in to Regional Policy.
  • Run a couple of one-day training events on Evaluation 2.0 focusing on some real hands-on workshops for evaluators and evaluation unit staff rather than just on information giving.
  • Check out with people responsible for evaluation in other DGs whether there is an opportunity for some joint development (a novel idea!) Unlike other ‘perspectives’ it is not tied to content or any particular theoretical approach.
  • Think about developing some mobile phone apps for evaluators and stakeholders around content specific issues – I can easily think of 5 or 6 possibilities to support both counterfactual, quantitative approaches and theory-based qualitative approaches. Although the ideas are generic, customizing the content means evaluators would have something concrete to work with rather than just ideas.
  • Produce an easy-to-use handbook on evaluation 2.0 for evaluators / evaluation units who want practical information on how to do it.
  • Ring fence a small amount of funding to support one-off explorations into innovative practice and new ideas around evaluation.
  • Encourage the evaluation unit to demonstrate leadership in new approaches – for example, try streaming a live internet radio programme around the theme of evaluation (cheap and easy!); set up a multi-user blog for people to post work in progress and interesting observations of ongoing projects using a range of media as well as text-based major reports; make some podcasts of interviews with key players in the evaluation of Regional Policy; set up a wiki around evaluation rather than having to drill down through the various Commission websites; try locating projects using GPS data so that we can all see where the action is taking place! Keep a twitter stream going around questions and issues – make use of crowd sourcing!
  • Advertise the next European Evaluation Society biennial conference, in Helsinki, October 1st – 5th 2012 “Evaluation in the networked society: New concepts, New challenges, New solutions” (There you go Bob, I just did!)
  • Broaden the idea of Evaluation 2.0 and maybe get rid of the catchphrase! We are already using the power of the semantic web in evaluation to mash open and linked data, for example. Should we be now be talking about Evaluation 3.0?? Or should we find another name – Technology Enhanced Evaluation? We could have TEE parties instead of conferences – Europe’s answer to the American far right ; )

P.S. Message to the large numbers of English delegates at the conference

When you left Heathrow yesterday to come to Brussels, I do hope you waved to the English Rugby team arriving home from the Rugby World Cup in New Zealand.

(Just as well this conference was not a week later or I’d have leave a similar message for the French delegates…..)

Open Badges, assessment and Open Education

August 25th, 2011 by Graham Attwell

I have spent some time this morning thinking about the Mozilla Open Badges and assessment project, spurred on by the study group set up by Doug Belshaw to think about the potential of the scheme. And the more I think about it, the more I am convinced of its potential as perhaps one of the most significant developments in the move towards Open Education. First though a brief recap for those of you who have not already heard about the project.

The Open Badges framework, say the project developers, is designed to allow any learner to collect badges from multiple sites, tied to a single identity, and then share them out across various sites — from their personal blog or web site to social networking profiles. The infrastructure needs to be open to allow anyone to issue badges, and for each learner to carry the badges with them across the web and other contexts.

Now some of the issues. I am still concerned of attempts to establish taxonomies, be it those of hierarchy in terms of award structures or those of different forms of ability / competence / skill (pick your own terminology). Such undertakings have bedeviled attempts to introduce new forms of recognition and I worry that those coming more from the educational technology world may not realise the pitfalls of taxonomies and levels.

Secondly is the issue of credibility. There is a two fold danger here. One is that the badges will only be adopted for achievements in areas / subjects / domains presently outside ‘official’ accreditation schemes and thus will be marginalised. There is also a danger that in the desire to gain recognition, badges will be effectively benchmarked against present accreditation programmes (e.g. university modules / degrees) and thus become subject to all the existing restrictions of such accreditation.

And thirdly, as the project roils towards a full release, there may be pressures for restricting badge issuers to existing accreditation bodies, and concentrating on the technological infrastructure, rather than rethinking practices in assessment.

Lets look at some of the characteristics of any assessment system:

  • Reliability

Reliability is a measure of consistency. A robust assessment system should be reliable, that is, it should yield the same results irrespective of who is conducting it or the environmental conditions under which it is taking place. Intra-tester reliability simply means that if the same assessor is looking at your work his or her judgement should be consistent and not influenced by, for example, another assessment they might have undertaken! Inter-tester reliability means that if two different assessors were given exactly the same evidence and so on, their conclusions should also be the same. Extra-tester reliability means that the assessors conclusions should not be influenced by extraneous circumstances, which should have no bearing on the evidence.

  • Validity

Validity is a measure of ‘appropriateness’ or ‘fitness for purpose’. There are three sorts of validity. Face validity implies a match between what is being evaluated or tested and how that is being done. For example, if you are evaluating how well someone can bake a cake or drive a car, then you would probably want them to actually do it rather than write an essay about it! Content validity means that what you are testing is actually relevant, meaningful and appropriate and there is a match between what the learner is setting out to do and what is being assessed. If an assessment system has predictive validity it means that the results are still likely to hold true even under conditions that are different from the test conditions. For example, performance evaluation of airline pilots who are trained to cope with emergency situations on a simulator must be very high on predictive validity.

  • Replicability

Ideally an assessment should be carried out and documented in a way which is transparent and which allows the assessment to be replicated by others to achieve the same outcomes. Some ‘subjectivist’ approaches to evaluation would disagree, however.

  • Transferability

Although each assessment is looking at a particular set of outcomes, a good assessment system is one that could be adapted for similar outcomes or could be extended easily to new learning.  Transferability is about the shelf-life of the assessment and also about maximising its usefulness.

  • Credibility

People actually have to believe in the assessment! It needs to be authentic, honest, transparent and ethical. If people question the rigour of the assessment process, doubt the results or challenge the validity of the conclusions, the assessment loses credibility and is not worth doing.

  • Practicality

This means simply that however sophisticated and technically sound the assessment is, if it takes too much of people’s time or costs too much or is cumbersome to use or the products are inappropriate then it is not a good evaluation!

Pretty obviously there is going to be a trade off between different factors. It is possible to design extremely sophisticated assessments which have a high degree of validity. However, such assessment may be extremely time consuming and thus not practical. The introduction of multiple tests through e-learning platforms is cheap and easy to produce. However they often lack face validity, especially for vocational skills and work based learning.

Lets try to make this discussion more concrete by focusing on one of the Learning Badges pilot assessments at the School of Webcraft.

OpenStreetMapper Badge Challenge

Description: The OpenStreetMapper badge recognizes the ability of the user to edit OpenStreetMap wherever satellite imagery is available in Potlatch 2.

Assessment Type: PEER – any peer can review the work and vote. The badge will be issued with 3 YES votes.

Assessment Details:

OpenStreetMap.org is essentially a Wikipedia site for maps. OpenStreetMap benefits from real-time collaboration from thousands of global volunteers, and it is easy to join. Satellite images are available in most parts of the world.

P2PU has a basic overview of what OpenStreetMap is, and how to make edits in Potlatch 2 (Flash required). This isn’t the default editor, so please read “An OpenStretMap How-To“:

Your core tasks are:

  1. Register with OpenStreetMap and create a username. On your user page, accessible at this link , change your editor to Potlatch 2.
  2. On OpenStreetMap.org, search and find a place near you. Find an area where a restaurant, school, or gas station is unmapped, or could use more information. Click ‘Edit’ on the top of the map. You can click one of the icons, drag it onto the map, and release to make it stick.
  3. To create a new road, park, or other 2D shape, simply click to add points. Click other points on the map where there are intersections. Use the Escape to finish editing.
  4. To verify your work, go to edit your point of interest, click Advanced at the bottom of the editor to add custom tags to this point, and add the tag ‘p2pu’. Make its value be your P2PU username so we can connect the account posting on this page to the one posting on OpenStreetMap.
  5. Submit a link to your OpenStreetMap edit history. Fill in the blank in the following link with your OpenStreetMap username http://www.openstreetmap.org/user/____/edits

You can also apply for the Humanitarian Mapper badge: http://badges.p2pu.org/questions/132/humanitarian-mapper-badge-challenge

Assessment Rubric:

  1. Created OpenStreetMap username
  2. Performed point-of-interest edit
  3. Edited a road, park, or other way
  4. Added the tag p2pu and the value [username] to the point-of-interest edit
  5. Submitted link to OpenStreetMap edit history or user page to show what edits were made

NOTE for those assessing the submitted work. Please compare the work to the rubric above and vote YES if the submitted work meets the requirements (and leave a comment to justify your vote) or NO if the submitted work does not meet the rubric requirements (and leave a comment of constructive feedback on how to improve the work)

CC-BY-SA JavaScript Basic Badge used as template5.

Pretty clearly this assessment scores well on validity and also looks to be reliable. The template could easily be transferred as indeed it has in the pilot. It is also very practical. However, much of this is due to the nature of the subject being assessed – it is much easier to use computers for assessing practical tasks which involve the use of computers than it is for tasks which do not!

This leaves the issue of credibility. I have to admit  know nothing about the School of Webcraft, neither do I know who were the assessors for this pilot. But it would seem that instead of relying on external bodies in the form of examination boards and assessment agencies to provide credibility (deserved for otherwise), if the assessment process is integrated within communities of practice – and indeed assessment tasks such as the one given above could become a shared artefact of that community – then then the Badge could gain credibility. And this seems a much better way of buidli9ng credibility than trying to negotiate complicated arrangements that n number of badges at n level would be recognized as a degree or other ‘traditional’ qualification equivalent.

But lets return to some of the general issues around assessment again.

So far most of the discussions about the Badges project seem to be focused on summative assessment. But there is considerable research evidence that formative assessment is critical for learning. Formative assessment can be seen as

“all those activities undertaken by teachers, and by their students in assessing themselves, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged. Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet the needs.”

Black and Williams (1998)

And that is there the Badges project could come of age. One of the major problems with Personal Learning Environments is the difficulties learners have in scaffolding their own learning. The development of formative assessment to provide (on-line) feedback to learners could help them develop their personal learning plans and facilitate or mediate community involvement in that learning.Furthermore a series of tasks based assessments could guide learners through what Vygotsky called the Zone of Proximal Development (and incidentally in Vygotsky’s terms assessors would act as Significantly Knowledgeable Others).

In these terms the badges project has the potential not only to support learning taking place outside the classroom but to build a significant infrastructure or ecology to support learning that takes place anywhere, regardless of enrollment on traditional (face to face or distance) educational programmes.

In a second article in the next few days I will provide an example of how this could work.

OERs, communities and openness: A Paradox?

June 17th, 2011 by Graham Attwell

I am intrigued by this abstract is for a Symposium that will be presented at ALT-C 2011 by Frances Bell, Cristina da Costa, Josie Fraser, Richard Hall and Helen Keegan and is published on Frances Bell’s blog. I am reproducing it in full below.

This symposium will examine the paradoxes of giving and receiving online in education in a changing economic climate.  Each of the panelists will briefly address topic areas within the symposium theme, followed by an opportunity for present and at distance audiences to contribute, concluding with a 25 minute plenary discussion.

Symposium delegates will be provoked to reconsider the costs of participation online by paid and unpaid participants in ‘open’ discussion and sharing of resources.

Open Educational Resources exist within communities that create, use and sustain them (Downes 2007). When ‘communities’ in Higher Education break down due to redundancy and casualisation of labour what happens to OERs? Are they sustained? Can they reach out to other contexts?

All areas of education, including the school sector, currently face significant financial challenges and uncertainties. Institutions are increasingly reviewing the provision of devices and services, and looking at learner owned devices and commercially owned ‘free’ cloud-based services. What is the real price of an education system supported and transformed by embedded learning technologies?

Ownership in the age of openness calls for clarity about mutual expectations between learners, communities and ourselves. Ideas and content are shared easily through open platforms, and yet attributions can be masked in the flow of dissemination: does credit always go where it is due?

Openness in the production, sharing and reuse of education/resources is meaningless in the face of neoliberalism. Where coercive competition forms a treadmill for the production of value, openness/OERs are commodified. Control of the educational means of production determines power to frame how open are the relations for the production or consumption of educational goods or services, in order to realise value. The totality of this need, elicited by the state for capital, rather than the rights of feepayers, parents, communities or academics, shapes how human values like openness are revealed and enabled within HE.

Scarce research monies focus attention on impact factors, arguably stagnating practice. For publications, Open Access can increase wider societal impact but at the expense of career progression.We explore the tensions, paradoxes and professional costs on societal benefits, individual agency and academic progression.

Obviously this is a bit of a mash up proposal but it does raise a lot of questions. I think there is a tension between the idea of communities and institutions. Communities of practice, and for that matter the communities in which Open education Resources are being produced and shared, cross institutional boundaries. Furthermore the use of OERs may be within an institutional setting but may also be outside.

This again is reflected in recognition and reward structures. Whilst reward structures within institutions are based on either monetary compensation or in terms of progression, rewards within the community may reflect a wider understanding of recognition, especially respect or standing within that community. Does credit always go to who it should? Probably not, but this is taking a very individualistic view of research. Surely credit should be shared in the community rather than in the closed door offices of academic researchers.

Are OERs being commodified? Presumably the term “coercive competition” refers to the growing practice to require academics and researchers to publish their work as OERs. I don’t really understand what the authors mean in saying “The totality of this need, elicited by the state for capital, rather than the rights of feepayers, parents, communities or academics, shapes how human values like openness are revealed and enabled within HE.” Of course the idea of openness is being hijacked by institutions. But at the same time the movement towards openness is contradictory and I am not sure this is reflected in the abstract. Especially missing is a discussion of the nature of OERs in allowing reuse and modification and the impact this has on (commodified) relations of production and intellectual property. And at the same time, the spread of OERs is allowing new open forms of learning and knowledge production outside the confines of the institution. thus whilst the movement towards OERs may be becoming commodified the use of OERs is challenging traditional understandings of those very commodities.

I don’t think this reveals a paradox but a dialectical contradiction. Present schooling models of education are being found to be wanting. The discussion about open education is important in that it could provide an alternative to privatisation. The discussion over OERs forms an important part of this debate and to this extent the debate over this symposium is extremely interesting.

Pedagogic Approaches to using Technology for Learning – Literature Review

May 31st, 2011 by Graham Attwell

The proliferation of new technologies and internet tools is fundamentally changing the way we live and work. The lifelong learning sector is no exception with technology having a major impact on teaching and learning. This in turn is affecting the skills needs of the learning delivery workforce.

Last September, together with Jenny Hughes I undertook a literature review on new pedagogical approaches to the use of technologies for teaching and learning. You can access the full (86 pages) document below.

The research was commissioned by LLUK to feed into the review then being undertaken of teaching qualifications in the Lifelong Learning sector in the UK. The review was designed to ensure the qualifications are up to date and will support the development of the skills needed by the modern teacher, tutor or trainer.

However, we recognised that the gap in technology related skills required by teaching and learning professionals cannot be bridged by qualifications alone or by initial training and a programme of opportunities for continuing professional development (CPD) is also needed to enable people to remain up to date.

The literature review is intended to

  • identify new and emerging pedagogies;
  • determine what constitutes effective use of technology in teaching and learning
  • look at new developments in teacher training qualifications to ensure that they are at the cutting edge of learning theory and classroom practice
  • make suggestions as to how teachers can continually update their skills.

Pedagogical Appraches for Using Technology Literature Review January 11 FINAL 1

  • Search Pontydysgu.org

    Social Media




    News Bites

    Cyborg patented?

    Forbes reports that Microsoft has obtained a patent for a “conversational chatbot of a specific person” created from images, recordings, participation in social networks, emails, letters, etc., coupled with the possible generation of a 2D or 3D model of the person.


    Racial bias in algorithms

    From the UK Open Data Institute’s Week in Data newsletter

    This week, Twitter apologised for racial bias within its image-cropping algorithm. The feature is designed to automatically crop images to highlight focal points – including faces. But, Twitter users discovered that, in practice, white faces were focused on, and black faces were cropped out. And, Twitter isn’t the only platform struggling with its algorithm – YouTube has also announced plans to bring back higher levels of human moderation for removing content, after its AI-centred approach resulted in over-censorship, with videos being removed at far higher rates than with human moderators.


    Gap between rich and poor university students widest for 12 years

    Via The Canary.

    The gap between poor students and their more affluent peers attending university has widened to its largest point for 12 years, according to data published by the Department for Education (DfE).

    Better-off pupils are significantly more likely to go to university than their more disadvantaged peers. And the gap between the two groups – 18.8 percentage points – is the widest it’s been since 2006/07.

    The latest statistics show that 26.3% of pupils eligible for FSMs went on to university in 2018/19, compared with 45.1% of those who did not receive free meals. Only 12.7% of white British males who were eligible for FSMs went to university by the age of 19. The progression rate has fallen slightly for the first time since 2011/12, according to the DfE analysis.


    Quality Training

    From Raconteur. A recent report by global learning consultancy Kineo examined the learning intentions of 8,000 employees across 13 different industries. It found a huge gap between the quality of training offered and the needs of employees. Of those surveyed, 85 per cent said they , with only 16 per cent of employees finding the learning programmes offered by their employers effective.


    Other Pontydysgu Spaces

    • Pontydysgu on the Web

      pbwiki
      Our Wikispace for teaching and learning
      Sounds of the Bazaar Radio LIVE
      Join our Sounds of the Bazaar Facebook goup. Just click on the logo above.

      We will be at Online Educa Berlin 2015. See the info above. The stream URL to play in your application is Stream URL or go to our new stream webpage here SoB Stream Page.

  • Twitter

  • Recent Posts

  • Archives

  • Meta

  • Categories