What is Evaluation 2.0?
Graham Attwell interviews Jenny Hughes about Evaluation 2.0
Just what is Evaluation 2.0?
Evaluation 2.0 is a set of ideas about evaluation that Pontydysgu are developing. At its simplest, it’s about using social software at all stages of the evaluation process in order to make evaluation more open, more transparent and more accessible to a wider range of stakeholders. At a theoretical level, we are trying to push forward and build on Guba and Lincoln’s ideas around 4th generation evaluation which is a constructivist approach incorporating key ideas around negotiation, multiple realities and stakeholder engagement. But this is the first part of the journey – ultimately, I believe that e-technologies are going to revolutionise the way we think about and practice evaluation.
In what way do you think this is going to happen?
Firstly, the use of social media gives stakeholders a real voice – irrespective of where they are located. Stakeholders can create and publish evaluation content. For example, in the past I might carry out some interviews as part of an evaluation. Sometimes I recorded it, sometimes I just made notes. Then I would try and interpret it and draw some conclusions about what it meant. Now I set up a web page for each evaluation and I podcast the interviews using audio or video and put them on the site. (Obviously this has to be negotiated with the interviewee but so far, no one has raised any objections.) There is the usual comment box so any stakeholder with access to the site can respond to the interview, add their interpretations, agree or disagree with my conclusions and so on.
Secondly, I think it is challenging our perceptions of who are evaluators. Everyone is now an evaluator. Think of the software that you use every day for on-line shopping from Amazon or Ebay or any big chain store. If I want to buy a particular product I check out what other people have said about it, how many of them said it and how many stars it has been given. These are called recommender systems and I think they will have a big impact in evaluation. We have moved from the paradigm of the ‘expert’ collecting and analyzing data into a world of crowd sourcing – harnessing the potential of mass data collection and interpretation.
Thirdly, the explosion of Web 2.0 applications has provided us with a whole new range of evaluation tools that open up new methodological possibilities or make the old ones more efficient. For example, if I am at the stage of formulating and refining the evaluation questions – I put it out as a call on Twitter. It’s amazing how restricting evaluation questions to 140 characters can sharpen them up!
I did an evaluation of a community capacity building project in an inner city area recently and spent quite a long time before I went to the first meeting walking around the streets, checking out the community facilities, the state of the housing, local amenities and so on, to get a ‘feel’ for the area – except I did it on Google Earth and with street-view on Google maps. There are about 20 or so other applications I use a lot in evaluation but maybe they will have to wait for another edition!
Fourthly, I think the potential of Web 2.0 changes the way we can visualize and present data. Why are we still writing long and indigestible text-based evaluation reports? Increasingly clients are preferring short, sharp evaluation ‘articles’ on maybe one outcome of an evaluation which they can find on a ‘newsy’ evaluation webpage – with hyperlinks to more detailed information or raw data or back up evidence if they want to check it out. We can also create ‘chunks’ of evaluation reporting and repurpose them in different ways for different stakeholders or they can be localized for different cultures – for example, I have started doing executive summaries as downloadable podcasts. I think evaluation 2.0 is about creating a much wider range of evaluation products.
Following on from that, I think Evaluation 2.0 breaks down the formative-summative divide and notions of ‘the mid-term report’ or ‘the ex-ante report’. Evaluation 2.0 is continuous, it is dynamic and it is interactive. For example, I use Googledocs with all my clients – I add them as readers and editors on all the folders that relate to their evaluations. At any time of the day or night they can see work in progress and add their comments. I keep their evaluation website up to date so they get evaluation information as soon as it is available.
So do you think all evaluators will have to move down this road or will there always be a place for evaluators using more established methods?
Personally, I think massive change is inevitable. Apart from anything else, our clients of the future will be the digital natives – they will expect it.
There will always be a role for the evaluator but that role will be transformed and the skills will be different. I think a key job for the specialist evaluator will be designing the algorithms that underpin the evaluation. The evaluator will also need to be the creative director – they will need skills in informatics, in visualizing and presenting information, the creative skills to write blogs and wikis. They will need networking skills to set up and facilitate online communities of practice around different stakeholder groups and the ability to repurpose evaluation objects.
The rules of engagement are also changing – in the past you engaged with a client, now you engage with a community. We also have to think how stakeholder created content might change our ideas about copyright, confidentiality, ownership, authorship.
So do you think evaluators as we know them will become extinct!!
Well, as Mark Halper said
“Dinosaurs were highly successful and lasted a long time. They never went away. They became smaller, faster, and more agile, and now we call them birds.”
[...] Bridge to Learning – Educational Research So do you think evaluators as we know them will become extinct!! Well, as Mark Halper said The rules of engagement are also changing – in the past you engaged with a client, now you engage with a community. We also have to think how stakeholder created content might change our ideas about copyright, confidentiality, ownership, authorship. [...]