Why can’t great review papers be 4*?

I was recently involved with a study of evidence use in Parliament, a piece of research led by UCL STEaPP and POST. One of the main conclusions from this piece of work was that policy-makers weren’t interested in the latest, shiny, new academic papers. Since time is precious, they wanted accessible summaries of what the body of evidence said; in other words, nicely summarised syntheses of the evidence. I’ve written about this already here and make similar points in this blog written by a member of the Science in Policy group at the University of Sheffield.

DP48B4eXUAAWEAG

Imagine my surprise then in my first ever REF progress meeting as a new permanent member of Faculty. When asked to propose my top papers, I immediately suggested a review that I did in Methods in Ecology and Evolution, which I thought was probably my most useful paper. The dismissal was instant – ‘sorry, that’s a review paper, that doesn’t make an ‘original, novel, significant contribution’ to knowledge so that can never be 3* or 4*’. What nonsense is this? We have policy-makers desiring syntheses of evidence, while at the same time criticising academia for not providing evidence in a usable format, and the powers-at-be shaping the REF are actively discouraging evidence synthesis? And as Neal Haddaway said to me after reading this, the novel part is the synthesis! Madness!

Now you might tell me that the impact part of the REF rewards such activity. Well yes, maybe it does. But academic career progression is still mainly linked to your ability to write a 4* paper. As I’ve written here, this seems to motivate academics to spend a good deal of time making up some fancy-words, upping the ‘bull****’ in a paper, and focusing on novelty. The consequence – we end up swimming in an ocean of new information without being able to make the most of what we already have.

The way to make the most of the evidence we have is to synthesise. Now I’ve had some experience of working with Bill Sutherland’s Conservation Science group at Cambridge who are real pioneers of synthesising, and then summarising conservation evidence, in a user-friendly way. This takes great skill and effort. Occasionally I’ve been met with slightly disappointed comments from colleagues when I say that I’ve written a review paper – a sort of patronising tone that might be saying ‘you did a review paper, that’s nice, you weren’t clever enough to come up with something novel, but well done anyway, that will be policy relevant’. This perception that systematic reviews and summaries of evidence are somehow easier than other types of academic papers is nonsense. It is exactly the kind of snobbish attitude that means that great review papers can’t be 4*.

So, answers in the comments section please everyone – why can’t a great review paper be worth 4*? It takes great skill, and it is what policy-makers want.

Advertisements

Call my Geography Bluff…

This week on the train home from UEA, I started reading a Geography paper. To be honest, it’s been a little while since I read something from a Geography journal because much of my recent work has been based either in the agricultural, conservation, or political sciences. As I read the first page, however, the memories of my Geography degree came flooding back…’why’, I thought to myself, ‘have the authors made up so many complicated words and why have they used ten words when one or two would have sufficed?’. Now this is a gross over-simplification – I have seen many examples of great and bad writing in all fields, and there are certainly papers that I’ve written which are unnecessarily complicated in part.

But as I carried on through this paper, all I could think of was the BBC’s old gameshow ‘Call my Bluff’ . For those unfamiliar with this BBC daytime classic, which involved the brilliant Sandi Toksvig in later series, the gameshow was centred around obscure words. One team had to read out an obscure word with a definition and claim that their one was truthful (only one card was the truth). The other team had to guess which definition was truthful.

So let’s try that with Geography. Let’s start with a Foucauldian classic ‘Governmentality’. Is that:

(a) the way in which the state exercises control over, or governs, the body of its populace

(b) the capacity for intellectual thought in a government

(c) David is just really jealous that he can’t make up complicated words to gain a 4* paper and should really shut up

Answers in the comments maybe?

Let’s try another. ‘Glocalization’

(a) Someone mis-spelled globalization

(b) the practice of doing business making both global and local considerations

(c) David still hasn’t written a Geographical 4* paper and is getting increasingly jealous

Another. ‘Tricontinentalism’

(a) “An alternative term to post-colonialism that emphasizes the transnational locations and the political implications of critiques of colonialism and imperialism.” (Dictionary of Human Geography)

(b) “Speaking about three continents at once” (Dictionary of Academic Optimism)

(c) You guessed it…I’ll say no more

Last one. ‘Social constructivism’. Ah we can bring good old Marx in here – I say good old Marx, I tend to have significant disagreements with every Marxist Geographer that I meet. Never been a fan of Marx.

(a) social interaction and processes create knowledge and order in the world

(b) it’s definitely (a), this theory has some merits

(c) still definitely (a), but if you go too far down the road of social constructivism you might think that everything in the world is socially constructed and you’ll think nothing is really real.

OK that was a largely pointless exercise in which I listed a few complicated Geographical words or phrases – I could list many further words that are far more difficult! In response to the first draft of this blog, someone sent me this paper  . I bet this is 4*, prizes for anyone who can tell me what it means. There are some spectacular sentences in it!

My point is simple, however, and it is a criticism not just labelled at Geography. Most of us are using taxpayers money to fund our research, so surely we have some sort of duty to write things in ways that people understand. Too many papers that I read in the discipline are not written in an accessible way. Indeed, papers that I’ve written, and papers from colleagues, have sometimes been criticised by Geography reviewers for being written in too much of a policy relevant, tangible way. We’ve been advised to make the paper more theoretical, and less policy relevant – surely that’s code for less useful. And I knew a former colleague who used the phrase ‘policy relevant’ as a marking code on dissertations that he felt were useful, but not theoretically elegant – yes, they really used the phrase ‘policy relevant’ as a criticism! Madness! I wager that a fair few papers rated 4* are actually no better, nor more useful, than papers ranked 3*, but they are ranked as such because they’ve managed to make up some fancy-sounding word or theory and convinced the reviewer that it sounds cool.

This reminds me of some other conversations I’ve had over the years. I remember a session held at Cambridge by a famous visiting professor on how to communicate more effectively outside of the discipline. This academic arranged a session for early career researchers based on one of the academic’s papers – after speaking for nearly an hour on their own paper, they finally asked for contributions from the early career researchers. My first contribution, perhaps rudely, but it needed saying – was any strategy that doesn’t involve writing an incomprehensible paper like that is good! Surprisingly that started a chain of emails to me after the session in which some of the other ECRs wrote to me privately saying that they were also frustrated about how Geographers sometimes wrote in difficult ways and they felt duty-bound to follow suit. It was disappointing that they felt unable to share this critique in front of big-name Geographers, but I’m never a shrinking violet.

Second story, I’ll always remember the job interview in which I was asked to describe what I did. The interviewer then replied to me with ‘OK that’s a nice bit of consultancy you’ve just described there, but how is it academic?’. To which I answered ‘well it was published in Nature Climate Change, and it’s very policy relevant, so I’m not sure what your definition of academia is, perhaps it is publishing obscure papers in journals no one reads, but that isn’t my definition of academia’. I didn’t get that job but it was fun to render the interviewer completely silent…..

Rant from me over! But let’s try and write more accessible and understandable papers. It isn’t clever to write in a way that no-one bar a few so-called ‘clever’ Geographers understand – mind you that’s made the careers of quite a few big-name Geographers that I won’t mention! It’s far cleverer to write in a clear, concise, and understandable way! The best compliment I had this week was from AHDB, who I wrote a report for – they said that they found it clearly written! They had expected not to be able to read it because it had been written by an academic, but were surprised that they’d been able to read the whole thing in one go! Winning!

Overcoming the barriers to the use of conservation science in policy: time for action

Our open access accepted article on the barriers and solutions to the use of conservation science in policy is now online with Conservation Letters. The main data used in the paper are from a global multi-lingual survey filled in by 758 research scientists, practitioners, or people in policy positions. Do bear with us though – we still need to tidy a little bit of the wording (e.g. last sentence abstract) and add multi-lingual abstracts, the latter of which is such an important thing to do in order to make research more accessible. Stay tuned for the more updated version in a matter of weeks! In the typeset version, the article will be easier to read, as the figures should be in more logical places rather than at the end! We will also be publishing opinion pieces written by several members of the fantastic author team in the near future – hopefully in news outlets all over the world and in several languages! From a personal viewpoint, it has been amazing to work with a diverse list of authors from multiple academic departments and conservation NGOs associated with the Cambridge Conservation Initiative (including the University of Cambridge Conservation Research Institute and the Conservation Leadership programme) in the David Attenborough Building. Collaboration between, and outside of, academic disciplines is really where research should be these days!

 

 

 

The most interesting result from our study is that there is agreement (perhaps surprisingly!) between research scientists, practitioners, and people in policy positions about the main barriers preventing the use of conservation science in policy. Although barriers such as lack of policy relevant science, lack of understanding of science on the part of policy-makers, and limited awareness of policy processes from researchers, featured in the top-ten barriers included in the online survey, they were not the most highly ranked.

 

 

Rather, all groups ranked barriers relating to the limited importance of conservation on the policy agenda as most important, as well as the related barrier of lack of funding for science, and finally poor communication between scientists and policy-makers. This suggests that while communication problems, and lack of policy relevance or understanding (less so), are partially to blame for lack of science use, they are not the biggest barriers. Instead, the fact that conservation is not a political priority, or that private sector agendas often dominate and contradict long-term pro-environmental management, seem to be the key reasons why conservation science is not influencing the policy agenda.

Unsurprisingly, therefore, the most pressing solutions identified by our respondents do not relate to the production of more science, but rather to winning the hearts and minds of people (both the public and policy-makers; related!) for the conservation cause. One of our respondents said that this does not mean targeting messages to people already converted to the conservation cause. Instead, we need to be inspiring people who are not currently convinced by the need to conserve our environment, who can then put pressure on policy-makers to make good evidence-informed decisions. In the paper, we speculate about whether we need a new kind of conservation science, as suggested in recent papers, that moves beyond a narrow technical approach towards a more inclusive system that involves people in the co-production of knowledge.

 

Figure3
Solutions to top-ranked barriers. Under a CC-BY licence, Rose et al. 2018, see details in first sentence

 

One personal reflection from this piece of work. The list of barriers and solutions in this paper is really nothing new. The fact that different groups from research, practice, and policy backgrounds seem to agree on what the problems and solutions are may initially seem optimistic. If we all recognise how to overcome the problem of lack of science use, then surely this makes it easier to do something about it. Yes, that seems logical. But, this is one of many papers that identify similar barriers and similar solutions. If the same issues continue to be noted, then would it not suggest that progress isn’t being made? If we all agree on solutions, why haven’t we solved the problems?

I can see two logical reasons. Firstly, perhaps the barriers to science use are just too hard to overcome. Certainly, improving the use of science in policy is challenging in non-linear democracies where it is right that values and other forms of knowledge are included in decision-making. Furthermore, convincing people of the need to conserve the environment long-term is challenging when many are understandably concerned about their own lives in the short-term, and may not see benefits in changing their behaviour. In this case, I do not agree that the challenges are too great to overcome. We can present more convincing, optimistic messages about conservation, which can be persuasive and can make the environment relevant to people’s lives. We can present science in better ways so that it is as easy as possible for policy-makers and practitioners to use it.

The second potential barrier to action may be institutional/organisational/even cultural, and here I will talk about academia because of the nature of this blog platform and my own work. I suspect that many research scientists, policy-makers, and practitioners know about the things they can do to improve the use of science. Research scientists could present evidence in clearer, user-friendly ways, and seek to co-produce knowledge with others. But, do they have time to do this and are they rewarded for it? Perhaps not, although this is changing, but can we find ways of doing it anyway? To some extent, I’m sure we can. There are likely to be many other institutional and organisational barriers elsewhere.

What is clear from this piece of work is that there is agreement on what the problems and solutions are to improve the use of conservation science in policy. It would be nice if another survey in five or ten years’ time didn’t find the same barriers and proposed solutions. For this to be the case, action is required from all of us.

When the full version is published, we will include translated abstracts in several languages in the supplementary material.

 

 

 

No luck stopping that erosion then? Slapton social science stories

I’m just back from a one-week fieldtrip at Slapton Ley in Devon. We took a mix of first year Geographers, Environmental Scientists, and Climate Change students (and a few more!) to take part in a mixed programme of field skills training. Since this is an optimistic blog, I’ll focus on the positive stories here, so you won’t here much about the quality of my bed in the accommodation!

I was responsible for the social science elements of the week, which was a great opportunity to hone the skills of those who had experienced such methodologies before, but also to introduce new skills to those nearer the natural science end of the spectrum. First and foremost, the location of the field course was fantastic, if a little remote for one or two of the students! The Field Centre at Slapton hosted us wonderfully and the food was a particular highlight! And there were lots of Cirl Buntings!

On the first weekend we had two sessions; firstly on experimental design showing how you could design an experiment to investigate the effect of an intervention by splitting groups into ‘control’ and ‘intervention’ populations. Firstly, I split the group in two and asked them to identify some photos of birds seen in the Slapton area using a field guide as help. Unbeknown to them, one group had clues telling them which ‘family’ to look at in the bird guide. Similarly, we then asked the two groups to name the 50 states of the USA, but again one group had some clues and the control group didn’t. Then the students were asked to design their own experiments and perform them on their peers using a control and intervention group – we had some interesting ideas! Experiments involved testing whether signs encouraged people to switch off the light or turn off the tap (one group didn’t have the signs in each case) and whether people would be more likely to take biscuits from a plate if a sign said ‘help yourself’. Some of them worked as expected, some didn’t, probably because baseline knowledge/behaviour in each group wasn’t the same. I was really pleased that one group decided to use experimental design for their independent project – testing whether a video of ocean plastic pollution would change people’s opinions on taking part in beach cleans. The control group received no such intervention!

Next we did some surveys on visitor’s perceptions of the Slapton coast. It was great to see some students, who were very nervous about approaching people, get over their fears and actually conduct surveys with several people! I’ve not forgotten the students who said that they couldn’t possibly talk to people, who then chose to do surveys as part of their independent project! In fact, it was heartening to see this all week – students saying they couldn’t do something, but learning that they could after trying it out!

As part of this session, we went down to look at the A379 road which had been affected by coastal erosion. I had asked my demonstrator to prepare a visual aid showing the erosion, but I was quite surprised when I got there to realise that they had taken the brief a little too seriously and 2/3 of the road was missing…..not really, the road was washed away by Storm Emma! The management of the Slapton road formed the basis of at least two student projects later in the week!

The best days were devoted to the issue of plastic pollution in our oceans and on our beaches, which involved a little bit of Blue Planet. After conducting surveys on local awareness of plastic pollution in Kingsbridge (where there was a public plastic art installation with milk bottles), we collected litter on Slapton beach. Sadly we found quite a bit, even though the beach is on the cleaner end of the spectrum! The crisp packets we found were all dated before 2003, which shows that we have a big problem – no degradation at all, listen carefully manufacturers and supermarkets! We recorded the data on what we found for the Marine Conservation Society, and then we tried to display the material creatively.

The point of this exercise was never to make beautiful, award-winning art (a point lost on some people on Twitter! #hatersgonnahate), but rather to show that science communication and research impact are key aspects of academia! Let’s face it, members of the public do not engage with environmental issues by reading academic papers. They do so by watching things like Blue Planet instead! Us academics aren’t always good at presenting our work for people who aren’t like us (which is most people!). Hopefully it will plant the seed that future research needs to be communicated well, perhaps through artwork (maybe spend more than 15 mins on it!).

Overall it was a great week! I got to enjoy some time off while students did their independent projects and found Dartmouth looking like the South of France! I’m sure they’ll be more reflections in the coming days, but for now I’m happy to be home, but also happy to have experienced #envslapton18.

In case you’re wondering about the title, one group had a long running fieldtrip joke related to Hot Fuzz. They used this as a group name ‘No Luck Catching Them Killers Then?’ – it was quite amusing to see the fieldtrip leader try to work out what this had to do with their plastic pollution project, but I was rather more up with pop culture than him!

New paper – a method for designing usable decision support systems!

Cite as Rose, D. C., Parker, C., Park, C. Fodey, J., Sutherland, W. J., and Dicks, L. V. 2018. Involving stakeholders in agricultural decision support systems: improving user-centred design, International Journal of Agricultural Management 6 (3-4): 80-89

***Do contact me if you are unable to access the paper ***

Our new paper is out today! The paper looks at how we can encourage designers of decision support systems to adopt user-centred design practices – in other words, focusing on the needs of the intended end user so that systems are relevant, usable, and credible. Our motivation for this work came from a study as part of Defra’s Sustainable Intensification Platform – this research found that many decision support systems in agriculture were simply never used in practice. What a waste of time and money!

We suggest a six-stage process for designing decision support systems and term this a ‘decision context assessment’. We argue that funders of projects should require researchers to produce a plan which pays careful attention to each of these six stages. Ultimately, this will improve the chances of the system being used in practice – if it isn’t actually used by the intended user community, then lots of funders’ money and scientific time is wasted! The six stages are:

Picture1

Common sense will tell you that in order to carry out each stage effectively, you need to involve (or at least think about) the user at every single stage!!!

“1) Who is the user? – identify a clear user, understand their workflows, and ask about their needs. Needs and workflows differ between audiences – you can’t assume that a one-size-fits-all approach will work! – result: tool designed for a clear audience!

2) Why should they want to use it? – scientifically, the system might be robust and impressive, but ask whether there is a need for it from a user perspective. Ask whether it is better than how decisions are currently made (you’ll need to ask users how they currently make decisions!). You need to prove that it adds value to the user, either financially, or it saves time, or it helps them meet compliance or market requirements etc. – result: tool has a unique selling point and is needed!

3) Can they use it? – test whether users are able to use it effectively, also find out whether users can practically use it in a given setting (e.g. is there internet access on-farm?) – result: tool works in the intended use setting (e.g. on a farm)

4) Is it easy to use? – related to point 3, however there is a distinction between merely being able to use it, and the ability to use it easily – ask about user design preferences and test it on actual end users rather than colleagues! – result: tool is easy to use, users actually want to use it. 

5) Is there a delivery plan? – ask how users will find out about the system. This might involve making use of existing trusted peer and adviser networks. – result: tool becomes well-known to its intended users. 

(6) What is the legacy? – if the tool needs to be consistently updated to maintain relevance, then consider how to do this once funding ends. Do you need to maintain a technical support helpline for users? – result: tool continues to work long after implementation.”

(Rose et al., 2018)

We hope that this paper fosters a greater interest in user-centred design methods, and ultimately to the design of usable systems which make a difference in policy and practice! If funders made sure that tool designers constantly reported against the criteria above, then tools would have a clear audience and unique selling point, work on the ground and be easy to use, as well as being well-known to users and continuing to work after implementation.

***Do contact me if you are unable to access the paper ***

 

 

Lecturing about sustainability

In the last week or so I’ve been able to do lots of lectures/talks on sustainability. For me, the opportunity to lecture and tell students about the latest research is one of the great attractions of being an academic! Below, I’m going to give you a flavour of what we covered this week, also covering a school outreach talk that I gave when UEA let me loose in Worcestershire.

Green growth versus de-growth

My first workshop of last week tried to get students to think about two possible ways of achieving sustainable development – a ‘Green Growth’ mindset, which seeks to work within current capitalist structures to protect the environment, and a ‘De-growth’ mindset, which would challenge the existing emphasis on economic indicators as a means of progress. After showing an innovative video produced by the fantastic ‘Comedy in the Classroom‘ team at UEA, I split the students into groups and asked them to list the pros and cons of both approaches, and if any changes needed to happen to adopt them successfully. Below are the flip charts that were produced (note here and elsewhere that the IP belongs to the first-year UEA BA Geography students, or later on also to the UEA Environmental Scientists, or BSc Geographers!).

IMG_3619Clear themes emerged – green growth was seen as realistic, acceptable, less radical (some thought not radical enough), but improvements were needed in technology, which needed to be affordable to all. De-growth was universally seen as radical, and perhaps even impossible to achieve. One student did write, however, that de-growth ‘would liberate us from the burden of pursuing material excess’ – now that’s a quote fit for a first-class exam mark! I was surprised that everyone thought that de-growth was radical (I don’t necessarily disagree) so we had a bit of discussion about social constructivism and that was that!

Playing politics

In our next class, I talked about the challenge of making evidence-based environmental policy within a messy policy-making process. This very much resonates with my own work. I thought the best way to expose students to the messy reality of policy-making was to make them play a computer simulation, helpfully built by the BBC. Each group of students had to follow different strategies – such as an environment-focused approach, and an economy-focused approach (similar in many ways to green growth v de-growth). The challenge was to balance money, food, water, and energy resources whilst adhering to climate change pledges. As can be seen by the photos below, some students fared better than others! Anyway, it was certainly a nice break from me talking!

Technologies of agricultural sustainability

I also got to do a week’s worth of lectures on technologies of agricultural sustainability. I told the students about the idea of sustainable intensification, and of the recently ended Defra project on the topic. The rest of the week was spent discussing how technologies could help, but also about how they could create problems of their own. We also discussed whether we could do other things like changing our diets and consuming less in order to reduce the pressure to grow more food! One interesting exercise in which Toby Townsend from the British Beet Research Organisation helped was to split students into small groups to discuss the ongoing controversy about neonics, which is an important chemical given to crops, but it may also be harming our pollinators. Each group was given an identity – either the public, media, farmers, scientists, policy-makers, or an environmental pressure group. Each was asked to discuss what the position of their group would be on the issue, what their motivations would be, and how you would seek to communicate evidence and information to that group. The flip charts below show that there were lots of good ideas, but it was really hard. One of the reasons was because groups cannot be thought of as homogenous – e.g. there are all kinds of publics, so a one-size-fits-all communication strategy was unlikely to work. I think the students got the message about why environmental policy-making can be so ‘wicked’ – because a variety of stakeholders are involved in decision-making, all with different interests, and all with different beliefs.

Malvern St. James

Finally I was dispatched to Great Malvern to visit Malvern St. James Girls School to speak to years 10-13 (some students from other schools were also present). I posed the question of whether population growth was the biggest threat to our environment. In one sense, yes, population growth is a challenge, but so are other things like over-exploitation, over-consumption, unhealthy diets etc. The audience were fantastic and I received some really interesting questions, some of which I didn’t really know the answer to! Hopefully that illustrated the oft-forgotten problem of thinking that someone is an expert outside of their main field of expertise! I also hope the lecture made them think and perhaps it will inspire famous Geographers and Environmental Scientists of the future (hopefully at UEA!). Thanks to all at the school for their welcome, for the bottle of wine – excellent branding on the water in reception by the way!

Aims for 2018

It’s always hard to imagine where you will be, and what you’ll be doing come the end of the year. It is possible, however, to start each year with a few academic goals that you are determined to achieve by New Year’s Eve! The trick here is to be optimistic, but not over-demanding. In other words, goals must be realistic in the time available, but they should also challenge you to work hard.

This year I have set a goal of achieving 10 first-author outputs, with an upper bound of 15 papers overall. Now this might seem rather enthusiastic since I’ve never achieved that many in a year before. But, the year thus far has been rather like how Hollywood movie releases sometimes go for actors, i.e. you film several films over a number of years, before finding that they are all released at the same time. Rather like waiting for a London bus for 15 minutes, before finding that three come along at once. I’ve had three papers out in January, all first author (one co-lead), two more are on the cusp of acceptance (one first author), and I’ve been asked to write two book chapters this year. With other papers in preparation and review, it therefore seems an achievable target.

IMG_3603

Second goal – perhaps this is somewhat meaningless, as many people think academic metrics aren’t particularly useful. It’s true, they don’t tell you everything, but I still keep an eye on the h-index now and again. At the moment it’s languishing at four, but at least the citation graph is showing a good trend! By the end of the year I want that closer to ten, which will involve more good-quality paper writing, and good dissemination of existing papers. With a h-index of nearer ten, and with 10+ papers written in the year, I’d be happy with my progress at PhD+4 in December 2018.

Screen Shot 2018-01-15 at 06.58.05.png

Thirdly, I’d love to gain a big research grant and/or a PhD student. I absolutely love teaching and I would be so excited to guide a PhD student optimistically through their studies. I am going to explore collaborative studentships and other opportunities, which would mean that I gain the funding required to attract an excellent student to do their PhD with me. Roll on 2018!