Measuring Impact

The number of scientific publications and citations have traditionally been the primary indicators to evaluate research and its impact. But measuring research and its impact is more challenging than that; you can’t meaningfully compare two disciplines with significantly different publishing and citing patterns. Furthermore, indicators to measure impact beyond the scientific are very limited.

The difficulty of measuring the impact of research projects does not only derive from the lack of indicators, it also lies in the complex nature of impact and research itself: impact happens over a long period of time and is not a linear process - this is why it is often impossible to say which results lead to which changes. Research groups often work across borders in multinational teams, which basically makes it impossible to say whether a certain change (=impact) should be attributed to research or to other activities.

Impact happens over a long period of time and is not a linear process - this is why it is often impossible to say which results lead to which changes.

Since the 2000s science is seen more as an interactive process, where impact is a result of collaborative efforts between science and society. This also reflects the current thinking of universities third task, to contribute to the society and economy. Research funders are also expected to present figures that depict the return on their investment (=research funding) to governments (and taxpayers). Thus, they have each come up with their own sets of indicators that should be taken into consideration when writing the funding application. These indicators help you understand what the funder expects you to deliver in terms of results, outcomes, and impact. Since measuring the actual impact is, as stated earlier, at least very challenging, funders usually want you to list project results that can lead to impact during or after the project lifetime. For example, to verify the scientific impact of your research, you need to list results such as publications, degrees, research data, methods, tools, and software as well as research mobility derived from the funding in the final report. To show that your research has also societal and possibly even economic impact, you should list events and openly available documentation targeted at non-scientific audience, interactions with stakeholders and possible IPRs. Obviously, all this requires that you collect the relevant data during your project. That is why it is highly recommended that you plan how you will verify your impact.

When you are writing the impact section of your application, you should be aware of the funder's requirements regarding various types of impact. If you are writing an application for the European Research Council (ERC), the sole funding criteria is scientific excellence, thus there is no need to plan how you will verify the societal impact of your research project. Should you be working on an application for the Horizon Europe programme, the expected outcomes per project are listed in the call text (topic) and you should make sure that you are covering as many of them as possible in your proposal and provide a credible narrative, how the wider impacts, set out in the work programme and in the strategic plan, are reached through your project results.

Key Performance Indicators (KPIs) are a good way to show your objectives regarding various outcomes and impacts and set clear measurable targets. They also showcase the scale and significance of your project, a required element of Horizon Europe Pillar II proposals. You can use KPIs in multiple ways. The starting point is to formulate your research objectives to be SMART: Specific, Measurable, Achievable, Relevant, and Time-Bound. ​This allows you to set meaningful KPIs to showcase your project’s progress and success.​ KPIs can also outline your project’s scale and significance.​

When setting up your KPIs think about the following:

• Objectives: What are you aiming to achieve?
• Indicators: What can be measured in numbers and seen as the direct outcome of your project when successful?
• Stakeholders: Who will benefit from your measures? How many stakeholders will you reach with your dissemination, exploitation, and communication (DEC) activities?
• Scale & significance: How many end-users will uptake your results? What could be the change to the current situation in 10 years?

Choose a couple of well-defined Key Performance Indicators (KPIs) for your project.​ As a rule of thumb, every time you say that your project results will improve, reduce, increase, etc., you should be able to make an educated guess by how much and when. For example, if your project aims to lower the climate impact of electronics manufacturing through improved waste management, tell the reviewers what the situation is now, and how could it look like in 10 years. Use statistics and strategy papers relevant to your funding call. One good starting point is to look at the EU strategies or the UN Sustainable development goals and relevant targets and indicators.

You can do different tables for the project's success, impact, and outreach activities.

It is often necessary to demonstrate the impact of your research in numbers. There are unlimited quantitative indicators that can be used, the difficulty is to choose the right ones for your research project.

Useful quantitative indicators include:

• number of IPRs, patents, spin offs, new products, new jobs
• uptake of results by x number of end users
• savings in production costs, energy, time etc...
• number of seminars/workshop organised
• number of appearances in social media/press
• number of stakeholder contacts
• number of courses/seminars organized based on research results
• participation in committees, councils and working groups (incl. in companies), parliamentary hearings.

Traditionally research impact has been measured by bibliometric indicators such as:

• number of publications
• quality of publications, i.e. Impact Factor value
• visibility of publications, i.e. the number of publications in citation databases (Scopus and Web of Science)
• impact of publications, i.e. the number of citations they have received
• researchers’ cooperation network, i.e. the number of co-authors and affiliated organizations in publications.

Altmetrics is an interesting addition to the traditional bibliometrics. Altmetrics measure the impact of publications (e.g. articles, books) in social media and other social networking services by tracking for example the number of downloads, mentions on news sites, tweets, likes and traditional citations. Altmetrics help assess the impact of publications quicker than traditional citation-based metrics. Tweets and mentions on news sites may accumulate very quickly after publication, whereas for citations this may take months, even years.

UN SDG Indicators

As the connection between research and the United Nation Sustainable development Goals (UN SDGs) becomes more and more important, it might be useful to take a look at the indicators the UN has set to measure the achievement of each goal and their targets. For example:

• Goal number 3 Ensure healthy lives and promote well-being for all at all ages
  • Target: By 2030, reduce the global maternal mortality ratio to less than 70 per 100,000 live births,
    • Indicator (one of many): maternal mortality ratio.
• Goal number 12 Responsible consumption and production
  • Target: By 2030, ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature
    • Indicator (one of many): Extent to which (i) global citizenship education and (ii) education for sustainable development (including climate change education) are mainstreamed in (a) national education policies; (b) curricula; (c) teacher education; and (d) student assessment.

If you can find the link between your research and the UN SDGs, why not check whether the indicators that measure the advancement of the targets could be used to measure also your research impact.

What kind of qualitative indicators can we use to measure research impact? The most powerful tool to describe your impact is often a narrative (impact story), enhanced with quantitative indicators.

Impact story

When measuring societal or other impacts of research, qualitative indicators can also include interviews, observations and corroborations of third parties that express the perceptions of the changes that have occurred during and after a research project. Often the most effective qualitative tool to verify your research impact is a narrative (case study/impact story) describing the effect your research has had on the society, economy or culture etc.. It allows you to critically assess how you succeeded in fulfilling the aims you had regarding research impact. Recently impact stories have been used also in research evaluation, for example, in the UK as part of the Research Excellent Framework 2014. Of course, stories can and should, when possible, be supported with data and quantitative indicators. Impact stories very often present the results in favorable light, nevertheless they do give valuable insight about the results of the research. An effective impact story includes:

• A short summary of the research and the impact
• Description of the conducted research (what was done, where and by whom)
• Description of the impact (What happened and when? What and where was the change and why and to whom did it matter?)
• Description of the process that lead to impact (independent, verifiable evidence. ) Add quantitative indicators to strengthen your case. (Source: LSE Impact Story guidance).

The Strategic Research Council in Finland uses impact stories in reporting the results of the SRC funded projects. They have published a very comprehensive template for impact stories here. Impact stories are also a great tool to communicate your research to external stakeholders and society at large. Check out for example University of Oulu's project Bright Clouds Dark Clouds (funded by SRC) stories on their website: BCDC impact narratives.