Every five years, the Portuguese Science and Technology Foundation (FCT) reviews its research institutes from Astronomy to Zoology. But this year, for the first time, the FCT contracted the European Science Foundation (ESF) for the job. FCT's resulting reallocation of funding will impact upon the career of more than 5,000 researchers – a third of the total in the country.
The methods of the review and revelations about the contract signed between the two foundations have mired the process in controversy. And the results might affect science funding policies across Europe.
It is not the first time that the ESF has been brought in for an evaluation. For instance, it was involved in a 2009 evaluation of the Bulgarian Academy of Sciences, which led to the decision to review all Bulgarian universities. According to Research Europe, following the evaluation, the academy felt it needed an "honest picture of where it stood in global research performance".
Miguel Seabra, president of FCT, gave a similar reason for bringing in ESF. He argues the official reason was that an experienced and independent assessment from ESF would help find research units that are productive at an internationally competitive level.
However, there is a case to be made that some times global competitiveness is not the best metric. Some countries, even in Europe, need to focus on capacity building which would lead to excellence in the long term. For instance, FCT's investment in the past 30 years brought the number of PhDs in Portugal closer to the European average. In that period, the level of productivity in research and science has grown exponentially. All this progress might be undone by the new review.
Two steps to disaster
The purpose of the ongoing review in Portugal is to identify the research units which meet these standards and redistribute funding until 2020. It is set up in two stages, the first of which is a knockout round. ESF set up seven review panels composed of international experts.
Results of this first phase, published in June this year, were devastating. About a fifth (22%) of all research units – institutes and university departments – were graded "fair" or "poor". These units will receive no funds until 2020. About a quarter (26%) of such units were graded "good", and they will get limited funding depending on their size. The remaining (52%) move to the second stage of the evaluation and stand to gain greater funding, based on money that will be taken away from the rest.
Even if the "good" units receive some funding, they will die out in the long run, because the allocated funds are insufficient to pay the most basic expenses. These units will also be disadvantaged in competition for other funds, because of this grade. That means, a total of 154 research units cannot continue their research, hire new people or train new graduate students until 2020. These units make up a third (5187) of the researchers. Naturally, this has caused much criticism among the research community.
According to the FCT and Nuno Crato, the minister for education and science, the money for research in Portugal has not decreased. This raises questions about the justification of the cuts to half the research units in the country.
Unusual review process
There are various objections to the first stage of the review. One major criticism is the lack of specialisation of the review panels. In contrast to previous evaluations for example, there are no longer separate panels for Materials Science, Physics, Mathematics or Chemistry, but only one "Exact Sciences Panel" consisting of 11 members. In comparison, the 2007 FCT review of research units had 25 different subject panels, some of which had more than 11 review members.
Nicolas Walter, who supervised the evaluation at the ESF, argues that specialist reviewers made up for the shortcoming: "While the seven panels each covered a broader spectrum of science, their members were assisted by two external referees for each application."
Many research units, however, complain about review panels lowering the grades given by external reviewers, according to a blog founded by a physics professor. There are cases of huge differences in how external referees and panel members give grades, with external experts approving a unit and panel members failing to take on the advice, or vice-versa. For instance, this happened to the Philosophy Centre of the University of Lisbon, where I work, and with the Centre for Applied Statistics at the University of Lisbon.
The Portuguese Physics Society (SPF) and the Portuguese Chemistry Society (SPQ) have released official statements of concern with the process and its effect on Portuguese science. The Portuguese Philosophy Society intends to make a similar statement, as its president, João Cardoso Rosas, confirmed.
Consequences beyond Portuguese science
The first stage of the review was officially aimed at weeding out underperforming research units. FCT's president Miguel Seabra denied that a 50% approval rate was recommended to the panel – "neither ESF nor FCT would have agreed to anything of the sort". Beyond allegations of bias and factual errors, it has now emerged, after FCT finally disclosed the contract signed with ESF, that there was a quota stipulated from the very beginning.
The redistribution in funding will cut resources from many research units based in public universities, and concentrate resources in fewer institutions (including private foundations). There is no good financial or scientific reason to take away funds from already productive centres. FCT's scientific councils, not involved in the ongoing review, publicly condemned the process, too.
The ESF's review conveniently agrees with the recent shift in research funding policy in Portugal, which previously resulted in drastic cuts to PhD and postdoctoral grants. Other European countries in the same situation as Portugal, particularly those in the eastern part, should pay attention if they are considering an ESF review.
Explore further: What lesson do rising retraction rates hold for peer review?