Oct 22, 2024 | Scholarly publishing
Towards the end of September, Peer Review Week hosted a whole host of content around the subject of peer review, its processes, and editorial policies, as well as highlighting updates and current threats to the system that underpins academic publishing.
Editage, in partnership with EASE, organised a discussion entitled “Envisioning a Hybrid Model of Peer Review: Integrating AI with reviewers, publishers, & authors”, led by Chris Leonard (Director of Strategy and Innovation, Cactus Communications). He was joined by Serge P.J.M. Horbach (Institute for Science in Society, Radboud University, Netherlands), Hasseb Irfanullah (independent consultant on environment, climate change, and research systems) and Marie E. McVeigh (Lead, Peer Review Operations and Publication Integrity, Mary Ann Liebert Inc).
The discussion started with highlighting problems with peer review today; noting that we are engaging with a mid-20th Century system of “gatekeeping” which has developed into one setting standards for research, a space to develop work and ideas, where communities can collaborate and a place to discuss what it means to create “good” standards. A major issue with the peer-review process, as highlighted by Serge, is finding quality reviewers – the number of invitations required has increased due to subject specificity and interdisciplinary fields overlapping. The volume and vast number of articles exacerbates this issue and small communities of interest can no longer support growing areas of academic research alone. This can lead to exclusivity and the network of reviewers within it is diminishing.
Hasseb raised the question of whether peer review has become overrated? If a manuscript’s decision can be made on the outcome of two reviewer reports – does this undermine the whole of the research? As peer review is not valued necessarily by the publisher in a financial sense, the value of its contribution is lost. However, it is important to understand that the communication around the research does not end with the peer-review process, it starts upon publication.
As the discussion progressed, the focus turned towards a hybrid approach to peer review and what that means. A hybrid approach could equate to a generative AI and human reviewer contributing towards a reviewer report, while the Journal Editor provides a final commentary. This assistance would provide “free labour” and generative AI is a good bibliographical research tool. Marie posed the suggestion that in cases such as this, it is ideal to allow machines to do what machines can do and allow humans to engage with the outcomes. For example, AI would be great at screening manuscripts, analysing citations and identifying peer groups. Such routine and rule-based tasks can be done quickly and efficiently. Evaluations can then be conducted by humans, who are able to decipher and assess whether the article adds to the scholarly record. As human reviewers cannot be located quickly enough, this dual aspect might be the quickest, cost-efficient way to support peer-review processes going forwards.
As journals look to lean on AI technologies, we need to understand what a journal is and what it does. Is it still simply a facet to share and disseminate work and ideas, or is it becoming more than that – a place where communities engage and develop their insights? By involving AI, do we consider it a peer? If community is at the core of journal publishing, surely humans will be required to keep that sense of togetherness ongoing. Without it, it’s just computers talking to each other.
May 9, 2024 | Scholarly publishing
The “text recycling in research writing” COPE Lightning Talk in April 2024 was presented by Professor Cary Moskovitz (Director of the Text Recycling Research Project at Duke University, NC, USA).
Text recycling can be defined a number of ways. Essentially it is the reuse of material – whether it is prose, visuals or equations – in a new document where material is used in the new document as it was in the original source. The material is not presented in the new document as a quotation and usually one author on the new document is the same as the original.
Recycling material in published papers is common, however authors do need to be mindful of the copyright law that they are submitting their paper under. Under US copyright law, authors may find that generally they can reuse portions of text under the Fair Use clause, and in many STEM journals publishers ensure their policies work to accommodate this, so as not to infringe the copyright/legal ownership.
The ethics of recycling are different and it requires appropriate transparency. Legalities and ethics may differ and the author must ensure that permission has been requested if their use of copyrighted material falls outside of the Fair Use guidance and journal policies. Text recycling should be done within best practice – we cannot expect authors and editors to be experts in copyright law, however publishers’ policies should adapt to and support their communities. It is also important to highlight that authors should check their publisher agreements as this is where the details lie. These will vary from publisher to publisher and the rules around text recycling are not always consistent.
It is recommended to authors that if they are using work which is previously published, a statement could be included in the paper to highlight this, and this would act in accordance with best practice. This will allow the reader to know that the work is not original, but it will also signpost accordingly.
Apr 2, 2024 | Scholarly publishing
The March 2024 COPE Forum began with a discussion on ‘ethical considerations around watchlists’, hosted by Daniel Kulp (Chair, COPE Council).
During the discussion, the benefits and potential risks associated with watchlists were explored from the position of both individuals and journals, as well as how publishers should use them. In principle, a watchlist could be used as a quality control tool; usually kept private as a measure of accountability. If, however, the list was exposed/leaked, the publisher’s trustworthiness and maker of transparency can be held to account.
It is important to note what is included on a watchlist – is information recorded about an individual, including personal data? Do COPE offer any guidance on what kind of criteria individuals should meet to be on a watchlist? And how can an individual defend themselves or “remove” themselves from a list? These were some of the key questions raised during the conversation regarding individual authors or account holders. The threat is much greater to be named and listed; as careers, research programs and scholarly publications can all be potentially be disputed/disrupted if this extremely sensitive information is exposed in the public domain.
At an individual level, abhorrent practices to circumnavigate peer-review processes could be observed through “flagging” an author’s account – for example, to collate information about suspicious activity. However individuals have a higher right to privacy and such lists need to be regulated or kept private within the publishing house. Academic publishing is witnessing a systematic effort to undermine peer review; the rise of papermills, reviewer cartels and bad actors exploiting the system often for financial gain mean that individuals should be aware of this and make due considerations during submission.
Sharing nefarious behaviours is important and if publishers share systematic problems regarding behaviour amongst one another, they can collectively disrupt the ways papermills and peer-review rings operate. COPE has guidance on how EICs can share information. However, how can we share this information more broadly? The STM Integrity Hub offers a duplicate submissions check across publishers, which could be a useful tool to manage misconduct. This is being developed and the STM Integrity Hub are exploring managing this in an ethical and legal way.
One of the biggest risks with watchlists posed at an individual level is that account holders are added incorrectly or without proper reason. This could be very detrimental to an individual in terms of their career and may also possibly be defamatory and could have legal ramifications. Watchlists must aim to be transparent but not give away details of what we are looking for. The risk is higher for individuals as it can directly impact future employment and ability to do research. Should there be mitigation circumstances for appeal for minor infractions? A similar name could be a risk, for example.
In the absence of an agreed decision amongst the industry on what should and shouldn’t be on a watchlist, the main risk appears to be a legal one. STM have already advised that the industry needs to be careful regarding the type of data which is shared. Legally underpinning this stage is crucial to ensure an individual’s privacy and GDPR-compliance is maintained. It was suggested that if private watchlists are maintained, legal counsel should be able to advise how to evoke sanctions or prevent someone from publishing. The publisher however must be mindful of any reputational damage or litigation brought against them.
If an individual finds themselves on a watchlist they do of course have the right to reply, to ensure that the process is fair and transparent and that they can defend themselves and their position. It was raised that it would be good practice to notify an individual that concerns have been identified, which will allow them to ask for an appeal. It is important to consider and understand what the watchlist represents.
As the risks increase and the systematic manipulation of peer review is attacked, the ways in which this data is captured in order to identify patterns of misconduct has to be carried out quickly, ethically and legally in order for the publisher to maintain integrity and be commercially viable as well as managing its own trustworthiness.
Mar 19, 2024 | Events, Scholarly publishing
On Tuesday 12th March we headed off to one of the industry’s leading events in academic publishing. This year the London Book Fair was buzzing; the main hall was filling up as soon as the doors opened, and the Tech Theatre and Main Stages had consistent queues. Meetings, networking and seminars are all key elements of the fair and this year we were lucky enough to participate and find out more about the challenges facing the industry.
The hot topic as ever was AI, and we attended a talk hosted by Sureshkumar Parandhaman (AVP of Publishing Solutions and Pre-Sales, Integra) entitled “Embracing AI in Publishing: Transform Editorial Excellence and Enhance Downstream Efficiency”. Needless to say it’s evident that tools are being developed to tackle many of the administrative tasks that editorial services provide, however we know there will still be an important role for us to play as integrity, quality and efficiency are all integral within our roles.
During a key discussion about AI and copyright it was stated that when King Charles’ recent surgery was announced, over 250 AI-generated texts filled the virtual Amazon bookshelves the very next day! Content is quick to make, and the session “Global Discussion of Machines, Humans, and the Law” hosted by Glenn Rollans (President and Publisher, Brush Education Inc.), Nicola Solomon (CEO, Society of Authors), Porter Anderson (Editor-in-Chief, Publishing Perspectives), Dan Conway (CEO, The Publishers Association UK) and Maria A. Pallante (President and CEO, Association of American Publishers) highlighted how important it is for the copyright holder to be accredited and financially rewarded. Whilst Government bodies manage policy and next steps with regards to AI, we need to be mindful of what we “feed” to LLMs and how “Fair Use” comes into play.
The Editorial Hub at LBF 2024
A panel discussion entitled “AI & Publishing – Navigating the Impact of Large Language Models” was hosted by Lucy McCarraher (Founder, Rethink Press Limited), Nadim Sadek (Founder & CEO, Shimmr AI Ltd), and Sara Lloyd (Group Communications Director & Global AI Lead, Pan Macmillan). The group discussed the ways that LLMs can spark creative thinking and should be used within workshops as a tool to support creativity. In terms of optimising the features of AI it was recommended that the industry looks to use it for creating the structural analysis needed for promotion work. An AI bot can extract book DNA far quicker than a pair of human eyes, so let’s use its skillset appropriately and it can enhance our work, rather than hinder progress. As well as creating copy, it can also be useful for generating advertising and images. As well as connecting couriers and customers, it can scroll databases to highlight potential consumers. The panel really encouraged the audience to engage with the features of AI bots and embrace how useful it can be to enhance and build on capabilities.
Feb 6, 2024 | Scholarly publishing
The ALPSP panel for this discussion on AI and the impact it is having within academic publishing was made up of Nicola Davies, IOP Publishing (Co-chair), Helene Stewart, Clarivate (Co-Chair), Meurig Gallagher, University of Birmingham (Speaker), Matt Hodgkinson, UKRIO (Speaker) and Jennifer Wright, CUP RI Manager (Speaker).
It was fascinating to see how the conversation around AI has moved on within a few months regarding this technological advancement. Institutions, publishers and journal stakeholders all have a concept of AI and are developing policies and guidance about how we should be using it and are underpinning the “what for?”.
Many of us by now will have tried asking a Large Language Model (LLM) to write a paragraph or create an image using Artificial Intelligence. It’s brilliant to watch how quickly tasks can be created, large blocks of text are generated at an immense speed, and right before us we see how quickly human intelligence can be mimicked. This notion was highlighted in Meurig Gallagher’s presentation; that essentially AI is trying to act as human as possible using the instructions it has been provided with. However, when these tools are posed with mathematical equations it does not have the knowledge to apply the learning and therefore can spectacularly fail! These “gaps” therefore build into the guidance stakeholders need to be aware of when creating policy around AI – it cannot be relied upon solely to do the work. Matt Hodgkinson developed this further and shared many caveats that researchers and general users might come up against when using chatbots:
- Many LLMs are unvalidated for scholarly uses.
- References should be fact-checked as they can be falsified, therefore it is important to check sources and supporting literature.
- The quality of evidence is not assured.
- Outputs may be based on using out-of-date information based on “old” training material.
The ominous but noteworthy warning was circulated that “if you are not an expert, you will be fooled by fluent but incorrect outputs”. Therefore, all of us involved in scholarly publishing need to be mindful of these contributions and check author statements within articles to assess whether an LLM has been used. Of course, one of the largest threats we are witnessing is the output of paper mills and their use of AI could lead to the tool’s collapse as its knowledge bank is infiltrated with “fake” data, which if left undetected will pollute the pool where the data is extracted from.
Nonetheless the principles of Research Integrity can be applied to the use of AI-generated content and Matt shared this slide to disclose how these principles are applied:
UKRIO presentation at ALPSP 2024
Dr Jennifer Wright from Cambridge University Press shared with the audience how to implement transparency which is really the crux of what many VEOs are looking at. It was suggested that AI declarations should be included within image captions, acknowledgements, and methodologies if applicable and the details that should be shared include the type of model that was used, eg: CHATGPT, and how and when it was accessed. It is also important to include any additional COI statements because of the use of the model. Looking forwards, Dr Wright elaborated on future considerations and posed some important questions around reporting standards: What will the impact be of AI on the scholarly record? How could/should/will research and publication practices change? How will concepts such as retractions be enforced? Can a bot retrain itself?
The challenges are still clearly evident with AI. However the more we progress and understand how it can be used, trust markers can be identified to validate the outputs. As long as scholars use and do not abuse the tech, we could watch something incredible unfold!
Dec 2, 2023 | Scholarly publishing
AI is inevitably going to infiltrate all our lives in some way or another in the near future; learning how we shop, communicate, write, create and plan our lives. We therefore also need to look at adapting our ways of working to benefit from these technological advancements. Working alongside this adaptive new tech, and generating new guidance and principles, will enable us to harness and nurture it. We can create preventative methods to stop bad actors abusing and infiltrating the systems we have in place to educate and teach.
At the NEC Annual Publishing Conference (7 November 2023, London), the keynote was delivered by David Smith from the IET who looked “Back to the Future!”; highlighting the importance of an article published by Darcy DiNucci. Fragmented Futures (published in print, 53.4, 1999) demonstrates technological growth and that the Web she wrote about was only the beginning…how things have changed in 20 years! Essentially it is thought we are at a similar point with AI; it is new, raw and ready to be refined and developed.
Leslie Lansmann, Global Permissions Manager from Springer Nature, discussed how Large Language Models (LLM) such as ChatGPT are ingesting content, and this is not yet fully disclosed by AI companies. This is important to monitor as we must maintain the stewardship for the content and protect copyright and protected manuscripts. As much as AI is currently learning – it probes and reiterates content – it does not understand the deeper context behind the language. The publishing industry is however having to react to the developments in technologies, many publishers are imposing bans on AI content, and introducing new and different policies.
The discussion around authorship is constantly developing and debated – should research be done using AI? Can it help an author whose first language isn’t English produce a more succinct piece of work? If the data is accurate and the same research principles are adhered to, maybe we should move towards incorporating it into our practices. This notion was delivered by Anastasia Toynbee from the Royal Society of Chemistry who was looking at the problem of non-native English speakers and how these tools could help. The key feature of this was that a problem had been highlighted and AI was being used to support it – not the other way around.
It became clear from all the speakers how important it is to identify the problem initially and use AI tech/tools to help with it, rather than decide how to harness and squeeze new systems into processes that are working well. Ian Mulvany at BMJ really brought home this idea that we as an industry need to balance risk vs opportunity. AI has perception, however no intention to act; therefore we are in a position through governance, policy, and stewardship that we can lead AI to improve processes and not be reactive and in fear of the unknown! Andy Halliday , Product Manager at F100 iterated the benefits and pitfalls of AI and how humans can help harness this tech and enable it to support our ecosystem and develop a sense of AI preparedness.
We are in the awakening of AI. The box has been opened and we all have access to create new and exciting content, images and access information much easier than ever before. As the discussions continue it will be really exciting to see how developments are made, what fixes it can be used for, and how policy and guidance are updated to meet the demands of users.
