2:AM Amsterdam: Setting the standard
This is a guest post from Adam Dinsmore, a member of the Wellcome Trust’s Strategy Division. Adam describes the importance of rigorous data standards and infrastructure to funders who wish to use altmetrics to better understand their portfolios, and looks ahead to the standards session at next month’s event.
It was a moment of some personal and professional pride last September when the Wellcome Trust played host to the inaugural altmetrics meeting (1:AM London). As a large funder of biomedical research the Trust is always keen to better understand the attention received by the outputs of the work that it supports, and over the two days delegates were given much cause to consider the potential of altmetrics to help us gather intelligence on the dissemination of scholarly works.
Among the biggest developments in the UK’s metrics debate since 1:AM was the publication of The Metric Tide; a three volume report detailing the findings of the Higher Education Funding Council for England’s (HEFCE) Independent Review of the Role of Metrics in Research Assessment and Management (or IROTROMIRAAM for short). The review, commissioned by then Minister of State for Universities and Science David Willets in Spring 2014, sought to bring together thinking on the use of metrics in higher education from across the UK’s researchscape. A call for evidence launched in June 2014 attracted 153 responses from funders, HEIs, metric providers, publishers, librarians, and individual academics.
Attendees at last year’s 1:AM event heard an update on the review’s progress from the report’s eventual lead author James Wilsdon (viewable at our Youtube channel) who described the group’s aims to consider whether metrics might support a research environment which encourages excellence, and crucially how their improper use might promote inefficient research practises and hierarchies.
The full report expounds further, crystallising more than a year of thoughtful consultation into an evidence base from which several important recommendations proceed. Among them is a call for greater interoperability between the systems used to document the progression of research – from funding application to scholarly inquiry to publication and re-use – and the development of appropriate identifiers, standards, and semantics to minimise any resulting friction. Fortunately for those with a vested interest in an efficient research ecosystem (i.e. everyone) some very clever people are working to make these systems a reality.
In 2 weeks the second annual altmetrics meeting (2:AM Amsterdam) – which this year is being hosted at the Amsterdam Science Park – will open with a session on Standards in Altmetrics, featuring a presentation from Geoff Bilder on a newly announced CrossRef service potentially able to track activity surrounding research works from any web source. First piloted in Spring 2014, the DOI Event Tracker will capture online interactions with any scholarly work for which a DOI can be generated (articles, datasets, code) such as bookmarks, comments, social shares, and citations, and store these data in a centralised clearing house accessible to anyone. Critically, CrossRef have stated that all of the resultant data will be transparent and auditable, and made openly available for free via a CC-0 “no rights reserved” license. The service is currently stated for launch in 2016.
The session will also feature an update on the National Information Standards Organization’s (NISO) Alternative Assessment Metrics (Altmetrics) Initiative. Since 2013 NISO has been exploring ways to build trust in metrics by establishing precise, universal vocabularies around altmetrics to ensure that the data produced by them mean the same things to all who use them. In 2015 NISO has convened three working groups tasked with the development of specific definitions of altmetrics, calculation methodologies for specific output types, and strategies to improve the quality of the data made available by altmetric providers.
The continuing work of these groups speaks to the challenges inherent in establishing consistent, transparent data provision across the altmetric landscape. Zohreh Zahedi of CWTS-Leiden University will present the findings of a study of data collection consistency among three altmetrics providers, namely Altmetric.com, Mendeley, and Lagotto. The study examined data provided by these vendors for a random sample of 30,000 academic articles, finding several discrepancies both in terms of coverage of sources like Twitter, CiteULike, and Reddit and the scores derived from them. These findings provide an important indication that the use of altmetric data remains laden with caveats regarding the context in which they were derived and exported.
It is heartening that real attention is being paid to the issues of interoperability and consistency often raised by funders, publishers, and HEIs, and drawn together by the Metric Tide report. The presentations from CrossRef, NISO, and the CWTS-Leiden group are bound to stimulate much thought and discussion, which will then be built upon in a standards-themed workshop session later in the day. These discussions portend a time when a rigorous data infrastructure allows altmetrics to approach their hitherto unrealised potential. I look forward to hearing about it in Amsterdam!
 Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363