Developing Meaningful Metrics

Thursday 27th September


by Lily Troia, Engagement Manager at Altmetric LLP


The second day of 5AM kicked off with a session on Developing Meaningful Metrics moderated by Hans Zijlstra (Elsevier). The three presenters offered unique perspectives on how our community can encourage the development of more meaningful metrics and what potential insights can be uncovered to help inform better tools and services.

David Sommer (Kudos) began with a call to broaden the definition of altmetrics to include closed channel exchanges, like private digital forums, email shares, and in-person/ephemeral conversations. Sommer identified two axes of communication to categorize these interactions, emphasizing the value of data around how these audiences access this information.

Kudos conducted an online survey asking researchers to identify ways they share research materials and foster visibility. Their results revealed conferences was the most common exchange forum, noting many digitally visible forums were not being utilized as much as private, “under-the-radar” interactions.

Sommer explained Kudos’ “share labels” tool to help users categorize different information exchanges, and their “shareable pdf” feature for researchers to track communications on private channels, the latter of which is the platform’s most popular service. He then announced their newest development: the upcoming launch of Kudos for Research Groups, aimed at maximizing efficiency and efficacy of research communications.

Next up Kevin Boyack (SciTech Strategies) shared his perspectives as a bibliometrician new to the field of altmetrics, focusing on aligning metrics with the outcomes we need measured. His organization creates visualizations around specific research communities, and found most altmetrics publications tend to fall into the research evaluation or social media cluster.

Cluster maps for non-profit organizations in the United States center around altruistic goals: impacting culture, community, citizenship, etc. Boyack suggested the need for a new topic/research community cluster based on these societal goals, and how to measure these desired outcomes.

He pointed out the difficulty in developing metrics around broader impact -- citing blog posts, news/media, and clinical and policy posts as having the best potential in this realm. Measurable outcomes might include indicators like efficiency gains, mortality rates, crop yields, and literacy.

Currently metrics are often driven by available data, versus developed in response to our desired research outcomes.

Boyack closed by mentioning a new project with the NIH working to develop “virtuous” indicators that might address things like replicability, translation potential, and data availability.

Philippe Mongeon (CWTS - Leiden) closed the session with his in depth study characterizing social media events, examining twitter engagement around approximately 3500 library and information research articles published over the past five years. Previous research in this sphere emphasizes the heterogeneity of altmetric data points: diverse forums for sharing carry diverse nuance and meaning. Philippe’s research extends this concept to the relationships amongst users and documents, proposing a new dimension of heterogeneity for analysis.

He pointed out the wide variance in types of online engagements with research outputs -- acts related to access versus appraisal, for example. Liking a tweet is not identical to commenting upon the same, and similar acts can function in different ways, even within the same community. The relationship between the sharing agent and the publication itself holds meaningful, valuable insights.

Philippe used Twitter data, Web of Science author information, and CTWS’s custom author-disambiguation algorithm to express these relationships in terms of social or topical distance. His results identified several types of agent/publication relationships:

  1. Agent is topically related to content but not connected to its creator
  2. Agent is related to the content-creator, but publication is outside of the scope of agent’s topical area
  3. Agent does not relate to content creator or topic
  4. Agent is the content creator

He suggested by examining social and topical distance patterns we can start to uncover the diversity of otherwise seemingly-homogeneous phenomena. Philippe posed the heterogeneity of research sharing as a reality to be better understood, versus a problem to be solved -- and that hopefully his approach and methodology can serve as a new tool available to our community as we work towards developing meaningful metrics.