Evaluation tools

Tools such as Google analytics have existed for many years, and continue to provide metrics for websites and the field of altmetrics or measuring the online activity surrounding research articles (or article level metrics), continues to develop. There is also now a range of tools available to enable a researcher to evaluate the impact of their social media presence.

These tools can measure qualitative or quantitative data or a combination of both. Some tools are free, some are paid for, and others have both free and paid for options.

The altmetrics manifesto, developed by Jason Priem, Dario Taraborelli, Paul Groth  and Cameron Neylon is a good starting point to learn more about the field. Examples of dashboard type services that have been developed to measure this overall impact (both traditional impact and altmetrics) include Altmetric, Impact Story, Plum Analytics and Science Card.

Many journal publishers now collect article level metrics (for example PLOS – new data sources added to article level metrics), and make it easy to share details about a paper on social media services such as Twitter, Facebook or Linkedin. Social bookmarking, citation and reference management tools such as Mendeley, Cite-U-Like, Zotero and Papers, as well as social networking tools such as Twitter, Academia and more enable you to share papers with other people. It is also easy to count within social media tools such as retweets (RT) and favorites in Twitter or likes and shares in Facebook. Blog dashboards and comments have also been a quick way to identify which posts are most popular and generating discussion.

There are also tools available that attempt to connect all of your social media accounts and give you a score to measure your influence, such as Chartbeat, Crowdbooster, Klout, Peer Index, Kred and socialmention.

However do remember that none of these tools are perfect and should not be used in isolation.

things to think about BEFOre you measure

  • What do I want to measure?
  • Why am I measuring?
  • Which tools am I using?
  • How and what is the tool / service measuring?

Altmetrics everywhere but what are we missing? from Dr Alan Cann (University of Leicester) and Understanding the limits of altmetrics slideshare statistics from Brian Kelly, (UKOLN) are two blog posts that raise questions about the usefulness of these evaluation tools.

tools available

Tool Paid for / free Description
Chartbeat Paid for (30 day free trial) Visits, device used to access, links – direct, search etc., Tweets, Facebook likes, page shares, how long people spend on your site
Crowdbooster Three different levels of paid account Real time Twitter and Facebook analytics, down to granular detail e.g. who shared your page/site/info. Download data, schedule tweets
Klout Free Connects your various social media accounts to calculate your overall influence (in a score out of 100)
Peer Index Free Measures social interactions and gives you a score. Can add Twitter, Facebook, LinkedIn, Quora and your website
socialmention Free Aggregates a variety of social media account information into one stream
Topsy Free trial then paid for Range of products. Pro and free analytics – analysing on topic, term and/or hashtag; data and social search
Altmetric Can embed for free and two products available at a cost Article level metrics – aggregates mentions of a research paper on Twitter, Facebook, Mendeley and more and get given a score
Impact Story Can register 1000 items for free Not only measures traditional impact but also social media impact
Plum Analytics Paid for (though can get access to PlumX – beta launch) Captures article level metrics
Science Card Free Collects article level metrics for research papers
Reddit Free Add story to reddit and people vote on whether they are good or not – this generates a Reddit score
Kred Free Measures your influence from Twitter and Facebook
t-index Free Measure influence on Twitter
followerwonk Basic account free. Paid for account gives more options. 30 day trial available Twitter Analytics: Find, Analyze, and Optimize for Growth
SocialBro Free trial version and different levels of paid for version Find out about your followers, tools to help you use Twitter more effectively (e.g. best time to Tweet) from other Twitter accounts, Twitter lists, text files, Twitter search
Blog statistics Free WordPress dashboard
Google Analytics Free Measuring visits, source of traffic, popular pages of your website

Current projects happening in higher education relating to evaluating online engagement

Online engagement is becoming more important, with some funders requiring applicants for publicly funded research grants to demonstrate how they will communicate research results with members of the public.

The Jisc has made funding available to get research groups to consider online engagement via the Jisc Impact Analysis: Facilitation and Synthesis Project. One of these projects is Public Engagement with Research Online at the University of Warwick.


Sign up to two of the tools mentioned above. What do you think about what they are measuring? Would they be useful?

References / Useful links

The creation of this module was inspired by sessions at SpotOn London 2012 including Assessing Social Media Impact, Can we work better together to evaluate online engagement and Altmetrics: Beyond the numbers.

Fenner, M. (2012) Altmetrics: first we need the for what? and only then the how? OK? Gobbledegook Blog [Online]. Available from: http://blogs.plos.org/mfenner/2012/11/09/altmetrics-first-we-need-the-for-what-and-only-then-the-how-ok/

Oxford Internet Institute (nd) Toolkit for the Impact of Digitised Scholarly Resources [Online]. Available from: http://microsites.oii.ox.ac.uk/tidsr/

Priem, J., Piwowar, H.A. &  Hemminger, B.M. (2012) Altmetrics in the wild: using social media to explore scholarly impact. Arxiv [Online]. Available from: http://arxiv.org/abs/1203.4745v1

Tait, E. & Holden, J. (2012) Tracking digital impact: The challenge of evidencing impact. LSE Impact of Social Sciences Blog [Online]. Available from: http://blogs.lse.ac.uk/impactofsocialsciences/2012/07/19/tracking-digital-impact-the-challenge-of-evidencing-impact/


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s