DocuBricks

Open Impact: A Toolkit for Measuring Impact for Open Source Hardware Projects

Hi there! We are a handful of attendees of GOSH 2016 at CERN who wanted to make an attempt at putting together a set of guidelines for measuring the impact of open source hardware projects. We felt that there is, as yet, no existing model for measuring impact specifically catered to the open source hardware community, and so we set out to curate some key metrics and tools that project managers, developers, grant writers, scholars, etc. could refer to.

We realized that each project has its own way of defining impact, which makes total sense! Each project comes with its own goals and aims, targeting different communities, funded by different funders (or some none at all).

The way that each project team tells its story varies widely. So at the conference, we asked around as many people as we could to get their feedback on the matter. We held two separate sessions about the topic and also spoke to people informally about their thoughts.

So this is a tiny toolkit we created based on what we found. By no means is it complete, so we would love your feedback on it too, which you can provide by e-mailing docubricks (docubricks at gmail dot com) or Lili Bui (lilybui at mit dot edu).

What is impact? A working definition.

First of all, what do we mean by 'impact' and why bother to measure it at all? It's tricky to give ‘impact' one definition, but many of us came to the consensus that it means to have changed the conversation in some way.

Why measure impact?

Measuring impact is allows you ...

To report on progress or potential to secure grants and funding for projects
To document history of projects in a social context
To construct stories about your project that you can pass on to others
To document methods, approaches, attitudes

Who needs to measure impact?

There are many stakeholders that have a need for measuring impact in some way. Here are a few we came up with. We're sure you can think of others.

Universities and institutions
Communities, specific publics
Press
Policymakers
Designers/builders
Investors, funders
Peers
Ourselves

How exactly do you measure impact?

That's the tricky part. There is no one exact way to do it. Like we mentioned before, everyone measure impact differently. Each project comes with different goals, audiences, stakes, etc. And what may count as impact in one project might not apply to the next. Here are some metrics that we came up with (both quantitative and qualitative) to capture impact. We don't recommend trying to use them all; rather, we recommend trying to identify which factors are the most important and relevant ones to you.

Quantitative

Nr. downloads of code, materials, tutorials
Nr. visitors on your website
Nr. participants or volunteers in your project
Nr. publications produced by your project
Nr. times publications have been cited
Nr. community members
Nr. success stories
Nr. of speaking engagements that came from the project
Nr. of successful replications based on documentation
revenue from sales of an OSH tool or kit
And more ...

Qualitative

Media clips and mentions
Transfer of project ownership to core community addressed
Recruitment of new project participant
Project team learning/understanding new things
Core community learning/understanding new things
'Did it change the conversation?'
More diversity or inclusion as a result of the project
Promoting tech/knowledge
Policy changes
Changing scientific standards
Feedback from users
Derivatives, add-ons, upgrades of the project
Inspiration for people to do things themselves
Effect on price baseline in commercial market/standards
New communities built around project
Successful or even unsuccessful case studies
Usage of OSH product, tool, kit, etc. How are people using it? Where?

What are some tools you can use to help measure impact?

There are a variety of tools for measuring some of the metrics above. We created a list of some that would be helpful. Not all of them are open source, but we tried to find ones that you can use for free.

Tool	Type	Link	What it does	How might we use it?
Groundsource	Survey	https://www.groundsource.co/	Survey form builder, free to start.	Get feedback from your community, users, or beta testers about the OSH tool or project.
Vojo	Survey, storytelling	http://vojo.co/	Set up a phone line where people can call in and record voice messages. People can also submit stories using SMS and online interface. Audio/media can be downloaded and edited or archived.	See Sandy Storyline example.
LimeSurvey	Survey	http://www.limesurvey.org	Open source survey form builder.	Get feedback from your community, users, or beta testers about the OSH tool or project.
Dragon	Transcription	[link]	Transcribes audio recordings of interviews.	Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.
PopUp Archive	Transcription	http://www.popuparchive.com	Transcribes audio recordings of interviews. Paid service, around $20/hr of audio.	Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.
Mechanical Turk	Transcription	http://requester.mturk.com/case_studies/transcription	Transcribes audio recordings of interviews. Paid service. You set the price you want to pay for 'turks' to transcribe audio. Turnaround usually within a few hours for 1 hr. audio. Average price is $10/hr for audio.	Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.
Keyhole	Social media analytics	http://keyhole.co/topsy-alternative	Lets you search for posts on Twitter an Instagram using key words, hashtags, handles.	Use to search for how people are talking about your project or related topics on social media. See if the conversations change through time in a 'before and after' comparison.
SumAll	Twitter analytics	http://sumall.com	Searches and visualizes tweets. Tool provides graphs and other ways of visualizing information.	Use to summarize tweets about your project or topic by key word.
Twitter Analytics	Twitter analytics	http://analytics.twitter.com	Twitter's own analytics dashboard. Still a little iffy, as it seems to be pulling information in from other sources (demographic, income level, brand preferences, etc.) Only works on Twitter accounts you own. Provides audience insights.	Use to summarize tweets about your project or topic by key word.
TweetDeck	Twitter account manager	http://tweetdeck.com	Alternate interface for using Twitter, useful for managing multiple accounts at a time.	Grab tweets, create streams specifically for hashtags, mentions, etc.
Tweetbot	Twitter account manager	http://tapbots.com/tweetbot/mac	Only for iOS, very similar to Tweetdeck.	Same as above.
Google Analytics	Web analytics	http://www.google.com/analytics	Dashboard for tracking user behavior on sites.	Use to grab insights about bounce rates, number of users who visit your site, how much time they spend on your site, how much time they spend on each page, which links are most popular, etc. Run 'campaigns' that track activity during specific events such as product launches or big announcements.
Alchemy API	Sentiment analysis	http://alchemyapi.com	Sentiment analysis, natural-language processing toolkit. Free for 1,000 API events, next tier is a set price per event.	Use to analyze sentiment of posts people are making about your project. Is the conversation positive? Negative? Etc.
GapMinder	Data visualization tool	http://gapminder.org	Data visualization tool	Visualize a data set you already have.
CartoDB	Mapping	http://cartodb.com	Free mapping platform. Upload data sets or link to data sets. Customizable with various base maps. SQL friendly.	Map where your community is, show where impact was made. Add media or tweets to a map to tell a story about your project.
Google Fusion Tables	Mapping	[link]	Free mapping platform. Turn a Google Sheet into a map with geospatial information.	Same as above.
Google nGrams	Look up how often a term is used	http://books.google.com/ngrams	Search through Google's database of books to track how the use of a term changed (increased, decreased) over time. Cross reference with other terms or major events. Also possible to search through frequency of search terms on Google.	Example here . Search for 'open hardware' to see the first time the term was mentioned in a book and track how the use of the term increased over time. Compare with the use of the word 'citizen science' in the same graph.
Facebook Insights	Facebook analytics	Tutorial: http://blog.kissmetrics.com/guide-to-facebook-insights	Only works for Facebook Pages, not personal accounts.	Use to get insights of user behavior on your Facebook page.
ChartBeat	Web analytics	https://chartbeat.com	Sends you daily digest of analysis/context for web traffic	Use to get webpage insights.
MixPanel	Web analytics	http://mixpanel.com	Similar to ChartBeat. Free for up to 25,000 data points. Next tier is $150/month for 500,000 data points.	Same as above but more geared toward users who are using site and analyzing their behaviours, retention, engagement
Altmetrics	Research publication insights	https://www.altmetric.com	Free for noncommercial use. Track research publications, data sets.	Find out who is talking about your research publications, datasets.
Storify	Storytelling	http://storify.com	Curate posts from across various social media platforms to capture how people are talking about your project.	Search by hashtag, key terms, etc.
Tumblr	Storytelling, blogging	http://tumblr.com	Publishing platform. Free to sign up. Customizable interfaces/themes. Easy to integrate with different forms of media.	Use as your main blogging website where your team or community members contribute posts to document the development of a tool.
Medium	Storytelling, blogging	http://medium.com	Same as above. Can connect to your Twitter account.	Same as above.
Presskit	Press kit template	http://dopresskit.com	Package of code you can download for free and populate with images, text, etc. from your project. Can be integrated into your website.	Use to get the word out about your project. Send package to press outlets, blogs, etc.

All right, what can I do with this kit?

Try it out! Our ultimate hope is that this information becomes useful to someone in the OSH community. Try it out, or try parts of it out. And if you're willing, contribute a case study for measuring impact.

Pass it along! If you know of a project or initiative that could benefit from this information, pass it on. You can even consider passing it onto a project that is outside of this community, since it's more of a framework of thinking. Measuring impact is something many fields and disciplines must wrestle with at some point.

Give us feedback! Like we mentioned, this is still brand new. It will have inconsistencies, missing pieces, and things to improve upon. We hope you'll help us make it even better. Here is a googledoc for you to easily modify it. Feel free to fork this toolkit and adapt it to whatever your needs are for your specific project.