Open Impact: A Toolkit for Measuring Impact for Open Source Hardware Projects

Hi there! We are a handful of attendees of GOSH 2016 at CERN who wanted to make an attempt at putting together a set of guidelines for measuring the impact of open source hardware projects. We felt that there is, as yet, no existing model for measuring impact specifically catered to the open source hardware community, and so we set out to curate some key metrics and tools that project managers, developers, grant writers, scholars, etc. could refer to.

We realized that each project has its own way of defining impact, which makes total sense! Each project comes with its own goals and aims, targeting different communities, funded by different funders (or some none at all).

The way that each project team tells its story varies widely. So at the conference, we asked around as many people as we could to get their feedback on the matter. We held two separate sessions about the topic and also spoke to people informally about their thoughts.

So this is a tiny toolkit we created based on what we found. By no means is it complete, so we would love your feedback on it too, which you can provide by e-mailing docubricks (docubricks at gmail dot com) or Lili Bui (lilybui at mit dot edu).

What is impact? A working definition.

First of all, what do we mean by 'impact' and why bother to measure it at all? It's tricky to give ‘impact' one definition, but many of us came to the consensus that it means to have changed the conversation in some way.

Why measure impact?

Measuring impact is allows you ...

  • To report on progress or potential to secure grants and funding for projects
  • To document history of projects in a social context
  • To construct stories about your project that you can pass on to others
  • To document methods, approaches, attitudes

Who needs to measure impact?

There are many stakeholders that have a need for measuring impact in some way. Here are a few we came up with. We're sure you can think of others.

  • Universities and institutions
  • Communities, specific publics
  • Press
  • Policymakers
  • Designers/builders
  • Investors, funders
  • Peers
  • Ourselves

How exactly do you measure impact?

That's the tricky part. There is no one exact way to do it. Like we mentioned before, everyone measure impact differently. Each project comes with different goals, audiences, stakes, etc. And what may count as impact in one project might not apply to the next. Here are some metrics that we came up with (both quantitative and qualitative) to capture impact. We don't recommend trying to use them all; rather, we recommend trying to identify which factors are the most important and relevant ones to you.


  • Nr. downloads of code, materials, tutorials
  • Nr. visitors on your website
  • Nr. participants or volunteers in your project
  • Nr. publications produced by your project
  • Nr. times publications have been cited
  • Nr. community members
  • Nr. success stories
  • Nr. of speaking engagements that came from the project
  • Nr. of successful replications based on documentation
  • revenue from sales of an OSH tool or kit
  • And more ...


  • Media clips and mentions
  • Transfer of project ownership to core community addressed
  • Recruitment of new project participant
  • Project team learning/understanding new things
  • Core community learning/understanding new things
  • 'Did it change the conversation?'
  • More diversity or inclusion as a result of the project
  • Promoting tech/knowledge
  • Policy changes
  • Changing scientific standards
  • Feedback from users
  • Derivatives, add-ons, upgrades of the project
  • Inspiration for people to do things themselves
  • Effect on price baseline in commercial market/standards
  • New communities built around project
  • Successful or even unsuccessful case studies
  • Usage of OSH product, tool, kit, etc. How are people using it? Where?

What are some tools you can use to help measure impact?

There are a variety of tools for measuring some of the metrics above. We created a list of some that would be helpful. Not all of them are open source, but we tried to find ones that you can use for free.




What it does

How might we use it?



Survey form builder, free to start.

Get feedback from your community, users, or beta testers about the OSH tool or project.


Survey, storytelling

Set up a phone line where people can call in and record voice messages. People can also submit stories using SMS and online interface. Audio/media can be downloaded and edited or archived.

See Sandy Storyline example.



Open source survey form builder.

Get feedback from your community, users, or beta testers about the OSH tool or project.




Transcribes audio recordings of interviews.

Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.

PopUp Archive


Transcribes audio recordings of interviews. Paid service, around $20/hr of audio.

Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.

Mechanical Turk


Transcribes audio recordings of interviews. Paid service. You set the price you want to pay for 'turks' to transcribe audio. Turnaround usually within a few hours for 1 hr. audio. Average price is $10/hr for audio.

Record interviews with community members and transcribe them to create a text version that can be cited in publications, grant reports, etc.


Social media analytics

Lets you search for posts on Twitter an Instagram using key words, hashtags, handles.

Use to search for how people are talking about your project or related topics on social media. See if the conversations change through time in a 'before and after' comparison.


Twitter analytics

Searches and visualizes tweets. Tool provides graphs and other ways of visualizing information.

Use to summarize tweets about your project or topic by key word.

Twitter Analytics

Twitter analytics

Twitter's own analytics dashboard. Still a little iffy, as it seems to be pulling information in from other sources (demographic, income level, brand preferences, etc.) Only works on Twitter accounts you own. Provides audience insights.

Use to summarize tweets about your project or topic by key word.


Twitter account manager

Alternate interface for using Twitter, useful for managing multiple accounts at a time.

Grab tweets, create streams specifically for hashtags, mentions, etc.


Twitter account manager

Only for iOS, very similar to Tweetdeck.

Same as above.

Google Analytics

Web analytics

Dashboard for tracking user behavior on sites.

Use to grab insights about bounce rates, number of users who visit your site, how much time they spend on your site, how much time they spend on each page, which links are most popular, etc. Run 'campaigns' that track activity during specific events such as product launches or big announcements.

Alchemy API

Sentiment analysis

Sentiment analysis, natural-language processing toolkit. Free for 1,000 API events, next tier is a set price per event.

Use to analyze sentiment of posts people are making about your project. Is the conversation positive? Negative? Etc.


Data visualization tool

Data visualization tool

Visualize a data set you already have.



Free mapping platform. Upload data sets or link to data sets. Customizable with various base maps. SQL friendly.

Map where your community is, show where impact was made. Add media or tweets to a map to tell a story about your project.

Google Fusion Tables



Free mapping platform. Turn a Google Sheet into a map with geospatial information.

Same as above.

Google nGrams

Look up how often a term is used

Search through Google's database of books to track how the use of a term changed (increased, decreased) over time. Cross reference with other terms or major events. Also possible to search through frequency of search terms on Google.

Example here . Search for 'open hardware' to see the first time the term was mentioned in a book and track how the use of the term increased over time. Compare with the use of the word 'citizen science' in the same graph.

Facebook Insights

Facebook analytics


Only works for Facebook Pages, not personal accounts.

Use to get insights of user behavior on your Facebook page.


Web analytics

Sends you daily digest of analysis/context for web traffic

Use to get webpage insights.


Web analytics

Similar to ChartBeat. Free for up to 25,000 data points. Next tier is $150/month for 500,000 data points.

Same as above but more geared toward users who are using site and analyzing their behaviours, retention, engagement


Research publication insights

Free for noncommercial use. Track research publications, data sets.

Find out who is talking about your research publications, datasets.



Curate posts from across various social media platforms to capture how people are talking about your project.

Search by hashtag, key terms, etc.


Storytelling, blogging

Publishing platform. Free to sign up. Customizable interfaces/themes. Easy to integrate with different forms of media.

Use as your main blogging website where your team or community members contribute posts to document the development of a tool.


Storytelling, blogging

Same as above. Can connect to your Twitter account.

Same as above.


Press kit template

Package of code you can download for free and populate with images, text, etc. from your project. Can be integrated into your website.

Use to get the word out about your project. Send package to press outlets, blogs, etc.

All right, what can I do with this kit?

Try it out! Our ultimate hope is that this information becomes useful to someone in the OSH community. Try it out, or try parts of it out. And if you're willing, contribute a case study for measuring impact.

Pass it along! If you know of a project or initiative that could benefit from this information, pass it on. You can even consider passing it onto a project that is outside of this community, since it's more of a framework of thinking. Measuring impact is something many fields and disciplines must wrestle with at some point.

Give us feedback! Like we mentioned, this is still brand new. It will have inconsistencies, missing pieces, and things to improve upon. We hope you'll help us make it even better. Here is a googledoc for you to easily modify it. Feel free to fork this toolkit and adapt it to whatever your needs are for your specific project.