D3.3 Report on Tool Validation

D3.3 Report on Tool Validation

Project Acronym I3
Grant Agreement Number 688541
Project Full Title I3 Impact Innovate Invest
Document Type Report
Document & WP No. D 3.3 WP3
Document Title Report on Tool Validation
Partner iMinds (imec), Eurokleis
Release date 22 February 2017


Review status Action Person Date
Quality Check Shenja van der Graaf 21 February 2017
Internal Review Simona De Rosa 20 February 2017
Distribution Public


Revision history
Version Date Modified by Comments
V0.1 13.02.17 Shenja van der Graaf ToC
V0.2 17.02.17 Iana Dulskaia, Francesco Bellini Full draft
V0.3 20.02.17 Simona De Rosa Internal Review
V1.0 21.02.17 Shenja van der Graaf Final version, internal review


Statement of originality:

This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgment of previously published material and of the work of others has been made through appropriate citation, quotation or both.

Table of Contents

List of Tables ………. 5

List of Figures ………. 5

Executive Summary ………. 6

  1. Introduction ………. 7
  2. Brief description of the methodology ………. 7
  3. Self-Assessment Toolkit Development and Validation ………. 9

3.1 First stage of the development ………. 9

Output on the first round of Tool validation ………. 10

3.2 Second stage of development ………. 10

Output on the second round of Tool validation ………. 13

  1. Next steps ………. 14
  2. Conclusions ………. 14

Annex 1 – List of participants of the first workshop ………. 15

Annex 2 – List of participants of the second workshop ………. 16

List of Tables

Table 1 First workshop participants ………. 16

Table 2 Webinar participants ………. 16

List of Figures

Figure 1 Areas of impact ………. 8

Figure 2 Presentation of the SAT first version ………. 10

Figure 3 Project assessment results – Report 1 ………. 11

Figure 4 Example of Business Model Canvas report per Output ………. 12

Figure 5 Webinar on the Self-Assessment Toolkit ………. 13

Executive Summary

This document presents a report on the Self-Assessment Toolkit validation. Indeed, one of the main i3 projects’ activities is development of Self-Assessment Toolkit (SAT) that helps projects to estimate their economic, social and technological impacts. The report describes different stages of SAT development as well as tool validation after each step of interaction with users and the other project’s stakeholders. Different types of stakeholders were included in the Tool validation beginning with the consortium partners for the internal check and continuing with ICT-19 projects by presenting SAT in the different events organised by i3 project. The first of the SAT was presented at first workshop in Brussels on the 7th of July 2016 where the i3 team received the first inputs which allowed to provide further improvements. The SAT functionalities were already discussed at the ICT proposers’ day in Bratislava (September 2016) and new functionalities were included also taking into account the latest modifications intervened in the Innovation Radar model. The final version of the tool was presented in the webinar held on the 3rd February 2017 “Self-Assessment Toolkit – how to estimate your project impact”.

1.          Introduction

This document describes the validation of the i3 Self-Assessment Toolkit (SAT) provided by the participants during the toolkit presentation at the different events organised by the i3 academy. SAT is an online platform that helps projects to self-assess their economic, social and technological impacts. At the end of the assessment, toolkit provides two infographic reports that consist of, first, of the project’s score in general according to three area of impact and secondly, SAT provides a business model of each output produced by the project. The platform can be found by following the link http://toolkit.i3project.eu/toolkit/

The report is structured as follows: Chapter 1 provides introduction to the document. Chapter 2 briefly introduces the i3 methodology. Chapter 3 summarises the main steps of Self-Assessment Toolkit development followed by its validation in each stage of modification. Chapter 4 presents the steps that will be taken further in order to accomplish the SAT. The document finalises with the main conclusions on the Tool validation.

Annex 1 presents the list of participants attended in the first workshop.

Annex 2 provides a list of participants that attended the webinar.

2.          Brief description of the methodology 

i3 methodology has its foundations mainly in the Cost-Benefit analysis, the Multi-criteria analysis, the Business Model Canvas and the Innovation Radar (IR). The assessment model is built by using a set of indicators and adapting them to the i3 operational context, namely the ICT 19 Convergence and Social Media. The i3 methodological framework adopts 3 indices related to specific areas of impact and related sub categories, visualised in figure 1. Each indicator is composed of sub-indices corresponding to specific subcategories; for example, the synthetic index Economic impact is composed of 7 sub-indices. All indices described here are visualised in the i3 self-assessment toolkit and constitute the core of the assessment analysis at the project and at aggregated/domain level.

Economic impact

This area of impact and associated indices consider all relevant economic results that projects develop in their lifetime. i3 provides an assessment of Convergence and Social Media projects by focusing on their economic, financial, organisational, generated value impacts at the level of projects partners and their stakeholders (micro level). The assessment is conceived in order to help projects (and their partners) to identify a Value Proposition model and related Business Model Canvas parameters.

   Figure 1 Areas of impact

Impact on society

This area of impact and associated indices consider all social results that projects develop during their lifetime. The i3 methodology provides that social impact is generally divided into following subsections:

  • Social capital
  • Impact on employment and working routine
  • Knowledge production and sharing

Each of the categories is divided into subcategories that can be adapted in relation to the project specifications.

Technological impact

As it was stated above in order to provide the assessment of technological impact SAT uses the Innovation Radar[1] methodology. Technological impact is related to the impact of project outputs on improving the state of the art, products and services, also outside of the observed domain. This impact analyses product, service and organisational innovation due to the technological outputs of the projects.

Technological readiness: The technology readiness level index describes how close to a potential exploitation a specific technology is. It has specific provisions and requirements to be fulfilled for each specific level, allowing projects to accurately assess their current position.

3.          Self-Assessment Toolkit Development and Validation

The aim of the i3 self-assessment methodology – is to make operational through a specific software tool by enabling ICT-19 projects to evaluate the potential impact of the innovations developed during their lifetime. The impact is measured not only in terms of socio-economic benefits for participating partners and the society, but also in terms of capability of participating projects to act as real innovators and build sustainable business models. The self-assessment activity is thus also preparatory to subsequent phases of the I3 project where investors’ attraction and acceleration of business initiatives is addressed.

SAT allows the acquisition of project information. It has been structured to guide the users in gathering the information with simple wizard (a guided procedure). The i3 team designed and developed the tool by dedicating particular attention to user experience in order to make it as simple and intuitive as possible.

The tool is used by project coordinators and by project partners. In order to access the dedicated online tool for data gathering, projects coordinators received a personal username and a password, then entered the information needed and were able to ask to specific partners (one or more) to fill-in specific sections. The wizard interface guides the user through the sections of information acquisition, at the end of which the user can set the parameters for the assessment and launch the project assessment.

3.1 First stage of the development  

The development of the first SAT version started in April 2016 after the first methodology was elaborated and the impact indicators that SAT assesses (social, economic and technological) were established. The work was accomplished by June 2016 and presented to the ICT-19 projects during the first Workshop in Brussels on 7th of July. 20 people have participated at the Workshop (see the list of participants in Annex 1). Most of the participants (a part of the i3 team) were represented by ICT-19 projects, as they are the main target of i3 project even though also some start-ups attended the workshop and were interested in the SAT. During the event, the functionality of using Toolkit was explained, guided by the following conceptual areas:

  1. Innovation, Economic and Social impacts
  2. Business model design: including questions on customer segments, channels, customer relationship, revenue streams, key resources, key activities, key partnership, cost structure
  3. Results and infographic reports

Figure 2 Presentation of the SAT first version

After the Tool demonstration, participants had the opportunity to provide their feedback and ask the questions regarding SAT.

Output on the first round of Tool validation

The feedback on SAT validation is considered positive, due to the information received at the end of the Workshop. The main conclusion regarding the SAT validation are:

  • participants were interested in the i3 Self-Assessment Toolkit and considered it useful and helpful;
  • participants have demonstrated the readiness to use the Toolkit, this was the result that the i3 team expected to achieve by presenting the SAT.

At the micro level, the participants identified a number of issues that they considered important for instance they were interested in security issues of a Toolkit as the data that they will provide according to projects are considered private and sensible.

3.2 Second stage of development

Lessons learned on the Tool validation from the first Workshop make it possible to modify the i3 methodology according to the participants’ suggestions, afterwards the SAT modification followed.

The modifications provided in the i3 methodology thereafter in the Self-Assessment Toolkit were the following:

  • application of the new version of the Innovation Radar which lead to the modification in the Innovation impact section
  • finalization of two infographic reports that SAT releases after providing the assessment of all the indicators.

Figure 3 Project assessment results – Report 1

Figure 4 Example of Business Model Canvas report per Output

The release of the final version of the SAT was made in December 2016; afterwards the series of the internal checks inside the consortium were made. The presentation of the SAT to the projects was decided to provide by organising a webinar – “Self-Assessment Toolkit – how to estimate your project impact” that was held on the 3d of February 2017 by the creators of the toolkit (Eurokleis). Thirteen participants took part in the webinar 11 of which were representing the ICT-19 projects and researchers (see the list of participants in Annex 2). The webinar duration was of about an hour and a half during which the participants had an insight in the SAT functionality and its goals. It should be noticed that couple of weeks before the webinar projects received their personal credentials to SAT in order to be familiar to it and have more concreate questions on its usability, functionality etc. during the webinar. The webinar started with the brief presentation of the i3 project its aims and methodology, then a step-by-step explanation and SAT demonstration followed.

Figure 5 Webinar on the Self-Assessment Toolkit

The webinar was interactive and participants had the opportunity to ask questions during the presentation. After the Toolkit demonstration, all the participants provided the feedback on the SAT.

Output on the second round of Tool validation

The tool validation process by users (projects) was one of the aims of the webinar. After the tool presentation, each of the participant provided his/her feedback and suggestions regarding the Tool.  In general, it can be noticed that SAT achieved positive feedback from the participants. All of them have demonstrated the willingness to use it as for the current ICT-19 projects as for the other types of projects. However, as in the first stage of Tool validation participants have demonstrated concerns about sensitive information that should be introduced to the SAT regarding partners, budget etc., the suggestions according these warries were to decide which data is really relevant to fulfil in the Self-Assessment Toolkit without including the private data. It was also suggested to provide a clear Data Policy before using SAT. Another advice was regarding the automatic data entry, for instance, automatic project partners’ entry in order to facilitate the SAT usage. The positive insights regarding Tool was the infographic report. Some participants have admitted that they already use the Innovation Radar, but they cannot see the results after completing the questionnaire, on the contrary the SAT that permits to visualise immediately the results of technological impact, in this case SAT is complementary to Innovation Radar methodology. The important suggestion regarding a SAT report was to make it possible to download; it can be useful for the projects for further deliverables development.

Some participants demonstrate interest in particular areas of impact and said that for them will be more important to concentrate on some impacts then on the others.

4.          Next steps

The feedback collected during the webinar was essential in order to organise the following actions of Self-Assessment Toolkit improvement. After a brainstorming on the participants’ suggestions, it was decided to take the following steps:

  • Preparation of Disclosure agreement in order to make transparent the Data Policy
  • Make it possible to download the infographic report in PDF
  • Make some functionalities of the SAT automatic (for instance partners’ data entry)

These modifications are foreseen by the end of the February 2017.

5.          Conclusions

The feedback provided by the projects during the SAT presentation in its different stages helped to have an overview on the Tool validation. In general, the SAT received a positive feedback as most of the participants demonstrated the wiliness to use it and also stated to recommend the platform to other projects, which was the result that i3 team wanted to achieve. Participants highlighted the usefulness of the tool, especially regarding the results that SAT provides (infographic report as on the project score in general as on particular area of impact in particular).  However, despite positive results participants provided their suggestions for the SAT improvement such as make the infographic reports downloadable, provide a disclosure agreement on data policy and make some functionalities of the SAT more automatic.


Annex 1 – List of participants of the first workshop

Table 1 First workshop participants

*List of participants has been removed for user privacy reasons.

Annex 2 – List of participants of the second workshop

*List of participants has been removed for user privacy reasons.

[1] https://ec.europa.eu/digital-single-market/en/innovation-radar