Filipe Carlos de Albuquerque Calegario · combinação, adaptamos um método de design tradicional,...

Pós-Graduação em Ciência da Computação

Filipe Carlos de Albuquerque Calegario

“METHOD AND TOOLKIT FOR DESIGNING DIGITAL MUSICAL

INSTRUMENTS: GENERATING IDEAS AND PROTOTYPES”

Universidade Federal de Pernambuco

[email protected] www.cin.ufpe.br/~posgraduacao

RECIFE 2017

1


METHOD AND TOOLKIT FOR DESIGNING DIGITAL MUSICAL

INSTRUMENTS: GENERATING IDEAS AND PROTOTYPES

Advisor: Geber Lisboa Ramalho Co-Advisor: Marcelo Mortensen Wanderley

RECIFE

2017

A THESIS PRESENTED TO CENTRO DE

INFORMÁTICA DA UNIVERSIDADE FEDERAL DE

PERNAMBUCO IN PARTIAL FULFILLMENT OF THE

REQUIREMENTS FOR THE DEGREE OF DOCTOR OF

PHILOSOPHY IN COMPUTER SCIENCE.

Catalogação na fonte

Bibliotecária Monick Raquel Silvestre da S. Portes, CRB4-1217

C148m Calegario, Filipe Carlos de Albuquerque

Method and toolkit for designing digital musical instruments: generating ideas and prototypes / Filipe Carlos de Albuquerque Calegario. – 2017.

160 f.: il., fig. Orientador: Geber Lisboa Ramalho. Tese (Doutorado) – Universidade Federal de Pernambuco. CIn, Ciência da

Computação, Recife, 2017. Inclui referências e apêndice.

1. Inteligência artificial. 2. Computação musical. I. Ramalho, Geber Lisboa (orientador). II. Título. 006.3 CDD (23. ed.) UFPE- MEI 2017-213

3


Method and Toolkit for Designing Digital Musical Instruments: Generating Ideas and Prototypes

Tese de Doutorado apresentada ao Programa de

Pós-Graduação em Ciência da Computação da Universidade Federal de Pernambuco, como requisito parcial para a obtenção do título de Doutora em Ciência da Computação

Aprovado em: 31/03/2017. __________________________________________________ Orientador: Prof. Dr. Geber Lisboa Ramalho

BANCA EXAMINADORA

________________________________________________ Profa. Dra. Patricia Cabral de Azevedo Restelli Tedesco

Centro de Informática / UFPE

_______________________________________________ Profa. Dra. Veronica Teichrieb Centro de Informática / UFPE

_________________________________________________

Prof. Dr. Jônatas Manzolli Departamento de Música / UNICAMP

_______________________________________________________

Prof. Dr. Andre Menezes Marques das Neves Departamento de Design / UFPE

_______________________________________________________

Prof. Dr. João Paulo Cerquinho Cajueiro Departamento de Engenharia Mecânica / UFPE

_______________________________________________________

Prof. Dr. François Pachet Computer Science Laboratory Paris / SONY

4

To Alissa =)

5

Acknowledgement I am immensely grateful for the great support and vibrant energy that the following people

gave me in the course of these four years of research. Sou imensamente grato ao

enorme apoio e a vibrante energia que as seguintes pessoas me deram em algum

momento ou no decorrer destes quatro anos de pesquisa.

Alissa Seixas, Francisca Albuquerque, Carlos Calegario, Rodrigo Calegario, Leonardo

Menezes, Lêda Carlos, Rosa Carlos, Ana Maria Gusmão, Maribenete Menezes,

Generina dos Santos, Marcelo Gusmão, Rosângela Seixas, João Tragtenberg, Geber

Ramalho, Giordano Cabral, Marcelo Wanderley, Stéphane Huot, Johnty Wang, Ian

Hattwick, Ivan Franco, Mailis Rodrigues, John Sullivan, Lígia Teixeira, Baptiste

Caramiaux, Carolina Medeiros, Gabriel Vigliensoni, Fernando Iazzetta, Darryl Cameron,

Jerônimo Barbosa, Vânia Pontes, Celio Eyng, Thais Fernandes, Helder Vasconcelos,

Rodrigo Medeiros, Eduardo Santos, Sofia Galvão, Ricardo Brazileiro, Jarbas Jácome,

Ricardo Ruiz, Simone Jubert, Tarciana Andrade, Hermano Ramos, Clara Arruda, Clara

Vasconcelos, Jeffeson Mandu, Carlos Montenegro, Miguel Mendes, Tomás Brandão,

William Paiva, Missionário José, Helder Aragão, Yuri Bruscky, Fernando Almeida,

Renato Barros, João Marcelo Ferraz, Leo Domingues, André Araújo, Sérgio Godoy,

Sofia Freire, Pedro Luiz, Ana Cecília Barbosa, Gabriella Martins, Madyana Torres,

Emiliano Abad, Luis Arthur Vasconcelos, Jaime Alheiros, Zaca Arruda e Preta Félix.

Thank you very much! Meu muitíssimo obrigado!

6

source: https://www.reddit.com/r/funny/comments/13vd1u/the_only_diagram_i_give_a_damn_about/

“When you step into an intersection of fields,

disciplines, or cultures, you can combine existing concepts into

a large number of extraordinary new ideas”

The Medici Effect Frans Johansson

7

Abstract Last decade witnessed a considerable rise in physical, programmable, interactive artifacts.

Sensors, devices, platforms and frameworks have become more accessible and more people are

programming the physical world beyond the screen. Interactive devices for artistic expression

present challenges that are worth investigating because the interaction often needs a high level

of skill that is hard to be obtained. Therefore, interactive artistic approaches can teach valuable

lessons applicable to other levels of interaction design and human-computer interaction. One

class of artistic, physical interactive objects is the digital musical instrument (DMI). DMIs are

artifacts in which gestural control and sound production are physically decoupled but digitally

mapped. It provides freedom for a DMI designer, since several combinations are possible, but

increases the complexity of the design space. Besides, structured methods and guidelines that

would help the design have not yet been established. To address this issue, prototyping seems

to be a promising approach as they are not only a tool for testing and communicating ideas, but

also for generating them. As a DMI is a means to produce music, its prototype should provide

real-time sound feedback for control gestures. For that reason, in DMI context, non-functional

prototypes are not entirely suitable. On the other hand, the development of functional prototypes

demands more time and effort, and consequently, can be a bottleneck of iterative design. How to

provide structured and exploratory paths to generate DMI ideas? How to decrease time and effort

of building functional DMI prototypes? To deal with those questions, we propose the concept of

instrumental inheritance, that is the application of gestural and/or structural components of

existing instruments to generate ideas of new instruments. As support for analysis and

combination, we leverage a traditional design method, the morphological chart, in which existing

artifacts are split into parts, presented in a visual form and then recombined to produce new ideas.

Finally, integrating the concept and the method in a concrete object, we developed a physical

prototyping toolkit for building functional DMI prototypes: Probatio, a modular system of blocks

and supports to prototype instruments based on certain ways of holding and gestural controls for

musical interaction. The evaluation of the toolkit showed that it contributed to reducing the time

to achieve a functional prototype, and also influencing the increase in the number of cycles of

idea exploration. Besides, the users reported more musical engagement with Probatio in

comparison to a generic sensor toolkit.

Keywords: New interfaces for musical expression. Digital Musical Instruments. Idea Generation.

Ideation. Prototyping. Prototyping Toolkit.

8

Resumo A década passada testemunhou um aumento considerável em artefatos físicos,

programáveis e interativos. Sensores, dispositivos, plataformas e estruturas tornaram-se mais

acessíveis e mais pessoas estão programando o mundo físico além da tela. Dispositivos

interativos para expressão artística apresentam desafios que valem a pena investigar porque a

interação geralmente precisa de um alto nível de habilidade difícil de obter. Portanto, as

abordagens artísticas interativas podem ensinar lições valiosas aplicáveis a outros níveis de

interação e interação humano-computador. Uma classe de objetos interativos artísticos e físicos

é o instrumento musical digital (DMI), artefatos em que controle gestual e produção de som são

fisicamente desacoplados, mas digitalmente mapeados. Este desacoplamento proporciona mais

liberdade para um designer de DMI, uma vez que são possíveis várias combinações, mas

aumenta a complexidade do espaço de design. Além disso, métodos estruturados e diretrizes

que ajudariam o projeto ainda não foram estabelecidos. Para abordar esta questão, a

prototipação parece ser uma abordagem promissora, pois não serve apenas como forma de

testar e comunicar ideias, mas também para gerá-las. Como um DMI é um meio para produzir

música, seu protótipo deve fornecer, a partir de gestos de controle, feedback de som em tempo

real. Por essa razão, no contexto DMI, protótipos não funcionais não são inteiramente

adequados. Por outro lado, o desenvolvimento de protótipos funcionais exige mais tempo e

esforço e, consequentemente, pode ser um gargalo no design iterativo. Como fornecer caminhos

estruturados e exploratórios para gerar ideias DMI? Como diminuir o tempo e o esforço de

construir protótipos DMI funcionais? Para lidar com essas questões, propomos o conceito de

herança instrumental, que é a aplicação de componentes gestuais e/ou estruturais de

instrumentos existentes para gerar ideias de novos instrumentos. Como suporte para análise e

combinação, adaptamos um método de design tradicional, a caixa morfológica, em que os

artefatos existentes são divididos em partes, apresentados de forma visual e depois

recombinados para produzir novas ideias. Finalmente, integrando o conceito e o método em um

objeto concreto, desenvolvemos um toolkit de prototipação física para a construção de protótipos

funcionais de DMI: o Probatio, um sistema modular de blocos e suportes para protótipos de

instrumentos baseados em certas maneiras de segurar e controles gestuais para a interação

musical. A avaliação mostrou que o toolkit contribuiu para reduzir o tempo para conseguir um

protótipo funcional e também influenciou o aumento no número de ciclos de exploração de ideias.

Além disso, os usuários relataram mais envolvimento musical com a Probatio em comparação

com um toolkit de sensores genéricos.

Palavras-chave: Novas Interfaces para Expressão Musical. Instrumento Musical Digital. Geração

de Ideias. Ideação. Prototipação. Ferramentas de Prototipação.

9

Table of Contents 1. INTRODUCTION ...................................................................................... 121.1. Context ......................................................................................................... 121.2. Objectives .................................................................................................... 131.3. Approach ...................................................................................................... 141.4. Document Outline ........................................................................................ 14

2. CHALLENGES IN DESIGNING DMIS ........................................................... 162.1. Digital Musical Instruments ........................................................................ 162.1.1. DMI Classification .......................................................................................... 182.2. The Challenge of Multiple Combinations .................................................. 182.2.1. Gestural Controller ........................................................................................ 192.2.2. Sound Output ................................................................................................ 202.2.3. Mapping ......................................................................................................... 212.2.4. Feedback ....................................................................................................... 222.2.5. Summary ....................................................................................................... 232.3. The Challenge of Expressivity and Virtuosity .......................................... 232.4. The Challenge of Evaluation and Evolution .............................................. 242.5. The Challenge of No Previous Knowledge ............................................... 252.6. The Challenge of Multiple Stakeholders and Contexts of Use ................ 252.7. Final Considerations ................................................................................... 26

3. DESIGN PROCESS ................................................................................... 283.1. Idea Exploration ........................................................................................... 303.2. Prototyping ................................................................................................... 333.3. Final Considerations .................................................................................... 37

4. STATE OF THE ART .................................................................................. 394.1. Frameworks and Approaches for DMI Design ............................................ 394.2. Functional Prototype in DMI Design ............................................................ 454.2.1. Tools for Physical and Functional Prototyping ................................................ 454.2.2. The Trade-off Area .......................................................................................... 494.3. Final Considerations ..................................................................................... 50

5. EARLY EXPLORATION ............................................................................... 515.1. Methodological Approach ............................................................................ 515.2. Project Batebit ............................................................................................... 525.2.1. Interviews ........................................................................................................ 52

10

5.2.2. Workshops ...................................................................................................... 535.2.3. Pandivá ........................................................................................................... 545.2.4. Sandbox Wow ................................................................................................. 565.3. Summary ........................................................................................................ 58

6. PROPOSITION .......................................................................................... 596.1. Scope and Basis ............................................................................................ 596.2. Instrumental Inheritance ............................................................................... 606.2.1. Related Concepts ............................................................................................ 606.2.2. Possible Evidences ......................................................................................... 626.2.3. Discussion ....................................................................................................... 646.3. Morphological Chart for DMI Idea Generation ............................................ 656.3.1. What is Morphological Analysis? ..................................................................... 656.3.2. Morphological Chart Based on Instrumental Inheritance ................................ 676.4. Development of the Functional Prototyping Toolkit for DMI ..................... 706.4.1. Guidelines ....................................................................................................... 716.4.2. Implementation Decisions ............................................................................... 726.4.3. Physical Structure ........................................................................................... 726.4.4. Connection Slots ............................................................................................. 756.4.5. Blocks .............................................................................................................. 776.5. Final Considerations ..................................................................................... 80

7. EVALUATION OF PROBATIO 0.1 .................................................................. 817.1. Description ...................................................................................................... 817.2. Evaluation ....................................................................................................... 82

8. EVALUATION OF PROBATIO 0.2 .................................................................. 888.1. Evolution from Probatio 0.1 ........................................................................... 888.1.1. Number of Blocks and Multiple Sensors .......................................................... 898.1.2. Changing Mapping Strategy ............................................................................. 898.1.3. Sound Output Module ...................................................................................... 908.1.4. Curved Shapes ................................................................................................. 908.1.5. Connection Arm Support .................................................................................. 908.1.6. Protection and Connections to the Hub ............................................................ 908.1.7. Friction of Blocks and Slots .............................................................................. 918.2. Experiment ...................................................................................................... 918.2.1. Objectives ......................................................................................................... 918.2.2. Design .............................................................................................................. 928.2.3. Methods for Data Collection ............................................................................. 93

11

8.2.4. Methods for Quantitative Analysis .................................................................... 958.2.5. Methods for Qualitative Analysis ...................................................................... 978.2.6. Setup ................................................................................................................ 978.2.7. Technical Test Pilot ........................................................................................ 1038.2.8. Participants ..................................................................................................... 1048.2.9. Experiment Protocol ....................................................................................... 1068.3. Results .......................................................................................................... 1088.3.1. Quantitative Analysis ...................................................................................... 1088.3.2. Qualitative Analysis ........................................................................................ 1228.4. Discussion .................................................................................................... 1338.4.1. About Probatio ................................................................................................ 1338.4.2. About GSToolkit ............................................................................................. 1348.4.3. Summary of Bugs and Errors ......................................................................... 1358.4.4. Different Engagements ................................................................................... 1368.4.5. Three Profiles ................................................................................................. 1368.4.6. Limitations ...................................................................................................... 1378.4.7. Final Considerations ....................................................................................... 138

9. CONCLUSION .......................................................................................... 1409.1. Research Question Revisited ...................................................................... 1409.2. Contributions ................................................................................................ 1409.3. Limitations .................................................................................................... 1419.4. Future Works ................................................................................................ 142

REFERENCES ......................................................................................... 144

APPENDIX A - RELATED PROJECTS ......................................................... 159

12

1. INTRODUCTION “In the grand scheme of things, there are three levels of

design: standard spec, military spec and artist spec. Most

significantly, I learned that the third, artist spec, was the

hardest (and most important). If you could nail it, then

everything else was easy.” (BUXTON, 1997)

1.1. Context Last decade witnessed a considerable expansion of physical, programmable, interactive artifacts.

Sensors, devices, platforms, and frameworks have become more accessible, fostered by the

proliferation of mobile technologies, the growth of the DIY and Maker communities, and the

spread of open source and open hardware philosophies. If the question used to be "How to make

it?”, now, it becomes “What can be made of what is available?” (SANDERS; STAPPERS, 2014).

More people are experimenting with sensors and actuators, programming the physical world

beyond the screen.

Among these new interactive devices, those devoted to artistic expression present challenges

that are worth investigating, since "the essence of the artist [...] is rooted in skill, [...] which is hard

earned" (BUXTON, 1997) and should be taken into account during the design process. Interactive

artistic approaches can teach valuable lessons applicable to other levels of interaction design and

human-computer interaction.

One class of artistic, physical interactive objects is the digital musical instrument (DMI). DMIs are

devices in which gestural control and sound production are physically decoupled but digitally

connected according to a mapping strategy (MIRANDA; WANDERLEY, 2006). Sensors translate

gestures into digital data that can then be processed and mapped to sound synthesis algorithms

or modules.

DMIs cover a variety of artifacts as these examples illustrate: The Hands (TORRE; ANDERSEN;

BALDÉ, 2016), hyperinstruments (MACHOVER, 1991), Jam-O-Drum (BLAINE; FORLINES,

2002), The Hyper-Flute (PALACIO-QUINTIN, 2003), Radio Baton (MATHEWS, 2005), AKAI EWI

(VASHLISHAN, 2011), Yamaha WX5 (MIRANDA; WANDERLEY, 2006), Reactable (JORDÀ et

al., 2007), Laser Harp, Tenorion (NISHIBORI; IWAI, 2006), Novation Launchpad, Monome,

Faderfox, QuNeo (PAINE, 2013), Karlax, and Eigenharp (PAINE, 2015) (the citations do not

necessary represent the author of the instrument, but where further information can be found).

DMI design and implementation encompass a number of issues (MEDEIROS et al., 2014). One

is that the sound-control dissociation provides more freedom for DMI designers in comparison to

builders of acoustic instruments, as there are no mechanical or physical constraints. However,

the multiple input-output combinations increase the complexity of the design space and can

sometimes lead to creative paralysis (MAGNUSSON, 2010).

13

Reportedly in literature, the DMI design process is idiosyncratic (BARBOSA et al., 2015a). The

instrument is usually conceived and developed for one performer, or the roles of performer and

designer are played by one person. The discussions cover the proposition of principles, mainly

based on personal experience, and conceptual frameworks that attempt to define the relationship

between components of DMI design space, as concepts and stakeholders. The few structured

methods and processes are commonly generic, presenting no fined-grain steps that would guide

the designer during the design phase of idea generation, for instance. Thus, normally, it is

necessary for the designer to find her own suitable inspiration and ideas.

In addition, the cycle of exploring new ideas and transforming them into prototypes is an important

part of the design process (BROWN, 2008). Prototypes help to identify flaws, redirect and adjust

decisions, improve understanding of the context, and generate new ideas (WARFEL, 2009). As

the criteria for DMI design success and for their formal evaluation are not clear (BARBOSA et al.,

2015b), the DMI design process must likely rely on multiple cycles of prototyping (DAHL, 2016).

However, because the DMI is not an end in itself (such as a table or a chair), but a means to

produce music, a DMI cannot be adequately evaluated without being played. For that reason,

conventional low-fidelity non-functional prototype tools and methods are not entirely suitable for

the DMI context. For a complete understanding, DMI prototypes should be functional, reacting to

player’s actions in real-time (DAHL, 2016). This aspect demands more time and effort during

development when compared with non-functional prototypes, and can become a potential

bottleneck for iterative design process (HUOT, 2013).

1.2. Objectives Inspired by this context, our objective is related to two questions that will address conception and

implementation of DMIs:

1. How can we provide structured and exploratory paths for generating new DMI ideas?

2. How can we reduce the time and effort needed to build functional DMI prototypes?

This project aims to provide designers with directions for conception, as well as to narrow the gap

between idea and prototype. With that, we intend to boost the cycles of idea exploration, fostering

more experimentation in less time. From the perspective of iterative design cycles, by using a

straightforward and structured set of steps, we expect that designers and users can achieve a

better convergence between (a) the user’s needs, intentions, and contexts of use and (b) the

resulting, evolving DMI.

In the long run, even understanding that it is hard to know if this kind of objective is reachable, we

hope to contribute to the acceleration of evolution cycles of musical instruments. These cycles

normally take decades until the evolving instruments are actually part of human culture, and

incorporated in artistic expression.

14

1.3. Approach Considering our first question, we propose a new concept in the context of DMI design:

instrumental inheritance, which is the set of components of physical structure or playing

techniques that an existing instrument transfers to a new instrument. We also explore a design

method for idea generation based on morphological analysis (CARD; MACKINLAY;

ROBERTSON, 1991) and morphological chart (CROSS, 2000), in which existing devices are split

into their fundamental parts and then recombined to generate new ideas. In our case, we analyzed

ways of controlling and ways of holding musical instruments.

Addressing our second question, we introduce a proof-of-concept physical and functional toolkit

for prototyping digital musical instruments named Probatio (Latin word for “test, experiment, trial”)

(see Figure 1.1). Probatio provides a modular environment where users can make functional DMI

prototypes by combining parts of existing instrument controls and supports, following the

morphological chart approach.

Figure 1.1: Example Probatio in use. For the video demonstration: https://youtu.be/Ge_aj5uMgOU

It attempts to provide designers with examples and directions to generate new ideas, as well as

to reduce the gap between an idea and its working prototype. By producing an environment that

combines exploration and implementation into a single tool, it is our intent to allow the user to

create functional prototypes in less time. We expect to benefit the dialog between designer and

performer, and enable the designer-performer to explore new ideas more easily with less effort.

The two cycles of evaluation of the toolkit showed that it contributed to reducing the time to

achieve a functional prototype, and it also influenced the increase in the number of cycles of idea

exploration. Besides, the users reported they had more musical engagement with Probatio in

comparison to a generic sensor toolkit.

1.4. Document Outline • Chapter 2: we present the concept of DMI with some examples; present a list of

challenges to help to discuss the complexity of DMI design, and concludes that cycles of

idea exploration and prototyping seem to bring positive points to DMI design process.

• Chapter 3: we work on the concept of design process, focusing on idea exploration and

prototyping.

• Chapter 4: looking through the lenses of the design process, we introduce the state of

the art on frameworks, principles, methods and prototyping tools for DMI design.

15

• Chapter 5: we show our early exploration that comprises, interviews, workshops, initial

prototypes, and the insights related to each step.

• Chapter 6: we define our methodological approach and present our threefold proposition:

(1) the concept of instrumental inheritance, (2) the method of morphological chart applied

for DMIs, and (3) the toolkit intended to be used for prototyping physical functional DMIs.

• Chapter 7: we describe the version 0.1 of the toolkit and its preliminary evaluation.

• Chapter 8: we explain the version 0.2 of the system, describe a comparative experiment,

present and discuss the results.

• Chapter 9: we conclude by discussing our contributions, limitations and future works.

16

2. CHALLENGES IN DESIGNING DMIS "As Robert Moog states [...] Musical Instrument Design is

one of the most sophisticated and specialized

technologies that we humans have developed [...] [W]hen

we speak of musical instruments today, we understand

that we are talking about precisely made and finely tuned

objects." (SYLLEROS; DE LA CUADRA; CÁDIZ, 2014)

In this chapter, we examine the very concept of DMI and discuss a list of illustrative challenges

one might face when conceiving and building these artifacts. This list is not meant to be

comprehensive or exhaustive, but it helps us to attest the complexity of the area and to conclude

why it is important to have various cycles of experimentation and implementation during the

development of a new DMI.

2.1. Digital Musical Instruments Digital Musical Instrument (DMI) is a class of artistic, tangible, interactive objects intended to

articulate sound by means of gestural control. Contrary to their acoustic counterparts which follow

physical constraints, in the DMIs the control input is decoupled from the sound output (MIRANDA;

WANDERLEY, 2006). As an intermediate layer connecting these two modules, there is the

mapping strategy (See Figure 2.1). Besides, feedback beyond sound, such as haptic, luminous

are also elements of a DMI.

Figure 2.1: Digital Musical Instrument Diagram

In an attempt to discuss the concept further, Gurevich et al. (2011) argue that not all forms of

musical interactions are instrumental, thus proposing another definition: Digital Musical

Interactions (GUREVICH; CAVAN FYANS, 2011). Despite considering the relevant discussion,

in this work, we will use the term Digital Musical Instrument as an artifact composed of parts that

can be independently analyzed, with which the user interacts to obtaining a musical result.

Another term found in the literature is New Interfaces for Musical Expression (NIME). Although

NIME implies a wider definition, as it is not unique either to “Digital” or to “Instruments,” in the

present work we will use the terms DMI and NIME interchangeably.

17

DMIs comprises a plethora of artifacts, which some examples are (Figure 2.2): Haken Continuum,

a continuous playing surface that tracks position and pressure of multiple fingers; AlphaSphere,

a set of pressure sensitive tactile pads disposed as a sphere; Reactable, a combination of a multi-

touch table and acrylic cubes that control synth elements and sequencers (JORDÀ et al., 2007).

Eigenharp Alpha, a DMI in a bassoon-like appearance that comprises a matrix of keys with three

degrees-of-freedom (up, down, right, left, pressed, released), a wind controller and a ribbon

controller which control a variety of sound synthesis parameters.

Figure 2.2: DMI examples: (a) Haken Continuum, (b) Oval, (c) Ableton Push 2, (d) Eigenharp Alpha,

(e) Karlax, (f) LinnStrument (g) AlphaSphere

However, in the area, "it is hard to find artifacts that have been widely or convincingly adopted by

musicians" (MEDEIROS et al., 2014). In fact, it is considered in literature only a few virtuosi or

professional musicians (JORDÀ; MEALLA, 2014). Although relevant to the discussion, the

virtuosic use of a DMI is only one facet of the many possible options based on intentions and

contexts of use of the user.

It is reported in literature that the design process of DMI is guided by idiosyncratic approaches

(RYAN, 1991), (WANDERLEY; ORIO, 2002), (BONGERS, 2007), (WARD; TORRE, 2014),

(BARBOSA et al., 2015a), which leads to little or no room for comparison, or evolution of these

instruments.

The advance in mobile technologies, besides the DIY, maker, and open source philosophies open

the world of sensors, devices, platforms, and frameworks for a broader audience, which enforced

the tendency of the users experimenting adequate configurations for their own need.

18

Instead of denying idiosyncrasies or finding other ways to DMI development, we assume the

position of the user in the center of the DMI design process, valuing their decisions, intentions,

and contexts of use.

2.1.1. DMI Classification

Miranda and Wanderley (2006) proposes a classification continuum for DMIs (Figure 2.3) based

on the similarity of the gestural controller compared to acoustic instruments (MIRANDA;

WANDERLEY, 2006). Along the continuum, the categories are:

• Augmented musical instruments: are acoustic instruments with sensors that expand the

gestural capabilities of the performer, but maintains the gestures repertoire references of the

traditional instrument;

• Instrument-like gestural controllers: simulators of acoustic instruments, i.e. MIDI controller

version of instruments. Comparing to acoustic instruments, they present less potential of

controlling nuances but offer an expanded set of possible sounds.

• Instrument-inspired gestural controllers: present some similarity with acoustic instruments

but do not intend to simulate them;

• Alternate gestural controllers: do not hold any resemblance with existing musical

instruments.

Figure 2.3: DMI classification adapted from (MIRANDA; WANDERLEY, 2006)

2.2. The Challenge of Multiple Combinations In this section, we attempt to show the multiple combinations between inputs, mapping strategies,

and output, which are the result of the decoupling from the gestural controller and the sound

production module. We based our analysis on the model initially proposed by Rovan et al. (1997)

and refined by Miranda and Wanderley (2006) (ROVAN et al., 1997), (MIRANDA; WANDERLEY,

2006). Our objective here is to understand the DMI design space from the point of view of DMI

parts.

19

2.2.1. Gestural Controller

To better comprehend the possibilities for the gestural controller, it is necessary to understand

some characteristics of the input gestures.

Cadoz and Wanderley (2000) defines the gestural channel as a human communication means

similar to the verbal communication (CADOZ; WANDERLEY, 2000). The author defines three

functions of this channel:

• Ergotic function: (from ergos, “physical work, energy” (LUCIANI, 2007)) related to an action

that modifies and transform the environment with energy transfer between the actor and the

environment (MUSER, 2015).

• Semiotic function: delivery of meaningful information to the environment, such as thumbs

up to tell you something is correct.

• Epistemic function: exploratory movements to acquire haptic or tactile information from a

particular artifact. It can be considered as a preliminary phase to the ergotic gestures, in which

the user is experiencing the space to understand it better.

Based on the presence or absence of physical contact, Jensenius et al. (2010a) present two

definitions: manipulative gestures, which are related to physical contact, and empty-handed

gestures, a synonym of free gesture, naked gestures, or semaphoric (JENSENIUS et al., 2010).

Instrumental Gestures, or gestures related to manipulating objects, have the three functions of

the gestural channel mentioned above. These gestures can be divided into three broad

categories:

• Selection: is the choice of a particular element of the instrument, and it is not related to

adding energy to produce a sound, e.g. the position of the fingers on a flute, or on the neck

of a violin or a guitar;

o Sequential: one selection is performed at a given time

o Parallel: multiple selections are carried out at once

• Excitation: energy is added to the system. For example, the movement of the bow of a violin,

blowing a flute, pressing the key of a piano, striking a percussion instrument;

o Instantaneous: the energy is provided in a single event. It can be further divided into

percussive, when the sound event happens as soon as the physical contact begins,

and picking when the sound event happens after the physical contact ends

(ACEITUNO, 2015).

o Continuous: the energy is provided during the sound event

• Modification: the form given to some control parameter. For example, when a vibrato is

made, which is the modulation of the height of a note or the tremolo, which is the modulation

of the amplitude or volume of this note.

o Parametric: if the controlled parameter varies during the event (e.g. vibrato, tremolo)

o Structural: if the there is a qualitative change in the structure of the instrument (e.g.

the mute on a brass instrument, the sustain pedal of the piano)

20

Focusing on possible human outputs to musical instruments, Bongers (2000) lists (BONGERS,

2000): (1) muscle action; (2) blowing; (3) voice; (4) biosignals: galvanic skin response,

temperature, blood pressure, heart rate, etc. Focusing on the muscle action, the author presents

two categories based on the physiology of the muscles: isometric contraction, in which there is

no change in the muscle length or the joint angle; and isotonic contraction, in which the tension

continue the same and the length of muscle changes, causing movement. Figure 2.4 presents

the classification of muscle action.

Figure 2.4: Muscle action classification. (Extracted from Bongers (2000))

Sensors are used to allow the possible human outputs to control sound parameters. These

devices convert physical energy into electricity, and then in digital values (BONGERS, 2000).

Examples of physical energy are: “kinetic (such as pressure, torque, inertia); light; sound;

temperature; smell; humidity; electricity; magnetism; electro-magnetism (radio waves)”

(BONGERS, 2000).

By using signal conditioning and processing techniques in hardware, or in software, “devoted to

adjusting, amplifying, filtering, selecting and transducing signals” (MEDEIROS; WANDERLEY,

2014), the raw data can be manipulated to achieve more stable and adequate results (MALLOCH,

2008) (STEINER, 2005).

2.2.2. Sound Output

The output module of the DMI is responsible for synthesizing the sound based on the inputs and

mapping strategy. Examples of possible synthesis methods are oscillators and wavetables,

additive synthesis, subtractive synthesis, modulation synthesis, frequency modulation synthesis,

physical modeling, granular synthesis, vocal/formant synthesis, sampling/PCM synthesis (COOK,

2002) (MIRANDA, 1998). In this work, we do not focus on the sound generation. Thus, we

approach it at a higher level of abstraction, being more concerned with defining the possibilities

of control over the sound generation than the actual method of sound synthesis.

How to decide on the elements of the gestural controller for a new DMI? How to choose the suitable sensors to translate the gestures into digital values?

21

Schloss (1990) presents a classification for musical controls based on three levels (SCHLOSS,

1990):

• Timbral level: it is the microscopic control, in which sound properties can be manipulated in

details. It is a level that demands continuous control of the synthesis parameters in different

dimensions. For example, changing parameters such as resonance on a modular synthesizer;

• Note level: it is related to triggering notes, and controlling their execution. For example, pressing

keys of a piano;

• Process level: it is the macroscopic level based on the control of a pre-defined musical event

or sequence of events. For instance, DJ sampling.

2.2.3. Mapping

Mapping strategies are the essence of DMI (ROVAN et al., 1997) (JORDÀ, 2005a). Hunt,

Wanderley and Paradis (2002) show that even if one keeps the same inputs and outputs, but

alters the mapping strategy, the performer perceives both configurations as two different instruments (HUNT; WANDERLEY; PARADIS, 2003).

Mappings, or mapping strategy, are a set of connections between the gestural control and the

sound module.

By the nature of the mapping, it can be classified as (WANDERLEY, 2006):

• Implicit mapping: connections between input and output modules are defined by a process,

for example, machine learning, neural networks. It is considered a "black box approach"

(NORT; WANDERLEY, 2006). Examples of implicit mapping are Wekinator ((FIEBRINK;

TRUEMAN; COOK, 2009)), based on machine learning, and (CONT; CODUYS; HENRY,

2004), based on neural networks.

• Explicit mapping: the user explicitly defines the relationship between input and output by

analytically, or graphically, connecting two sets of variables (NORT; WANDERLEY, 2006).

Examples of explicit mapping systems are libmapper (MALLOCH; SINCLAIR; WANDERLEY,

2014), OSCulator (OSCULATOR, [s.d.]), and junXion v5 (STEIM, [s.d.]).

In addition to these systems, there are projects such as LoM (NORT; WANDERLEY, 2006), and

MnM (BEVILACQUA; MÜLLER; SCHNELL, 2005), that, despite using an interpolation process to

help the user map input and output parameters, the final representation can be modified

analytically (MALLOCH, 2008).

The explicit mapping’s advantage is allowing the fine-grained adaptation of the mapping by clearly

presenting analytic means to do so. Although limiting detailed modifications, the implicit mapping

strength is to encapsulate technical details related to mapping for the user.

By the number of inputs and outputs, the mapping can be classified (ROVAN et al., 1997) as:

How to choose a fitting sound synthesis? How to decide a suitable level of musical control?

22

• One-to-one: relationship in which each independent control of gesture is associated with a

musical output parameter. It is the simplest of the mappings, but usually the least expressive;

• One-to-many (divergent mapping): relationship in which an input parameter simultaneously

controls more than one musical parameter. In this mapping, few controls can possibly

generate many sound outputs. This may interfere in the sense of more expressiveness, but

limit the expressive interaction by not allowing the individualized manipulation of sound

details;

• Many-to-one (convergent mapping): relationship in which more than one input parameter

is associated with only one sound parameter. In this case, certain input variables can

influence (or modulate) the values of other input variables as well. Although more complex

than the other previously presented mappings, this convergence proves to be more

expressive than the unit mappings.

Because of their simplicity, a one-to-one mapping is considered to yield superficial user

interaction with the instrument. Research reveals (HUNT; WANDERLEY; PARADIS, 2003) that

the user feels more compelled to use a challenging musical interface than a simple one.

In an attempt to facilitate the creation of more complex mappings than the one-to-one,

intermediate layers between the controls and the musical outputs can be used. These layers aim

to process raw data by adding more semantics to their behaviors, so they were termed semantic

layers (MALLOCH, 2008). Semantic layers allow simpler visualization alternatives for instrument

designer since they encapsulate complex mappings in a one-to-one approach (HUNT;

WANDERLEY; PARADIS, 2003).

By the number of abstraction layers between the inputs and outputs, the mapping can be

classified as:

• Direct or one-layer mapping: direct connection between the raw values of the sensors and

the parameters of the synthesizers;

• Multilayer mapping: connecting raw data to intermediate layers that process values and add

meaning to them.

2.2.4. Feedback

Besides the sound feedback, DMI also presents:

• Haptic feedback: related to tactile, or kinetic references, usually related to shape, textures,

surface, or generated by vibration motors;

• Visual feedback: can be visible marks, which indicate or delimited areas of the instrument;

light sources, such as light-emitting diodes (LEDs); or images and visualizations that are

generated based on gesture or the resulting sound.

How to define an appropriate mapping strategy for a new DMI?

23

The absence of haptic feedback in DMI projects is considered a weakness when comparing to

acoustic instruments (MAGNUSSON; MENDIETA, 2007a), as it affects the embodied relation

between the musician and the artifact (PAINE, 2013). For instance, “A pianist can see and locate

a specific key before playing it, can use the resistance of the key action mechanism to help know

how hard to press the key, and can use the feeling of adjacent keys to keep track of hand position”

(DOBRIAN; KOPPELMAN, 2006). The kinetic feedback and tactile feedback can unfold additional

input channels to the brain (STEINER, 2005), therefore, helping to understand better how the

system works.

Using the generator of the feedback as a criterion, they can be classified as:

• Passive: inherent in the interaction device itself, without using any actuator, such as the

weight of piano keys, the click of the mouse, or the feeling of displacement of a computer

keyboard key (WANDERLEY, 2006);

• Active: when there is an actuator device producing movement, force, light, such as a

loudspeaker, motor, LED, etc.

In addition, depending on its source, the feedback can be:

• Primary: generated directly by the gestural control;

• Secondary: generated from the sound module.

2.2.5. Summary

In contrast to acoustic instruments, which are physically constrained by their body form that

produces sound, the DMI design allows more freedom, since the sound is digitally synthesized

by an independent module of the control interface. In DMI design, "each link between the

performer and the computer has to be invented before anything can be played" (RYAN, 1991).

The decoupling input-output allows a broad range of possibilities which enlarges the design

space, and, consequently, its exploration. Paradoxically, these numerous possibilities can lead

to creative paralysis, or difficulty to find proper ways to begin ideas exploration (MAGNUSSON,

2010).

2.3. The Challenge of Expressivity and Virtuosity An important property of a musical instrument is to enable the performer to be expressive

(MEDEIROS et al., 2014), or to “effectively convey meaning or feeling” (MERRIAM-WEBSTER,

2004). For that, it is essential that the instrument allows the performer to have a subtle control

over some features of individual notes and musical phrases, such as timing, volume, timbre,

accents, and articulation (DOBRIAN; KOPPELMAN, 2006).

How to choose the suitable feedback modality for a new DMI?

In sum, how to deal with the multiple possibilities of the DMI design space exploration?

24

In fact, for musical expression, it is not only a matter of having an instrument with an excellent

control interface, but it should also provide room for the development of control intimacy. With

high levels of control intimacy, the player can embody the instrument, i.e. “there is a transparent

relationship between control and sound” (FELS, 2004).

Virtuosity is the high developed technical skill “that enables the player to master so well the subtle

controls of the instrument that he/she can perform other cognitive activities as the music

interpretation” (MEDEIROS et al., 2014). A virtuoso does not only successfully realize a highly

difficult task, but she does it with expressivity (GUREVICH, 2009).

Acquiring a level of virtuosity with an instrument demands several years of practice

(WANDERLEY; ORIO, 2002). This fact raises concerns about the adoption of a new DMI, and

what makes it attractive for new players. In fact, Wessel and Wright (2001) discuss that it is

important for the instrument to have a “low entry fee with no ceiling on virtuosity” (WESSEL;

WRIGHT, 2001), i.e. allowing newcomers to explore the instrument with immediate musical

results, and providing room for expansion of skills and expression.

2.4. The Challenge of Evaluation and Evolution Evaluation is a critical component of the design process (BUXTON, 2007) (LOWGREN;

STOLTERMAN, 2004). In DMI, it can help the evolution of an instrument and also aids the

comparison of one instrument with another. In literature, much has been discussed about

evaluation methods: from adopting techniques from HCI (WANDERLEY; ORIO, 2002), using

qualitative approaches (STOWELL et al., 2009), focusing on the performer (BARBOSA et al.,

2011), considering the audience (BARBOSA et al., 2012), or providing tools for classifying DMIs

based on dimension spaces (BIRNBAUM et al., 2005). O’Modhrain (2011) propose different

evaluation approaches depending on the stakeholder (O’MODHRAIN, 2011). Additionally,

Barbosa et al. (2015) discuss that the term “evaluation” has different understandings within the

community (BARBOSA et al., 2015b).

Evaluation in DMI design appears to be hard because there are many levels of interacting

components and layers of complexity causing the criteria of success to be not clear (DAHL, 2016).

For that, Medeiros et al. (2014) summarize some literature topics, suggesting two categories of

criteria for success (MEDEIROS et al., 2014). One category covers how effectively the artifact

matches a given context of use, e.g. ergonomics, sound quality, visual and haptic feedback

(MONTAG et al., 2011), fine-grained gesture control, embodied relationship (ESSL;

O’MODHRAIN, 2006), efficiency, learning curve (JORDÀ, 2005a). The other category covers

criteria that normally requires a long time to be assessed, such as, instrument adoption by a

How to design a DMI that allows a novice to obtain musical results and potentially become a new user? How to design a DMI that provides room for continuous evolution of player’s techniques?

25

community, lifetime (PAINE, 2009), integration with an existing genre/style or the creation of a

new genre/style (GUREVICH, 2009), and commercial success.

In fact, studies point that evolution of acoustic instruments is based on an evolutionary process

of trial-and-error (WASSERMAN A.; CULLEN, 2015). Centuries have forged the musical artifacts

to which we are used today. Will digital musical instrument follow this process?

How to evaluate DMIs given not established criteria for success? How to evaluate DMIs considering a long time? How to accelerate the DMI design process to create and learn more in less time?

2.5. The Challenge of No Previous Knowledge Moreover, there is almost no musical repertoire created to draw the attention of potential adopters

or to contribute to the technical advances of the instrument (OORE, 2005).

Understanding, adopting and learning a conventional instrument relies on an existing body of

knowledge that comprises: (1) advanced users or virtuoso players, who demonstrate the potential

of the instrument to new players and develop playing techniques that serve as reference; (2)

musical repertoire, the set of compositions for that instrument and performed using it; (3) methods

of learning the instrument, a compilation of playing techniques presented in a structured form

focusing on practice techniques that worked previously (MEDEIROS et al., 2014).

This body of knowledge is constructed, updated and consolidated over the years, turning the

instrument into a cultural object (KVIFTE, 1988). The artifact holds an aesthetic discourse or

attitude, a regional load, an event connection, and a specific genre or style association (and even

an emotional relation).

In fact, these aspects can be crucial to foster the instrument adoption, since "some started playing

after having been inspired by some music and, in particular, the sound of the instrument"

(GREEN, 2002).

2.6. The Challenge of Multiple Stakeholders and Contexts of Use

As mentioned by (WOOD, 1997), a critical ingredient for designing systems is "understanding

potential users". DMI design is a multidisciplinary area that assembles artistic and technical

creation (JORDÀ, 2001). Performer, composer, designer, audience, manufacturer, and customer

are some of the stakeholders in the DMI context (KVIFTE; JENSENIUS, 2006) (O’MODHRAIN,

2011). Therefore, we can consider that DMIs have not only a user but an ecosystem that should

be taken into account during design. According to Payne, “any implementation of a new musical

interface must, therefore, consider the ecology of this environment” (PAINE, 2013).

How to deal with the lack a body of knowledge of a new DMI?

26

For instance, designing significant and evident connections between gestural input controls and

sound is not only a matter for the musicians, but also for the audience (O’MODHRAIN, 2011).

The definition of transparency is “the psychophysiological distance, on the player and the

audience minds, between the input and output of a device mapping” (MURRAY-BROWNE et al.,

2011). Thus, if the transparency is “opaque”, it is hard for the audience to engage with the

performance (FELS; GADD; MULDER, 2002).

Considering the performer, “good musical instruments must strike the right balance between

challenge, frustration, and boredom” (JORDÀ, 2005b). Besides that, there are different contexts

of use: the performer can do a solo or play in an ensemble; can improvise, accompany others,

play a predefined score; and can play distinct songs, repertoires, and genres.

Performing simple gestures with little effort, the musician can trigger a set of notes, pre-recorded

samples, and sound effects. Paine (2013) presents two different activities that the performer

engages during the interaction with the DMI: creation, when triggering notes and altering

parameters of sound synthesis; and control when triggering process such as samples and loops

(PAINE, 2013).

This approach has a strong relation with the conceptual framework presented by Malloch et al.

(2006) that shows three abstraction levels to categorize the performance behavior (MALLOCH et

al., 2006). The skill-based level is the rapid coordination of movement to manipulate signals in a

continuous way (e.g. someone playing violin). The rule-based level is “selection and execution of

stored procedures” (e.g. drag-and-drop pieces of music). The model-based level is the highest

level of abstraction in which the behavior goes towards a conceptual goal (e.g. live coding).

Jordà (2004) describes three levels of instrument diversities: micro-diversity, or performance

nuances, measures how to performance of the same piece can differ; mid-diversity, or

performance diversity, given one instrument how diverse is two performances played with it; and

macro-diversity, or stylistic diversity, covers how flexible one instrument is in different contexts

(JORDÀ, 2004a).

Beyond the performance, there are other contexts of use such composition; musical education;

musical therapy; musical toys, musical installations etc. Each context may demand specific

requirements or properties of the instrument; it may influence the musician satisfaction, and,

consequently, the instrument refinement and its adoption (KVIFTE; JENSENIUS, 2006) (PAINE,

2013). Those multiple contexts can substantially influence the experience in use and deal with

them simultaneously during the design process can be complex.

How to consistently include the stakeholders’ views in the design process? How to design a DMI in view of different contexts of use and diversity of the artifact?

2.7. Final Considerations From the previously discussed challenges, we can conclude that the DMI design context presents

various levels of complexity: from the micro level of mapping possibilities, passing through

27

behaviors in performance, contexts of use, and stakeholders, to the macro level of cultural

aspects. Besides, the criteria for success are loose and depend on context.

In design community, much has been discussed about a class of problems called wicked, or ill-

formulated, or ill-defined problems (BUCHANAN, 1992) (CROSS, 2006). It consists of a “class of

social system problems which are ill-formulated, where the information is confusing, where there

are many clients and decision makers with conflicting values, and where the ramifications in the

whole system are thoroughly confusing” (BUCHANAN, 1992). In fact, there are already studies

that define DMI or NIME design as a wicked problem (DAHL, 2012) (DAHL, 2016).

A characteristic of these problems is that normally they do not have a definitive formulation, and

the problem definition always comes in pair with the problem-solving. In that situation, cycles of

idea exploration and prototyping play a major role in the process (JOBST; MEINEL, 2014) (VON

THIENEN; MEINEL; NICOLAI, 2014), as the cycles are not only meant to generate, test, and

communicate ideas but also to interpret and create meaning, sometimes redefining the problem

after learning in the process (FALLMAN, 2003).

In DMI context, the artifact is not an end, but a means to produce sound and music. To have a

clearer understanding of how the DMI behaves and validate if it is adequate to an intention or

context of use, it is important to have real-time audio feedback of the control gestures.

Accordingly, conventional low-fidelity non-functional prototype tools and methods employed in

HCI, such as the paper prototype or Wizard of Oz (BUXTON, 2007), are not entirely suitable for

the DMI context. For a richer comprehension, DMI prototypes should be functional, reacting to

player’s actions in real-time (DAHL, 2016).

However, the development of functional prototypes demands more time and effort, and

consequently, they can become a hindrance for iterative design (HUOT, 2013). During the design

process, more iteration leads to mature results to the context of use and intention. The more

cycles of prototyping the process has, the better the results are expected to be (BROWN, 2009).

To deal with the wicked nature of DMI design and to focus on designing DMIs for a broader

adoption, we raise two questions for guiding our research:

• Regarding the complexity of the DMI design space, how to provide structured and exploratory

paths for generating ideas of new DMIs?

• Considering the functional requirement of DMI prototypes, how to decrease the time and effort

of building them?

In the next chapter, we deepen our discussion in the design process based on design literature,

focusing on the phases of idea exploration and prototyping.

28

3. DESIGN PROCESS "Design is the creation process through which we employ

tools and language to invent artifacts and institutions. As

society has evolved, so has our ability to design." (OWEN,

1993).

The purpose of this chapter is to introduce the concept of design process and its phases, focusing

on idea exploration and prototyping. Metaphorically, we would like this chapter to be a pair of

glasses, through with we can analyze initiatives in DMI literature (discussed in the next chapter)

that address the conception and implementation of new DMIs. Diving into the metaphor, idea

exploration would be one of the lenses of our glasses and prototyping would be the other (Figure

3.1).

Figure 3.1: Understanding the design process focusing on idea exploration and prototyping

The design process can be understood as a sequence of activities, or methods, that are

performed in series, or in parallel, in order to design something (CROSS, 2000) (JONES, 1992).

Besides guiding the designer to take decisions, the design process can be used to explain the

designer’s activities to users, collaborators, and students (DUBBERLY, 2010).

Cross (1984) explains that logical analysis and creative thought are present and necessary for

the design process. By using structured and clear ways of understanding the process, these two

forms of thinking can take place without cognitively loading designer’ mind or relying on the

designer’s inner inspiration timing (CROSS, 1984).

In Figure 3.2, we present examples of processes from different areas such as engineering design,

mechanical design, creative thinking, user-centered design, and innovation. Finally, inspired by

these diverse design processes, we propose a set of phases that can be used to analyze the DMI

design context.

29

Figure 3.2: Examples of Design Processes’ Phases based on literature. (CROSS, 2000), (PAHL et al.,

2007), (DUBBERLY, 2008), (PUCCIO; CABRA, 2012), (BROWN, 2008), (IDEO, 2011), (MAURYA, 2012), (NEVES, 2014)

The proposed phases are used in this work for analysis only, and it is not a definitive or

comprehensive proposition for DMI design process. Additionally, we use the concept of design

space as a constrained set of design possibilities that leaves some dimensions open for

exploration (BEAUDOUIN-LAFON; MACKAY, 2000).

• Problem / Design Space Definition: related to understanding the concepts of musical

instruments, such as the stakeholders, the scenarios, the common knowledge of the area,

understanding the user’s intention and context of use, and defining the restrictions that define

the design space of the project.

• Idea Exploration: related to exploring possible paths in the design space, generating and

selecting ideas that conform to user’s intentions and contexts of use.

• Prototyping: related to concretizing the abstract ideas into an artifact that can be utilized,

and tested, with which the user can interact.

• Evaluation: related to validating whether the artifact is adequate to user’s intention and

context of use.

Figure 3.3 illustrates how we understand the relationship between the proposed phases of DMI

design process.

30

Figure 3.3: Proposed phases for analyzing DMI design process

In the next sections, we will focus on idea exploration and prototyping, since these phases are

emphasized in this work.

3.1. Idea Exploration “Design conceptualisation can be defined as creating an

idea, gradually maturing its meaning and eventually

expressing the understanding through representations like

words, drawings or models.” (CAPJON, 2005)

Ideation methods provide a prescription (normative

procedure) on how to overcome certain blocks to creativity

(HERNANDEZ; SHAH; SMITH, 2010)

Idea Exploration can be understood as the process of conceiving and testing ideas that can be

useful for the accomplishment of some desired result (REINIG; BRIGGS, 2008) (PUCCIO;

CABRA, 2012).

We consider that idea exploration comprises idea generation (also called “ideation” in some

references) and idea selection (also mentioned as “idea evaluation” (PUCCIO; CABRA, 2012)),

that are respectively divergent, and convergent ways of thinking about solutions based on the

design space. According to Puccio (2012), creativity results in novel and useful outcomes, and

idea generation and idea evaluation can be respectively related to the search for novelty and the

pursuit of usefulness.

It is important to make a clear differentiation between idea evaluation and the design process

phase, evaluation. The latter concerns the validation of a tangible artifact and its attainment to

the outcome intentions, and the former still on an embryonic, or abstract stage of exploration.

31

In an attempt to compare the idea generation methods, Shah et al. (2000) present a classification

based on the essence of the methods (Figure 3.4) (SHAH; KULKARNI; VARGAS-HERNANDEZ,

2000). For the authors, intuitive methods focus on fueling the unconscious thinking, and logical

methods rely on rational approach towards a problem.

Figure 3.4: Classification of idea generation methods adapted from (SHAH; KULKARNI; VARGAS-

HERNANDEZ, 2000)

Although understanding fewer details about how intuitive methods affect the designer’s mind,

these methods are related to inducing more novel results (SHAH; KULKARNI; VARGAS-

HERNANDEZ, 2000). The subdivisions proposed by the authors to classify intuitive methods are:

• Germinal: methods intended to be used when the designer has no previous solutions or

ideas, as an initial step;

• Transformational: methods that transform existing ideas to generate new ones;

• Progressive: methods based on applying repetitive steps;

• Organizational: methods that help the designer to group the already generated ideas;

• Hybrid: combination of the aforementioned methods.

For logical methods, the subdivisions are:

• History based: methods that leverage existing solutions, which are typically compiled in

catalogs or archives;

• Analytical: methods based on systematical exploration of relations, causes and effects, and

wanted or unwanted characteristics of the already generated ideas.

According to Shah et al. (2000), key components of an idea generation method are "mechanisms

that are believed to promote idea generation intrinsically or to help designers overcome specific

32

mental blocks". The authors surveyed literature in cognitive psychology and engineering design

to recognize these components from a variety of ideation methods and collect evidence of their

usefulness. The authors provide the following list of idea generation promoters:

• Combinatorial Play or Synthesis: allow the combination of parts, modules, components, or

other ideas to achieve new results.

• Use of analogies and metaphors: mapping between familiar aspects of an item into an

unfamiliar context;

• Imagery/Sketching: presence of pictorial representation during the ideation;

• Feedback: continuous feedback whether the generated ideas are leading towards the goals;

• Constraints: imposed limits that allow the designer to focus on specific aspects of the set of

possibilities;

Additionally, idea generation “tackles” are components that help to overcome mental blocks.

Some examples are mentioned by Shah et al. (2000) and Hernandez et al. (2010):

• Provocative Stimuli: Display correlated and uncorrelated materials to designers, in the form

of images, texts, sounds, objects.

• Suspend Judgment: Delay early decisions that may put ideas away;

• Flexible Representation: use means that can be easily understood changed such as

graphical representation.

• Frame of reference shifting: modify the way the goals of the project are being absorbed

visually or understood;

• Incubation: force the designer to delay some aspects of the ideation process to allow

unconscious processing to happen.

• Example exposure: related to Provocative Stimuli, but in this component, solutions for the

same problem are shown to the designer. Literature may indicate that the presence of this

component may cause design fixation, which is when the designer unconsciously focuses on

specific aspects of an artifact, neglecting others, thus negatively interfering on creative

outcome (VASCONCELOS; CRILLY, 2016).

• Random connections: allow random combinations or connections between the explored

elements;

• Emphasis on quantity: focus on the generation of a high number of ideas. However, Reining

and Briggs (2008) argues that relation between the number of ideas and the quality of ideas

are not always direct due to cognitive and solution space limitations (REINIG; BRIGGS,

2008).

• Emphasis on variety: focus on producing results that have diverse characteristics from each

other.

Figure 3.5 presents five examples of idea generation methods highlighting their key components.

33

Figure 3.5: Comparison between idea generation methods adapted from (SHAH; KULKARNI;

VARGAS-HERNANDEZ, 2000)

Smith (1998) presents the concept of active ingredients in idea generation methods. The author

analyzed 172 methods of idea generation and distilled 50 categories of prescribed actions by

which idea generation techniques affect the designer thinking (SMITH, 1998). The following mind

map (Figure 3.6) shows an abbreviated version of the ingredients.

Figure 3.6: List of Active Ingredients for Idea Generation adapted from (SMITH, 1998)

3.2. Prototyping “When something is truly novel, we cannot plan it into

existence, but we need experimentation to learn through

trial-and-error.” (PASSERA; KAERKKAEINEN; MAILA,

2012)

Prototypes are the reduced implementation of an idea focusing on some aspect such as function,

or form. Gill (2011) discusses how the term prototype is used in different areas. Industrial

34

designers tend to focus more on appearance, engineering designers normally are more

concerned with the artifacts’ functionality, and for an interaction designer or software engineer,

the prototype can mean a simulation of a user interface, for instance. (GILL; SANDERS; SHIM,

2011).

Due to these diverse areas, Gill (2011) concludes that trying to find a common understanding is

hard, but some aspects seem to be recurrent such as the objective (explorative or evaluative

prototypes), or the level of prototypes details (low-fidelity or high-fidelity) (GILL; SANDERS; SHIM,

2011). Exner (2015) proposes three categories to investigate the prototyping process (EXNER et

al., 2015): objectives (explorative, experimental, evolutionary), dimensions (form, material,

concept, principle, process, functions, requirements), and fidelity (high, low, mixed)

Describing dimensions in detail (EXNER et al., 2015):

• Form study: related to visual features of an artifact and its final look

• Material study: related to the experimenting different materials to accomplish the desired

look-and-feel

• Proof of concept: to verify feasibility of concepts (for example, the use a new kind of

technology)

• Proof of principle: to check the applicability of a principle (e.g. biological principles)

• Proof of process: in the context of service development, is related to checking the

procedure, and completeness of a service.

• Proof of function: to verify if specific functions or requirements have been satisfied.

Prototyping objectives:

• Exploratory: focus on the use of quick and straightforward prototyping methods, such as

paper prototype, in order to explore initial ideas and concepts. It is also called throw-away

prototypes because the important part is not the material, but the knowledge gained during

the process.

• Experimental: aims to verify if principles and requirements are being fulfilled. Normally,

happens in later stages in the design process.

• Evolutionary: is intended to be used as a scaffold that is constantly being modified and

enhanced. “e.g. by automatically digitalizing sketches of user-interfaces in a working

smartphone app” (EXNER et al., 2015).

Prototype fidelity is related to the level of details and similarity with the final product (EXNER et

al., 2015). Although fidelity is a term often used in literature, Beaudouin-lafon (2000c) discusses

that precision is preferable, because it focuses on the prototype itself and not on a product that is

not defined yet (BEAUDOUIN-LAFON; MACKAY, 2000). The level of fidelity or precision of

prototypes can be classified as:

• Low: simplified representation with limited details, focusing on few aspects of the idea.

Normally associated with initial stages of the design process, this level of prototyping is

35

considered to be useful for collecting usability data at a low cost (GERBER; CARROLL,

2011).

• High: a greater number of details, focusing on more than one aspects of the idea. In

engineering design or industrial design, it may be associated with the concept of not being

distinguishable from the final manufactured artifact (HORVAT, 2011).

• Mixed: combines elements of low-fidelity and high-fidelity prototypes in one approach. For

example, when sketches are used as a representation for an interactive prototype of a mobile

user interface (LIM; STOLTERMAN; TENENBERG, 2008).

Additionally, Nakamaru (2016) expands the concept of fidelity with a two-dimensions classification

considering the axes appearance and function (NAKAMARU, 2016) (Figure 3.7). With this

categorization, the author highlights two combination possibilities, appearance, and functional

prototypes, that probably were blurred in the one-dimensional fidelity classification. Further, in the

text, we decided to use this classification for comparing different prototyping tools for DMI.

Figure 3.7: Two dimension prototype classification proposed by (NAKAMARU, 2016)

Concerning the benefits of using prototypes, Angesleva (2016) enumerates eight topics (ÄNGESLEVÄ et al., 2016a):

• Help to understand complex concepts

• Allow the visualization of abstract ideas

• Enhance communication, since they remove cultural and linguistic barriers

• Exercise the focus, as they are built to experiment specific points

• Test functionalities and related them to requirements

• Build the ground on which other ideas can develop

• Refine users’ interests

• Allows a better comprehension of users’ interaction

36

According to (DOW, 2011), prototyping plays a central role in product design and interaction design, since it helps to identify flaws in the artifacts, to redirect and adjust them, to have a better

understanding of how they work and to generate new ideas (WARFEL, 2009).

As said by Gauntlett (2014), during conceptual design it is difficult to hold all detailed, and

complex information about an idea in our head at once (GAUNTLETT, 2014). Donald (2002)

introduces the concept of ‘external memory field’, which can be a note, a drawing, or a physical

model, that enhances our ability to deal with those complex details, and unburdening our minds

(DONALD, 2002).

Accordingly, Beaudouin-lafon (2000c) states the importance of prototyping by highlighting the

capability of generating “concrete representations of new ideas and clarifying specific design

directions” (BEAUDOUIN-LAFON; MACKAY, 2000). Besides, the author emphasizes the

tangibility of the prototype, stating that: “[prototype is] not an abstract description that requires

interpretation." (BEAUDOUIN-LAFON; MACKAY, 2000)”.

Additionally, prototypes play a central role in the visualization of ideas, since they transform

thoughts into tangible representations that can be easily read and shared (PASSERA;

KAERKKAEINEN; MAILA, 2012).

Much has been discussed in the literature about the benefits of producing numerous concepts

during the design process (ELSEN et al., 2012). Often, the quantity of ideas is associated with

quality of creative outcomes. However, studies suggest that there is a trade-off of quantity and

creative results. Relatedly, it seems to be well-stablished that more iteration leads to mature results to the context of use and intention (CAMBURN et al., 2015). The more cycles of

prototyping the process have, the better the results are expected to be (BROWN, 2009).

Israel et al. (2016) performed a focus group with prototyping experts from different areas in order

to discuss the future perspective of prototyping (ISRAEL; BÄHR; EXNER, 2016). Topics such

as hybrid and modular prototypes were associated with blurring the barrier between physical and

digital prototypes, leading to a deeper validation of user experience. Another highlight was the

importance of quick changes between physical and virtual prototypes, focusing on the evaluation

of the interaction between the user and the artifact.

In the dynamic contexts of interactive applications, where it is important to assess the responsive

nature of the system, the literature highlights limitations of the use of low-fidelity prototyping

approaches (LIM et al., 2013). Therefore, functional prototypes seem to be the most suitable

option to study interaction, due to the continuous cycles of action and reaction between user and

system.

However, the development of functional prototypes demands more time and effort (LIM et al.,

2013), besides requiring more technical expertise from the designer (ROECK et al., 2013).

Consequently, they can become a hindrance in the design process (HUOT, 2013).

To illustrate the time and effort to build functional prototypes, Sadler (2016a) studied an expert

during the prototyping process of a smart shoe. The prototyper used a wearable camera on his

37

chest during all the development. The results show that the expert took more than 15 hours to

produce a functional prototype, and used more than 30 different tools. Besides, there are shifts

of contexts such as dealing with structure, electronics, and programming (SADLER, 2016).

Nakamaru (2016) depicts the functional prototyping as a customer journey map divided into five

stages: Planning, Preparation, Development, Use, Keep/Destroy (NAKAMARU, 2016).

According to Sadler (2016a), that the number of technical interruptions during prototyping

negatively affects the creative process, since it generates an increase in cognitive load, and

deviate the user from the task of creative experimentation (SADLER, 2016).

“[Thomson's Rule for First-Time Telescope Makers] It is

faster to make a four-inch mirror then a six-inch mirror than

to make a six-inch mirror." (BENTLEY, 1985)

Isa et al. (2015) emphasize that literature is divided into whether prototypes should be used in the early stages of design or not. The authors who state the latter emphasizes the high cost

and time to build the prototypes, suggesting to using prototypes only when actually needed. On

the contrary, other researchers defend prototyping in early stages, because they enhance the

communication between the users, the designers, and possibly other stakeholders (ISA; LIEM;

STEINERT, 2015).

Elsen (2012) recommends that the “prototyping should begin as soon as possible during the

design process” (ELSEN et al., 2012). The author mentions that a commonly performed practice

in the preliminary stage of the design process is to use low-fidelity prototypes since they are quick

to build with associated low cost.

“Rapid and early prototyping enables learning through

making.” (SANDERS, 2013)

Furthermore, Valamanesh et al. (2013) defend that prototypes should be built in the early steps

of the design process, and can also be used as tools for idea generation, “since a physical artifact

enables designers to be exposed to unlimited perspectives and combinations” (VALAMANESH;

SHIN, 2013). Viswanathan et al.’s (2015) study confirms that prototyping physical artifacts aids

to remove incompleteness in initial ideas leading to better outcomes (VISWANATHAN et al.,

2015). Besides, Youmans et al. (2011a) emphasize the property of reducing the cognitive load in

artists and designers, as prototypes can store ideas for further development (YOUMANS, 2011).

3.3. Final Considerations In this chapter, we presented the definition of the design process, compared different descriptions

of its phases, and summarized them in four phases that we propose to use in the DMI context:

design space definition, idea exploration, prototyping, and evaluation. Based on the scope of this

work, and relating to our first question, we focused on the description of idea exploration, which

comprises the cycles of idea generation and idea evaluation, and prototyping. We highlight that

38

idea generation methods are a set of systematic steps whose components or active ingredients

foster the creative thinking and/or help to overcome mental blocks.

We discussed that prototyping is important to the design process because it does not only

concretize ideas into partial implementations that can be evaluated, but it also impacts the idea

generation, redefining initial impressions and understandings. Prototypes should be built in early

stages of the design process, as quick as possible, and with as many cycles as possible because

the number of cycles is often associated with mature outcomes. We also argued that functional

prototypes provide a better comprehension of the ideas, but they are often more difficult to build.

With the concepts discussed in this chapter, we aimed to be equipped with the necessary

understanding to analyze the DMI literature in search of initiatives that deal with the phases of

idea exploration and prototyping.

39

4. STATE OF THE ART In this chapter, firstly, we present initiatives in DMI literature such as concepts, principles,

frameworks and processes that can be related to DMI design, and specifically, we look for

structured steps for idea exploration. Then, we analyzed commercial products and DMI literature

in search for prototyping tools that can be suited for DMI prototyping phase. Following our glasses

metaphor, in sum, we will discuss initiatives and tools focusing on idea exploration and

prototyping (Figure 4.1).

Figure 4.1: Analysis of DMI literature and commercial products focusing on idea exploration and

prototyping

4.1. Frameworks and Approaches for DMI Design To help us analyze different approaches or initiatives in DMI literature, we opted to define some

categories inspired by definitions presented by O’Modhrain (2011) (O’MODHRAIN, 2011). We

use the term conceptual framework as a theoretical set that describes the relationship between

elements of a certain design context. We understand that a concept is an isolated piece of

knowledge, which can be part of a framework or be presented alone as the result of a study or

discussion. Principles, guidelines or recommendations (that can also be part of frameworks) are

attempts to transfer experience in the form of points to be achieved (often they do not show how

to achieve the point). Finally, we consider that processes and methods are a set of structured

steps towards a specific goal.

In early years, only academic laboratories had access to expensive computers and sensor

platforms that could provide the immediacy response between gestural controls and sound

production (BATTIER, 2000). With the popularization of personal computers and their increasing

processing power, the real-time interaction became available for a broader audience. This fact

seems to have influenced the growing discussion in the last years about DMIs, or NIME.

40

Jordà (2004) argues that low-level research that tries to solve only parts of DMI design is

necessary, but insufficient and integral studies that carry a holistic view of the area should be

conducted, since "very few attempts are being made" (JORDÀ, 2004b). Bongers (2007) states

that the literature in the area lacks established "guidelines and approaches for [the] complexity"

(BONGERS, 2007). Restating that this issue might not have changed in time, Jordà (2014)

mentions that "general and formal methods that go beyond specific use cases have probably not

yet emerged. Will these be the El Dorado or the Holy Grail of NIME research?" (JORDÀ; MEALLA,

2014).

"While standardisation may not be a must (maybe not

even desirable), highly idiosyncratic instruments which are

often used only by their creators may not be the best sign

or strategy for a serious evolution in this field." (JORDÀ,

2004c).

As a direct consequence of this lack of established guidelines or methods, the conception and

implementation of some instruments might not leverage the accumulated experiences and best

practices from the others.

By analyzing the literature, we could realize the effort of the community in defining frameworks,

concepts, principles, guidelines, and recommendations in DMI design. However, only a few

references present processes or methods that could guide the designer into an initial path in the

design process.

Some authors present individual contributions and reflections in the form of concepts, such as

Fels et al. (2002), who define transparency, or the ease of understanding the mappings of an

instrument, that can be achieve with the use of metaphors and it is related to instrument

expressivity (FELS; GADD; MULDER, 2002). Besides, Essl and O’Modhrain (2006) came to the

conclusion that the sensorimotor experience is an important point for taking into account when

developing engaging musical instruments (ESSL; O’MODHRAIN, 2006).

Wanderley and Orio (2002) provide a set of recommendations to formulate better ways to define

musical tasks that can be tested using established concepts and methods from HCI

(WANDERLEY; ORIO, 2002). Additionally, Blaine and Fels (2003) introduce a list of "elements of

design" related to collaborative musical interfaces (BLAINE; FELS, 2003).

Additionally, Cook (2001) and Cook (2009) present a set of principles (Figure 4.2) for designing

computer music controllers based on the author's experience with related projects (COOK, 2001)

(COOK, 2009). The principles cover topics related to artistic, and technical aspects of instruments

development. As explained before, these principles are abstract design goals, but they do not

present detailed information on how these goals should be accomplished (O’MODHRAIN, 2011).

41

Figure 4.2: Set of Principles proposed by Perry Cook. Adapted from (COOK, 2009)

Additionally, Birnbaum et al. (2005) present a graphical tool that aims to facilitate communication

and support design decisions (BIRNBAUM et al., 2005). Their proposed dimension space (Figure

4.3) consists of a radar graph with seven-axis representing comparative aspects of DMIs such as

required expertise to play, musical control, feedback modalities, degree of freedom, inter-actor,

distribution in space, and role of sound.

Figure 4.3: Dimension Space proposed by (BIRNBAUM et al., 2005)

The mentioned initiatives contribute to DMI design by attempting to explain the components of

the area and their relationship. They provide little or no initial structured path for a designer to

42

start conceiving and developing a DMI. These concepts and frameworks are more suitable for

validating ideas or classifying an already built instrument (a posteriori) than generating them (a

priori). In fact, Marquez-Borbon et al. (2011) highlight that NIME literature shows attempts to

“categorize and situate existing or newly designed musical devices in the growing body of

exemplars”, but emphasize that there only a few tries going towards generative approaches

(MARQUEZ-BORBON et al., 2011).

Miranda and Wanderley (2006) present a list of steps normally applicable for designing a new

DMI (Figure 4.4) (MIRANDA; WANDERLEY, 2006). The approach is more suited as a

recommendation for delimiting the design space than a method for idea exploration, as it gives

little guidance on how to accomplished the steps. It is important for the designer to know that is a

fundamental information to have, but she typically needs to find her proper inspiration for

generating the ideas.

Figure 4.4: Topics on designing new digital musical instruments extracted from (MIRANDA;

WANDERLEY, 2006)

Drummond (2009) studies the context of interactive music systems and formulates a set of

definition, classification, and models in an attempt to summarize different views for what the

author call a cross-disciplinary field (DRUMMOND, 2009).

O'Modhrain (2011) reflects about different evaluation methods for distinct stakeholders in the DMI

context, such as performers, audience, composers, designer, manufacturer, and customers

(O’MODHRAIN, 2011). The author guides the evaluation by proposing methods related to each

stakeholder. This is an example of a method used to validate and, logically, for validating one

expects to have already an idea. This is supported by Jordà et al. (2014), who mention that the

discussion regarding evaluation can influence instrument design (JORDÀ; MEALLA, 2014), but,

in general, the evaluation deals with a posteriori aspects, and possibly lacks elements of formative

or generative thinking.

43

Morreale et al. (2014) gather different concepts from the literature to build a unifying framework

for digital instruments and musical installations, called MINUET (MORREALE; ANGELI;

O’MODHRAIN, 2014). The authors organize the topics into two major groups: Goal and

Specifications. By describing people, contexts, activities, and technologies, the designer would

have covered the core elements in DMI conception. The framework diagram is shown in Figure

4.5. Although MINUET reduces the complexity of DMI design space, the designer has to generate

the ideas and then "plug" the ideas on the framework. For example, the framework does not

provide a set of examples of technologies, or contexts, (even if this initial description was

superficial) that the designer can choose from and start thinking about them.

Figure 4.5: MINUET framework extracted from (MORREALE; ANGELI; O’MODHRAIN, 2014)

Wallis et al. (2013) discuss the property of long-term engagement of musical instruments and

what can be learned to apply in HCI development process (WALLIS et al., 2013). The authors

present a set of heuristics meant to be used in HCI projects from idea generation throughout the

evaluation. The heuristics cover three aspects of long-term practice with musical instruments:

mastery, autonomy, and purpose. Although the authors present the heuristics as a method for

idea generation, the descriptive nature of the list makes it closer to the definition of the design

space.

44

Fyans et al. (2012) apply participatory design to engage performers and spectators into the design

process in an attempt to extract relevant information about the usage and perception of the

instruments (FYANS et al., 2012). The authors point out that since the design of DMI is an open-

ended process, the definition of a goal is generally subjective. This yields in issues when defining

the modifications within the design cycles.

Besides, (SYLLEROS; DE LA CUADRA; CÁDIZ, 2014) presents an instrument created based on

a cyclic design process centered in a group of users. The authors discuss concepts of user-

centered design, personal identities, and interactive behaviors to establish their method, and also

rely on users focus groups to discover requirements that inform the proposed solution.

Additionally, one of the design process phases takes into account the number of movements of

user's joints after a sound stimulation session. Considering the osteokinematic motions, user’s

movements are ranked as presented in Figure 4.6. Even though a structured method is

presented, and imposes some restrictions in the search space of user’s gestures, the authors do

not discuss the aspects that lead them to generate the ideas of the resulting instruments.

Figure 4.6: Osteokinematic motions and the percentage of motions performed by the users

(adapted from Sylleros et al. (2014))

In sum, there are numerous attempts to build up a body of knowledge on DMI design but few

structured processes and methods which lack paths on how to achieve the mentioned results

regarding instrument ideas. Although the discussions about defining elements of the area are

necessary to reduce the complexity of DMI design space, we believe that structured and

exploratory paths could help DMI designers to conceive instrument ideas, and also could

contribute to kick start and accelerate new DMI projects.

In the context of engineering design, Pahl et al. (2007) highlight the importance of having explicit

and structured steps during the design process (PAHL et al., 2007). The authors present that the

weaknesses of not following a structured method are: “the right idea rarely comes at the right

moment”, “the result depends strongly on individual talent and experience”, “there is a danger

that solutions will be circumscribed by preconceived ideas based on one’s special training and

experience”. That is why we find it important to provide methods for idea generation.

45

4.2. Functional Prototype in DMI Design There is a considerable amount of hardware and software tools already available that could be

used to build functional DMI prototypes. Some examples are:

• microcontrollers environments (Arduino, Raspberry Pi, Beaglebone, Teensy),

• sensor kits (Infusion Systems, littlebits, makey makey)

• MIDI controllers (keyboard, wind controllers, percussion controllers)

• general purpose programming languages (C, C++, Java)

• audio-oriented programming languages (CSound, SuperCollider, Chuck, Pure Data,

Max/MSP),

• creative programming environments (Processing, openframeworks, Cinder, Scratch)

• applications for mappings (libmapper, iCon, OSCulator, juxion, Wekinator), and

• digital audio workstations (Logic Pro, Ableton Live, Pro Tools, GarageBand, Reaktor,

Tassman).

The designer must, therefore, be a polyglot in a plethora of technologies to be able to "hack",

adapt and integrate specialized tools effectively to obtain non-trivial functional prototypes

(HARTMANN et al., 2005). Since the tools are fragmented, the shift of context between multiple

development areas (mechanical structure, electronics, programming, and sound design) and their

technical details directly affects the duration of prototyping cycles. For further analysis, we

selected a set of hardware tools that can be used to build physical functional prototyping.

4.2.1. Tools for Physical and Functional Prototyping

In this section, we analyze academic and commercial systems and devices that can be used to

help prototyping physical and functional DMIs, whose objective is to allow the experimentation of

musical interactions. The list of the projects and references can be found in Appendix A. Besides,

the detailed description of each tool is out of the scope of this analysis.

Our analysis criteria are inspired by the two-dimensional prototype classification proposed by

(NAKAMARU, 2016). However, instead of using the concept of appearance, we adopt the notion

of physical structure. For DMIs, this criterion is relevant given the intimate physical connection

between the musician and the instrument. Supporting that concept, Paine (2003) performed a

study with musicians that showed their “strong need for a physical connection with their

instrument”, and defines the physical relationship between performer and instrument as “crucial

step in the development of new musical interfaces” (PAINE, 2013).

Because of that, we decided that the scope of this analysis is restricted to systems whose

resulting prototypes’ interaction is based on manipulative gestures. Therefore, systems that deal

exclusively with empty-handed or free gestures are not considered in the present analysis.

We are interested in analyzing the features of the projects that help the designer to achieve

physical, functional prototypes. Although not being a continuous variable, didactically we

46

attempted to present the results in a continuum in both axes from characteristics that demand

more effort and time to less effort and time. According to Lidwell et al. (2010), flexibility is accepted

to be inversely proportional to usability (LIDWELL; HOLDEN; BUTLER, 2010) in a given system.

In our case, we consider that the usability of a prototyping tool is how quick and easy is to obtain

a functional prototype. For example, a textual programmed hardware microcontroller allows the

designer to build a great variety of functional prototypes. In our interpretation, we consider that

this is a case of high flexibility. However, the designer needs to dedicate a lot of time and effort

to overcome technical barriers to accomplish the tasks of programming and building the structure

of the prototype. On the other hand, a system with a fixed structure and a fixed set of interaction

can allow immediate experimentation but lacks possibilities to modifications, i.e. low flexibility.

For the physical structure axis, the points of the scale are presented in order, from more

demanding (more flexible, less immediate use) to less demanding (less flexible, more immediate

use) regarding quickness and ease to accomplish results:

• Only a board: no physical structure other than the board. In this case, all the components

that will serve as a physical interface to the user should be acquired elsewhere. The designer

has to build the physical structure, acquire the sensors or input devices, and define the layout.

• Board with sensors: the physical interface comprises the structure of raw sensors. The

designer has to build the physical structure and deal with arranging the layout of the sensors.

• Supports and modules: there are one or more physical supports where the modules can be

placed, but cannot be connected together. The designer has to define the layout of the

modules.

• Encapsulated sensors or modules: the sensors are encapsulated in a physical structure

as modules, and they can be connected together forming a composed interface. The designer

can use the system immediately and has the option of combining the position of the modules.

• Fixed structure: the structure of the system cannot be modified. The designer does not deal

with physical structure.

For the functionality axis, we considered features that help to configure (or program, or map) how

the prototype responds to user inputs. We considered a discrete scale varying from:

• Textual programming: the designer has to program the behavior of the system by code.

• Visual programming: the designer has to program the behavior of the system by

manipulating graphical elements, such as boxes, arrows, or diagrams.

• OSC and MIDI mapping: normally related to the feature that allows the device to be

automatically recognized by the computer, with which the designer has to use an external

software to map the input gestural controller to sound generation outputs.

• Configuration and Mapping using GUI: the system provides a graphical user interface, in

which the designer has to configure input parameters to define how the system reacts to

user’s input.

• Configuration using elements of the physical system (e.g. self-contained mapping): the

system can be modified without the use of external software because it already implements

47

mechanisms in its interface that allows such modification. The designer has to use these

physical parameters to define the behavior of the system.

• Fixed interaction with no configuration: the system presents a limited set of interactions,

and no modification is allowed. Although obtaining immediate results with this approach, the

experimentation is restricted by the pre-defined behaviors. The designer does not deal with

defining functionalities.

Figure 4.7 presents the axes we used to analyze the related tools for prototyping and shows the

trade-off line between flexibility and time-effort to achieve results.

Figure 4.7: Two-dimensional analysis with Functions and Physical Structure axes. In the middle,

the trade-off line between flexibility, and time and effort

Figure 4.8 shows the projects positioned in the axes according to their features. From the broad

spectrum of projects, we considered three major groups. The first comprises tools that require a

certain amount of time and effort to be programmed or to obtain a physical interface. Although

those tools provide wide flexibility, they do not provide rapid results that potentially boost the

cycles of idea exploration. On the other hand, the second group presents a good potential of

delivering fast results, but it is not flexible enough to allow a wide experimentation. Besides, this

low levels of flexibility may limit different contexts of use and possible intentions of the user.

48

Figure 4.8: Analysis of tools which have the potential to be used in physical, functional prototyping

of DMIs. Axis X: Function. Axis Y: Physical Structure

The trade-off spot (highlighted in green in Figure 4.9) may provide a good balance between

flexibility and immediate usability. Besides, the presence of a support opens up possibilities of

gestural explorations. In sum, these systems present a support that holds the modules, can be

configured by means of a GUI or can be used directly as a MIDI or OSC device.

49

Figure 4.9: The green area denotes the trade-off group of tools.

4.2.2. The Trade-off Area

Here, we present a more detailed description of the projects in the trade-off area.

Pin&Play&Perform is a physical interface composed of a set of sensors that can be attached,

removed and reattached, to a surface board (VILLAR; LINDSAY; GELLERSEN, 2005). Each

sensor has a pin that penetrates a multilayer surface, which is responsible for providing energy

and establishing communication with each module. The user can freely arrange the input devices

on the surface. The on-the-fly reconfigurability is similar to our approach. The elements can be

easily placed or removed. The position and the distance between the elements are not restricted

as long as they are on the surface. The diversity of the controls seems to be limited by buttons,

sliders, rotary potentiometers, and encoders. Finally, the surface appears to become limited after

several uses, because of the perforations. This directly affects this device as a tool for rapid and

iterative prototyping, because after a few cycles of experimentation, the surface might have to be

replaced.

Joué is a MIDI controller that consists of a wooden board with a pressure sensor and eight silicon

layers or modules with different forms. The modules can be placed on the board, and the pressure

of user’s fingers and hands is transmitted to the sensor. User’s gestures are constrained by the

shapes present in the layers, which are inspired by drum pads, piano keys, guitar frets. The

modules are held in place by magnets present on its bottom.

50

Mine is a modular MIDI controller based on a support with grid-based slots, on which unit control

modules can be attached. Until this thesis was written, there are six modules: a pad, buttons, a

potentiometer, a rotary encoder, a slider, and a blank module to fill the gaps. The connection

appears to be rigid since it is demonstrated that for removing the module, the user has to use a

special tweezer.

Modulares Interface comprises a frame made of aluminum and plastic, that can be placed on the

top of an iPad screen, and three kinds of aluminum modules: button, slider, and knob. On the

bottom of each module, there is conductive foam, which allows the transmission of the electrical

discharge of the hand to the iPad surface. In sum, the project is an attempt to create a modular,

physical layer to the iPad.

Pin&Play&Perform and Modulares Interface are academic projects that did not become available

publicly. The other two projects are commercial and available for purchase. Mine is strongly

related to the universe of conventional DJ controllers with the possible interactions limited by the

tabletop position and the traditional button-slide-knob paradigm. Although Joué presents a novel

approach that broadens the explorative horizons of DJ controllers, it is based on a pressure

sensor that still presents interactive and diversity limitations.

The trade-off area of the analysis of potential prototyping tools for DMIs presents opportunities

that will be explored further in this work.

4.3. Final Considerations In this chapter, we analyzed DMI literature in search of initiatives that could potentially be used in

the idea exploration phase. We conclude that there are few structured methods and processes

which seem not to focus on guiding the designer in the idea exploration phase.

Also, for prototyping phase, the tools are little integrated, demanding multiple expertise for the

designer to achieve functional prototypes. Besides, considering the specific tools for physical

prototyping, only a few are balanced in the trade-off flexibility and time-effort. The four projects in

the trade-off area present limited diversity of outcomes when we consider structural combinations.

Finally, the relationship between the conceptual frameworks, methods, and processes do not

seem to be integrated with the prototyping tools. This fact produces a gap between idea and

prototype, which can be a hindrance to the cyclic design process.

51

5. EARLY EXPLORATION In this chapter, we present our methodological approach, early exploration actions and the

resulting insights from the overall experience. We attempt to present the genesis of the ideas that

base our proposition.

5.1. Methodological Approach For this work, we followed the design thinking process, which comprises cycles of inspiration,

ideation and implementation (BROWN, 2008), and human-centered design, “that puts human

needs, capabilities, and behavior first, then designs to accommodate those needs, capabilities,

and ways of behaving” (NORMAN, 2013).

We are inspired by agile methodologies which have been widely used in software development

(MARTIN, 2003), with initiatives such as Scrum and eXtreme Programming being well-

established. In other areas, agile principles are also present such as Lean UX, Lean Innovation,

or Lean Manufacturing.

In agile approaches, providing detailed design information does not hold (ADIKARI; MCDONALD;

CAMPBELL, 2009) and the requirements are defined during the course of the project. The

concept of Little Design Up-Front emerges as a pattern that combines user-centered design and

agile development, and the focus is on doing the minimum necessary to bring value to the user

(BERTHOLDO et al., 2014), where the encouragement is to make mistakes fast, often, but early.

Because of that, we decided to follow a spiral, iterative and incremental approach (Figure 5.1),

searching breadth-first instead of depth-first, i.e. developing in small amounts of the whole idea,

instead of focusing on only one aspect of the idea in detail.

Figure 5.1: Our methodological approach: spiral, iterative, and incremental passing through

inspiration, ideation, and implementation phases

Insight A: The user decides what is better for her contexts and intentions Insight B: Focus on quick, iterative and evolutionary process

52

5.2. Project Batebit In the context of developing digital musical instruments for popular music, we took part in a one-

year project in which we conducted explorative interviews with local musicians, developed

preliminary prototypes, evaluated them through interviews, rehearsals and jam sessions.

The project "Batebit: Diálogos entre a Lutheria Digital e a Música Popular Pernambucana"

("Batebit: Dialogues between Digital Lutherie and the Popular Music from Pernambuco") was

funded by the Government of Pernambuco, Brazil, through its cultural fund, FUNCULTURA, and

was conducted by the author of this work, Filipe Calegario, and two other researchers: Jerônimo

Barbosa e João Tragtenberg (BARBOSA et al., 2015a). The project focused on understanding

how a community of popular musicians could adopt new digital musical instruments. Each

researcher was responsible for the exploration of one concept of musical instrument, which was

developed during the research project.

5.2.1. Interviews

Initially, six interviews (Figure 5.2) were conducted with: a Frevo (traditional genre of

Pernambuco) conductor; a DJ with no experience in playing traditional instruments; two musicians

with more than five years of practice; and two musician-luthiers who build their own instruments

also with more than five years of practice. The interviews focused on understanding the

instruments and tools the musicians used in their musical process. Also, trying to understand the

absorption of new technology in their practice. All the interviews were registered on the project's

website (http://batebit.cc).

Figure 5.2: Interviews with Brazilian Northeast popular musicians

A recurrent theme in the interviews was the intimate relationship between musician and

instrument. For instance, one musician mentioned: "I do not think about the gesture I am doing.

It is like a second voice".

The maestro highlighted the possibility of using generic objects, such as rocks and wood sticks,

as musical instruments and reflected how different kinds of musicians could possibly use these

objects in distinctive ways based on previous experience with their own instruments. Besides, the

53

search for alternative ways of expanding techniques with the instrument and explorative approach

towards new uses and new sounds were mentioned as well.

5.2.2. Workshops

The three researchers conducted two workshops of DMI creation with the attendance of twelve

participants in the first one and seven participants in the second one. The participants' profiles

were musicians with interest in digital technology and little or no previous knowledge of the

prototyping tools.

The first one (Figure 5.3) covered three tools for prototyping instruments: Arduino, an electronic

prototype platform; Pure Data, a visual programming language for sound manipulation; and

Ableton Live, a digital audio workstation with real-time control of processes, mainly used by DJs.

Figure 5.3: First workshop of DMI creation

Due to the complexity of covering a broad spectrum of topics, the researchers decided that the

second one (Figure 5.4) would cover only the Arduino platform and its capabilities of producing

sound without other tools.

Insight C: Leverage the existing intimate relationship musician-instrument to conceive new instruments.

54

Figure 5.4: Example of one of the instruments created with Arduino during the second workshop

In both workshops, the researchers showed the content in a hands-on approach, presenting

examples and encouraging the participants to experiment the use the tools for musical

expression. Despite this didactical effort and the availability of the tools as hardware platforms

and open-source software, the workshops showed that technical details were perceivably an

initial barrier for most people, who took a long time to achieve a musical result (in some cases

not even reaching it).

5.2.3. Pandivá

Following these insights, we developed an exploratory functional prototype which merged the

guitar-inspired posture, the way of triggering sounds by tapping a tambourine skin and the way of

altering the pitch using a trombone slide. The instrument was called Pandivá (reduction of

Portuguese words "pandeiro de vara", in English: slide tambourine) (Figure 5.5).

Figure 5.5: Pandivá #0, Pandivá #1, Pandivá #2.

For the first functional prototype (Pandivá #0, Figure 5.5), we used piezoelectric sensors inside

rubber layers as pads for the player to strike and a slide made of concentric pipes of PVC with

different diameters. In one tip, we placed a small LED and the other, a light-dependent resistor.

We connected the sensors to an Arduino Uno that sent MIDI messages to a computer, which was

Insight D: Encapsulate technical details to allow the users to reach a musical experimentation faster and with less effort.

55

responsible for synthesizing the sound. With this version, we could test sensor combinations and

possible mappings.

The second functional prototype (Pandivá #1 in Figure 5.5) incorporated a structure made of

laser-cut MDF. The instrument could be used on the lap or with a shoulder strap. This version

comprises fifteen contact buttons placed in three groups, used to detect a strike on its surface

and trigger notes. The groups of buttons attempted to represent the playable areas in the pandeiro

skin surface. Four piezoelectric sensors measure the intensity of the attack, and a slide

potentiometer coupled to the sliding PVC pipes form a piston that controls pitch.

The player triggers notes by striking on one of the three sets of buttons located on the instrument

body and alters their pitch by moving the slide. Using a mode button, the player could switch

between melodic mode, which triggers notes with pitch based on slide position; and harmonic

mode, in which the slide changes the chord that is being played and the three pads correspond

to the three notes of the chord.

The sound of Pandivá #1 was synthesized on the computer by GarageBand via a MIDI

connection. We chose the steel string guitar as the primary instrument for demonstration, due to

its evident attack that exemplifies the rhythmic characteristics of the device.

After testing this version of the Pandivá prototype with four users (Figure 5.6), three of them

mentioned that the way of altering the pitch could be improved to allow a discrete control because

managing to reach a specific note was difficult with the slide.

Figure 5.6: Testing Pandivá with three percussionists and a guitar player

To provide this functionality, we developed a new version of Pandivá (Pandivá #2) that had a set

of four buttons instead of a slide. Combining the buttons, the player could trigger up to sixteen

different notes. Both versions of Pandivá can be seen in Figure 9.

56

The Pandivá #2 was developed and tested with two performers. As soon as a left-handed

musician used it, he felt uncomfortable with the buttons positions. To solve this issue, we made

additional holes on the opposite side and moved the buttons to the left-handed position. The

adapted version can be seen in Figure 5.7.

Figure 5.7: Pandivá version 2 modified for left-handed users.

An overall reflection is that, for all the modifications, we had to switch our development mindset

several times from structure, electronics, programming, and sound design. Time and effort we

have expended while changing modes of operation have been a hindrance in our iterative design

process.

5.2.4. Sandbox Wow

Sandbox Wow was a prototype developed during Batebit project to experiment possible musical

interactions with: (1) eight surfaces made of a homemade conductive ink with graphite powder

and white glue; (2) eight pieces of a capacitive sensor with Arduino digital ports; and (3) eight DIY

sensitive pad made with a sandwich of two sides of EVA rubber and a piezo as filling. It could

function as a sequencer or a MIDI controller.

In the project, as Figure 5.8 shows, we had to deal with different development contexts such as:

structure (mechanical support, mechanisms, material choice), electronics (electronic

components, sensors, actuators), programming (communication protocol, coding), mapping

(operations, scaling, adaptation, connections), sound (synthesis, parameters choice, content

Insight E: Allow the user to dynamically modify the functional prototype to adapt it for her contexts of use and intentions.

57

choice). For each context, we used specific sets of tools, which required a spectrum of different

expertise. Besides, the often shifts in mindsets may have interfered in the cognitive load, since

these shifts spread the attention to aspects of different natures of the object or its behavior.

Figure 5.8: Example of different contexts (mechanisms, electronics, programming, mapping,

sound) to build a functional prototype of a DMI (Video can be seen: http://youtu.be/l2HnE3txKdc)

Insight F: There are multiple contexts during the development of a DMI functional prototype, which demand different kinds of expertise.

58

5.3. Summary The early exploration allowed us to collect relevant insights for the development of this project.

Here is the summary of the insights:

• Insight A: The user decides what is better for her contexts and intentions

• Insight B: Focus on quick, iterative and evolutionary process

• Insight C: Leverage the existing intimate relationship musician-instrument to conceive new

instruments.

• Insight D: Encapsulate technical details to allow the users to reach a musical experimentation

faster and with less effort.

• Insight E: Allow the user to dynamically modify the functional prototype to adapt it for her

contexts of use and intentions.

• Insight F: There are multiple contexts during the development of a DMI functional prototype,

which demand different kinds of expertise.

59

6. PROPOSITION In this chapter, we explain the decisions we made in terms of scope and present our proposition

that consists of a concept, a method, and a toolkit.

6.1. Scope and Basis Based on literature, in Figure 6.1, we present the scope that we decided for this work. Firstly, we

choose to explore the implementation of gestural controllers and to prepare them for mapping.

We aim to provide tools which aid the design of artifacts that would lay in the border between

instrument-inspired and alternate gestural controllers.

Furthermore, due to previous experience in the area, we will focus our efforts on manipulative

gestures, i.e. physical objects, and we will not deal with empty-handed or free gestures. We

expect that our approach will aid designers and performers to obtain exploratory prototypes,

focusing on idea exploration and prototyping phase of the DMI design process. Besides, we aim

to provide ways of implementing functional prototypes, which means that we are not concerned

to assess either appearance or the final product.

Figure 6.1: Scope of this work

Our proposition comprises a concept, a method, and a toolkit. In this chapter, we present each

point describing the cumulative relationship they have, i.e. the toolkit contains the method that

contains the concept (Figure 6.2).

60

Figure 6.2: Three elements of our proposition

We propose to explore the concept of instrumental inheritance, to adapt a design method for idea

generation, called morphological analysis, to be used in the DMI context, and a physical

prototyping toolkit based on the morphological chart.

6.2. Instrumental Inheritance Inspired by the Insight C (“Leverage the existing intimate relationship musician-instrument to

conceive new instruments”) and by the discussions in our research group on Music, Technology,

Interactivity, and Creativity (MusTIC) at Centre for Informatics - UFPE, Brazil, we started to

develop the concept of instrumental inheritance, which is the similarity in shape or playing

techniques a certain instrument transfers from other existing instruments. MusTIC’s discussion

covered a hypothesis that instruments with higher levels of inheritance have more chances to be

adopted due to the existing body of knowledge and gestures repertoire. Our understanding is that

instrumental inheritance is composed of two other inheritances: structural, physical elements of

the instruments, and gestural, ways of manipulating the instrument.

We propose the use of instrumental inheritance as an initial constraint to ignite the creative

process for new instruments ideas. Though the new resulting instruments will not be structurally

or gesturally restricted by acoustic laws, this approach will serve as a kick-start method to explore

and generate ideas based on common knowledge and existing cultural hooks, serving as an

initial, structured and exploratory path for idea generation. This decision is supported by idea

generation literature which defines the constraints and the use of analogies and metaphors as

idea generation promoters (SHAH; KULKARNI; VARGAS-HERNANDEZ, 2000).

6.2.1. Related Concepts

In literature, there are some related concepts that do not exactly describe the specificities we

attempt to communicate with “instrument inheritance”.

Related to the discussion, skeuomorphism is a concept that designates the incorporation of

elements of existing artifacts in new artifacts, even not presenting a functional importance for the

new one (NORMAN, 2013). It is originated in the field of archeology, where researchers call a

61

skeuomorph “an element of design or structure that serves little or no purpose in the artifact

fashioned from the new material but was essential to the object made from the original material”

(BASALLA, 1989). O’Hara (2012) mentions that the original concept of skeuomorphism can be

observed but not intentionally induced (O’HARA, 2012). That is why the author strongly disagrees

with the use of this concept in the context of user interface design, which is commonly adopted

by Apple to describe elements of interface based on the aesthetics of physical objects (such as

the bookshelf in iBooks) (CURTIS, 2015). According to O’Hara (2012), Apple is using visual

metaphors and not skeuomorphism.

In its turn, a metaphor is a conceptual tool used to interpret aspects of an idea or artifact based

on a comparison of the characteristics of another idea or artifact (HEY et al., 2008). It is a concept

that appears related to analogy, which can be understood as the transfer of information from

familiar, existing domain in order to explain or define elements of a different domain (DAHL;

MOREAU, 2002). In design, metaphors and analogies are commonly used to generate new ideas

based on existing ones (HEY et al., 2008). In DMI design, the use of metaphors is related to more

transparent communication between the performer and the audience yielding a more expressive

instrument (FELS; GADD; MULDER, 2002).

The three concepts can be summarized as a transfer between characteristics of an existing

artifact to another (in the case of skeuomorphism, observable but not intentional). In our case,

instrument inheritance defines elements that are passed or transferred from a predecessor to a

successor (MERRIAM-WEBSTER, 2004).

Other related concepts that we will use to refine ours is affordance and signifier. According to

Tanaka et al. (2012b), “affordances are a configuration of properties that provide a direct link

between perception and action.” (TANAKA; ALTAVILLA; SPOWAGE, 2012). The term was

defined in the context of psychology by James Gibson, who tried to present an alternative

ecological discussion about visual perception (KAPTELININ, 2014). For Gibson, affordances are

“action possibilities offered by the environment to the animal” (GIBSON, 1979), or as interpreted

by Norman (2013): “the physical objects conveyed important information about how people could

interact with them, a property […] named “affordance”’ (NORMAN, 2013). According to Norman

(2013), even if it is not visible, the affordance is present between the environment and the user.

Thus, it is important for the designer to signal how the components of an artifact should be used

or how they work to transform the invisible affordance into perceived affordance for the users.

Norman (2013) highlights that perceived affordances aid people to realize possible actions

without recurring to labels, marks or instructions (NORMAN, 2013). The author presents the

concept of signifier as “any perceivable indicator that communicates appropriate behavior to a

person. Affordances determine what actions are possible. Signifiers communicate where the

action should take place.” (NORMAN, 2013).

With our concept of instrumental inheritance, we aim to provide new instruments with signifiers

inspired by existing instruments. In this way, it seems to be possible to leverage a current body

of knowledge, playing techniques, and familiarity. For the audience, inheritance can provide

62

familiar elements which can possibly avoid a strange reaction and disconnection, causing

engagement, and perhaps more expressiveness.

6.2.2. Possible Evidences

In an attempt to illustrate the instrumental inheritance, we discuss possible cases where the

concept can be applied.

We can speculate that the success and popularization of Moog Synthesizers were related to the

implementation of an existing piano standard keyboard. On the other hand, Buchla Synthesizers,

which implement an alternative controller, are known for being used by a niche of experimental

musicians (FANTINATTO, 2014) (TEBOUL, 2017). However, it is important to highlight that there

are different factors and variables involved in instrument adoption, which we consider a complex

issue, and this speculation is one possible view in an attempt to understand the context.

In the same direction, we can conjecture that electric guitar players took advantage of the existing

familiarity, body of knowledge and playing techniques of acoustic guitars because of the inherited

instrument shape and ways of controlling (six strings, selecting and plucking the string with a

hand, and selecting the string and the fret with the other hand). It does not mean that the electric

guitarists remained imprisoned by playing techniques of acoustic guitar, but they have possibly

started from a common ground and developed other paths. It is different from a completely new

instrument which does not convey familiar information on how it works or how it can be used.

Following a similar trend, another possible evidence is related to the keyboard layout that was

transferred from instruments such as clavichord and harpsichord to the piano (SACHS, 1940).

6.2.2.1. Hybrid Instruments

We consider that hybrid instruments inherit from more than one musical instrument, combining

ways of holding, ways of playing, or shape. Looking through the lenses of instrumental

inheritance, instruments like the keytar (Figure 6.3) seems to benefit from the body of knowledge,

and the gestures repertoire from both keyboard and guitar.

63

Figure 6.3: James Brown playing a Moog Liberation keytar. Source: synthtopia.com

As Figure 6.4 shows, other examples are the Scratchocaster, developed by Viny’ Lourd Son,

merges shape of a guitar with a mixer and a turntable; the DRUMITAR, developed by Futureman,

member of the band Béla Fleck The Flecktones, guitar-inspired body for a drum set;

KIMOPHONE, developed by Kimo Lobo, which is a series of instruments that hybridize a sax-like

mouthpiece with a keytar; and Arduino Ribbon Synth, a DIY project by Dean Miller, that

implemented a drum pad and a fretless-inspired ribbon sensor that is hold like a guitar.

Figure 6.4: (a) Scratchocaster, (b) DRUMITAR, (c) KIMOPHONE, (d) Arduino Ribbon Synth

Furthermore, an example of a commercial project that incorporates the concept of instrument

inheritance is the Artiphon (Figure 6.5), which allows the user to hold the instrument in positions

64

inspired by cello, violin, keyboard, and guitar. The controllers are inspired by piano’s keys, guitar’s

strings, and frets.

Figure 6.5: Artiphon

The modulin (shortening for modular synth + violin) is an instrument developed by Martin Molin

(member of the band Wintergatan) that consists of a ribbon sensor connected to a modular

synthesizer. It is held and played like a violin (Figure 6.6).

Figure 6.6: Modulin, instrument developed by Martin Molin integrant of the Swedish band

Wintergatan

6.2.3. Discussion

One possible criticism about basing the development of new instruments on existing acoustic

counterparts is the difficulty of avoiding clichés, as the majority of playing techniques and search

for new sounds have already been explored (MAGNUSSON; MENDIETA, 2007b). In a survey

with musicians, Magnusson and Mendieta (2007) discussed that some participants negatively

mentioned the DMI that is “slave of the historical”, which can diminish its potential to be an original

65

and creative way to expand musical expression. For that, we argue that our intention of using

instrumental inheritance go in the direction of using the familiarity as inspiration, and leverage

existing cultural hooks. Considering the DMI classification, Miranda and Wanderley (2006) made

an important distinction between instrument-like and instrument-inspired gestural controllers

(MIRANDA; WANDERLEY, 2006). As mentioned before, we focus our work on inspiration. We

speculate that there is a creative potential on exploring combinations of instruments parts, and

we consider to use the inheritance to ignite initial ideas to be explored depending on the context.

6.3. Morphological Chart for DMI Idea Generation Gathering the insights from previous sections, here, we explore the potentials of combining

elements of existing instruments to generate ideas of new instruments. For that, we followed a

well-established design method called morphological analysis. Shah et al. (2000) classify

morphological analysis as a germinal method, i.e. methods used in the initial step of conception

when the designer has yet no previous ideas. This characteristic is strongly related to our first

question, making the morphological analysis suitable for our use. Besides, this method provides

a quick way of visualizing and combining existing parts.

6.3.1. What is Morphological Analysis?

The morphological analysis is an idea generation method in which existing artifacts are split into

their fundamental parts and then recombined to generate new ideas (RITCHEY, 1998). The

method was firstly proposed by Fritz Zwicky, a Swiss astrophysicist, and aerospace scientist, in

the context of generating alternatives for jet propulsion (ZWICKY, 1967). He analyzed and split

the propulsion system into six functions, and through combination, he demonstrated he could

achieve more than five hundred possible alternatives for jet propulsion design (VASCONCELOS

et al., 2016).

According to Smith (1998), the morphological analysis can be described as an analytical strategy

based on decomposition, in which wholes are divided into parts or attributes, and ends into means

(SMITH, 1998). The author states that this approach was the most often used in his analysis.

Cross (2000) describes the procedure to formulate a morphological chart (or matrix, table, box)

(CROSS, 2000) as:

1) “List the features or functions that are essential to the product

2) For each feature or function list the means by which it might be achieved

3) Draw up a chart containing all the possible sub-solutions

4) Identify feasible combinations of sub-solutions”

The advantage of using a morphological analysis approach is to have an overall picture of the

possible solution space that can be explored in a structural and systematic way. According to

Vasconcelos (2016a), this systematic approach consequently “forces designers to consider many

potential solutions that would otherwise be overlooked” (VASCONCELOS et al., 2016).

66

As said by Pahl et al. (2007), the problem of this method is to select which combinations are

compatible (PAHL et al., 2007). Besides, Cross (2000) mentions that depending on the number

of combinations; the search can become impossible (or really tedious) (CROSS, 2000). For that,

the author suggests discarding some incompatible combinations up-front, or taking a random,

intuitive approach to choosing the possible paths.

To illustrate how the morphological chart is implemented, Figure 6.7 shows a generic

representation of a morphological chart and Figure 6.8 presents a morphological chart for forklift

trucks.

Figure 6.7: Representation of Morphological Chart (extracted from Pahl et al. (2007))

67

Figure 6.8: Example of a morphological chart for forklift trucks (extracted from Cross (2000))

6.3.2. Morphological Chart Based on Instrumental

Inheritance

Here, we follow the morphological analysis steps described by Cross (2000) (CROSS, 2000) to

produce a morphological chart based on instrumental inheritance.

1) Features or functions that are essential to the product

For an acoustic instrument, an essential feature is the physical structure, not only for producing

the sound (as we are not interested here) but also for providing a support that can be held, on

which the hands can perform the three functions of the gestural channel: ergotic, semiotic, and

epistemic, and elements of manipulative gestures.

Considering instrument classification as an inspiration source to extract instrument’s features,

Hood (1982) criticizes the limited focus of the organology (or the science of musical instruments)

being practiced at the time on the description of physical features and acoustic properties of the

instrument, disregarding points such as techniques of performance, and musical functions.

Because of that, his approach to classification differs from established ones, such as (KNIGHT,

2015) and (MIMO CONSORTIUM, 2011), whose primary focus was on how the material

classification of the instruments (e.g. how the instrument is made or how it acoustically generates

sound), mainly used to organize the plethora of historical instruments in museums. The author

proposes elements that focus on the relation between the performer and the instrument, such as

the instrument support, which is the “manner in which the instrument is supported” (HOOD, 1982).

68

In the context of analyzing the appropriation and the emergence of stylistic variation between

different performers, (ZAPPI; MCPHERSON, 2014) and (GUREVICH; MARQUEZ-BORBON;

STAPLETON, 2012) present features that were the analysis results of the use of an overly

constrained musical interface. The features related to gestures are affordances, interaction

techniques, ways of holding, and ways of playing.

Inspired by these initiatives, we decided to focus on features that are related to instruments:

• Physical structure: focusing on the way the object induced player's postures, the way

the player holds or the object is supported.

• Gestural control: we are interested in the player's instrumental gestures to control

sound.

2) Means by which each feature or function might be achieved

Considering the physical structure, Figure 6.9 shows artifacts that are intentionally represented

as generic blank objects to highlight the ways of holding and the posture they induce when used.

Figure 6.9: Postures inspired by existing instruments. Drawings by Giordano Cabral

For our initial set, we explore supports with signifiers based on popular instruments. The postures

were inspired by guitar, tambourine, accordion, drum pedals, flute/saxophone/clarinet, cájon, and

piano (Figure 6.10).

69

Figure 6.10: Simple analysis of existing musical instruments associated with an abstract symbol of

control

Regarding ways of control, Figure 13 show elements inspired by the hurdy-gurdy crank, the DJ

turntable, the kalimba tines, the percussion instruments skin, the violin bow, the trombone

mouthpiece, the guitar strings, the trombone slide, and the piano keys. Although the inspiration

comes from existing instruments, we depicted the controls as simple generic drawings aiming to

70

present them in a more abstract way and broaden their uses beyond the existing instrument. We

focus on controls of instruments we considered that have potential to expand combinations.

3) Chart containing possible sub-solutions

4) Feasible combinations of sub-solutions

The current version of the chart comprises seven postures and nine controls organized in three

groups of control (excitation, modification, and selection) according to the instrumental gestures

classification by (CADOZ; WANDERLEY, 2000) (Figure 6.11).

Figure 6.11: Chosen path on the morphological chart: a mix of guitar-based posture, a slider, and

pads. Drawings by Giordano Cabral

6.4. Development of the Functional Prototyping Toolkit for DMI

In this section, we will present the development cycle of the proposed prototyping toolkit for DMIs,

the Probatio (the Latin word for "test, experiment, trial").

The morphological chart is a useful tool to visualize and navigate in the design space of musical

instruments, but as mentioned before, prototypes are a crucial part of the design process and, for

a better understanding of a DMI, functional prototypes are more suitable. As we aim to boost the

design cycle of DMIs, it is important to have the functional prototype, so the user, or designer,

can easily modify and evaluate the generated idea.

Concerning the process of implementing functional DMI prototypes, we decided to build a

prototyping toolkit that will work as a physical morphological chart based on instrumental

inheritance.

The toolkit approach seems promising as presented by (SADLER et al., 2016a) and (HELMINEN;

AINOA; MÄKINEN, 2015), which respectively discuss the benefits of encapsulating technical

71

details to achieve more creative results, and the importance of toolkits in transferring knowledge

from the user to the developer to achieve adequate alternatives during the design process.

With our prototyping toolkit, the user can change the controls and postures by using and

combining physical modules that are already responsive. Inspired by the related projects, the

basic idea is to build a system that has (1) blocks, the units of control that enable the gestures of

the player to produce sound, and (2) supports, in which blocks can be placed.

The objective of the prototyping toolkit is to provide an initial, structured and exploratory path for

the designer, or performer-designer, to generate instrument ideas and to reduce time and effort

to build functional prototypes.

6.4.1. Guidelines

Based on the insights that were collected from the literature review and the early exploration, we

formulate our design guidelines which help our decision making during the course of the

development.

• Tangibility: the toolkit should present tangible modules to relate to the “strong need for a

physical connection with their instrument.” (PAINE, 2013).

• Modularity: the toolkit should allow the user to explore combinations of a diverse set of

modules (MAESTRACCI; FRECHIN; PETREVSKI, 2011) and to easily perceive causal

properties of connections (GELINECK; SERAFIN, 2010a). This guideline is inspired by

Insight A (“the user decides what is better for her contexts and intentions”) and Insight E

(“allow the user to dynamically modify the functional prototype to adapt it for her contexts of

use and intentions”).

• Technical Encapsulation: the toolkit should encapsulate technical details to reduce time

and effort to build prototypes, allowing the user to focus on creative thinking (SADLER et al.,

2016b). Directly, the toolkit should “expose the functionality and abstract the underlying

technological complexity” (KNORIG, 2008). This guideline is inspired by Insight B (“focus on

quick, iterative and evolutionary process”), Insight F (“there are multiple contexts during the

development process of a DMI functional prototype, which demand different kinds of

expertise”), and Insight D (“encapsulate technical details to allow the users to reach a musical

experimentation faster and with less effort”).

Besides, we defined additional requirements that we believe will enhance the usability and the

users’ experience with the environment:

• Feedback: the environment should provide a clear, perceptible and real-time response to

actions and modifications performed by the users (JORDÀ, 2003) (O’MODHRAIN; CHAFE,

2000).

• Integration: the environment should be easily integrated with previous or legacy systems,

benefiting from their functionalities (SCHMEDER; FREED, 2008).

72

6.4.2. Implementation Decisions

To help to fulfill the requirements, we took the following implementation decisions.

• Build the environment using techniques of digital fabrication: use techniques of digital

fabrication to obtain tangible artifacts with a better accuracy and quality with less time and

effort (WALTER-HERRMANN; BÜCHING, 2014) as experienced during the exploratory cycle.

• Use of open-source electronic prototyping platforms: use established open-source

platforms, such as Arduino (BANZI, 2009), and sensors that are available and easy to

integrate due to increasing trend of DIY and Maker Movement (DOUGHERTY, 2012).

• Implement standard protocols: use standard protocols such as MIDI and OSC (WRIGHT;

FREED; MOMENI, 2003) and leverage existing systems for mapping and development

ecosystems such as libmapper (MALLOCH; SINCLAIR; WANDERLEY, 2014). With that, we

hope to reduce the time and effort for the sound production.

6.4.3. Physical Structure

As a preliminary validation, we prototyped a nonfunctional version of the system with MDF pieces,

and hooks and loops fastener (Velcro). The supports (pieces (a), (b), (c), and (i) in Figure 6.12)

had the hooks, and the control parts ((d), (e), (f), (g), and (h)) had the loops. With this early proof-

of-principle, one could already understand the potential of trying different combinations and

understand how the system would work. This prototype was important to communicate the idea

to potential users and to have a better grasp about the desired dimensions of the final toolkit.

Figure 6.12: Initial set of nonfunctional modules made of MDF and Velcro: (a) guitar-inspired body, (b) clarinet-inspired body that can also be a guitar-inspired neck, (c) tambourine-inspired body, (d)

73

guitar fretless neck inspired input, (e) tambourine-inspired three pads input, (f) guitar-inspired strings input, (g) wind instrument-inspired breath input, (h) piano keys-inspired keyboard, (i)

combination of part (a) and part (b) resulting in a guitar-inspired support, and (j) combination of inputs and guitar-inspired support.

In order to increase the granularity choice of possible positions to place the control blocks, we

decided to build slots in the supports that are presented in Figure 6.13.

Figure 6.13: Sketches of Probatio's supports

To visualize how the necks and bodies would fit together, we rendered 3D models as shown in

Figure 6.14. This step was important to realize how the neck would be connected to the body,

and how many slots would be the minimum necessary to implement functions following the

modularity guideline.

Figure 6.14: 3D renderings of Probatio’s supports and block

Aiming to obtain quicker results, we decided to build the structure with laser cut MDF (medium-

density fiberboard) 3mm-thick. MDF is an engineered wood made of residuals wood fibers, mixed

with wax and resin, and formed with high temperature and pressure. It is a plain and rigid material

with an inexpensive cost (MALONEY, 1996).

The rapid prototyping technique based on laser cutter favored our development process because

it is simple to model since it is basically based on 2D drawings, provide precise cuts, and a short

74

time to achieve the product. With the proliferation of the Maker culture, the cost of laser cutting is

decreasing, and the availability of machines is growing.

To build our tridimensional structures, we used a technique based on joining two pieces of wood

by cutting a set of matching rectangular cuts, called finger joints. This kind of joint makes the

connection between two boards stronger and easier to build since it helps to keep the two boards

in place and then glued together.

We used a commonly used web tool in the Maker community to generate the finger joints, the

MakerCase website (Figure 6.15). The website presents a simple interface where the user

provides the dimensions of the block, and it generates the basic 2D cutting plans.

Figure 6.15: MakerCase.com was used to generate the finger joints of the blocks

After processing the drawings in Adobe Illustrator to add the desired details, we produced, for

instance, the cutting plans of a four-slots support for Probatio (Figure 6.16).

75

Figure 6.16: Example of drawings used for laser cutting

To check if the plans were correctly generated, we imported the 2D drawings and modeled in 3D

using Dassault SolidWorks (Figure 6.17). This step was important to avoid rework because of

errors, besides reducing the wasted material and time.

Figure 6.17: 3D rendering for checking if all the components fit together before sending to laser

cutting

6.4.4. Connection Slots

Following our modularity and technical encapsulation guideline, we decided to expose the

functionality of the controllers but hide the technical details of blocks.

The blocks can be positioned on the supports by inserting them into the slots. The connection is

made through spring-loaded pins in contact with a metal surface (Figure 6.18).

76

Figure 6.18: The spring-loaded connector

To ensure that the block is held in place and also to increase the adhesive force, we used four

neodymium magnets placed in each slot, and the same amount placed in each block. These

magnets are stronger than the ferrite counterparts, but also more expensive. In our case, as the

blocks have restricted room within, we decided to reduce the needed volume for the components

by choosing neodymium magnets.

Our first attempt to build the metallic contact surface for the slots was based on etching copper

plates, a common technique for building a circuit board. The process involves drawing on the

copper board with a permanent marker and submerge the plate in a solution of ferric chloride

(Figure 6.19). The surface without the marks dissolves, keeping the metallic paths intact that will

serve as an electronic contact. The procedure was laborious and with a high risk of affecting the

needed accuracy for placing the blocks. Besides, this way of building was not modular,

constraining the possibilities of building supports with different shapes.

Figure 6.19: Building the contact slots by etching copper plates

77

As a better and modular alternative, we developed the slot module using a CAD for printed circuit

board (PCB) and order a few boards in a Chinese PCB manufacturer. The main disadvantage is

the time it takes for the package to arrive, but the gains are in quality and accuracy. With the PCB

unit-slot, it was only a matter of soldering and arranging the boards in the support, as can be seen

in Figure 6.20.

Figure 6.20: Connector based on printed circuit designed in EAGLE and manufactured in China

6.4.5. Blocks

Each block has its microcontroller (the slaves) which communicates with the central hub (the

master) through a wired connection and uses the I2C (Inter-Integrated Circuit) serial

communication bus, a master-slave-based protocol whose connection consists of four wires:

VCC, GND, SCA, SDL. Amongst other buses, such as SPI, CAN, UART, the benefit of using I2C

is the multi-slave with only two-wire per slave, i.e. there is no need of a dedicated wire connection

for each slave or additional circuitry. This reduces the number of wires to connect and becomes

simpler and cheaper to implement this prototype. Figure 6.21 illustrates the connection between

the block and the slot. The GND and VCC bus provides the electricity for the block to work, SCL

is the clock line, and SDA is the transmitted data.

An alternative option for communication and power supply was to transform each block in an

autonomous, battery-powered module that would wirelessly connect directly to the computer and

would eliminate the hub from the architecture. We considered that the management of batteries

charge would bring a layer of complexity that could negatively affect the time of our development.

Simply put, it would take some time to recharge the batteries, and it would take longer to develop.

Therefore, we chose the wired-block approach.

78

Figure 6.21: Block and slot’s I2C connections

The main loop of the central hub is illustrated in Figure 6.22. Firstly, the hub checks all the

registered slave address. If the slave replies, it means the block is connected to a slot. If the slave

does not reply, it means the block is not present in the support.

The hub gathers all the sensor values of each block and format all the data in one message.

Then, it sends this message to the computer via the serial port following the protocol which is

shown in Figure 6.23. The position of values in the message are fixed. If a block is not present,

the hub fills the position in the message with 0.

Figure 6.22: Sequence diagram of communication between the hub, the blocks, and the computer.

Between the hub and the blocks are I2C messages.

This approach guarantees that the size of the message remains constant, even with the varying

number of blocks in the support. This avoids irregularities in the time each message takes to

reach the computer, reducing jitter, which is the deviation of a periodic transmission. We

performed an informal test to assess the latency of the system, and the result was approximately

5 milliseconds.

79

Figure 6.23: Communication Protocol used between Probatio's hub and the computer via Serial

Port

Software written in Java receives the serial messages and plot the values of each connected

block for monitoring purposes. Probatio’s overview is illustrated in Figure 6.24.

Figure 6.24: Probatio's communication overview. The supports are highlighted to show they are simply an extension of the physical communication bus of I2C. The logical parts are the blocks

which communicate to the hub.

80

6.5. Final Considerations In this chapter, we presented our threefold proposition that addresses our research questions.

For the structured and exploratory steps for idea generation, we propose the use of a

morphological chart based on the concept of instrumental inheritance. As we aim to boost the

cycles of idea exploration and prototyping, the morphological chart appeared to be distant for the

user to experiment the functional instrument. Therefore, we decided to incorporate the

combinatory nature of the morphological chart into a toolkit for physical, functional DMI

prototyping. In this sense, the toolkit embeds a method, which already embeds a concept.

Following our guidelines, the physical toolkit was implemented to encapsulate technical details in

the format of modules that can be combined in different ways to achieve the intended user’s

result.

81

7. EVALUATION OF PROBATIO 0.1 In this chapter, we present a description of the version 0.1 of Probatio and perform an exploratory

evaluation (CALEGARIO et al., 2017).

7.1. Description The first functional prototype of Probatio comprises five blocks and two bases (for an online video

demonstration: https://youtu.be/Ge_aj5uMgOU). As can be seen in Figure 7.1, the first support is

inspired by the player’s posture when playing the piano, also known as a tabletop posture, and is

laid out in a three by three square grid. The second support is inspired by the posture of the

performer when playing the clarinet or the saxophone and it has four slots in a single row. Each

slot can fit a block either from the top or sides, allowing multiple block orientations (except for the

center slot in the three by three grid). As an example, the 4-slots and 3-by-3 supports can be

combined to create a compound support that resembles a guitar body and neck.

Following the support options mentioned by (HOOD, 1982), the bases and their combinations can

be used on a stand or table, player's lap, between legs, across legs, hold by hands.

Figure 7.1: Probatio's bases

As a feasible alternative that we believe would not compromise the concept of the environment,

we developed the following five blocks (Figure 7.2):

• Fretless: inspired by fretless necks of string instruments. It is made of a resistive touch tape

and a force-sensing resistor and measures the position and force of the finger on the surface.

• Turntable: inspired by DJs' turntables. It is made of a circular piece of MDF attached to a

rotary encoder.

• Bellows: inspired by the bellows of a harmonium. It is made of a moving top connected to

the body of the block by a central pin and a pair of springs. It has a small magnet on the

moving top and a hall effect sensor on the internal wall of the body. As the top is depressed,

the distance between the magnet and sensor changes, which indirectly allows the rotation of

the top to be measured.

• Buttons: inspired by discrete controls in several instruments such as accordions' bass

switches, piano keys or brass instrument valves. It comprises four buttons.

• Crank: inspired by the crank of the hurdy-gurdy. It consists of a crank attached to a rotary

encoder.

82

Figure 7.2: Probatio's blocks

Figure 7.3 presents a possible combination using the five blocks and a compound base.

Figure 7.3: Possible combination of a DMI prototype using Probatio

As can be seen in Figure 7.4, combining the supports in different positions and orientations

provides diverse ways of holding Probatio, inducing various postures.

Figure 7.4: Example of possible postures using Probatio

7.2. Evaluation In this section, we present a preliminary evaluation of the system. We consider this first cycle of

evaluation as the first opportunity to collect initial impressions about Probatio’s shape and

functionalities.

83

7.2.1.1. Objective

We conducted a preliminary study to collect feedback on Probatio and its intrinsically associated

design method. The main purpose is explorative.

7.2.1.2. Participants

We enrolled 10 volunteers (3 women and 7 men), ranging in age between 24 and 50 years old,

with one of them having no formal musical training. All of the participants had experience using

digital technology in music, and nine had already developed at least one DMI and were familiar

with the development of digital technology for music.

7.2.1.3. Design

The sessions (Figure 7.5) were performed with one participant at the time and lasted

approximately one hour each. The participants were first introduced to the prototyping toolkit and

asked to explore it freely for 10 minutes as a practice session. After that, the participants were

instructed to perform a set of five simple tasks: 1) "insert a block in the side slot and play with it",

2) "make the movement of a block interfere with the sound of another block", 3) "use more than

two blocks at once", 4) "make a compound base", and 5) "hold the base in a different way". The

tasks were independent of each other, and they were given following the same order for all the

participants. For each task, we notified the participant as soon as we observed that he or she

achieved it.

We chose these 5 tasks since (a) they are easily observable, (b) they are likely to be performed

consistently by all the participants whatever their skills in the design of DMIs, and (c) they are

representative of the basic assembly tasks that can be performed with the current version of the

toolkit. Moreover, the tasks acted as a stimulus for the participants to understand the features of

the system and then enable a proper discussion about it.

Following this, the participants took part in a semi-structured interview covering their impressions

about the current version of the system and an overall discussion about the prototyping toolkit

concept. The interviews were video recorded, transcribed, and analyzed according to the thematic

analysis method used in (TANAKA et al., 2012).

84

Figure 7.5: Participants experimenting Probatio

7.2.1.4. System Setup

To focus on the structure and use of the physical prototyping toolkit, we decided to exclude the

possibility for the participants to create their own mapping with a computer. Though one-to-one

mappings have been shown to be a limiting factor in DMI design (HUNT; WANDERLEY;

PARADIS, 2003), we consider that they are effective in the context of this experiment. Therefore,

each Probatio block was uniquely mapped to a sound, generated via Musical Instrument Shield

85

for Arduino by Sparkfun. A demo video of this implementation can be seen at

https://youtu.be/Ge_aj5uMgOU.

7.2.1.5. Analysis

In this section, we grouped the participants' responses into four major groups: (1) positive points,

(2) negative points, (3) suggestions, and (4) contexts for use. For each of these groups, we

present the main points raised and quotes from the interviews with the indication of the participant

code in parenthesis.

7.2.1.5.1. Positive Points

The most recurrent positive points during the interviews were: (1) the physical affordances that

invite to use the system (mentioned by 50% of the participants). Illustrating this, one participant

said, "They [the blocks] just suggest: pick me up and do something with me" (P01); (2) the clarity

of the way it works (40% of the participants). For instance, one participant mentioned: "A cool

thing about it is that it is self-explainable" (P02), and two participants highlighted that they felt the

system to be "quite intuitive" (P03, P04); and (3) the convenient and functional connection

between the blocks and the bases (30% of the participants). For example, a participant

highlighted "I think it is brilliant" (P05) and another participant described the connection as "pretty

solid" (P06) since the magnets held the blocks in place.

Other points mentioned were: the system's immediate response ("you put it together, and it works"

(P07)), and the possibility to perform quick cycles of trial-and-error; the system's versatility ("you

can build different shapes, [...] connect the blocks in various orientations, [...] explore

combinations" (P05)).

Finally, three participants said that the system could be used as a tool to help to have ideas (P01,

P06, P08), and two participants mentioned it could potentially work as a "creative trigger" (P01,

P03).

7.2.1.5.2. Negative Points

The most highlighted negative points were: (1) the current mapping and sound (60% of the

participants). One participant described his experience with the current mapping as "slightly

disappointing" (P07), while another commented on the limited bank of sound and controls: "As

far as sonic exploration, I reached the end very fast" (P01); (2) its ergonomic features (mentioned

by 50% of the participants). One participant said: "It is big enough to pick up and move around,

but small enough that you do not feel you can drop it" (P01). Another participant mentioned that

the cubic shape does not invite one to touch it "because it is uncomfortable" (P09), a fact

mentioned by four other participants; and (3) the fragility of the current prototype (50% of the

participants). Five participants felt that the system was not robust, and, amongst them, two

mentioned they were afraid of breaking it somehow (though this never happened in the trials).

One participant mentioned: "I think that in performance, my feeling is that there are already

interfaces with which you can interact better" (P10).

86

7.2.1.5.3. Suggestions

Three participants mentioned that even though the system presents various controllers, "all you

have is two hands" (P06), and, one participant suggested richer combinations of control on a

single block by attaching different sensors to it. Two other participants suggested the addition of

a sequencer or a looper, because, as one of the participants said, "you don't need to have hand

busy, you [can] build [musical] layers" (P02). Another participant mentioned: "you can let the

system run instead of being in charge all the time, actually making a gesture for creating the

sound" (P10).

7.2.1.5.4. Contexts of Use

When asked about possible contexts of use in future version, the participants mentioned:

"brainstorming tool" (P01), "creative trigger" (P03), "creativity support tool" (P07), "quick prototype

physical design ideas" (P01), "mapping testbed" (P05), "preliminary test phase" (P05), "explore

ideas" (P08), "generate ideas" (P01), "experiment mappings" (P03). All of these answers are

encouraging since they denote how the toolkit and its associated methodology promoted the

generation and quick exploration of ideas to our participants, which was our main initial

hypothesis.

Besides, the toolkit was also interpreted as an instrument that could be used on performances,

for "artistic practice" (P05), as a "DJ controller" (P10), to "control effects" (P02). It was also related

to "versatile generic controller" (P08), for "video editing" (P05) or "light control panel" (P08).

Additionally, the participants commented about its use with children as a music learning tool or

as a musical game, at a classroom, or with people with disabilities during rehabilitation sessions.

7.2.1.6. Discussion

Although this first experiment is not a formal evaluation of our approach, the results indicate that

we were on a promising path. Participant responses concerning positive points and context of

use (e.g., "creative trigger", "brainstorming tool", "explore ideas", etc.) support our hypothesis that

Probatio provides structured paths for generating ideas through fast, functional prototyping ("you

put it, and it works", "reduce time to trial-and-error", "[the connection] is pretty solid"). There are

also confirmations that the proposed system could lower the entry barrier for designing or

customizing DMIs thanks to appropriate physical affordances ("pick me up and do something with

me", "quite intuitive", "self-explainable", etc.). Although we intended to provide only a toolkit for

prototyping DMI, some participants suggested that it could be used as the final DMI itself. We

believe that if the prototype were non-functional or did not properly react to input commands in

real time, this perception would not exist.

The ergonomic issues and the lack of robustness mentioned by the participants are directly

related to the straight-angled shapes and the material chosen for the current version of the

system. Following the approach of quicker cycles of prototyping, we designed this version of

Probatio to be the simplest possible concretization of our approach in order to validate its founding

87

hypotheses. We opted for simpler shapes and materials in detriment to ergonomics and

robustness only for the sake of speed. These are clearly issues that should be taken into account

for the next development cycles.

Finally, the negative mentions and participants' frustration about the current sound and mapping

produced by the prototype were expected due to our choice to exclude computer-based mapping

tools from this experiment in order to focus on the physical toolkit and design methodology.

Obviously, designing and evaluating an appropriate mapping interface for Probatio is mandatory

for the completeness of the approach, but it is a research problem per se that we are currently

working on.

88

8. EVALUATION OF PROBATIO 0.2 In this chapter, we present the version 0.2 of Probatio and an experiment comparing this version

with a generic sensor toolkit.

8.1. Evolution from Probatio 0.1 For Probatio 0.2 (see Figure 8.1), we analyzed the comments from the exploratory study and

implemented new features. As mentioned before, the most common negative point raised by the

participants regards the mapping and the sound. Participants mentioned that the sound result

was disappointing and the total combination possibilities were easily reachable. Mainly due to its

cubic shapes and the repealing edges, another weakness pointed by the participants concerns

the ergonomics of the supports and blocks. As a suggestion, the participants mentioned that it

could be interesting to aggregate different sensors in one block.

Related to those issues, development actions taken for this second version focusses on four

aspects: (1) the increase of blocks quantity; (2) the possibility of changing the mapping strategy;

(3) the enhancement of sound output module; (4) the addition of curved shapes on components;

(X) the integration of multiple sensors in one block.

Additionally, due to observation during the first evaluation, we made other enhancements: (5)

modification on the way the arm supports are connected; (6) adaptation on the way the supports

are connected to the hub; (7) hub enclosure; (8) modifications to decrease the friction between

the blocks and the slots border.

Figure 8.1: Probatio 0.2: (a) hub, (b) three taps, (c) fretless, (d) limiters for the sides, (e) four-slots support with curved edges, (f) four-slots support, (g) locks for four slots support, (h) cradle with

connector on the side, (i) cradle with connector on the bottom, (j) three-by-three support, (k) breath, (l) bellows, (m) buttons, (n) one tap, (o) turntable, (p) knobs, (q) crank, (r) joystick.

89

8.1.1. Number of Blocks and Multiple Sensors

For this version, we built more eight blocks and modified the four buttons blocks to have two

buttons instead. The new blocks are:

• Breath: inspired by the mouthpiece of wind instruments. It is made of a wind sensor based

on temperature measurement, which is commercially available for drones. The user's breath

rises the temperature, and the value can be measured.

• Two buttons: still inspired by instruments that have keys, switches or valves. We built more

three blocks with this configuration in an attempt to increase the possible ways of selecting

or triggering discrete events.

• One tap: inspired by percussion instruments. It consists of a piezo element attached to the

upper face of the block. The user strikes the top of the block and the piezo generates a current

that can be measured. The dynamics of the strike interferes on the output value: the stronger

the strike, the higher the value.

• Three taps: works in a similar way as one tap. This block tries to incorporate a suggestion

collected in the exploratory evaluation which was to integrate into one block multiples

sensors. It is inspired by the tambourine skin.

• Knobs: inspired by the presence of knobs in instruments such as electric guitars,

synthesizers, etc. It has two potentiometers with knob head attached to them.

• Joystick: it is related to the integration of different sensors into one block. It is made of a

game controller spare part and comprises two small potentiometers attached to a metal stick.

The user moves horizontally; the horizontal potentiometer changes its value. Vertically, the

vertical potentiometer is activated. Its normal position is in the middle, thus, with no user

action, both vertical and horizontal potentiometers marks half of the value each.

8.1.2. Changing Mapping Strategy

Probatio 0.2 uses the libmapper ecosystem (MALLOCH; SINCLAIR; WANDERLEY, 2014).

Libmapper is a library that can be used in different software languages and allows an application

to define input and output signals that can be manipulated through the network and mapped to

various devices on-the-fly without recompiling the code, or resetting any system. By using

libmapper, Probatio can leverage existing sound synthesizers that are already built as part of the

libmapper environment and can be easily integrated with musical software such as Max and Pure

Data (via the library), and digital audio workstations (via MIDI).

Using a web browser, the user can change the mapping strategy by using a graphical user

interface developed for libmapper called Webmapper, which consists of a tabular interface on

which the user can connect gestural input parameters to sound output parameters.

90

8.1.3. Sound Output Module

The sound output was developed in Max 7, using the percolate library and a simple

implementation of a modular synthesizer developed with Max/MSP default objects.

Besides other features, percolate (TRUEMAN; DUBOIS, 2015) implements a series of STK

(Synthesis Toolkit) (COOK; SCAVONE, 2004) physical models of sound synthesis and allows the

manipulation of synthesis parameters through Max 7 interface.

The libmapper externals allow the integration of libmapper environment with Max. Therefore, the

parameter inputs of both Probatio and the sensors toolkit could be mapped into the output

parameters of the physical models in Max.

We chose the physical models of a mandolin and a flute. For the mandolin, the participant could

control the following parameters: pluck attack, detuning, body size, general volume, continuous

frequency, and discrete frequency inside a chosen musical scale. For the flute, the parameters

available were: breath pressure, tone hole state, register state, general volume, continuous

frequency, and discrete frequency inside a chosen musical scale.

Additionally, for a simple synthesizer, the available parameters were: ADSR envelope, an

oscillator with three types of waveforms (sine, sawtooth, square), resonance, cutoff frequency of

a lowpass filter, note trigger, general volume, continuous frequency, and discrete frequency inside

a chosen musical scale.

8.1.4. Curved Shapes

Attempting to avoid the use edges, we experimented with adding curved shapes in a new arm

support. We chose to experiment on an arm in order to allow the user to perform smooth hand

translation between different positions. We used a technique of digital fabrication that allowed to

bend the MDF by placing kerfs on the board. As a downside of this new configuration, the new

arm support lost the side connections.

8.1.5. Connection Arm Support

In the previous version, the arm support could just connect to the communication bus through the

4x4 support. This fact limited the users to use it standalone. For broadening the possibilities and

allowing this use, we introduced in this version two cradles in which any block (and therefore, the

base of the arm) can be placed.

8.1.6. Protection and Connections to the Hub

We improved the connection from the supports to the hub by replacing the jump wires headers

by RJ-12 connectors. This improvement gave more stability to the connection allowing the user

to move the components without the risk of damaging the connection.

91

Also to improve the connection and avoid loose contacts, the hub was inserted in a MDF box.

8.1.7. Friction of Blocks and Slots

The spring-loaded pins of the block's side surface have a ledge of less than a millimeter, which

made difficult the insertion in the slot. For solving this issue, we opened a two-millimeters cavity

on the slot's border that allows the block to pass freely without sticking.

We noticed that the MDF material suffers from a sensitive interference of humidity which causes

its dimensions to increase. Due to this fact, some block's insertion became difficult. To address

this issue, we sanded the block's surfaces that enter made contact to the slot's border.

Also, we observed that the fretless object, which was originally designed as a 2-unit block with

one block with a hot connection and other only as an additional support, was hard to be inserted

having due to misalignment of the blocks, which provoked friction. For that, we removed the

additional block, decreasing the contact area.

8.2. Experiment In this section, we present the second cycle of evaluation considering Probatio 0.2.

8.2.1. Objectives

We conducted a controlled experiment to analyze the effect of using Probatio: a) on the time

duration to achieve a DMI functional prototype, b) on the cycles of idea exploration and evaluation,

c) on the diversity of the explored possibilities, and d) on the user involvement with the system.

Following the Interaction Design approach (PREECE et al., 2015), in this experiment, we focused

on usability and user experience goals. For usability, we considered three principles:

effectiveness (the ability to accomplish results with quality), efficiency (used resources to achieve

the results), and satisfaction (user’s subjective reactions) (BROOKE, 1996). Regarding user

experience, we observed positive and negative aspects of the interaction, such as user’s

engagement and frustration with the system.

As Probatio is a prototyping toolkit for idea experimentation, we consider that a measure of

effectiveness is the number of cycles of idea generation and idea evaluation, and the diversity of

the exploration of possibilities. According to Camburn (2015a), Beaudouin-lafon (2000c), and

Von Hippel (2001), the quality of the outputs can be related to the number of these cycles,

because the user will be able to modify the prototype in order to achieve adequate results

(CAMBURN et al., 2015) (BEAUDOUIN-LAFON; MACKAY, 2000) (VON HIPPEL, 2001).

Efficiency can be evaluated taking into account the time the user needs to obtain a functional

prototype. Satisfaction is assessed by understanding the engagement, and frustration of the users

when using the system.

92

8.2.2. Design

We used a within-subjects design, i.e. all participants used Probatio, and a generic sensor

system. The task was to experiment musical interaction ideas by building and modifying functional

DMI prototypes.

According to Preece et al. (2015), besides reducing the interference of individual differences, the

advantage of the within-subjects design is that the sample of participants can be reduced to half

when comparing to other approaches such as between-users design, where different groups are

submitted to different conditions. However, the order of the conditions can lead to biased results

due to learning from the first condition (PREECE et al., 2015). To deal with that, counterbalancing

measures should be taken. For instance, one approach is to randomly choose half of the

participants to start with one condition and the other half to start with the other condition.

The independent variable was the use of a system, and it was evaluated with two levels:

• Condition X (Probatio) and

• Condition Y (a generic sensor toolkit).

As it will be described in details in section 8.2.6.3, the generic sensor toolkit presents a set of

sensors that can be connected to a microcontroller using jumper wires and breadboard. It will

function as a baseline to compare aspects such as the presence of physical structure, and way

of connecting items.

Our dependent variables (DV) are:

a) the time duration to achieve a functional prototype,

b) the number of cycles of idea exploration,

c) the number of distinct items used during the idea exploration, and

d) the overall user experience.

Variables (a), (b), and (c) are intrinsically quantitative, can be measured through observation.

However, we decided not to reduce user satisfaction to quantifiable values only. Considering

discussions on the HCI literature (LAW et al., 2009), we opted to use qualitative methods to

discuss the user experience.

Probatio is a tool for helping DMI designers to experiment ideas through a hands-on approach by

having immediate functional prototypes. It is expected that the cycles of idea exploration become

shorter and more numerous in times. Our hypothesis is that by using Probatio, the user achieves

prototypes in less time and performs diverse modifications or adaptations in these prototypes.

Considering the user experience, our hypothesis is that the user feels more engaged, and less

frustrated when using Probatio as compared to the generic sensor toolkit. We summarize our

hypotheses in Figure 8.2.

93

Figure 8.2: Summary of hypotheses

8.2.3. Methods for Data Collection

8.2.3.1. Video Recording

In order to measure the time and the number of experimentation cycles, we decided to record video and audio of the participants’ interactions with each system.

8.2.3.2. Questionnaire

To help us understand user experience, we decided to use a 5-points Likert-based scale

questionnaire, a comparative questionnaire, and a semi-structured interview.

8.2.3.2.1. System Usability Scale

The questionnaire is based on the System Usability Scale (SUS), which is intended to be a “quick

and dirty usability scale” (BROOKE, 1996). Bangor et al. (2008) present four reasons for using

SUS to assess the usability of a product: (1) the scale is not dependent on the technology used

in the system, (2) the set of questions is easily understandable by researchers and participants,

(3) the result is a single score number that makes comparison simpler, and (4) the set of question

and the scale is not restricted by property or trademarks (BANGOR; KORTUM; MILLER, 2008).

According to Lewis et al.’s (2009), for comparative within-subject experiments, a sample size of

at least 12 participants is recommended (LEWIS; SAURO, 2009).

The 10-questions SUS survey is:

1) I think that I would like to use this system frequently.

2) I found the system unnecessarily complex.

3) I thought the system was easy to use.

4) I think that I would need the support of a technical person to be able to use this system.

5) I found the various functions in this system were well integrated.

6) I thought there was too much inconsistency in this system.

7) I would imagine that most people would learn to use this system very quickly.

8) I found the system very cumbersome to use.

9) I felt very confident using the system.

10) I needed to learn a lot of things before I could get going with this system.

94

8.2.3.2.2. Exploratory Questions

Additionally, we formulated three exploratory questions to understand user’s impression about

how fast a resulting prototype can be achieved. Although there is a recommendation in literature

that Likert-based questions should be extensively tested before the main application to avoid

misinterpretation (BROOKE, 1996), due to constraints in time and participants’ numbers we

decided to present the question without a prior test. Therefore, we label these questions

exploratory, and we report the results with caution because of the chance of misinterpretation.

• I think that the system allowed me to experiment diverse ways of interaction

• I think that the system allowed me to explore in a fast way different interactions

• I think that the system showed me possibilities that I had not seen

8.2.3.3. Comparative Questions

Besides the SUS survey and the exploratory questions, we decided to present a comparative

questionnaire which comprises 18 fill-the-gap sentences whose multiple choice answers were:

“Both”, “Neither”, “Condition X”, and “Condition Y”. The sentences attempted to assess negative

and positive aspects of the experience (e.g. frustration, engagement, and perceived level of

difficulty). We decided to use a comparative approach to contrast the user’s impressions about

the system in an attempt to extract more information. The sentences were:

1) I think I achieved more interesting musical results using _____.

2) I felt I tested more musical interactions using _____.

3) I felt I could try more things using _____.

4) I would imagine that most people would learn to use _____ faster.

5) I think the process was more laborious when I used _____.

6) I think I achieved faster results using _____.

7) I felt more engaged using _____.

8) I felt more bored using _____.

9) I felt more frustrated using _____.

10) I felt more confident using _____.

11) I felt that _____ were the most challenging for me.

12) Comparing the two systems, I felt more creative using _____.

13) I felt that _____ were the most inspiring for me.

14) I think I've explored everything that _____ had to offer me.

15) I think I'd like to use _____ more often.

16) I think I understood the operation of _____ more.

17) I thought that _____ were the most complicated of the two systems.

18) I thought that _____ were the easiest to use.

95

8.2.3.4. Semi-structured Interview

To have an in-depth understanding of the participants’ experience and better comprehend the

impression they had of both systems, we performed a semi-structured interview that covered

topics such as strengths and weakness of each physical interface, challenge, frustration,

engagement, and impressions about musical interaction.

The semi-structured interview followed the topics below:

1) Describe the experience with each system. Please, explain.

2) Describe strengths and weakness of each system. Why?

3) If applicable, describe difficulties during each interaction. Why?

4) If applicable, describe moments that you felt frustrated. Why?

5) If applicable, describe moments that you felt engaged. Why?

6) If available, which interface would you use in the future? Why?

7) Which system provided a better way for exploring musical interactions? Why?

8) Which system provided a faster way for testing an idea? Why?

9) If you wish, give suggestions, comments, or improvements.

8.2.4. Methods for Quantitative Analysis

8.2.4.1. Video Analysis

To analyze the quantitative variables, we adopted video analysis based on instrumental

interaction analysis (JORDAN; HENDERSON, 1995). Instrumental interaction is a set of “activities

driven by the manipulation of physical objects” (XAMBÓ, 2015), this concept must not be

confused with the instrumental interaction proposed by Beaudouin-Lafon (2000), which is an

interaction model to describe the coupling instruments between physical world and on-screen

objects (BEAUDOUIN-LAFON, 2000). According to Jordan and Henderson (1995), the video is

beneficial over basic written annotation during observations because it provides ways of revisiting

sequences multiple times (JORDAN; HENDERSON, 1995).

To annotate, navigate, and visualize the recorded videos, we used ChronoViz, which is an open-

source tool for annotating and navigating through time-coded data (FOUSE et al., 2011).

Considering the concept of segments presented by Jordan and Henderson (1995), we analyzed

the data and associated codes (or categories) to sub clips of video in which the users presented

a similar intention during the interaction.

For instance, the user placing a block in the slot is coded as “Mounting”. For the definition of the

coding scheme, we followed a bottom-up approach: we watched all the videos, found recurrent

actions, marked the beginning and end time, assigned preliminary codes, revisited the data, and

adapted the codes.

96

8.2.4.2. System Usability Scale Score

To measure the SUS value of each system, we followed the procedure presented by (BROOKE,

1996), which aims to balance the positive and negative sentences, presenting an overall scale

from 0 to 100. This scale should not be confused with percentage (BROOKE, 2013).

Bangor et. al (2008) presents an anecdotal way of interpreting SUS values: products which SUS

scores above 70 are “passable”, between the high 70s and upper 80s are better and truly superior

products score better than 90. Products below 70 should be enhanced (BANGOR; KORTUM;

MILLER, 2008).

A recommendation by Brooke (1996) is that the individual answers of the SUS survey do not

represent a particular result, because of their correlation. Therefore, only the combined result

should be considered (BROOKE, 1996).

8.2.4.3. Paired-samples t-test

To determine whether there was a statistically significant difference between the SUS means of

condition X and condition Y, we used a paired-samples t-test. Paired t-test are used in within-

subjects design to reveal if the mean difference between paired observations is statistically

significantly different from zero (SHESKIN, 2003). To use this method, the assumption is that the

dependent variable is measured at the continuous level, and the independent variable comprises

two categorical related groups (in our case, the same participant tests two systems). Therefore,

we chose this method because it suits our experiments variables and within-subjects design.

This hypothesis test was also used to determine the significance of the mean difference of the

duration to build the prototype, as well as the number of cycles in the two conditions.

8.2.4.4. Wilcoxon Signed-Rank Test

For the three exploratory Likert-based questions, we decide to use to Wilcoxon Signed-Rank Test to determine whether there was a statistically significant median difference between the two

conditions. Wilcoxon Signed-Rank Test is a hypothesis test for within-subjects design which

considers ordinal data. It can be considered the nonparametric equivalent to the paired t-test

(SHESKIN, 2003). Instead of considering the mean difference between the paired observation,

the method considers the median difference.

Much has been discussed in HCI literature that one should or should not treat Liker-based

questions results as interval data (KAPTEIN; NASS; MARKOPOULOS, 2010). A possible bias,

for instance, is that the participants’ perception of the Likert-scale may differ: for one participant,

the hypothetical distance between “Strongly Agree” to “Agree” may differ from another participant.

This makes the continuous comparison unmatchable. Clason (1994) suggests that Likert-scale

results should be treated as ordinal value (CLASON; DORMODY, 1994). For that, the commonly

used t-test should be replaced by a corresponding nonparametric method. Therefore, we opted

to use Wilcoxon Signed-Rank Test with Likert-based questions.

97

8.2.5. Methods for Qualitative Analysis

8.2.5.1. Thematic Analysis

Thematic analysis is a commonly used method in qualitative research in which themes or patterns

are analyzed and identified within the data (BRAUN; CLARKE, 2006). The themes groups similar

responses, or impressions across different sources of data. We used a top-down approach to

identify recurring themes and topics from the interviews’ transcripts. Besides, we determined the

prevalence of themes using the number of different participants whose quotations were related

to the themes. Instead of identifying the themes at a latent or interpretative level, we adopted the

semantic or explicit level of interpretation, which considers what was said by the participants, and

not what could be beyond what the participant said or what is written.

8.2.5.2. Coding Methods

According to Saldaña (2009), a code is “a word or short phrase that symbolically assigns a

summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based

or visual data” (SALDAÑA, 2009). Coding is the basic activity to perform a thematic analysis, and

we used the following coding methods presented by Saldaña (2009): Structural Coding,

Evaluation Coding, and Emotion Coding, that will be discussed in details in the Qualitative Results

section.

8.2.6. Setup

The experiment setup comprises three groups of materials: the software, the hardware, and the

experimental physical interfaces.

In an attempt to develop an environment of prototyping DMIs, we followed the DMI model

presented by Miranda and Wanderley (2006) (MIRANDA; WANDERLEY, 2006). The experiment

setup covered gestural control input, with the experimental physical interfaces; mapping, with

libmapper and its GUI (called Webmapper); and sound output module, with synthesizers

developed in Max/MSP.

8.2.6.1. Hardware

The software ran on an Intel Core i7 MacBook Pro Retina. A 24” Dell U2413 LCD Monitor with a

1920x1200 pixel resolution was used as the primary display, and the screen of the MacBook used

as the secondary one. On the primary display, right in front of the participant, the windows of

Webmapper and Max were placed, and, on the secondary display, the window of the plotter was

positioned. The speaker system was composed of a subwoofer and two satellites speakers. A

wireless Magic Mouse 2 was made available for the participant to interact with the GUI.

98

For registering the experiment, a DSLR Camera Canon Rebel T4 with an 18-55mm lens was

placed on the left side of the participant in a tripod, far enough for not interfering in the experiment.

Figure 8.3 and Figure 8.4 illustrate the desk layout for Probatio and GSToolkit, respectively.

Figure 8.3: Desk layout for Probatio: (a) Probatio, (b) LCD monitor, the main screen, (c) MacBook

Pro, the second screen, (d) Magic Mouse 2, (e) chair for the participant, (f) camera

Figure 8.4: Desk layout for GSToolkit: (a) GSToolkit, (b) LCD monitor, the main screen, (c) MacBook Pro, the second screen, (d) Magic Mouse 2, (e) chair for the participant, (f) camera.

8.2.6.2. Software: Mapping and Sound

We adapted the software in Java which is responsible for the communication between Probatio

and the computer via serial port to work also with GSToolkit. As mentioned before, the software

is also accountable for plotting the input values of on the screen.

Kept the same for both experiment conditions, the Webmapper was used as the mapping

graphical user interface.

Webmapper GUI and the Max patches in Presentation Mode were placed on the main screen.

The window with sensor values plots, developed in Processing, was located in the secondary

screen on the left-hand side of the participant. Figure 8.5 presents the main screen configuration,

and Figure 8.6 shows the window with plotted input values of the system (in this example,

Probatio’s items).

99

Figure 8.5: Graphical user interface on the main screen

Figure 8.6: Secondary screen with the plotter values of Probatio's items

For performing the mapping in Webmapper, the participant had to drag the mouse pointer from

the input-parameter located on the left-hand side and release it over the output parameter located

on the right-hand side. To delete a mapping connection, the participant had to click on the curve

which represented the connection and press the Delete key.

8.2.6.3. Generic Sensor Toolkit

The generic sensor toolkit (or GSToolkit to simplify for further description) is a system that we

developed exclusively for this experiment (Figure 8.7).

100

Figure 8.7: GSToolkit's components: 8 buttons, 4 piezos, 1 membrane potentiometer, 1 force-sensing resistor, 1 joystick, 2 rotary encoders, 2 rotary potentiometers, 1 hall-effect sensor, 1

breath sensor

The toolkit consists of 21 sensors, an Arduino Mega 2560, a 400-tie-points clear solderless

breadboard, and a set of A5-size cards with instructions (Figure 8.8) on how to assemble the

circuit of the sensors.

The sensors were presented with their circuit pre-assembled. In other words, all the needed

electronic components (such as resistors) were bundled together with the sensor and sealed with

hot glue. Our objective was to succinctly enclose technical details that would require for the

participants to have skills related to identifying electronic components and correctly using them.

This approach is similar to the iCubeX, a sensor toolkit developed by Mulder (1995) (MULDER,

1995). We believe that this complexity layer would reduce the ability to perform a comparative

experiment between Probatio and GSToolkit.

Figure 8.8: GSToolkit's instructions cards

101

Each sensor had a correspondent instruction card which presented the wiring connections

between the sensors, the breadboard, and the Arduino board. We used this approach as a

simplification of the internet search for information regarding a specific sensor. Our main objective

was to create a controlled environment with a reduced complexity regarding outside variables

such as internet speed, individual strategy to search such information online, previous knowledge

about existing source of such information, or other factors that would interfere in the session

duration.

Furthermore, we pre-programmed the Arduino microcontroller, so the participant did not have to

deal with coding or to use Arduino IDE. We chose to use gate pins that would activate or

deactivate the reading of a specific analog or digital port of the Arduino. Each sensor had its own

associated gate pin. Instantaneously, after following the instructions card, the sensor values were

plotted in a graphical user interface on the secondary display.

8.2.6.4. Correspondence Between Systems

Probatio presented all the correspondent sensors used in GSToolkit, but Probatio’s items

encapsulated more than one item in one block, besides having the physical structure. Figure 8.9

presents the correspondence between the two systems.

102

Figure 8.9: Correspondence between GSToolkit's items and Probatio's items

The structure and encapsulation cause perceptible differences in the interaction with each

system. For example, the block Two Buttons is equivalent to using two buttons in GSToolkit at

once, i.e. in Probatio with just one connection the user obtains two degrees of freedom (DoF),

and in GSToolkit, to obtain the same number of DoFs the user has to perform two assembly

actions. The block Fretless combines two sensors of GSToolkit (membrane potentiometer and

force-sensing resistor) in just one surface. Although using a Hall Effect Sensor inside, the block

Bellows is mechanically constrained by the upper part movement, while the user in GSToolkit can

experiment different uses with the sensor. The blocks Turntable and Crank present elements (the

disc and the crank) that change the interaction with the Rotary Encoder.

Although we maintain a sensor type equivalence in both systems, because of the structure and

the encapsulation we obtain different possible uses. In sum, in this experiment, we attempt to

assess the impact of these differences in the overall prototyping process, and in the user’s

engagement with the system.

103

8.2.7. Technical Test Pilot

To test the system setup with Probatio and GSToolkit in an attempt to discover some technical

issues that could interfere with further experiments results, we organized a workshop with the

students of the Computer Music The discipline presented a mixed group of programmers and

musicians from Computer Science, Computer Engineering, and Music courses.

For all the participants, we briefly explained how Webmapper and Max objects work. Then, the

class was divided taking into account programming skills, and musical skills and two mixed groups

were formed, group A and B, both with 9 participants. Group A received the GSToolkit system,

and group B used Probatio. The groups were then sent to different rooms, and they freely

experimented the systems for one hour and a half. They were asked to experiment musical

interactions focusing on finding ways to control the parameters of sound objects in Max/MSP. The

sessions were recorded on video for later analysis. Figure 8.10 shows the two groups

experimenting the systems.

Figure 8.10: Group A (GSToolkit) and Group B (Probatio)

The highlights of this preliminary test were:

• Probatio presented several errors mainly due to loose contacts between the spring-loaded

pins and the slots connectors. It caused the software to crash, which demanded the system

to be reset more than 5 times. The participants demonstrated frustration by sentences such

as or “the idea is exciting, but there is a lot of errors”. Besides, they were limiting their

interaction to avoid causing the system to crash, as illustrated by the sentence “It is better to

keep it there [talking about a block] to avoid causing errors”.

• The sensor kit presented no major errors.

• The group B dynamics can be described as a person in the middle doing all the work and the

other members around giving suggestions and instructions. The same person who was

mounting the sensors make the connections on the mapping GUI.

• Group A more people participated in the hands-on activity. More than five participants

grabbed Probatio’s blocks and inserted them in the slots. Only one person though controlled

the mapping GUI.

This pilot showed:

104

• The vulnerability of Probatio’s connections in two aspects: (1) the loose contacts between the

spring-loaded pins and the support’s connectors, and (2) the jumper wires connections

between Probatio’s supports and the hub.

• The lack of fault tolerance and system recovery in Probatio setup after the occurrence of a

bad connection. Several times, the system fully reset after simple errors.

• The stability of the sensor toolkit

For the main experiment, we reviewed Probatio structure and connections. Regarding the fault

tolerance issue, as an urgent solution, we used a second Arduino (Uno version). It works like a

facade: it is always connected to the computer even if the Arduino Mega resets due to Probatio

errors. It is not the most elegant solution, but we managed to deal with the errors in the main

experiment with this buffer-like bypass.

8.2.8. Participants

For the main experiment, we selected our participants based on a list produced in the early

exploration phase (Batebit project (BARBOSA et al., 2015a)) of popular musicians and musical

producers from Recife, Brazil, with interest or experience in music technology.

We enrolled 19 volunteers ranging in age from 19 to 50 years old (mean age 34.74, SD 8.87), 17

males and 2 females. All of the participants had experience in digital technology in music, played

at least one musical instrument, and had little or no experience with microcontrollers, digital

electronics, sensors, and programming languages (Figure 8.11).

Figure 8.11: Participants' profile. List of covered topics in order of mentioned experience: Musical

Experience, Digital Audio Workstations, Digital Instruments, Sound Synthesis, MIDI Keyboard,

105

Electric Instruments, Acoustic Instruments, Musical Theory, Mapping, Electronic Instruments, MIDI Controller Non−keyboard, Digital Electronics, Physical Interfaces Music, Artistic Programming

Language, Microcontrollers, Self−made MIDI, General Programming Language

Five participants were not able to perform the complete session due to schedule unavailability,

so we reduced the duration of their experiment sessions. Instead of not considering all the results

from these participants, we divided the 19 participants into two groups (G05 and G14) (Figure

8.12). For G14, we performed quantitative and qualitative analysis, and, for G05, we analyzed

only qualitatively the transcript of their interviews. Figure 8.13 and Figure 8.14 present the

participants using Probatio and GSToolkit respectively.

Figure 8.12: Group division. The participants’ codes follow the order of experiment sessions.

Figure 8.13: 19 participants using GSToolkit

106

Figure 8.14: 19 participants using Probatio

8.2.9. Experiment Protocol

The sessions took place with one participant at the time and lasted approximately 100 minutes

each (some adaptations were made for the group of five participants that could not spend long

durations, G05). As a counterbalancing measure, we randomly defined the order of the conditions

X (Probatio) and Y (GSToolkit) prior to the beginning of the experiment.

The participants were warned that the experiment would cause no harm and by continuing in the

session, they agreed of allowing the use of their image for research analysis purpose.

Each session followed the subsequent steps (illustrated in the Figure 8.15):

Figure 8.15: Within-subjects design with group X->Y starting with Probatio (condition X) followed by generic sensors (condition Y). And, group YX, using the generic sensors followed by Probatio

1. Brief Introduction (2 minutes): the researcher introduces the participant to the research,

explaining that the experiment is a part of a Ph.D. project which investigates new interfaces

for musical expression. The researcher states that the objective of the experiment is not to

107

measure individual abilities with digital technology and music but to understand the use and

the experience of using two different physical interfaces for musical interaction.

2. Profiling questionnaire (5 minutes): the participant answers a questionnaire that covers

experience in music, background in musical instruments, digital electronics, and

programming languages.

3. Explanation about the GUI (5 minutes): the researcher presents the graphical user

interface to the participant and gives some examples of how the mapping from input

parameters can be made to the sound output parameters using the Webmapper. The

researcher also presents the three sound devices developed in Max and mentions their

parameters. For the didactic purpose and to be used in this initial explanation about the GUI,

two Max GUI-based input devices were developed: a low-frequency oscillator with six waves

in different frequencies generating six input parameters for mapping; and a graphical

interface with three buttons and three sliders.

4. Interaction with GUI (5 minutes): the participant freely explores the GUI by her/his own.

5. Brief explanation about System [X or Y] (5 minutes): the researcher places the first

system on the table at which the participant is sitting. The researcher mentions all parts by

name and briefly describes what each one does. The researcher mounts one part and

presents how the sensor graph will plot the value. Then, the researcher shows how the

parameters will appear on Webmapper. Finally, the researcher says that the participant can

freely explore and use the system to experiment musical interactions.

6. Interaction with System [X or Y] (25 minutes): the participant freely explores the physical

interface and the mapping GUI. The video and audio of this interaction session are recorded

for later analysis.

7. Questionnaire considering System [X or Y] (10 minutes): the participant is asked to

answer a questionnaire on the computer in front of her/him. In the first question of the

questionnaire, she/he identifies which system she/he firstly used. The questions follow the

Likert scale from 0 to 5 (strongly disagree - strongly agree). While the participant answers

the questions, the researcher removes the physical interface from the table.

8. Brief explanation about System [Y or X] (5 minutes): step 6 is repeated for the second

system.

9. Interaction with System [Y or X] (25 minutes): the participant freely explores the second

physical interface using the same mapping GUI. The video and audio of this interaction

session are recorded for later analysis.

10. Questionnaire considering System [Y or X] (10 minutes): the participant answers the

same questionnaire from step 8, identifying the second system in the beginning question.

11. Comparative questionnaire (5 minutes): the researcher presents a comparative

questionnaire to the participant. The questionnaire is similar to the previous set of questions,

108

but instead of Likert-scale answers, the questions have only four options: “Both”, “Neither”,

“System X”, “System Y”.

12. Semi-structured interview (10-15 minutes): the researcher conducts a semi-structured

interview with the participant covering points such as strengths and weakness of each

system, engagement, frustration and perceived challenge and time spent, difficulties,

integration with her/his own musical tools and suggestions for enhancement. The video and

audio of the interview are recorded for later analysis.

8.3. Results

8.3.1. Quantitative Analysis

8.3.1.1. Video Analysis

After cycles of analyzing the video data and assigning preliminary descriptions to video segments,

we defined the following list of codes, which represent the interaction phases:

8.3.1.1.1. Code Scheme

• Mounting: the participant arranges the physical setup using the system’s components. For

example, placing a block in a slot using Probatio or connecting sensor wires using elements

of GSToolkit.

• Mapping: the participant changes the focus from the physical interface to the computer

screen, grabs the mouse and uses the Webmapper interface to map inputs to outputs

parameters, or adjusts some elements such as musical scale in Max.

• Testing: the participant uses the elements of the physical interfaces and expects either visual

feedback from the sensor value plots or sound feedback from the sound output modules.

• Thinking: the participant does not perform any action either using the physical interface or

the graphical interface.

• Asking: the participant stops what she/he is doing to ask the researcher about some doubt

of the physical interface or GUI.

• Bug: the system stops working due to an error and the researcher intervenes to solve the

problem.

8.3.1.1.2. Duration of the Interaction Phases

In order to help to visualize the data, we plotted the video segments of each session as bar

graphs, presented in Figure 8.17 (legends placed separately in Figure 8.16).

109

Figure 8.16: Legend for the video segments

Figure 8.17: Time segments of the interaction phases

As can be seen, two sessions took much less than 25 minutes: P09_X and P19_Y. It happened

because the participants said they had tested everything they wanted and asked to stop.

Figure 8.18 presents only the Mounting phases bars of Figure 8.17, and it shows that the larger

area of purple-colored bars in condition Y indicates that GSToolkit demands more time for the

user to build a functional prototype when compared to Probatio as expected. This preliminary

visual analysis is supported by paired t-test performed to determine the statistically significant

mean difference between the duration of Mounting phases in both conditions (t = -8.5796, df =

13, p = 1,03E-06). It appears that participants take longer to build a functional prototype when

using GSToolkit (M = 675.14, SD = 235.17) as opposed to using Probatio (M = 150.14, SD =

64.18). The mean increase is 525 seconds, 95% Confidence Interval [392.8031, 657.1969].

Figure DUR05 shows that the mean duration of Mounting phases in condition Y is slightly over

four times the mean duration of the same phase in condition X.

110

Figure 8.18: Mounting phases are highlighted, and other phases are grayed out

No statistical significant difference was identified in Bug phases duration between Probatio and

GSToolkit (t = 0.57981, df = 13, p = 0.572). However, Figure 8.19 shows that the incidence of

bugs was larger in Probatio sessions than GSToolkit sessions (six times more errors occurred in

Probatio than in GSToolkit).

Figure 8.19: Bug phases are highlighted, and other phases are grayed out

To have a better understanding of these errors, we revisited the videos and gathered more details

(Figure 8.20). As can be seen, the majority of bugs is related to loose contacts between the blocks

111

and the connection slots. The errors that happened in GSToolkit seems to be related to user

misinterpreting the instruction cards and placing the wires in wrong position, and a sudden serial

port failure in the computer. Therefore, the errors are not directly related to the system

malfunctioning. Additionally, the longer duration of a bug segment is in session P06_Y. The error

took longer to be discovered by the researcher because of the number of wires presented in the

participant’s layout. P06 developed a strategy of mounting all the sensors he wanted first and

then testing them.

Figure 8.20: List of Bugs

Figure 8.21 shows the visual differences between the duration of the two conditions. Paired-

sample t-test revealed a statistically significant mean increase of 183.71 seconds, 95%

Confidence Interval [47.01276, 320.41581], (t = 2.9033, df = 13, p-value = 0.01233) in Mapping

phases between X and Y. Besides, there was also a significant mean increase of 269 seconds

95% Confidence Interval [156.3708, 381.6292] (t = 5.1598, df = 13, p-value = 0.0001835) in mean

duration of Testing phases of Probatio when compared with GSToolkit.

Figure 8.21: Mapping and Testing phases are highlighted, and other phases are grayed out

112

The analysis of the following phases did not present any significant difference according to

performed paired t-tests: Asking (t = 1.4488, df = 13, p-value = 0.1711), Bug (t = 0.57981, df =

13, p-value = 0.572), and Planning (t = 0.63864, df = 13, p-value = 0.5341).

Figure 8.22 displays the mean of interaction phase durations in seconds. As revealed by the

graph, the Mounting phase duration is highly affected by the condition X and Y. It was somehow

expected due to the simplicity of Probatio’s way of connecting the components in the support that

did not require either any instructions to be followed nor the slightly elevated level of attention to

assembling the circuits.

Figure 8.22: Mean of Duration in Seconds

113

8.3.1.1.3. Quantity of Transitions Between Phases

In order to assess the impact of the conditions X and Y in the quantity of cycles of idea generation

and idea evaluation, we computed the transitions between the interaction phases. To help the

visualization, we plotted a directed cyclic graph considering the total numbers of transitions

between the phases (Figure 8.23). The nodes represent the interaction phases, and the edges

represent the transitions. The color scale of the edges is based on a progression from a cold color

(blue) to a hot color (red). Also, the thickness of the edges is related to the number of transitions.

The flow of the edge follows a clockwise direction between the nodes, for example, considering

the edges between Testing and Mapping, the edge of the right represents the transitions from

Testing phase to Mapping phase, and the edge of left represents the transitions from Mapping to

Testing.

Figure 8.23: Transitions between phases. The origin-destination flow of the edges follows a

clockwise direction. The thickness represents the number of the transitions.

As the graph displays, there is a visual difference (in color and in thickness) in the Mapping-

Testing and Testing-Mapping transition between the two conditions. This is supported by the

following paired t-tests that determine the statistically significant mean difference between the

conditions (Figure 8.24):

• Mapping-Testing (t = 6.6458, df = 13, p-value = 1.60E-05), mean difference: 26.57143, 95%

confidence interval [17.93377, 35.20909].

• Testing-Mapping (t = 5.1988, df = 13, p-value = 0.0001715), mean difference: 22.42857, 95%

confidence interval [13.10833, 31.74881].

114

Figure 8.24: Mean of Transition's Number

As presented in the Figure 8.25, the number of Mapping-Testing, as well as Testing-Mapping,

transitions in X is slightly over twice as many transitions of the same kind in Y. Again, we can

argue that the reduced duration of Mounting phase might have influenced the increase of

Mapping-Testing and Testing-Mapping transitions. As the time and effort are not being consumed

in the Mounting phase, we may speculate that the participants able to perform more tests with

what they have.

Based on the data of this experiment, the use of Probatio seemed to contribute to the increase in

the number of cycles between interactions. If we consider the relevant correlation between the

interaction phases and the idea exploration-evaluation cycles, we can assume that the use of

Probatio facilitates in the increase of cycles exploration-evaluation.

Figure 8.25: Number of transitions between interaction phases

115

8.3.1.1.4. Quantity of Distinct Items Used

Additionally, to the dependent variables described before, we also tested a different variable that

can be related to the diversity of the exploration in the restricted design space of both systems.

This variable is the number of distinct items used in the sessions. Items for Probatio are the blocks

and for GSToolkit are the sensors. If the participant used one item several times, it was counted

as one.

A paired t-test revealed a statistically significant difference between the mean of numbers of

distinct items used in X and in Y (t = 4.6186, df = 13, p-value = 0.0004811). The mean difference

is 2.714286, 95% confidence interval [1.444678, 3.983893]. It means that the participants used

on average almost three items more in Probatio than in GSToolkit. We can speculate that the

ease of connection contributed to the expansion of exploration. Figure 8.26 illustrates the mean

difference between the two conditions.

Figure 8.26: Number of Distinct Items

8.3.1.1.5. Postures and ways of holding

As a secondary measure of diversity, we also analyzed the number of postures and ways of

holding the items of the systems. The tabletop position, where the items rest on the table, was

the only one observed in GSToolkit. Beyond the tabletop position, participants using Probatio

experimented different ways of placing blocks and holding the supports. The different positions

are described in the list below with the number of appearances and presented in Figure 8.27.

• Position A: (1x) Block [bellows] placed on the side of support [3x3]

• Position B: (1x) Block [breath] placed on the side of support [3x3]

116

• Position C: (1x) Block [knobs] placed on the side of support [3x3]

• Position D: (1x) Block [threetaps] placed on the side of support [3x3]

• Position E: (1x) Support [four-slots with curved edges] placed on the side of support [3x3]

• Position F: (1x) Support [four-slots] placed in support [3x3] as tower

• Position G: (2x) Support [four-slots with curved edges] placed in support [3x3] as tower

• Position H: (1x) Support [four-slots with curved edges] placed in support [cradle] and held

as clarinet

• Position I: (3x) Support [four-slots with curved edges] placed on the side of support [3x3] and held as guitar

Figure 8.27: Positions different than tabletop experimented by the participants

We may conclude that in Probatio the participants expanded the structural exploration, while in

GSToolkit they maintain the same tabletop position. The lack of structure in GSToolkit seems to

be the cause of this limited exploration. Since the participants had to assemble the circuit in a

reduced space, they had little options to enlarge their searching space.

8.3.1.2. Questionnaires

8.3.1.2.1. Comparing the System Usability Scale

We followed the method for calculating the System Usability Scale (BROOKE, 1996) for each

session was computed: ((Odd questions - 1)+(5 - Even questions))*2.5. Table SUS01 shows the

computed values.

A paired t-test showed a significant mean difference in the SUS values between conditions X and

Y (t = 2.5117, df = 13, p-value = 0.02601). In Figure 8.28, the means are presented with 95%

confidence interval, calculated according to (MOREY, 2008), which takes into account the within-

subjects experiment design. The mean difference is 13.03571 with a 95% confidence interval of

[1.823559, 24.247870].

117

Figure 8.28: Means of SUS Score

With a SUS score of 75.53, 95% CI [83.46, 67.61], according to (BANGOR; KORTUM; MILLER,

2008), Probatio is considered to be above average, which is 70. If we consider the lowest value

of the confidence interval, Probatio is slightly below average. Oppositely, GSToolkit scored 62.5,

95% CI [70.43, 54.57], and, therefore it would be considered below the average and candidate

for modifications. In sum, if we consider that SUS measures the subjective opinion of the usability

of a given system, and this opinion leads to the willing of using this system or not, Probatio would

be ranked in a better position than GSToolkit.

8.3.1.2.2. Explorative Questions

Due to the reduced sample size, we used the Exact Wilcoxon Signed-Rank Test following the

recommendation by Sheskin (2003) (SHESKIN, 2003). Exact calculation for large samples can

demand a costly computation, but for small samples, it is a more accurate method. The tests for

the three questions are presented below:

• Q18: Z = 1.889, p = 0.125

• Q19: Z = 1.354, p = 0.242

• Q20: Z = 2.111, p = 0.063

Thus, neither of the median differences of the three questions reached statistical significance. We

retain the null hypothesis that their median difference is equal to zero. Because of this result, we

can assume that the questions do not provide room for a comparative discussion. However, for

118

general reference only, the results are presented in Figure 8.29. As can be seen, in general, the

participants tended to evaluate the three questions with high values.

Figure 8.29: Results of Exploratory Questions

119

8.3.1.2.3. Comparative Questions

Figure 8.30 shows the percentages of answers for the six first questions of the comparative

survey. As can be seen, the majority of the participants chose Probatio for issues such as

interesting musical results, more musical interactions, and faster results. When asked about the

most laborious experience, almost 90% of the participants chose the GSToolkit. We can probably

state that, in terms of user’s perceptions, Probatio seems to be more associated to providing

quick ways to achieve more results.

Figure 8.30: Comparative questions: "01 - I think I achieved more interesting musical results using _____.", "02 - I think I achieved faster results using _____.", "03 - I felt more engaged using _____.",

"04 - I think I understood more the operation of _____.", "0

120

Figure 8.31 displays questions related to participant’s feelings, e.g. engagement and frustration

with the systems. As it is shown, the engagement and inspiration sensation measured by question

07 and 13, respectively, presents an almost equal distribution, with the highest percentage of both

systems. With that, we may conclude that both systems are equally engaging and inspiring for

the participants in this experiment. For the inspiration, the participants were more inclined to

Probatio.

Additionally, the majority of the participants responded that neither one of the systems was boring

and that GSToolkit was the most challenging of the two systems. We may summarize that even

though the participants found GSToolkit laborious, they engaged with the system. However,

almost one-third of the participants said that GSToolkit made them fell more frustrated. Frustration

is a feeling associated with not achieving what one desires. Therefore, we could say that although

GSToolkit was engaging and not boring, part of the participants did not accomplish the intended

results. Question 12 plot (Figure 8.31) shows that the user’s perception of creativity tends to be

greater using Probatio, but we cannot conclude that GSToolkit did not provide such feeling

because the majority of the responses was that both equally contributed to this feeling.

Figure 8.31: Comparative Questions: "07 - I felt more engaged using _____.", "08 - I felt more bored

using _____.", "09 - I felt more frustrated using _____.", "10 - I felt more confident using _____.", "11 - I felt that _____ were the most challenging for me.", "1 2 - Comparing the two systems, I felt

more creative using _____.", "13 - I felt that _____ were the most inspiring for me."

121

According to Figure 8.32, with both systems, the majority of the participants felt they did not reach

the level of total exploration. This extra room for exploration may contribute to the interpretation

that there is more potential for future engagement in both systems. Answers to questions 17 and

18 show that participants tend to say that the most complicated system is GSToolkit and the

easiest to use is Probatio. Considering the will of using the system more often, Probatio is better

ranked with almost half of the participant’s answers. These answers may contribute to the

conclusion that, due to the ease of use, quicker and faster ways of achieving results, Probatio is

a better candidate for future explorative incursions.

Figure 8.32: Comparative Questions: "14 - I think I've explored everything that _____ had to offer me.", "15 - I think I'd like to use _____ more often.", "16 - I think I understood the operation of _____ more.", "17 - I thought that _____ were the most complicated of the two systems.", "18 - I thought that _____ were the easiest to use."

Although the comparative questions may have limitations due to participants possibly

misinterpreting details, Probatio seems to be more associated with a quicker way of achieving

results and achieving more results.

122

8.3.1.3. Summary

In this Section, we present the summary of the quantitative analysis.

The results are:

• Mounting phases in GSToolkit were longer than in Probatio

• Bugs were more numerous in Probatio than in GSToolkit

• Testing and Mapping phases were longer in Probatio than in GSToolkit

• Mapping-Testing and Mapping-Testing transitions were more numerous in Probatio than in

GSToolkit

• The number of distinct items used in Probatio was greater than in GSToolkit

• System Usability Scale score of Probatio was above the considered average of 70

• SUS score of GSToolkit was below the considered average of 70

• SUS Score of Probatio was greater than GSToolkit’s

• Probatio seemed to be more associated with providing quick results

• Both systems seemed to be equally related to engagement and inspiration

• Neither system was considered boring

• The sensation of creativity tended to be greater using Probatio

The majority of the results corroborated with our expectative. However, the number of errors in

Probatio did not happen as projected. Surprisingly, participants using GSToolkit did not make as

many mistakes as we expected due to the harder connections.

8.3.2. Qualitative Analysis

The interviews were performed in Portuguese, transcribed and also analyzed in Portuguese. After

that, in order to present the results, we translated the used quotations and codes to English.

Our thematic analysis was cyclic and iterative. We chose to start with a top-down approach, in

which we defined the concrete elements present in the experiment. We used the method of

Structural Coding (SALDAÑA, 2009), which follows an initial structure to label and index the data

in order to make the access quicker.

Our top-down structural codes were:

• BLOCKS: covers comments about Probatio.

• SENSORS: comprises comments about GSToolkit.

• GUI: is composed of quotations about the graphical user interface.

• SOUND: is related to comments about the sound output.

After identifying each quotation with these codes, we started to categorize them in positive, or

negative comments. This is based on the Evaluation Coding, which defines positive and negative

magnitude codes to data (SALDAÑA, 2009).

123

In this first cycle, we also defined the following codes in a bottom-up approach, which means they

appeared during the coding:

• COMPARISONS: covers the sentences or paragraphs that were related to explicit

comparisons between BLOCKS and SENSORS.

• GENERAL: comprises general statements about the participant’s musical practice or

experience.

• PERSONAL: is related to sentences about personal characteristics of the participant.

• EXPERIMENT: consists of impressions about the experiment session.

Additionally, we followed the Emotion Coding method, which attempts to find parts of the text

related to feelings or sensations. This method is appropriate for exploring “interpersonal and

interpersonal participant experiences and actions” (SALDAÑA, 2009). The following codes were

defined and are presented with the number of participants in parenthesis that mentions related

sentences to the code: engagement (14), frustration (8), challenging (3), creativity (3), fun (3),

feeling like teacher sparrow or mad scientist (2), flow (2), rewarding (2), boredom (1), curious (1),

possessiveness (1), worried (1). The elevated number of “engagement” and “frustration” mentions

is related to the questions of the interview. Suggestions and contexts of use were also codes

defined during the first cycle.

Codes are the basic element of the thematic analysis (BRAUN; CLARKE, 2006). The theme is a

more abstract construction, and it is the result of the integration of related codes. For that, we

performed a second major cycle to identify common themes. The identified themes were:

8.3.2.1. Number of Components

Although one participant positively mentioned the number of blocks in Probatio as a pleasant environment to experiment ("It's a cool feeling, and since it has a lot of stuff, it's a really fun

feeling." (P12: 2)), two participants found that was too much information to process at first

("Because the blocks were a lot of things, a lot of information, a lot of possibilities" (P10:24)). One

of them also mentioned that the experience was overwhelming ("But I felt intimidated by a

number of things. I was a bit overwhelmed." (P15:3)).

8.3.2.2. Thinking Before Building and Experimenting Fewer Options

Three participants (P10, P15, P19) comment that GSToolkit process for building the prototype

induce them to follow a logic ("[The sensors] compel you to a logic." (P10:25)), and, because it

was laborious to connect the circuit ("The work required to assemble the sensors is much larger

than in the blocks, of course." (P19:6)), they took a strategy of thinking what they want before

and then building the prototype ("I think I thought a bit before in what I was going to do, as

opposed to fitting things." (P15:5) and "Then the process was, instead of trying something, I

thought first what I wanted" (P19:4)).

124

Because of that, some participants also mentioned they felt more focused ("There I was much

more focused" (P15:6) and "I felt more compelled to already have a focus than in a scattered

way" (P19:6)). Due to this focus, one of them stated that he experimented fewer options

("Instead of trying various things, I was more directed." (P19:3)).

In the opposite direction, Probatio might have provided an expansion in the experimentation space, as one participant said he could experiment more things at the same time ("I think one

took me a lot more to experiment with several things at the same time [the blocks], while the other

[the sensors] was more closed." (P19:7)).

Other participant felt that he experimented more possibilities, which caused him to feel more excited, and more engaged ("I think that, with the blocks, you can try more, so it gets you more

excited. It makes you more engaged in using that device." (P11:9)).

8.3.2.3. Following Instructions vs. Free Exploration

Regarding GSToolkit, the process of reading the instruction cards was mentioned as

challenging and rewarding ("Seeing the drawing and connect, it gives a stimulus .... 'Damn it,

I'm connecting a few wires here, and it's working.'" (P03:15)).

One participant felt that the cards help him attenuate the learning curve of using GSToolkit ("I

did not have a learning curve. I knew I had the cards each explaining what each sensor was and

how to connect them." (P15:8)). The participant suggests that with the cards, he did not have to

spend energy trying to understand the system, he just had to follow the instructions ("I had

to find out how those things were working" (P15:9)).

However, some participants mentioned the opposite. Even though Probatio did not have

instruction cards, in a comparative manner, one participant mentioned that with Probatio they did

not need to follow instructions ("You do not have to be looking at a manual to connect the stuff."

(P11:3)). Because, intuitively ("When I started interacting with the blocks system, I found them

quite intuitive." (P19:1)), they could mount a simple setup, and quickly achieve musical results.

8.3.2.4. Potential of the GSToolkit

In total, eight participants (P01, P04, P06, P07, P08, P09, P10, P14) made comments about the

potential of customization and more freedom to arrange the position of GSToolkit

components ("Sensors can give greater freedom because they are not already structured."

(P06:5)).

One participant felt they could arrange things the way he wanted ("It gave you the chance to

assemble the way you want." (P10:1)). Related to this, another participant mentioned that he

enjoyed more the experience with GSToolkit ("As I very much like to [...] adapt to my form, I

found the sensors more interesting" (P04:1)).

For some participants, the manipulation of wires ("I liked that feeling with the wire." (P03:9))

evoked a good sensation and feeling like a mad scientist ("Sense of openness to a world of

125

possibilities that you can develop with that there, a mad scientist thing" (P16:8)) or Gyro Gearloose (Disney’s fictional character, an inventor of Donald Duck’s universe. In Portuguese:

“Professor Pardal”) ("Here the sensors ended up being [...] I felt more like a Gyro Gearloose"

(P03:8)).

These positive impressions regarding GSToolkit’s structural freedom seem to go against the

negative positions about the lack of physical support and the restricted space of interaction

around the breadboard, mentioned by nine participants (P01, P02, P03, P08, P10, P11, P13, P14,

P16). This contradiction can be possibly explained by the fact that the participants seemed to be

talking about GSToolkit’s potential and not only about the implemented version for the

experiment (“The experience with sensors, it's more open, it gives you a sense of openness. It's

as if you knew, there was a very clear potential there" (P16:2)).

Comparing to this GSToolkit’s potential, Probatio seemed to be more limited ("It has a limit

because it already has pieces already pre-made. It has how to customize, but it cannot go on

such a deep level." (P10:6)).

8.3.2.5. Shape Limitations

One participant mentioned that Probatio was not ergonomic ("It is not very ergonomic." (P08:14))

and some reasons may have emerged, such as the cubic shapes and the pointy edges ("I think

the shapes, the very sharp edges, the square thing, is playful, but it needs to evolve a lot so I find

it comfortable to interact with really" (P16:24)).

Also concerning shape limitations of Probatio, one participant felt frustrated with the blocks that

did not fit together in Probatio ("Having a stuff that is bigger or smaller and does not fit right."

(P09:15)).

8.3.2.6. Physical Support

The presence of physical support in Probatio was positively highlighted by nine participants

(P01, P02, P03, P04, P05, P10, P11, P14, P15). For some participants, the object seemed to

cause an attraction ("It is here [the blocks] has a lust for the object, for the thing." (P08:31)).

Due to the structure, the Probatio’s blocks were related to musical instruments ("But I think the

blocks, they look more like an instrument" (P08:25)). As a consequence, one participant

mentioned that Probatio’s supports could awaken in the user a familiarity with instruments that already exist, and, even if the user does not play a certain instrument, one can connect with

known elements, be it shape or gesture. ("It has a lot of similarity to a wind instrument by the

format. So you kind of automatically make a reference, it connects with these canonical

languages, things that even people who do not have previous experience with musical

instruments can understand that there are languages there. I think this is cool"(P16:11)).

Because Probatio seemed to be closer to a final product, some participants stated their

experience was more engaging ("With the blocks, you engage more, you have a greater

126

engagement, because everything is ready, you do not have to assemble. It generates a greater

engagement" (P12:11)).

Additionally, a participant felt he had more time to explore ideas ("It is more ready, so give more

time to explore things." (P02:8)), which is supported by another participant ("It's ready, and I use

it faster" (P16:13)).

One participant felt that there was a good balance in adaptability and readiness (the perception

that the system presents well-defined structure, closer to a final product) of Probatio ("So this

balance is cool: it's very customizable and, at the same time, it's very ready too. You can quickly

fit it" (P16:10)).

One participant mentioned that Probatio was robust ("The block I felt it robust, I did not feel it

fragile. The weight, the way they are stuck I felt something rigid, something firm. I did not feel it

could break" (P07: 27)).

8.3.2.7. Restricted Space

In the opposite direction, because of the limited length of the wires in GSToolkit, depending on

the number of sensors, the interaction could become difficult ("The extension of the wire, then

you start to get a lot of things close" (P07:22)) in different aspects, such as restricted space,

fragility, disorganization, lack of fixed place.

Firstly, in terms of space ("So, I found it less ergonomic because everything is too tight" (P02:1)),

the component tends to be restricted by the length of the wire, and close to the breadboard.

The connection between the wire and the connectors is only an insertion with no additional lock

mechanism, and it was considered by one participant as fragile ("So, the wire is fragile. If you

move, with the touch it will release and disconnect there [in the GUI]." (P07:21)). With that, the

risk of making a mistake increases ("The way the sensors were, they're open, so any bullshit

you pull a wire, you have to put it back." (P10:3)).

8.3.2.8. Disorganized, Fixed in Place and Musical Interaction

Because the sensors were hanged by the wires and with no additional physical support, some

participants mentioned that the setup become disorganized ("The big problem is that things get

disorganized, hang by the wires." (P01:3)), directly impacting on the usability of GSToolkit ("The

fact that things are hanging by wires makes the usability it very difficult" (P01:5)). On the other

hand, one participant said the set up with Probatio was more organized ("It was much easier,

more organized for you to use." (P13:5)).

Without the physical support, the sensors tend not to be fixed in a certain place ("Each time you

use, those sensors will be in different places." (P10:8)), which may have caused difficulties in the

interaction ("To interact, things are loose on the table, so it was bad to maneuver, to interact with

the sensors." (P13:4)). One of the consequences may be the interference in the musical

127

precision ("But the thing of them being thrown to the side of the board, they end up becoming

less accurate musically speaking." (P14:23)).

Regarding Probatio, one participant mentioned that the structure could benefit musical interaction ("So when you have the controls already stuck rather than loose with the wires,

musically speaking, to get into musical notes, I found the blocks better" (P14:24)).

8.3.2.9. Reduced Dimensions

Due to the reduced dimensions of GSToolkit’s components, some participants mentioned that

it was difficult to assemble the circuits (P01, P02, P06, P08, P13, P14, P15, P16).

They stated that the wires and connectors were small ("That fucking little wires, you want to get

to the sound!" (P16:5)), the breadboard was small ("The little board being small makes it a bit

difficult" (P14:9)), and the size of the labels on the board was small ("You keep looking at these

little letters." (P02:3)).

8.3.2.10. Long Time

Because of the reduced dimensions, some participants felt they made some mistakes ("In the

case of the sensor, I took off my glasses so I could see right and put it in. I was wrong sometimes"

(P15:12)), and sometimes felt confused ("I found the two similar, but with the sensors I had more

difficulty because of the connections on the Arduino. It confused me a little." (P18:1)). As a result,

some participants felt they were losing time ("I put the pin in the wrong place, I lost time with it."

(P13:2)).

Additionally, reading the instruction card also interfered with the time spent to build the

prototype in GSToolkit ("Because it takes some time for you to read the card, it takes some time

for you to pick up those little things and connect ..." (P16:4) and "The Arduino's for me was slower

because I had to search the card to find out where are the connections." (P01:8) and "You spend

some time to see this thing: how do I connect it? Where is it? "(P02:2)).

All of these individual comments may lead to an overall perception of difference duration to

achieve results in the two systems ("We take a lot more time to test than with the blocks." (P13:3)

and "I would say that with the sensors you take longer." (P14:25)).

With GSToolkit, some participants mentioned that the time to achieve something satisfying was longer ("But the working time is much higher for us to generate a cool thing." (P08:21) and

"it took me a long time to produce more sound results" (P17:3)). Probably, one of the causes is

the effort of performing numerous tasks to add only one input parameter ("The tricky thing

about it [the sensor interface] is because there are many connections you have to make to use a

single sensor." (P11:4)).

This excessive time to achieve a musical result might have caused a feeling of frustration in

some participants ("Maybe at most with the sensors I have felt a little frustration about the work

of assembling." (P19:9)). And, as quoted before, also anxiety ("That fucking little wires, you want

128

to get to the sound!" (P16:5)). In some cases, even distress ("Because what interests me is the

sound, I want to see the sounds right away... And then those stages of having to connect the

wires and having to be right, it will take longer and cause a little more angst." (P12 :5)).

Because of that, some participants mentioned that the approach required to deal with GSToolkit

should be different, calmer ("Actually, [with] the Arduino you have to be calm. It's another timing."

(P08:5)).

However, despite all of the negative comments, some participants find that GSToolkit was

engaging and funny to use ("I had a lot of fun with this one [the sensors]. I would spend the

afternoon traveling around here [sensors]." (P02:18)). This might be explained by two different

kinds of engagements: technical vs. musical, that will be discussed later.

8.3.2.11. More Parameters in Just One Hand

In both systems, participants seem to want to control more parameters with just one hand

("Sometimes you want to interact with three things at the same time." (P13:8)).

The lack of a physical support in GSToolkit causes the participants to use one hand for support and the other for manipulation ("I had to use both hands for each one." (P01:12)). The participants

expected to use one hand to control more than one musical parameter, and with GSToolkit it

seemed to be impossible ("In the second [the sensors], it is almost impossible to touch and have

the same hand to generate various information" (P08:27)).

Although some participants found that Probatio was better to use than GSToolkit ("Physically,

they [the blocks] are more comfortable to use." (P03:13)), the system caused frustration to other

participants due to the difficult interaction of using one hand to control various parameters ("It

was more of a frustration in the matter of trying to coordinate the commands there [...] I wanted

to do several things at once, and I could not do it" (P18:9)).

One participant explained that this limitation is due to the distance between the blocks ("In the

blocks, this distance between one object and another ... sometimes makes it difficult, for example,

to create forms of performing [...] by taking advantage of the movement of a single hand"

(P08:36)). The same participant even attempted to stretch his hands with no success ("And I

already have a big hand, I already had to stretch a lot and sometimes I could not." (P08:26)). He

concludes that allowing the user to control more parameters with just one hand may contribute to

developing a performance repertoire of gestures for some combinations of blocks ("Because

then it would end up being so experimental and you would already start to create a culture of

performance." (P08:35)).

8.3.2.12. Easy and Quick

The quick way of connecting the blocks was positively highlighted by fourteen participants (P01,

P02, P04, P07, P08, P09, P10, P12, P13, P14, P15, P16, P17, P18). Some participants

commented that they could reach musical experimentation with less time ("Here [the blocks] I

129

move to solve musical things faster" (P02:19) or "I felt the sound results of my connections more

quickly" (P17:17).

There seems to be an initial barrier to using GSToolkit ("Actually, it's difficult to start using. It's

not of using, it's to start using." (P09:6)). One participant mentioned the assemblage as a barrier to achieve musical results ("With the sensors, you get this feeling, that you first have to make a

connection that has nothing to do with sound." (P.12:6)), and he concludes Probatio might reduce

this barrier ("It is a feeling of not having to make an assembling before you start to make sound"

(P12:7)). Other participant did not consider GSToolkit boring, but that it took longer to test musical interactions ("This thing of having to connect the wires, it's fun too, but it takes longer."

(P12:4)).

One participant also mentioned that quality of the musical results with the blocks would be

higher (“I would get better and faster results with the blocks" (P12:14)). Related to this, other

participant said that one could reach a higher level of musical experimentation with Probatio

("The blocks have already been more practical, artistically, they already comprise languages, try

languages, try clichés, try performative musical organizations" (P17:20)).

One participant made a correlation between the quickness and the creative flow, mentioning

that it was easier to reach a creative state ("For me, I felt ... working faster, which then lets

creativity flow more easily with the blocks" (P14:12)).

8.3.2.13. Urgency in Reaching Musical Results

One participant highlighted the urgency of the musician in general to reach a musical result ("At

the time that we are creating, at the time that we are composing something, we want to touch C

and play C" (P18:6)). It is a creative necessity ("So the faster you can do what you imagine, the

better." (P14:7)).

Another participant said that was easier to have a musical experience with Probatio than with

GSToolkit ("For me, the musical experience was better on the blocks because I got to get to a

musical thing more easily." (P14:22)).

Some participant mentioned that they could reach musical experimentation faster with Probatio

("You jump right into the sound issue, musical." (P02:22) and "I think that with the blocks I could

go more directly to what I wanted to do." (P18:2) and "On the block, it was a most immediate thing

to fit in and get started, to get somewhere "(P17:5)).

Participants mentioned that Probatio was not only fast to build prototypes ("I have tried many

more things and gained much more" (P12:16)) but also to modify them ("[In blocks], it is much

easier to modify." (P10:15) and "The other was one more like putting, and it did not work, I would

change the other, and I would try it until I found the cool one" (P07:7)). Other participant compared

his experience with both systems and highlighted that placing and removing a block was easier than managing the cable connections ("With the blocks, due to the ease of experimentation.

You put a block, it did not work, you remove it, you put another," ah, here I can control what I

130

want to control", so it's easier than you take four wires and put four more. "(P11:12)). Another

participant summarizes ("Everything is easier to put on, to test." (P13:17)).

Because of that, some participants seemed to enjoy the interaction ("With the blocks, this thing

is much faster. Thought, imagined, picked up, fit, it already appears on screen, that's really cool."

(P14:10) and "I liked the quick connectivity." (P08:1)). Besides, some considered that the musical experimentation was better with Probatio ("There is the musical engagement, of the musical

interaction, which was better with the blocks because I arrived faster" (P16:16)). With Probatio,

some participants reached a sound result more quickly ("I felt more quickly the sound results

of my connections" (P17: 17)). A participant summarized his impression of Probatio if he could

use it for a musical presentation ("Imagine and quickly assemble." (P14:21)).

One participant was excited to describe the easy and quick use of Probatio ("I found it all very

intuitive. The blocks I found to be fantastic, phenomenal. The duration [of the session] makes you

want to spend more time experimenting" (P07:4)). Other participant mentioned that Probatio is

efficient, effective, feasible ("In the blocks, it is much more efficient, much more effective, much

more possible." (P12:17)).

Due to a more direct approach towards musical interaction, one participant concludes that the

blocks would be interesting for musicians ("At first, for a musician, the block is much more

interesting." (P13:10)).

8.3.2.14. Cognitive Load and Creative Flow, Kinds of Engagement

Five participants (P02, P09, P13, P16, P17) explained they felt two kinds of engagements while

using the systems ("So for me, I feel engaged by the two. Now, for different environments,

different circumstances." (P17:13) and "I have been involved with both of them, but they are

different engagements" (P02:17) and "Then the two gave me different engagements" (P07:16)).

Other participant contrasted the engagement of assembling something in general and the

musical engagement ("Because there is the engagement that is of the assembly, that is you

prepare the setup that you are going to use, this is very stimulating, very engaging. That was the

same in both. There is the musical engagement, the musical interaction, which was better with

the blocks because I have come more quickly." (P16:15)). One participant distinguished the two

experiences in two perspectives: aesthetic and structural, or artistic and engineering ("So this is

not an artistic, aesthetic issue, this is a more structural, engineering issue, so to speak." (P17:21)).

The sensors seemed to be more associated with the engagement of dealing with an assembly kit ("It has the kit thing for you to assemble" (P14:12)), and one participant highlighted the rational aspects of using it ("Making the sensor work, for me, is more rational than sensory.... It is, in my

view, more logical, rational" (P17:9)).

These engagements seem to differ in time ("It's another kind of involvement, which I also enjoy,

but it's a different time, a different timing." (P17:7)), and in objectives ("But the idea here [in the

sensors] is an idea of interface, the idea here [in the blocks] is an idea of sound." (P02:35)).

131

On the other hand, Probatio may have contributed for the participants to have a more intuitive experimentation ("I had a moment in the use of the X [the blocks] that I entered into another

mode. Let's say, a less cognitive and more intuitive way." (P02:26)). Here, intuitive is not related

to the ease of use, but to exploring sensorial stimuli instead of solving problems. He also

mentioned that when he reaches this intuitive level, he does not want to deal with technicalities ("And then I do not want to think about the connection anymore ... I want to do

something else, I want to put it there, map and feel it" (P02:30)). Another participant supports that

("Making the sensor work for me is more rational than sensory" (P17:9)).

Solving problems may be associated with GSToolkit ("But here I have to solve more technical

and ergonomic things" (P02:36)).

Probatio might have interfered in the perception of engagement of some participants ("I think if

one watches the video, I'm talking much less. I'm more absorbed in that process." (P02:32)), and

in the time perception ("The second moment [the blocks] passed much faster. I was really more

involved." (P05:3)). This seems to be related to the definition of “distortion of temporal experience”

described as one of the characteristics of being in creative flow (NAKAMURA;

CSIKSZENTMIHALYI, 2014).

In this dichotomy between engagements, one participant mentioned the Maker movement, and

associate it with the technical engagement in contrast to the musical engagement ("I had a good

deal of engagement in the two of them. In the first [sensors], more by a maker thing than a musical

thing." (P13:12)).

One participant said he was always aware if he had made some wrong connection or whether

everything is working properly ("You do not have to wonder if you have connected the ground in

the right place or the right slot for it to be recognized." (P01:14)). Thus, other participant mentioned

he was always alert ("In this guy here [the sensors], you are thinking all the time" (P02:33)). In

this context, the cognitive overload can probably become higher ("This option is a very

interesting option [the sensors], but it has this cognitive cost" (P02:4)), which is related to effort ("It has more intellectual wear [...] mentally demands more, I think" (P07:13)). Physical and mental effort ("It's fun, [...] but it's frustrating because you feel you have a more physical, strategic

clash there to have a sound." (P07:23)).

The constant change between these two kinds of thinking can probably interfere with the

creative flow ("So when you move to this other level something goes there and slaps you, and

you have to go back... Then your creative part gets stuck." (P02:31)).

The participant explicitly illustrated his experience with GSToolkit with the metaphor of jumping trees ("You always keep jumping from side to side" (P02:34)), where you mount the circuit and

tries to play it.

With Probatio, the technical encapsulation seemed to contribute to reaching musical

experimentation more quickly ("Here [the blocks] the technique you do not think very much. You

forget it fast... You jump at once for a sonorous, even musical question" (P02:16)).

132

One participant said it is important to not have interruptions during the flow ("That there really

is the moment that the system needs to be ready for the person to have a flow ... Because,

otherwise, you stop. And then, fuck, you come back to ... " (P02:27)). Therefore, it seems to be

important to divide the technical and the musical processes apart ("There's a moment that I

want to forget about this, and I want to know that I'm going to play the timbre, here I'm going to

close the filter, here I'm going to play the pitch ..." (P02:29) and “I do not want to think about the

connection ... I want to do something else; I want to put it there, map and feel" (P02:30)).

However, the technical engagement seems to be empowering ("But it's more empowering, for

me, you put the wire there and ‘fuck, it went wrong, it worked’, sense of belonging." (P15:17)),

and contributes to the sense of ownership or authorship towards the process ("You gain greater

possessiveness, a greater authorial feeling for what you are doing, because you are in a way

setting up the stuff." (P15:15)).

Besides, the sense of challenge may have engaged some participants in the technical part ("But I felt more engaged even with the sensors because I wanted to make it work, so I fought

harder for that to work than for the blocks." (P18:7)).

Furthermore, individual interests may also have played a role in the technical engagement

("[The sensors] get with other things [interests] of mine." (P07:14) and "I felt that I was very

engaged with this part [of the wires, the connections]. It is something I like. I stood there [wiggling

the arms simulating the assembly action of the wires and sensors] I wanted to explore a lot more.

"(P09:10)).

8.3.2.15. Bug, Errors, Feedback

Some participants felt frustrated with the loose contact issue in Probatio. As a consequence of

this error, they lost all their mapping strategies after the error ("I got frustrated with some crashes,

the things that came loose and lost contact ..." (P16:20)). The mapping software, Webmapper,

did not provide a way of saving the connections or recovering from a failure, which forced the

participants to reconnect everything relying on her/his memory ("It falls, and you have to

remember." (P02:24)). Due to errors in Probatio, one participant stated that he felt angry when

he lost all the mappings (“Frustrated. Pissed off!" (P19:10)).

Regarding GSToolkit, the participants did not negatively highlight any major error during the

interview. However, one participant felt the lack of feedback indicating that the connection was

properly working or not ("Yes, sometimes I made the connections, and it did not work!" (P18:8)).

A participant felt compelled to compare GSToolkit and Probatio saying that he did not face any system error with the former ("And the sensors no. The drawing is there, I fit it, and it did not

cause a bug." (P03:5)).

133

8.4. Discussion With both quantitative and qualitative evaluation, we could assess user’s interaction following two

aspects respectively: (1) external observation, and (2) user’s perception about the systems. In

this section, we attempt to correlate both results. Comparing Probatio and GSToolkit helped to

understand the user’s engagement during the interaction with a prototyping toolkit for DMIs. The

systems differ in two major aspects: (1) physical structure and (2) way of connecting the modules.

Considering our first three hypotheses (H1, H2, H3 in Figure 8.2), the results confirmed that: (1)

the duration to achieve a functional prototype was shorter in Probatio and longer in GSToolkit; (2)

the cycles of idea exploration was more numerous in Probatio and less numerous in GSToolkit;

(3) the number of distinct items used was greater in Probatio and lesser in GSToolkit. However,

regarding the overall user experience, the results demonstrated that there were different kinds of

engagements with the systems, which did not confirm our expectation that the participants would

not engage with GSToolkit, feeling more frustrated with this system. Additionally, we did not

expect the various errors that happened in Probatio, and we expected that the participants would

have many more errors in GSToolkit, what did not confirm.

8.4.1. About Probatio

Both quantitative and qualitative results suggest that Probatio provided a quicker and easier way

to reach musical interaction (as shown in section 8.3.1.1.2 and 8.3.2.12). Besides, if we consider

that the mounting phase is a technical barrier that has to be overcome to enter the cycles of

mapping and testing, which are the actual exploration of musical interactions, Probatio’s shorter

mounting phases and longer testing and mapping phases (see section 8.3.1.1.2) may indicate

that the system reduced the implementation barriers and fostered the musical experimentation.

Qualitative results suggest that Probatio’s physical support presented a good balance in flexibility

and immediate usability (section 8.3.2.6). With it, the participants reported had more time to

explore ideas, and their experience was commonly mentioned as engaging. Additionally, the

number of components and the easy way of connecting them caused the participants to feel that

they experimented more combinations in less time. This is supported by the higher number of

distinct items (section 8.3.1.1.4) used in Probatio in comparison with GSToolkit which suggests

that the system allows a broader exploration of the possible combinations.

In the context of designing construction kits for kids, Resnick and Silverman (2005) propose the

following guideline: “low floor, high ceiling, wide walls”, respectively meaning that the kit should

be easy for beginners, allow an increase of complexity for more experienced users, and permit

the exploration of different directions based on creativity and imagination (RESNICK;

SILVERMAN, 2005). The results suggest that Probatio might fulfill the low floor, and wide walls

guidelines. We believe that, for assessing high ceiling, we would need to define longer periods of

experiment both duration of sessions and multiple sessions over weeks or months.

134

Participants using Probatio mentioned that the number of items could be overwhelming because

it was too much information to process at once (section 8.3.2.1). It seems to be contradictory to

the fact that, actually, the number of individual components of GSToolkit was higher. A possible

explanation is that, visually, the volume of GSToolkit’s sensors is smaller than the blocks and

supports of Probatio. Besides, the presence of instruction cards probably attenuated the amount

of information to process, since the stack of cards presented a sequential way of exploring the

space of possibilities.

Additionally, they also indicated some shape limitations in Probatio, mentioning that forms were

not ergonomic due to the pointy edges, and cubic shapes, and also that some of the components

did not fit together (8.3.2.5). Besides, the distance between blocks caused the participants to

stretch their hands to perform some gestures, as they wanted to control more than one input

parameter with just one hand (8.3.2.11). They said that possibilities in controlling more

parameters with one hand could probably influence the formation of a gesture repertoire for blocks

in Probatio.

Additionally, the presence of a physical structure contributed to the perception of robustness of

Probatio (8.3.2.6). Besides, because the components rested in fixed places, the participants could

perform more accurate gestures, which benefited the musical interaction (8.3.2.8).

These points highlight the importance of experimenting with the physical structure in a flexible

way. Normally, the majority of the prototyping tools (see section 4.2.1) focuses on providing

sensors and actuators, but the designer has to deal with building the suitable physical support.

We find it is a hindrance for idea exploration in DMI design since the designer keeps changing

the contexts of attention. According to Sadler (2016b), technical interruption negatively affects

the creative thinking (SADLER, 2016).

Some participants highlighted the resemblance of the physical support with existing musical

instruments, and maybe because of that, it yields attraction for the object (8.3.2.6). The video

analysis shows that with Probatio participants experimented different ways of holding the

prototype (8.3.1.1.5). It is difficult to state that this happens due to the signifiers present in the

objects. However, it seems to be a promising feature for further development in the future.

8.4.2. About GSToolkit

Participants indicated that GSToolkit process induced them to follow a certain logical path, which

is mainly related to reading the instructions cards (8.3.2.2). The cards seem to have attenuated

the learning curve, saving user’s energy on understanding how the system works.

Different from Probatio, participants mentioned that GSToolkit had a great potential for

customization, allowing the users to arrange the position of the items with more freedom in the

desired way (8.3.2.4). Additionally, the wire manipulation appeared to have caused a good

sensation of authorship and made the participants in control of the process.

135

In fact, the authorship and ownership feeling were related to both systems, and this appears to

be in accordance with other result reported in the literature: the co-design experience in

Angesleva et al.’s (2016a) study. The authors conclude that the user seems to give more value

to a self-made product and the experience of creating or modifying a product “generates a sense

of creativity and enjoyment in the user, in the accomplishment of a task." (ÄNGESLEVÄ et al.,

2016b).

About the negative aspects of GSToolkit, it was often mentioned that the laborious way of

connecting the items was a barrier to achieving musical results (8.3.1.2.3 and 8.3.2.2). This

caused the participants to think before building a prototype. In the context of designing physical

and virtual interactions, Wiethott et al. (2012) argue that the design process should focus on the

concept of “working it through rather than thinking it through”, (WIETHOFF, 2012). In our

experiment, the “thinking it through” turned the participants to be more focused, and ended up

causing them to feel that they experimented fewer options - a fact that is confirmed by the

quantitative results.

Furthermore, the lack of physical support caused the sensors to be hanged by the wires, and the

interaction was limited to an area closed to the breadboard (8.3.2.8). It made the environment

disorganized, what we can assume that affected the usability. The components did not rest in

fixed positions, and one of the user’s hands had to be always busy as a support for the

components. The video analysis shows that most of the participants were curved towards the

sensors, and the only position tested was the tabletop (8.3.1.1.5).

Due to the fragile connections and the reduced dimensions of the header connectors (8.3.2.7 and

8.3.2.9), the risk of making mistakes made the participants be always aware of their actions, in

constant alertness. This caused the sensation of losing time with actions that did not relate to

musical experimentation. The numerous tasks for achieving just one input result made the

process longer. This caused anxiety, frustration, and even distress in some participants.

8.4.3. Summary of Bugs and Errors

After the changes we made after the technical pilot test, we did not expect the high number of

bugs that happened in Probatio during the experiment sessions. However, mainly due to the

overall positive impressions, we believe that the error did not play a major role in the user’s

interaction with the system. In its turn, we expected that the users would have much more

problems with GSToolkit, because of the possible wiring mistakes. However, in general, the users

could manage to perform the wire connections, solving the problems themselves without the need

of external help. This might be due to the presence of instructions cards, which provided a quick

visual reference for the user.

136

8.4.4. Different Engagements

The participant’s perception of engagement was twofold: the technical engagement of assembling

parts of a system, which is more related to rational aspects of solving problems, and the musical

engagement, which is associated with the intuitive exploration of artistic aspects of sound and

musical interaction. These engagements differ in time, objectives, and nature. This dichotomy

seems to be related to the two thinking styles defined by Epstein (1996): one intuitive-experiential,

and the other analytical-rational (EPSTEIN et al., 1996).

The shift between the two was reported to affect the awareness of the creative flow. Participants

mentioned that they engaged technically with both systems, but highlighted that felt more

musically engaged with Probatio. This probably happened due to the technical encapsulation and

the ability to rapidly obtain musical results.

Another point is that the more numerous transitions of mapping and testing in Probatio imply more

cycles of idea exploration, which according to Camburn (2015a), Beaudouin-lafon (2000c), and

Von Hippel (2001) can lead to design maturity and trial-and-error learning (CAMBURN et al.,

2015) (BEAUDOUIN-LAFON; MACKAY, 2000) (VON HIPPEL, 2001). In fact, participants

commented that they could accomplish higher levels of musical experimentation with Probatio,

resulting in better musical results. Indirectly, we may conclude that with Probatio the users had a

better understanding of the how the items could be combined and used.

8.4.5. Three Profiles

A possible interpretation of the results led us to three categories of user’s profiles. Although these

profiles might not be generalizable to other contexts, we will use them as a conceptual scaffold

to deepen our discussion.

The first profile has the customization of the instrument as a priority, focusing on the freedom to

use the sensors, to place them where they prefer and define the instrument structure by their

own. We will call this group the builders.

The second group is characterized by concentrating on musical experimentation and the

combination of input devices with sound outputs, trying to identify elements that fit together. This

group will be called experimentalists.

The third group has more focus on virtuosity, i.e. on the instrument's ability to provide precise,

nuance-controlled gestures that allow more than one parameter to be controlled with just one

hand. The name of this group can be virtuosi.

We interpret that these three groups are not exclusive, that is, a single participant can be related

to these three categories.

For the builders, what seems to be the most important is the concept of a white box, or transparent

box, as opposed to the black box, or the technical encapsulation (discussed by (SADLER et al.,

2016a)). The main motivation is to reach an instrument made by the hands and to be proud to

137

understand everything that was done in the development of that instrument. This case seems to

relate to the feeling of being a mad scientist, a Gyro Gearloose, dealing with wires, going in the

intricacies, the details of the instrument. For this first category, Probatio seems to draw less

attention than GSToolkit, even though the latter demands more development time than the

former. The structural and functional constraints of Probatio seem to evoke a sense of already-

defined instrument. While GSToolkit, even in potential, provides a sense of unfolding the world

not yet explored. GSToolkit appears to be more challenging, and this may be interesting for

people with more skills, but discourages those with fewer skills. Here comes an interesting point

about layers of abstraction, since GSToolkit itself is already an encapsulated version of the raw

sensors. What is the ideal balance between technical encapsulation and freedom of

implementation?

For the experimentalists, what matters is to achieve the musical result in the most immediate way

possible. There seems to be an anxiety in musical experimentation. The quick connection of the

Probatio suited this group better. With Probatio, participants were able to easily and quickly

explore combinations of gesture control and sound output, while GSToolkit imposed some

barriers.

For the virtuosi, the main thought seems to be how to achieve maximum fine-grained parameter

control with just one hand. Also, we could see that some participants were interested in creating

a repertoire of gestures, somehow start to think in the gestural point of view and not only in the

prototype development. This fact seems to relate to a discussion raised by Vertegaal et al. (1996),

in which the authors defend that the digital instruments should allow that their properties can be

“frozen” to provide ways to develop gestural techniques for its use (VERTEGAAL; UNGVARY;

KIESLINGER, 1996). For this third category, GSToolkit did not seem to be interesting due to: (1)

the constant need to use a hand as a support for the sensors (as explained by (GUIARD, 1987)),

(2) because of the reduced interaction area, (3) because the sensors were not fixed and were

always disorganized on the table (thus, it makes difficult to practice muscle memory (LEVITIN;

MCADAMS; ADAMS, 2002)). On the other hand, the Probatio is not yet at a fine-grain level to

satisfy the cravings of this group. The blocks are still large, the hands are still far apart, and the

controls are still very unitary, i.e. an item can control only one parameter. We considered that to

please this group, the blocks need to be more ergonomic, smaller to be closer to each other and

with a greater amount of parameter controls on just one device.

8.4.6. Limitations

A potential criticism of our experiment can be related to the following topics, which we present

with our comments below.

Comparison between two different systems: the difference between the systems is important

to stimulate the participants to compare unique features that are important for the development

of Probatio.

138

Duration of the sessions: we considered that the duration of each session was balanced

between schedule availability of the participants and the amount of data and information we

gathered to analysis. Perhaps, with longer duration, the number of participants would be reduced.

Number of sessions: it would be valuable to analyze the evolution of user’s experience over

time. For keeping the sample size, it would be necessary to make several toolkits available, which

would be currently unfeasible. Altering the sample size would reduce the depth of our results.

Sample size: for that, we followed the HCI literature recommendation and achieved a number of

participants that is considered reasonable.

Users’ gender: although trying to contact more female participants, women that matched our

profiling requirements are not numerous in Recife.

In this experiment, we focused on variables and impressions related to building a functional

prototype of a DMI. However, another direction would be on the resulting instrument. For future

experiments, it would be interesting to evaluate the outcomes taking into account the opinion of

external evaluators about the generated ideas of DMIs.

Although we focused on the individual interaction, from the pilot test, we could perceive the

potential of Probatio to be used in collaborative experimentation. It allowed that multiple users

could experiment at once. On the contrary, maybe the reduced area for assembling the circuit,

and perhaps the dimensions of the components, caused a reduction in the number of people

experimenting at once with GSToolkit.

8.4.7. Final Considerations

Considering our initial hypotheses, we realize that the use of Probatio reduced the assembly time,

increased the cycles of ideas exploration and also the time of this exploration. These specific

hypotheses of the experiment are directly related to our research questions, since by reducing

assembly time we are reducing the time and effort to build a prototype. Consequently, with shorter

duration, the cycles of exploration of ideas are strengthened.

As for the user experience, the experiment allowed us to access a piece of knowledge about the

various behavioral profiles regarding the construction of a DMI. Each behavior demands specific

system requirements.

We have seen that for builders, flexibility, freedom, and the feeling of being inside the process

are more important than immediate usability. For the experimentalists, urgency and immediate

musical outcome are more important than freedom and flexibility. For the virtuoso, the direct result

is important, but the interface needs more refinement.

The Probatio seems to be suitable for the “experimentalists”, but it proved inadequate for the

“builders” and still needs improvements to satisfy the “virtuosi”.

We can conclude that Probatio is not a general-purpose prototyping tool. The system has

limitations that at the same time serve as the initial path of exploring ideas. Its use is directed

139

towards a specific user behavior and at a particular stage of the design process. The toolkit seems

to relate better to a phase where there is still no pre-established idea or to test simple ideas.

Surprisingly, the GSToolkit system that was developed to serve as a counterpoint in the Probatio

evaluation appeared to be interesting for part of the participants. The adjustment in the

encapsulation of technical details seemed to bring positive impressions.

The equivalence between the two systems and their use together during the prototyping process

can bring compelling possibilities to the various profiles of users' behavior. The user can start by

exploring early ideas with Probatio and, after the idea is consolidated, GSToolkit can be used to

expand this idea to levels that go beyond the structural limitations of Probatio. All of this is related

to the equalization of the levels of abstraction to maximize the exploration and concretization of

ideas. A restricted but faster start may encourage the exploration of ideas. With the idea defined,

the more focused use of a slower but more flexible tool is a good mix between agility and

suitability.

140

9. CONCLUSION As interest in designing interactive physical devices increases, it is important to refine the

methods and tools that improve the design process. In this perspective, artistic devices are useful

because their creation and development involve a variety of challenges.

This work focused on the cycles of idea exploration and prototyping as an approach to deal with

the complexity of DMI design.

9.1. Research Question Revisited 1) How to provide a structured and exploratory path for generating new DMI ideas?

To address this question, we followed one possible solution that was the adaptation of an existing

idea generation method, combining it with the concept of instrumental inheritance. The concept

leverages the familiarity and cultural hooks of existing instruments, and the method provides a

systematic and exploratory way to combine elements. We expect that this combination works an

ignition for generating new instruments ideas.

2) How to reduce the time and effort needed to build functional DMI prototypes?

Considering the second question, we propose a modular toolkit that embedded the

aforementioned concept and design method, aiming to provide the DMI designer with ways of

achieving physical, functional prototypes faster and with less effort.

Probatio’s evaluations quantitatively confirmed our objective of reducing time and effort to achieve

functional prototypes. Also, qualitative results indicate the suitability of Probatio in the initial

phases of design, for users interested in rapidly generating and evaluating musical interactions

ideas. The users appear to appreciate the value of Probatio as a tool for the fast design of

functional prototypes, allowing them to realize their musical interaction ideas with a low entry

barrier.

The results led us to believe that we have advanced in the exploration of our research questions

by understanding possible ways to provide structure and exploratory steps for idea generation

and to reduce the time and effort of building functional DMI prototypes.

9.2. Contributions We consider that our main contributions to the DMI design are:

• The introduction and exploration of the concept of instrumental inheritance: we

presented the concept of transferring structural or gestural elements from existing

instruments as an initial constraint to ignite the creative process for generating new

instruments ideas. Based on common knowledge and existing cultural hooks, the new

instruments can leverage the existing intimate relationship musician-instrument and provide

141

the audience with familiar elements, avoiding disconnection and perhaps more engagement

during the performance.

• The development of a prototyping toolkit that embeds an idea generation method and a concept: Since the toolkit was built based on the method and the concept, we believe it

has the potential to transmit the inherent knowledge through the artifact and allows the

designer to easily and quickly generate, implement, test, and modify her ideas. Supported by

evidence in the literature, with the increasing number of cycles, we expect that designers and

users can achieve adequate results considering user’s needs, intentions, and contexts of use.

We believe that our originality resides on the literature’s lack of structured and integrated

methods and tools for DMI design that helps the designer to generated ideas and immediately

evaluate them with a functional prototype.

As additional contributions, we can mention:

• The use of morphological chart in DMI design: to our knowledge, the morphological chart

based on instrumental inheritance is a novel approach towards the design of DMIs. We

believe that the other idea generation methods can be further explored to foster new

exploratory paths in DMI design.

Focuses on boosting the cycles of idea exploration and prototyping during the design process

and comprises, ultimately, we hope that our approach can contribute on the cycles of musical

instruments’ evolution.

With further development, we also expect that the concept, method, and toolkit can contribute not

only to DMI design but also to the conception and quick testing of interactive physical objects in

general, since the approach of analyzing existing products and representing their parts as

physical real-time-reactive easily combinable building blocks may provide alternatives to boost

design cycles.

9.3. Limitations Although our work is mainly focusing on exploring the gestural controllers and mappings for DMI

design, there is still elements that can be particularly developed in these regards.

Concerning mapping strategies, the graphical user interface based on Webmapper do not allow

the combination of input values, for example, trigger a value if the other passes a threshold. This

fact seems to reduce the experimentation to simple interactions.

Moreover, we did not go further in considering other important aspects that constitute the DMI,

such as feedback and sound module. We consider that Probatio still misses important visual and

haptic features that could contribute to a better user’s experience. Besides, the options of sound

production.

Considering the evaluation, both cycles assessed users that did not have previous knowledge of

the systems and spent a small amount of time using them. This might have caused the users to

142

maintain a superficial level of interaction, focusing on simpler elements of the systems. With that,

we may have a limited understanding of some aspects of Probatio.

Specifically, Probatio showed its potential suitability for the profile of experimentalists, which focus

on immediate sound results. However, the toolkit demonstrated its restrictions for the builder’s

profile, given the demand for more freedom to define the structure and position of the sensors. In

addition, for the virtuosi’s profile, Probatio lacked elements that could provide the development of

a gestural repertoire and execution techniques.

9.4. Future Works From the point of view of the concept, we believe that instrumental inheritance, both from a

gestural and structural point of view, can be used in other areas as a lens for generating

alternatives. An area whose application seems to be suitable is in the design of complex

interaction artifacts ((RAMALHO, 2017)). This class of artifacts is based on the high degree of

dexterity and gestural evolution required to perform tasks. Some examples are tools for surgery,

aircraft cockpits, game controls, Formula 1 steering wheel. Therefore, gestural inheritance

appears to be a promising approach to leverage existing techniques, transferring them to new

artifacts.

Considering the method, the morphological chart we built can be expanded by further exploring

functions to find more details that can be explored to generate ideas for new instruments. Due to

our iterative and incremental methodology, a possible postmortem interpretation is that the chart

ended up being built as an intermediate stage because our main focus was on the development

of the prototyping toolkit. In fact, for our purpose of providing an immediate way for the DMI

designer to achieve functional prototypes and rapidly test them, the morphological chart seems

to be a means and not an end.

There are many fronts that can be explored in the technical evolution of Probatio. One of these is

ergonomic suitability to provide a better user experience. Instead of using cubic shapes with

pointy edges, it may be interesting to try out more rounded or organic shapes that can fit the

user's hands. In addition, to allow better manipulation, the dimensions of the blocks can be

reduced. This demands a work of miniaturization of the electronic components, which we do not

see as impossible or difficult, but it just was not the focus of this work.

Although the Probatio was well evaluated for its robustness, the MDF proved to be an unreliable

material due to variations in dimension with air humidity. In addition, over time the material

becomes porous. Therefore, it is interesting to evaluate alternatives such as applying a coating,

paint or use alternative materials such as metal or plastic.

Furthermore, we can think of a graphical interface more integrated with the physical forms of

Probatio. An interface that in real time identifies and represents the shape of the physical world

in the virtual world. We believe this would facilitate the next steps in the design process. Let us

imagine that after defining a satisfactory configuration of blocks and supports, the user could

143

generate the plans of a laser cutting with a simple push of a button. The system would also provide

a list of materials and a tutorial explaining the step-by-step construction of that one non-modular,

"frozen" version of the prototype. In this way, the user could begin to create a gesture repertoire

for this instrument-specific instantiation. Perhaps this approach fits the profile of virtuosi discussed

earlier.

In addition to Probatio's technical development details, we can reflect on the appeal of

incorporating a certain type of knowledge into the object. To access this knowledge, it would be

necessary to have a high level of technical expertise. The use of sensors, actuators, and

microcontrollers in the music still has a lot of space to be explored, but the entry barrier may deter

users whose focus is not technical but rather artistic. On this side, we have the musicians with

their individual needs, intentions, and contexts of use specific to their artistic expression.

Probatio's role is to unite access to the technical world without losing the focus on the creative

path. It is the encapsulation of the technical details to allow a quick and direct access to the

functionalities. We believe that this exploration of new knowledge can reconfigure the musician's

initial ideas since we can consider it as a cognitive expansion.

Let us look at a possible use of Probatio as a knowledge transmission tool. Let us say there is

Alice, who is a designer of DMIs, and Bob, who is a musician who is interested in new interfaces

for musical expression. After scheduling a meeting, Bob visits Alice's workshop. Bob has his own

artistic intentions and already comes up with some ideas of instruments that he would like to build.

Alice disposes the Probatio on the table, and Bob begins to explore the possibilities of

combination. At this point, Bob begins to discover types of sensors and input devices he had not

yet known. With the freedom to map any input into the outputs, Bob begins to reconfigure his

initial ideas of possible instruments. Alice has a role as simply mediator or facilitator (in fact, at

this early stage, Alice may even be removed from the scenario) because it is as if the object

already communicates the possible paths of exploration and passes quickly a knowledge that

would take a long time for Bob to access alone. Despite being interested in the technical parts,

Bob really feels fulfilled with the musical interaction, which is his main focus. Finally, when feeling

satisfied, Bob then defines a set of combinations that most pleased him. These combinations can

be related to structure, gestural controllers or mapping strategies. With this set, Alice now can

explore with Bob the more specific possibilities for his context and intentions. In this example, the

benefit of using Probatio is to clarify ideas and reduce the time to come up with an interesting

prototype for the musician. Perhaps, without using Probatio, Bob would not have been able to

form ideas that were not in his head and immediately test them and check their suitability for his

context and intention. The Probatio as an experimentation catalyst.

144

REFERENCES ACEITUNO, J. Direct and expressive interaction on the desktop: increasing granularity, extent, and dimensionality. [s.l.] Université Lille 1, 2015.

ADIKARI, S.; MCDONALD, C.; CAMPBELL, J. Little Design Up-Front: A Design Science

Approach to Integrating Usability into Agile Requirements Engineering. In: JACKO, J. A. (Ed.). .

Human-Computer Interaction. New Trends: 13th International Conference, HCI International 2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, Part I. Berlin,

Heidelberg: Springer Berlin Heidelberg, 2009. p. 549–558.

ALLISON, J. T.; PLACE, T. Teabox: A Sensor Data Interface System. Proceedings of the 2005

International Conference on New Interfaces for Musical Expression (NIME05).

Anais...Vancouver, BC, Canada: 2005

ÄNGESLEVÄ, J. et al. The Results of Rethinking Prototyping. In: Rethink! Prototyping. Cham:

Springer International Publishing, 2016a. p. 201–210.

ÄNGESLEVÄ, J. et al. Beyond Prototyping. In: Rethink! Prototyping. Cham: Springer

International Publishing, 2016b. p. 161–199.

BALLAGAS, R. et al. iStuff: a physical user interface toolkit for ubiquitous computing

environments. Proceedings of the SIGCHI conference on Human factors in computing systems, v. 16, n. 5, p. 537–544, 2003.

BANGOR, A.; KORTUM, P. T.; MILLER, J. T. An Empirical Evaluation of the System Usability

Scale. International Journal of Human-Computer Interaction, v. 24, n. March 2015, p. 574–

594, 2008.

BANZI, M. Getting Started with Arduino. [s.l.] “ O’Reilly Media, Inc.”, 2009.

BARBOSA, J. et al. Towards an evaluation methodology for digital music instruments considering performer’s view: a case study. Proceedings of 13th Brazilian Symposium on

Computer Music. Anais...2011

BARBOSA, J. et al. Considering Audience’s View Towards an Evaluation Methodology for Digital Musical Instruments. Proceedings of the International Conference on New Interfaces for

Musical Expression. Anais...2012

BARBOSA, J. et al. Designing DMIs for Popular Music in the Brazilian Northeast : Lessons Learned. Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...Bâton Rouge, US: 2015a

BARBOSA, J. et al. What does “ Evaluation ” mean for the NIME community? Proceedings

of the International Conference on New Interfaces for Musical Expression. Anais...Bâton Rouge,

US: 2015b

BARRACLOUGH, T. J.; MURPHY, J.; KAPUR, A. New Open-Source Interfaces for Group

145

Based Participatory Performance of Live Electronic Music. (B. Caramiaux et al.,

Eds.)Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...Goldsmiths, University of London, UK: Goldsmiths, University of London, 2014

BASALLA, G. The Evolution of Technology. [s.l.] Cambridge University Press, 1989.

BATTIER, M. Electronic Music and Gesture. Trends in Gestural Control of Music, n. 1920, p.

328–330, 2000.

BEAUDOUIN-LAFON, M. Instrumental interaction: An Interaction Model for Designing Post-

WIMP User Interfaces. Proceedings of the 18th international conference on Human factors in computing systems - CHI ’00, v. 2, n. 1, p. 446–453, 2000.

BEAUDOUIN-LAFON, M.; MACKAY, W. E. Prototyping Tools and Techniques. Prototype Development and Tools, p. 1–41, 2000.

BENTLEY, J. Programmimg pearls. Communications of the ACM, v. 28, n. 9, p. 896–901, 1985.

BERDAHL, E.; JU, W. Satellite CCRMA: A Musical Interaction and Sound Synthesis Platform. Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...Oslo, Norway: 2011

BERTHOLDO, A. P. O. et al. Agile Usability Patterns for UCD Early Stages. In: MARCUS, A.

(Ed.). . Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience: Third International Conference, DUXU 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part I. Cham:

Springer International Publishing, 2014. p. 33–44.

BEVILACQUA, F.; MÜLLER, R.; SCHNELL, N. MnM: a Max/MSP mapping toolbox.

Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...2005

BIRNBAUM, D. et al. Towards a dimension space for musical devices. Proceedings of the

2005 International Conference on New Interfaces for Musical Expression (NIME05).

Anais...Vancouver, BC, Canada: 2005

BLAINE, T.; FELS, S. Collaborative Musical Experience for Novices. Journal of New Music Research, v. 32, n. 4, p. 411–428, 2003.

BLAINE, T.; FORLINES, C. JAM-O-WORLD: Evolution of the Jam-O-Drum Multi-player Musical Controller into the Jam-O-Whirl Gaming Interface. Proceedings of the 2002

Conference on New Interfaces for Musical Expression (NIME-02). Anais...2002

BONGERS, B. Physical Interfaces in the Electronic Arts: Interaction Theory and Interfacing

Techniques for Real-time Performance. In: WANDERLEY, M. M.; BATTIER, M. (Eds.). . Trends in Gestural Control of Music. [s.l.] Ircam Centre Pompidou, 2000. p. 41–70.

BONGERS, B. Electronic Musical Instruments: Experiences of a New Luthier. Leonardo Music Journal, v. 17, p. 9–16, 2007.

146

BONGERS, B.; HARRIS, Y. A Structured Instrument Design Approach: The Video-Organ.

Proceedings of the International Conference on New Interfaces for Musical Expression, n.

May, p. 18–23, 2002.

BRAUN, V.; CLARKE, V. Using thematic analysis in psychology. Qualitative Research in Psychology, v. 3, n. May 2015, p. 77–101, 2006.

BROOKE, J. SUS: A quick and dirty usability scale. In: Usability Evaluation in Industry. [s.l:

s.n.]. p. 189–194.

BROOKE, J. SUS : A Retrospective. Journal of Usability Studies, v. 8, n. 2, p. 29–40, 2013.

BROWN, T. Design Thinking. Harvard Business Review, v. 86, n. 6, p. 84–92, 2008.

BROWN, T. Change by Design: How Design Thinking Can Transform Organizations and Inspire Innovation. [s.l.] Collins Business, 2009.

BUCHANAN, R. Wicked Problems in Design Thinking. Design Issues, v. 8, n. 2, p. 5–21, 1992.

BUXTON, B. Artists and the art of the luthier. ACM SIGGRAPH Computer Graphics.

Anais...1997

BUXTON, B. Sketching user experiences: getting the design right and the right design. [s.l.]

Morgan Kaufmann, 2007.

CADOZ, C.; WANDERLEY, M. M. Gesture - Music. Trends in gestural control of music, p. 71–

94, 2000.

CALEGARIO, F. et al. A Method and Toolkit for Digital Musical Instruments: Generating Ideas

and Prototypes. IEEE MultiMedia, v. 24, n. 1, p. 63–71, jan. 2017.

CAMBURN, B. et al. A Systematic Method for Design Prototyping. Journal of Mechanical Design, v. 137, n. 8, p. 81102, 2015.

CAPJON, J. Engaged collaborative ideation supported through material catalysation. Nordic Design Research Conference, 2005.

CARD, S. K.; MACKINLAY, J. D.; ROBERTSON, G. G. A morphological analysis of the design

space of input devices. ACM Transactions on Information Systems, v. 9, n. 2, p. 99–122, 1991.

CLASON, D. L.; DORMODY, T. J. Analyzing Data Measured by Individual Likert-Type Items.

Journal of Agricultural Education, v. 35, n. 4, p. 31–35, 1994.

CONT, A.; CODUYS, T.; HENRY, C. Real-time gesture mapping in pd environment using neural networks. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...2004

COOK, P. R. Principles for designing computer music controllers. Proceedings of the

International Conference on New Interfaces for Musical Expression. Anais...2001

COOK, P. R. Real Sound Synthesis for Interactive Applications. [s.l.] A K Peters/CRC Press,

2002.

147

COOK, P. R. Re-Designing Principles for Computer Music Controllers: A Case Study of SqueezeVox Maggie. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...Pittsburgh, US: 2009

COOK, P. R.; SCAVONE, G. P. Audio Anecdotes: A Cookbook of Audio Algorithms and

Techniques, chapter The Synthesis ToolKit (STK) in C++. AK Peters, Natick, MA, 2004.

CROSS, N. Developments in Design Methodology. [s.l: s.n.].

CROSS, N. Engineering Design Methods: Strategies for Product Design. Third Edit ed. [s.l.]

John Wiley & Sons, 2000.

CROSS, N. Designerly ways of knowing. [s.l.] Springer, 2006.

CURTIS, A. Rhetoric of Flat Design and Skeuomorphism in Apple’s iOS Graphical User Interface. [s.l.] University of Rhose Island, 2015.

DAHL, D. W.; MOREAU, P. The influence and value of analogoical thinking during new product

ideation. Journal of Marketing Research, v. 39, n. 1, p. 47–60, 2002.

DAHL, L. Wicked Problems and Design Considerations in Composing for Laptop Orchestra. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...University of Michigan, Ann Arbor: 2012

DAHL, L. Designing New Musical Interfaces as Research: What’s the Problem? Leonardo, v. 49,

n. 1, p. 76–77, fev. 2016.

DOBRIAN, C.; KOPPELMAN, D. The E in NIME: Musical Expression with New Computer Interfaces. Proceedings of the 2006 International Conference on New Interfaces for Musical

Expression (NIME06). Anais...Paris, France: 2006

DONALD, M. A mind so rare : the evolution of human consciousness. [s.l.] Norton, 2002.

DOUGHERTY, D. The Maker Movement. innovations, v. 7, n. 3, p. 11–14, 2012.

DOW, S. How prototyping practices affect design results. Interactions, v. 18, n. 3, p. 54, 1 maio

2011.

DRUMMOND, J. Understanding Interactive Systems. Organised Sound, v. 14, n. 2, p. 124, 29

ago. 2009.

DUBBERLY, H. How do you design: A compendium of models. [s.l: s.n.].

DUBBERLY, H. The Space of Design. ACM’s Interactions, v. 17, n. 5, p. 1–9, 2010.

ELSEN, C. et al. Representation in early stage design: an analysis of the influence of sketching

and prototyping in design projects. ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, p. 737–747, 2012.

EPSTEIN, S. et al. Individual differences in intuitive-experiential and analytical-rational thinking

styles. Journal of Personality and Social Psychology, v. 71, n. 2, p. 390–405, 1996.

148

ESSL, G.; O’MODHRAIN, S. An enactive approach to the design of new tangible musical

instruments. Organised Sound, v. 11, n. 3, p. 285–296, 2006.

EXNER, K. et al. A transdisciplinary perspective on prototyping. 2015 IEEE International Conference on Engineering, Technology and Innovation/ International Technology Management Conference, ICE/ITMC 2015, 2015.

FALLMAN, D. Design-Oriented Human-Computer Interaction. Proceedings of the International Conference on Human Factors in Computing Systems (CHI’03), n. 5, p. 225–232, 2003.

FANTINATTO, R. I dream of wiresCanada, UK, USA, GermanyMonoduo Films, , 2014.

FEIL, M.; TUNG, F. Modular Toolkit Of Interface Components For Students To Efficiently And Easily Test Design Concepts. IDSA 2013 EDUCATION SYMPOSIUM. Anais...Chicago:

2013

FELS, S. Designing for intimacy: Creating new interfaces for musical expression. Proceedings of the IEEE, v. 92, n. 4, 2004.

FELS, S. S.; GADD, A.; MULDER, A. Mapping transparency through metaphor: towards more

expressive musical instruments. Organised Sound, v. 7, n. 2, p. 109–126, 17 jan. 2002.

FIEBRINK, R.; TRUEMAN, D.; COOK, P. R. A metainstrument for interactive, on-the-fly machine learning. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...Pittsburgh, US: 2009

FOUSE, A. S. et al. ChronoViz : A system for supporting navigation of time-coded data. CHI

2011. Anais...2011

FYANS, A. C. et al. Ecological considerations for participatory design of DMIs. Proceedings

of the International Conference on New Interfaces for Musical Expression. Anais...University of

Michigan, Ann Arbor: 2012

GAUNTLETT, D. The LEGO System as a tool for thinking, creativity, and changing the world. In:

LEGO Studies: Examining the Building Blocks of a Transmedial Phenomenon. [s.l: s.n.]. p.

1–16.

GELINECK, S.; SERAFIN, S. A practical approach towards an Exploratory Framework for

Physical Modeling. Computer Music Journal, p. 51–66, 2010a.

GELINECK, S.; SERAFIN, S. PHOXES - Modular Electronic Music Instruments based on Physical Modeling Sound Synthesis. 7th Sound and Music Computing Conference.

Anais...2010b

GERBER, E.; CARROLL, M. The psychological experience of prototyping. Design Studies, v.

33, n. 1, p. 64–84, jan. 2011.

GIBSON, J. J. The Ecological Approach to Visual Perception. New Jersey, USA: Lawrence

Erlbaum, 1979.

149

GILL, C.; SANDERS, E.; SHIM, S. Prototypes as inquiry, visualization and communication.

International Conference on Engineering and Product Design Education, n. September, p.

672–677, 2011.

GREEN, L. How Popular Musicians Learn: A Way Ahead for Music Education. [s.l.] Ashgate

Publishing, 2002.

GUIARD, Y. Asymmetric division of labor in human skilled bimanual action: the kinematic chain

as a model. Journal of Motor Behavior, v. 19, n. 4, p. 486–517, 1987.

GUREVICH, M. Designing for style in new musical interactions. Proceedings of the

International Conference on New Interfaces for Musical Expression. Anais...Pittsburgh, US: 2009

GUREVICH, M.; CAVAN FYANS, A. Digital Musical Interactions: Performer–system relationships

and their perception by spectators. Organised Sound, v. 16, n. 2, p. 166–175, 28 jun. 2011.

GUREVICH, M.; MARQUEZ-BORBON, A.; STAPLETON, P. Playing with Constraints: Stylistic

Variation with a Simple Electronic Instrument. Computer Music Journal, v. 36, n. 1, p. 23–41,

2012.

HADJAKOS, A.; WALOSCHEK, S. SPINE: A TUI Toolkit and Physical Computing Hybrid. (B.

Caramiaux et al., Eds.)Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...Goldsmiths, University of London, UK: Goldsmiths, University of London,

2014

HARTMANN, B. et al. d.tools: Visually Prototyping Physical UIs through Statecharts. Extended Abstracts of UIST 2005, p. 23–26, 2005.

HELMINEN, P.; AINOA, J.; MÄKINEN, S. Designing user innovation toolkits: exploring the

interrelation between solution space and module library. International Journal of Design Creativity and Innovation, n. May, p. 1–19, 2015.

HERNANDEZ, N. V.; SHAH, J. J.; SMITH, S. M. Understanding design ideation mechanisms

through multilevel aligned empirical studies. Design Studies, v. 31, n. 4, p. 382–410, 2010.

HEY, J. et al. Analogies and metaphors in creative design. International Journal of Engineering Education, v. 24, n. 2, p. 283–294, 2008.

HOLLINGER, A.; THIBODEAU, J.; WANDERLEY, M. M. An Embedded Hardware Platform for Fungible Interfaces. ICMC 2010. Anais...2010

HOOD, M. The Ethnomusicologist. New York: Kent State University Press, 1982.

HORVAT, I. Theoretical Framework for Comprehensive. International Conference on Engineering Design, n. August, p. 1–12, 2011.

HUNT, A.; WANDERLEY, M. M.; PARADIS, M. The Importance of Parameter Mapping in

Electronic Instrument Design. Journal of New Music Research, v. 32, n. 4, p. 429–440, dez.

2003.

150

HUOT, S. Designeering Interaction: A Missing Link in the Evolution of Human-Computer Interaction. [s.l.] Université Paris Sud - Paris XI, 2013.

IDEO. Human-Centered Design Toolkit: An Open-Source Toolkit to Inspire New Solutions in the Developing World. [s.l.] Authorhouse, 2011.

ISA, S. S.; LIEM, A.; STEINERT, M. the Value of Prototypes in the Early Design and Development

Process. 20th International Conference on Engineering Design (ICED 15), n. July, p. 1–8,

2015.

ISRAEL, J. H.; BÄHR, B.; EXNER, K. Perspectives on Future Prototyping—Results from an

Expert Discussion. In: Rethink! Prototyping. Cham: Springer International Publishing, 2016. p.

11–22.

JENSENIUS, A. R. et al. Musical Gestures. Musical Gestures: Sound, Movement, and Meaning, p. 12, 2010.

JOBST, B.; MEINEL, C. How Prototyping Helps to Solve Wicked Problems. In: Design Thinking Research. Cham: Springer International Publishing, 2014. p. 105–113.

JONES, J. C. Design methods. London, New York: Wiley-Interscience, 1992.

JORDÀ, S. New Musical Interfaces and New Music-making Paradigms. Proceedings of the

CHI’01 Workshop on New Interfaces for Musical Expression (NIME-01). Anais...2001

JORDÀ, S. Interactive music systems for everyone: exploring visual feedback as a way for creating more intuitive, efficient and learnable instruments. Proceedings of the Stockholm

Music Acoustics. Anais...2003

JORDÀ, S. Digital instruments and players: part I---efficiency and apprenticeship. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 59–63, 2004a.

JORDÀ, S. Digital Instruments and Players: Part II – Diversity , Freedom and Control. Proceedings of the 2004 International Computer Music Conference. Anais...2004b

JORDÀ, S. Instruments and Players: Some Thoughts on Digital Lutherie. Journal of New Music Research, v. 33, n. 3, p. 321–341, 2004c.

JORDÀ, S. Digital Lutherie. [s.l.] Universitat Pompeu Fabra, 2005a.

JORDÀ, S. Digital Lutherie Crafting musical computers for new musics’ performance and improvisation. [s.l.] Universitat Pompeu Fabra, 2005b.

JORDÀ, S. et al. The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. Proceedings of the 1st international conference on Tangible and

embedded interaction. Anais...2007

JORDÀ, S.; MEALLA, S. A Methodological Framework for Teaching , Evaluating and Informing NIME Design with a Focus on Expressiveness and Mapping. Proceedings of the

International Conference on New Interfaces for Musical Expression. Anais...Goldsmiths,

151

University of London, UK: 2014

JORDAN, B.; HENDERSON, A. Interaction Analysis: Foundations and Practice. Journal of the Learning Sciences, v. 4, n. 1, p. 39–103, 1995.

KAPTEIN, M. C.; NASS, C.; MARKOPOULOS, P. Powerful and consistent analysis of likert-type

rating scales. Proceedings of the 28th international conference on Human factors in computing systems - CHI 2010, p. 2391–2394, 2010.

KAPTELININ, V. Affordances. In: SOERGAARD, M.; DAM, R. F. (Eds.). . The Encyclopedia of Human-Computer Interaction 2nd Ed. [s.l.] Interaction Design Foundation, 2014.

KNIGHT, R. C. The KNIGHT-REVISION of HORNBOSTEL-SACHS : a new look at musical instrument classification. [s.l: s.n.].

KNORIG, A. Design Tools Design. [s.l.] FH Potsdam, 2008.

KVIFTE, T. Instruments and the Electronic Age. Oslo: Solum forlag, 1988.

KVIFTE, T.; JENSENIUS, A. R. Towards a Coherent Terminology and Model of Instrument

Description and Design. Proceedings of the conference on New interfaces for musical expression, p. 220–225, 2006.

LAW, E. L.-C. et al. Understanding, scoping and defining user experience: a survey approach. Proceedings of the SIGCHI conference on human factors in computing systems.

Anais...ACM, 2009

LEVITIN, D. J.; MCADAMS, S.; ADAMS, R. L. Control parameters for musical instruments: a

foundation for new mappings of gesture to sound. Organised Sound, v. 7, n. 2, p. 171–189, jan.

2002.

LEWIS, J. R.; SAURO, J. The factor structure of the System Usability Scale.dot.pdf. v. 1, p. 1–

10, 2009.

LIDWELL, W.; HOLDEN, K.; BUTLER, J. Universal Principles of Design. [s.l.] Rockport

Publishers, 2010.

LIM, Y. et al. Discovery-Driven Prototyping for User-Driven Creativity. p. 74–80, 2013.

LIM, Y.-K.; STOLTERMAN, E.; TENENBERG, J. The anatomy of prototypes. ACM Transactions on Computer-Human Interaction, v. 15, n. 2, p. 1–27, 1 jul. 2008.

LOWGREN, J.; STOLTERMAN, E. Thoughtful interaction design: A design perspective on information technology. [s.l.] The MIT press, 2004.

LUCIANI, A. Ergotic / epistemic / semiotic functions. In: Enaction and enactive interfaces: a handbook of terms, Enactive System Books. [s.l: s.n.]. p. 96–97.

MACHOVER, T. Hyperinstruments: A composer’s approach to the evolution of intelligent musical

instruments. Organized Sound, p. 67–76, 1991.

MAESTRACCI, B.; FRECHIN, J.; PETREVSKI, U. Modular Musical Objects Towards

152

Embodied Control Of Digital Music. Proceedings of the International Conference on Tangible,

Embedded and Embodied Interaction (TEI). Anais...2011

MAGNUSSON, T. Designing constraints: composing and performing with digital musical systems.

Computer Music Journal, v. 34, n. 4, p. 62–74, 2010.

MAGNUSSON, T.; MENDIETA, E. H. The Acoustic, the Digital and the Body : A Survey on

Musical Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 94–99, 2007a.

MAGNUSSON, T.; MENDIETA, E. H. E. The acoustic, the digital and the body: A survey on

musical instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, p. 94–99, 2007b.

MALLOCH, J. et al. Towards a New Conceptual Framework for Digital Musical Instruments.

Proceedings of the 9th International Conference on Digital Audio Effects. Anais...2006

MALLOCH, J. A Consort of Gestual Musical Controllers: Design, Construction and Performance. [s.l.] McGill University, 2008.

MALLOCH, J.; SINCLAIR, S.; WANDERLEY, M. M. Distributed tools for interactive design of

heterogeneous signal networks. Multimedia Tools and Applications, p. 1–25, 2014.

MALONEY, T. M. The family of wood composite materials. Forest Products Journal, v. 46, n.

2, p. 18, 1996.

MARQUEZ-BORBON, A. et al. Designing Digital Musical Interactions in Experimental Contexts. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...Oslo, Norway: 2011

MARTIN, R. C. Agile Software Development: Principles, Patterns, and Practices. [s.l.]

Prentice Hall PTR, 2003.

MATHEWS, M. Radio Baton, 2005.

MAURYA, A. Running Lean: Iterate from Plan A to a Plan That Works. [s.l.] O’Reilly Media,

2012.

MEDEIROS, C.; WANDERLEY, M. M. A Comprehensive Review of Sensors and Instrumentation

Methods in Devices for Musical Expression. Sensors, v. 14, n. 8, p. 13556–13591, 25 jul. 2014.

MEDEIROS, R. et al. Challenges in Designing New Interfaces for Musical Expression. In: Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience. [s.l.] Springer, 2014. p. 643–652.

MERRIAM-WEBSTER. Merriam-Webster’s collegiate dictionary. [s.l.] Merriam-Webster,

2004.

MIMO CONSORTIUM. Revision of the Hornbostel-Sachs Classification of Musical Instruments by the MIMO Consortium. [s.l: s.n.].

153

MIRANDA, E. R. Computer Sound Synthesis for the Electronic Musician. [s.l.] Butterworth-

Heinemann, 1998.

MIRANDA, E. R.; WANDERLEY, M. M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Middleton: A-R Editions, 2006.

MONTAG, M. et al. A Low-Cost , Low-Latency Multi-Touch Table with Haptic Feedback for Musical Applications. Proceedings of the International Conference on New Interfaces for

Musical Expression. Anais...Oslo, Norway: 2011

MOREY, R. D. Confidence Intervals from Normalized Data: A correction to Cousineau ( 2005 ).

Tutorials in Quantitative Methods for Psychology, v. 4, n. 2, p. 61–64, 2008.

MORREALE, F.; ANGELI, A. DE; O’MODHRAIN, S. Musical Interface Design: An Experience-oriented Framework. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...Goldsmiths, University of London, UK: 2014

MULDER, A. The I-Cube System: moving towards sensor technology for artists.

Proceedings of the ISEA. Anais...1995

MURRAY-BROWNE, T. et al. The medium is the message : Composing instruments and performing mappings. Proceedings of the International Conference on New Interfaces for

Musical Expression. Anais...Oslo, Norway: 2011

MUSER, S. Gestures in Human-Computer-Interaction. [s.l: s.n.].

NAKAMARU, S. Tangible Modeling Methods for Faster Rapid Prototyping. p. 518–523, 2016.

NAKAMURA, J.; CSIKSZENTMIHALYI, M. The concept of flow. In: Flow and the foundations of positive psychology. [s.l.] Springer, 2014. p. 239–263.

NEVES, A. Design Thinking Canvas. [s.l: s.n.].

NEWTON-DUNN, H.; NAKANO, H.; GIBSON, J. Block Jam: A Tangible Interface for Interactive Music. Proceedings of the 2003 Conference on New Interfaces for Musical

Expression (NIME-03). Anais...Montreal, Canada: dez. 2003

NIELS, B.; GELINECK, S.; SERAFIN, S. PHYSMISM : A control interface for creative exploration of physical models . Proceedings of the 2007 Conference on New Interfaces for

Musical Expression (NIME07). Anais...2007

NISHIBORI, Y.; IWAI, T. Tenori-on. Proceedings of the 2006 International Conference on New

Interfaces for Musical Expression (NIME06). Anais...Paris, France: 2006

NORMAN, D. A. The Design of Everyday Things: Revised and Expanded Edition. [s.l: s.n.].

NORT, D. VAN; WANDERLEY, M. M. The LoM Mapping Toolbox for Max/MSP/Jitter. Proceedings of the International Computer Music Conference. Anais...2006

O’HARA, D. Skeuomorphology and Quotation. In: ROUSSEL, M.; BORKENHAGEN, C. (Eds.). .

Creativity of Finding: Figurations of the Quotation. [s.l.] Wilhelm Fink Verlag, 2012. v. 2p.

154

281–293.

O’MODHRAIN, S. A framework for the evaluation of digital musical instruments. Computer Music Journal, v. 35, n. 1, p. 28–42, 2011.

O’MODHRAIN, S.; CHAFE, C. Incorporating haptic feedback into interfaces for music applications. Proceedings of the International Symposium on Robotics with Applications, World

Automation Conference. Anais...2000

OORE, S. Learning Advanced Skills on New Instruments (or practising scales and arpeggios on your NIME). Proceedings of the 2005 International Conference on New Interfaces

for Musical Expression (NIME05). Anais...Vancouver, BC, Canada: 2005

OSCULATOR. OSCulator.

OWEN, C. Considering Design Fundamentally. Design Processes Newsletter. Anais...1993

PAHL, G. et al. Engineering design: a Systematic Approach. [s.l: s.n.].

PAINE, G. Towards Unified Design Guidelines for New Interfaces for Musical Expression.

Organised Sound, v. 14, n. 2, p. 142, jun. 2009.

PAINE, G. New Musical Instrument Design Considerations. MultiMedia, IEEE, p. 76–84, 2013.

PAINE, G. Interaction as Material: The techno-somatic dimension. Organised Sound, v. 20, n.

1, p. 82–89, 2015.

PALACIO-QUINTIN, C. The Hyper-Flute. Proceedings of the 2003 Conference on New

Interfaces for Musical Expression (NIME-03). Anais...Montreal, Canada: 2003

PASSERA, S.; KAERKKAEINEN, H.; MAILA, R. When, how, why prototyping? A practical

framework for service development. n. Brown 2008, p. 1–16, 2012.

POTIDIS, S.; SPYROU, T. Spyractable: A tangible user interface modular synthesizer. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), v. 8511 LNCS, n. PART 2, p. 600–611, 2014.

PREECE, J. et al. Interaction design: Beyond human-computer Interaction. 4th. ed. [s.l.]

John Wiley, 2015.

PUCCIO, G. J.; CABRA, J. F. Idea generation and idea evaluation: Cognitive skills and deliberate practices. [s.l.] Elsevier Inc., 2012.

RAMALHO, L. F. DE O. Herança Gestual Aplicada Em Design de Novos Artefatos de Interação Complexa. [s.l.] Universidade Federal de Pernambuco, 2017.

REINIG, B. A.; BRIGGS, R. O. On the relationship between idea-quantity and idea-quality during

ideation. Group Decision and Negotiation, v. 17, n. 5, p. 403–420, 2008.

RESNICK, M.; SILVERMAN, B. Some Reflections on Designing Construction Kits for Kids.

Proceeding of the 2005 Conference on Interaction Design and Children (IDC ’05), p. 117–

122, 2005.

155

RITCHEY, T. General Morphological Analysis A general method for non-quantified modelling.

16th EURO Conference on Operational Analysis, p. 1–10, 1998.

ROECK, D. D. E. et al. Beyond Static : Ideation Using Interactive Prototyping Toolkits.

International Conference On Engineering And Product Design Education. Anais...2013

ROVAN, J. B. et al. Instrumental gestural mapping strategies as expressivity determinants in computer music performance. Proceedings of Kansei-The Technology of Emotion

Workshop. Anais...Citeseer, 1997

RYAN, J. Some remarks on musical instrument design at STEIM. Contemporary Music Review,

v. 6, n. 1, p. 3–17, 1991.

SACHS, C. The History of Musical Instruments. [s.l.] W. W. Norton & Company, Inc., 1940.

SADLER, J. et al. Bloctopus: A Novice Modular Sensor System for Playful Prototyping.

Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI ’14, p. 347–354, 2015.

SADLER, J. The Anatomy of Creative Computing: Enabling Novices to Prototype Smart Devices. [s.l.] Stanford University, 2016.

SADLER, J. et al. Building Blocks of the Maker Movement: Modularity Enhances Creative

Confidence During Prototyping. In: PLATTNER, H.; MEINEL, C.; LEIFER, L. (Eds.). . Design Thinking Research. [s.l.] Springer International Publishing, 2016a. p. 141–154.

SADLER, J. et al. Can Anyone Make a Smart Device?: Evaluating the Usability of a Prototyping

Toolkit for Creative Computing. In: AL., H. P. ET (Ed.). . Design Thinking Research, Understanding Innovation. [s.l.] Springer International Publishing Switzerland 2016, 2016b. p.

147–160.

SALDAÑA, J. The Coding Manual for Qualitative Researchers. [s.l: s.n.].

SANDERS, E. B.-N. Prototyping For The Design Spaces Of The Future. Prototype: Design and Craft in the 21st Century, p. 1–8, 2013.

SANDERS, E. B.-N.; STAPPERS, P. J. From Designing to Co-Designing to Collective Dreaming:

Three Slices in Time. Interactions, p. 24–33, 2014.

SCHLOSS, W. Recent advances in the coupling of the language Max with the Mathews/Boie Radio Drum. Proceedings of the International Computer Music Conference. Anais...1990

SCHMEDER, A.; FREED, A. uosc: The open sound control reference platform for embedded devices. Proceedings of the 2008 Conference on New Interfaces for Musical Expression

(NIME08). Anais...Citeseer, 2008

SHAH, J.; KULKARNI, S.; VARGAS-HERNANDEZ, N. Evaluation of Idea Generation Methods

for Conceptual Design: Effectiveness Metrics and Design of Experiments. Journal of Mechanical Design, v. 122, n. 4, p. 377–384, 2000.

156

SHESKIN, D. J. Handbook of Parametric and Nonparametric Statistical Procedures: Third Edition. [s.l: s.n.].

SMITH, G. F. Idea-Generation Techniques : A Formulary of Active Ingredients. The Journal of Creative Behavior, v. 32, n. 2, p. 107–134, 1998.

STEIM. junXion. Disponível em: <http://steim.org/product/junxion/>.

STEINER, H. Building your own instrument with Pd. Proceedings of the 1st International Pd

Conference. Anais...2005

STOWELL, D. et al. Evaluation of live human–computer music-making: Quantitative and

qualitative approaches. International Journal of Human-Computer Studies, v. 67, n. 11, p.

960–975, nov. 2009.

SYLLEROS, A.; DE LA CUADRA, P.; CÁDIZ, R. Designing a Musical Instrument: Enlivening

Theory Through Practice-Based Research. Design Issues, v. 30, n. 2, p. 83–96, abr. 2014.

TANAKA, A. et al. A Survey and Thematic Analysis Approach as Input to the Design of Mobile Music GUIs. Proceedings of the International Conference on New Interfaces for Musical

Expression. Anais...University of Michigan, Ann Arbor: 2012

TANAKA, A.; ALTAVILLA, A.; SPOWAGE, N. Gestural Musical Affordances. Proc. of Sound and Music Computing Conference (SMC), p. 318–325, 2012.

TEBOUL, E. The Transgressive Practices of Silicon Luthiers. In: Guide to Unconventional Computing for Music. Cham: Springer International Publishing, 2017. p. 85–120.

TORRE, G.; ANDERSEN, K.; BALDÉ, F. The Hands: The Making of a Digital Musical Instrument.

Computer Music Journal, v. 40, n. 2, p. 22–34, jun. 2016.

TRUEMAN, D.; DUBOIS, R. L. Percolate: a collection of synthesis, signal processing, and image processing objects. Disponível em: <https://github.com/Cycling74/percolate>. Acesso

em: 21 fev. 2017.

UDELL, C.; SAIN, J. P. eMersion: Sensor-controlled Electronic Music Modules & Digital Data Workstation. (B. Caramiaux et al., Eds.)Proceedings of the International Conference on

New Interfaces for Musical Expression. Anais...Goldsmiths, University of London, UK:

Goldsmiths, University of London, 2014

VALAMANESH, R.; SHIN, D. Tangible ideation: How digital fabrication acts as a catalyst in the

early steps of product development. Proceedings of the 15th International Conference on Engineering and Product Design Education: Design Education - Growing Our Future, EPDE 2013, n. September, p. 635–640, 2013.

VASCONCELOS, L. A. et al. Investigating real-world design fixation using morphological analysis. Design Creativity Workshop. Anais...Chicago: Northwestern University, Evanston,

2016

VASCONCELOS, L. A.; CRILLY, N. Inspiration and fixation: Questions, methods, findings, and

157

challenges. Design Studies, v. 42, p. 1–32, 2016.

VASHLISHAN, M. J. The Akai Electric Wind Instrument (EWI4000s): A Technical and Expressive Method. [s.l.] University of Miami, 2011.

VERTEGAAL, R.; UNGVARY, T.; KIESLINGER, M. Towards a musician’s cockpit: Transducers, feedback and musical function. Proceedings of the International Computer

Music Conference. Anais...1996

VILLAR, N.; LINDSAY, A.; GELLERSEN, H. Pin & Play & Perform: a rearrangeable interface for musical composition and performance. Proceedings of the 2005 International Conference

on New Interfaces for Musical Expression (NIME05). Anais...Vancouver, BC, Canada: 2005

VISWANATHAN, V. et al. Prototyping: A key skill for innovation and life-long learning.

Proceedings - Frontiers in Education Conference, FIE, v. 2015–Febru, n. February, 2015.

VON HIPPEL, E. PERSPECTIVE: User toolkits for innovation. Journal of Product Innovation Management, v. 18, n. 4, p. 247–257, 2001.

VON THIENEN, J.; MEINEL, C.; NICOLAI, C. How Design Thinking Tools Help To Solve Wicked

Problems. In: Design Thinking Research. Cham: Springer International Publishing, 2014. v.

36p. 97–102.

WALLIS, I. et al. Amateur Musicians, Long-Term Engagement, and HCI. In: HOLLAND, S. et al.

(Eds.). . Music and Human-Computer Interaction. [s.l.] Springer London, 2013. p. 49–66.

WALTER-HERRMANN, J.; BÜCHING, C. FabLab: Of Machines, Makers and Inventors. [s.l.]

transcript Verlag, 2014.

WANDERLEY, M. M. Instrumentos Musicais Digitais: Gestos, Sensores e Interfaces. In: Em Busca da Mente Musical. [s.l.] Editora da Universidade Federal do Paraná, 2006. v. 60.

WANDERLEY, M. M.; ORIO, N. Evaluation of Input Devices for Musical Expression: Borrowing

Tools from HCI. Computer Music Journal, v. 26, n. 3, p. 62–76, 2002.

WARD, N.; TORRE, G. Constraining Movement as a Basis for DMI Design and Performance. Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...Goldsmiths, University of London, UK: 2014

WARFEL, T. Z. Prototyping: A Practitioner’s Guide. [s.l.] Rosenfeld Media, 2009.

WASSERMAN A., E.; CULLEN, P. Evolution of the Violin: The Law of Effect in Action. Journal of Experimental Psychology: Animal Learning and Cognition, 2015.

WESSEL, D.; WRIGHT, M. Problems and Prospects for Intimate Musical Control of Computers. Proceedings of the CHI’01 Workshop on New Interfaces for Musical Expression

(NIME-01). Anais...2001

WIETHOFF, A. Prototyping Tools for Hybrid Interactions. [s.l.] Ludwig-Maximilians-Universität

München, 2012.

158

WOOD, L. E. Semi-structured interviewing for user-centered design. interactions, v. 4, n. 2, p.

48–61, 1997.

WRIGHT, M.; FREED, A.; MOMENI, A. OpenSound Control: State of the Art 2003.

Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03).

Anais...Montreal, Canada: 2003

XAMBÓ, A. Tabletop Tangible Interfaces for Music Performance: Design and Evaluation.

[s.l.] The Open University, 2015.

YOUMANS, R. J. The effects of physical prototyping and group work on the reduction of design

fixation. Design Studies, v. 32, n. 2, p. 115–138, 2011.

ZAPPI, V.; MCPHERSON, A. Dimensionality and Appropriation in Digital Musical Instrument Design. Proceedings of the International Conference on New Interfaces for Musical Expression.

Anais...Goldsmiths, University of London, UK: 2014

ZWICKY, F. The Morphological Approach to Discovery, Invention, Research and Construction.

New Methods of Thought and Procedure, p. 273–297, 1967.

159

APPENDIX A - RELATED PROJECTS Tool Reference

Arduino http://arduino.cc

Bareconductive Touch Board https://www.bareconductive.com/shop/touch-board/

BeagleBoard https://beagleboard.org/

BELA http://bela.io/

bitalino http://www.bitalino.com/

Block Jam (NEWTON-DUNN; NAKANO; GIBSON, 2003)

Bloctopus (SADLER et al., 2015)

d.tools (HARTMANN et al., 2005)

eMersion (UDELL; SAIN, 2014)

fungible interfaces (HOLLINGER; THIBODEAU; WANDERLEY, 2010)

i-cubeX http://infusionsystems.com/catalog/

Interface Z http://www.interface-z.fr/

iStuff (BALLAGAS et al., 2003)

Joué http://www.play-joue.com/

Keys https://igg.me/at/keys/x

LightPad https://roli.com/products/blocks

littleBits Korg https://littlebits.cc/kits/synth-kit

Makey Makey http://www.makeymakey.com

mbed https://www.mbed.com

microduino mCookies https://www.microduino.cc/

Mine Modular Controller http://special-waves.com/

MODI http://www.luxrobo.com/

Modulares Interface https://vimeo.com/108885687

Modulome (BARRACLOUGH; MURPHY; KAPUR, 2014)

mogees http://www.mogees.co.uk/

160

Molecule Synth http://moleculesynth.com/

Nascent Objects http://www.nascentobjects.com/

Neo http://www.lolagielen.nl/neo.html

OpenDeck https://github.com/paradajz/OpenDeck

oplab https://teenage.engineering/products/oplab

Ototo http://www.ototo.fm

Palette https://palettegear.com/

Phidgets http://www.phidgets.com/

PHOXES (GELINECK; SERAFIN, 2010b)

PHYSMISM (NIELS; GELINECK; SERAFIN, 2007)

Pin&Play&Perform (VILLAR; LINDSAY; GELLERSEN, 2005)

PowderBox http://yoshihito-nakanishi.com/works/device/powder-box/

Pulse Controller http://www.tetmusic.com/

Raspberry PI https://www.raspberrypi.org/

Reactable http://reactable.com/

SAM https://www.samlabs.com/

Satellite CCRMA (BERDAHL; JU, 2011)

Sifteo http://www.sifteo.com

Sound Clippys Platform http://modular-muse.com/digital-music-instruments/

SPINE (HADJAKOS; WALOSCHEK, 2014)

Spyractable (POTIDIS; SPYROU, 2014)

TeaBox (ALLISON; PLACE, 2005)

Tessel https://tessel.io/

TinyDuino https://tiny-circuits.com/tinyduino_overview

UI Prototyping Kit (FEIL; TUNG, 2013)

Video-Organ (BONGERS; HARRIS, 2002)

x-OSC http://x-io.co.uk/

Filipe Carlos de Albuquerque Calegario · combinação, adaptamos um método de design tradicional,...

Documents

Transcript of Filipe Carlos de Albuquerque Calegario · combinação, adaptamos um método de design tradicional,...