In the course Invention Design II, we developed a guitar learning app that uses augmented reality to make the learning process as easy as possible. In the app, the user’s hand on the guitars fingerboard is recorded and displayed on the screen. On the mirrored image, the chords to be learned are drawn directly on the fingerboard.This gives the user real-time visual feedback on how to grab a chord.This makes it possible to learn guitar without sheet music and to have a fast success.
In a design sprint, we looked for a technology and an existing problem at the same time in order to solve the same with the technology. We choose the technology “Mixed Reality”, more concretely then the spin-off “Augmented Reality” (AR). The initial challenge was to improve the a learning experience through AR. In the Sprint we came up with the Idea that we could enable playing the guitar without learning the notes first with Augmented Reality. We designed a first prototype to simulate mixed reality glasses. This was to test whether a mixed reality app would enrich the experience of learning to play the guitar. For this purpose, a click prototype was created on an iPad. This was held up in the air for testing where the user was looking. Through this, we learned that a mixed reality app works well and allows guitar playing without notes.
To learn more about the learning experience of instruments and more specifically the guitar, we conducted numerous interviews and examined established solutions such as music books and learning apps for instruments. We also observed the use of existing learning apps and a classical lesson with a guitar teacher. Here it was important for us to get a realistic picture of the current situation and possibilities. The statements of a guitar teacher with 30 years of experience were just as relevant as the assessment of a beginner.As a result, we were able to identify the following advantages of an AR app in the learning context of “guitar”:Chords are visible on the guitar, direct feedbackUpright posture when practicing, as you do not have to lean over the guitarApp recognizes how well you play and adapts to the skill with speed and difficultyNavigating the app via voice, direct visual response
Since the view of the guitar strings with glasses is only possible with a bad posture, we decided to use an augmented reality app. This should be available for the iPad or a PC with a webcam. To test whether playing via a mirrored view with the marking of chords makes it easier for beginners to get started, another prototype was created. The prototype AR app consisted of colored paper snippets marking chords and the iPad’s inner camera turned on. Users were quickly able to correctly grasp the first chords and practice chord changes.
The idea was further elaborated in another prototype. For this, an interface was designed and displayed in the prototyping app Protopie over the image of the inner camera. The successful recognition of correctly played chords was implemented using the Wizard of Oz method. Thus, the animations were triggered remotely using a keyboard. For testing, the guitar had to be fixed so that all animations were displayed at the correct position above the guitar.
The design sprint helped us to not get lost in research and to quickly find a concrete idea for a problem. The prototype created in one day was definitely enough to test the concept developed in the sprint. The testing was very honest, as we had only invested five days in the concept and there was clearly the possibility to discard everything and start again. Besides the design sprint, we learned a lot about prototyping augmented reality applications. With the realization that it is important to think in advance about what insights should be gained with the prototype. Based on that, the simplest prototype that can support or refute the thesis should be designed. In doing so, we found that sometimes a simple prototype with tape and a selfie camera is enough to test an augmented reality idea.