Capstone Project Presentation - Imitation Lab

Summary

This project presents a web application called Imitation Lab which allows experimentation with melodic exploration and remixing by imitation, variation and interpolation.

• Application: https://phhu.org/imitationlab
  • Use Chrome or Chromium only
  • https://phhu.org/imitationlab#basic will load faster
• Source code: https://github.com/phhu/imitation-lab
• Documentation: https://phhu.org/imitate

The project draws on previous Creative Coding work such as my previous, eponymous p5.js based imitation lab application for video. It is similarly written in Javascript, but uses React, Redux and Google Magenta instead of P5.js. See the source code for more details.

It attempts to create a conversational system, which melodies can be developed - and expressed using a suitable MIDI instrument - by goal-based interaction with artificial intelligence. This reflects an interest in Conversation Theory, an off-shoot of cybernetics developed by Gordon Pask and Paul Pangaro. (See also my previous project Melody Conversation for development of this theme).

In doing so, it allows copyrighted melodies to be blended with public domain ones, generating potentially novel melodies interpolated between these, and questioning some of the legal issues brought up in US law by the recent Katy Perry legal cases, as discussed by Adam Neely: can a melody be commercially exploited if it has been explicitly developed under the influence of other copyrighted melodies? Is such remixing not how culture has always developed? (See my previous work Melody Interpolation for further discussion of the cases involved)

The project is a remix on (at least) two levels:

• It combines, develops and critiques a number of previous projects, both the author’s and Google Magenta demos such as AI duet, Latent Loops and Beat Blender. Maestro Vocal Coach is also particularly interesting because it uses Magenta for vocal (singing) learning: this project is perhaps an instrumental version of this.
• It allows users to remix existing melodies into potentially novel ones. It is thus design for remixing.
• There is also an implicit critique of remix culture: remix culture can sometimes seem to be about recombining elements which the artist could not create himself (trivially, creating “internet memes” pictures with slogans). Here, the demand is to train the human musical mind to remix and create, on a more sensory level.

In terms of design, the interface seeks to allow the user to choose the level of complexity required, hiding and showing controls. In particular, visual prompts such as scores can be hidden in order to insist upon the primacy of the human ear in musical learning and development. In principle, the whole application can be controlled using a MIDI keyboard, foot pedal and sound, following Pangaro’s manifesto “Less Interface / More Dance!”.

There is also a critique of the role of Artificial Intelligence. This project seeks to force the user to learn to play music using an instrument, rather than by pressing buttons, thereby insisting upon the completeness of a sense-act cycle within the human mind, and viewing technology as a crutch to this process rather than a replacement of it: AI should engage with our curiosity and help us expand our perceptual and expressive possibilities, rather than replacing them.

Conversation Theory

This project is was influenced by conversation theory, as presented by Paul Pangaro. The following diagram summarises theory:

Conversation theory posits the development of ideas by interaction between two agents, based on a shared language. In the context of this project, the agents are presumably Google Magenta, which generates the melodies, and the user. The langauge is music notes, which can be represented easily digitally (using MIDI, and Google’s iNote JSON format), as well as graphically (in score) and in the human mind (or the mind’s ear). The context is set by the starting melodies chosen…

The user presumably has a goal: perhaps to improve their ability to play music by ear; or to explore variations of a melody for composition or improvisation; or perhaps to better understand the absurdity of recent court cases on copyright, or otherwise to find new melodies to exploit commerically… More prosaically, we might consider the goal to be to explore the chess board like grid of melodies… this is a kind of gamification perhaps, the sort of crutch that people sometimes use to progress.

The exchange happens in the playing back and forth of melodies: user and computer take turns to play and listen. Action and Transaction…

The computer and the user are both responsible for evaluating: both can judge is a melody was correctly reproduced by the user. Ultimately, it is to be hoped that the user will improve in their ability to evaluate (sense) as well as to act (play). Hopefully not just to a computer!

This theory is richer than the theories of memes I had looked at previously when considering imitation in design (see e.g. Imitation Games). It insists upon agency and purpose, and allows progress towards goals rather than seeing an arbitrary world.

Dark Horse and Joyful Noise

Listen to Joyful noise: https://www.youtube.com/watch?v=QPtynHTDlC0&t=57s

And Dark Horse: https://www.youtube.com/watch?v=QPtynHTDlC0&t=67s

Is there a difference?

Curiously Google Magenta finds it hard to create interpolated forms between these two:

Notice how the A and B icons, for Joyful Noise and Dark Horse respectively, are along side each other. Other varieties mentioned by Adam Neely (Go Down Moses and Godzilla) don’t have such a hard time…

But what would happen if we combined melodies deliberately to create new ones? Would that infringe copyright, since there’s a clearly traceable course from the copyrighted melody to the non-copyrighted one.

Indeed we can develop this question further by deriving a melody from Dark horse, Joyful Noise (copyrighted) and Mozart (public domain).

Who would get the royalties for a banal melody like 27 below, derived from all these artists work?

Of course, here we’re only concerned with melody, and not the other aspects of music: an argument presented by Adam Neely at https://www.youtube.com/watch?v=0ytoUuO-qvg&t=131s

Experimental System

In my proposal, I mentioned my desire to create an experimental system: that is to say a system rich enough produce unexpected results. Have there been any?

It’s still early and hard to tell. However, I would note that I find the system more absorbing to use that the other demos on Google Magenta. I think this is because it’s more meaningfully interactive than the other demos - though this may be creator’s bias! There’s a gradient of development, and crucially this gradient is under the user’s control (the user can choose their path across the grid to some extent, or explore it arbitrarily). This capacity to regulate felt difficulty (…shades of American Pragmatism) is an important quality in design.

I note too that giving the option to remove the musical score makes the system much harder for me to use! Where I though I might want more than 2 bars of music to work with, it turns out that without score it’s much harder for me to reproduce melodies. Sometimes less is more, and interfaces shouldn’t be visual.

Also, while I’m used to reading traditional score, the alternative visualisation do have value, as shown in the diagrams above, which allow easier graph-like comparison.

It’s also interesting to experiment with generating music in the style of a given composer. Here some passable artificial mozart, creating using the “Sonata in C 3” set:

36 is not brilliant, but it does show characteristics of its progenitors.

And we can try combining registers. Here some of the results and musically interesting…

Perhaps, if I test some more, I will come up with some of my own - and perhaps face a court case of my own ;-).

Further work

It would be interesting to develop this idea further. Some ideas:

• Allow the integration of this digital (MIDI) medium with analogue sources. An interesting example of this is Aqwertyon, by NYU music experience design. The theory screen is worthy, but the integration of youtube backing with the instrument is very fun - for a while.
  • Perhaps we could convert somehow between the digital and analogue words (Google Magenta can now convert piano audio to media
• The application could be expanded to include other composerly tricks for varying melodies (e.g. inversion, arpeggiation - some of which are covered by Google Magenta).
  • and we could introduce notions of harmony, rhythm and polyphony (or counterpoint)
• Video and audio could be used to allow comparison to real musicians (MIDI here has many limitations in terms of expression: using analogue - or richer Midi - would allow comparison of details of expression, dynamics, phasing etc).
• Perhaps some testing on (young?) musicians other than me?
  • The interface still needs to be streamlined, and bugs removed, before this would work well.
• Further minimisation of the interface might be interesting. For example, we could set the recorder to run all the time (like a listening human ear?), and have it recognise when you’ve played the melody correctly, rather than having the user need to trigger a recording. Then the user would only need to trigger to hear (a) the melody or (b) what s/he has recently played (e.g. last similar attempt)
• Import and export of melodies and sets of melodies could be facilitated (e.g. allow scanning of scores, import of MIDI files, drag and drop GUI score editor…)