The rough idea of this application is to explore a space of short melodies by listening to them and imitating them on an instrument.

It works (best) in Google Chromium or Chromium-based browsers. It is best with an external MIDI instrument, but a QWERTY instrument is provided. You need a MIDI driver set up on your computer.

There are four main elements:

1. a TARGET, a melody which is to be imitated,
2. a RECORDER which will record what a user plays (via MIDI),
3. a KEYBOARD / MIDI instrument, which allows the user to play melodes, and
4. A grid of INTERPOLATIONS (and variations) between a set of four defined melodies, generated by Google Magenta VAE, which can be navigated for use as targets. The user can therefore navigate a space of variations between known and generated melodies, hopefully developing musical fluency in the process.

The application is divided into sections. Press the “More…” / “…Less” button in the top right hand corner to see them all:

From top to bottom:

• The TOP RIGHT SECTION contains overall controls:

  • “State” will print the appliation’s state to the browser console (via dev tools. Useful for debugging and saving melodies)
  • “Reset” reloads the application with default settings
  • “MIDI reset” can (sometimes) help with hanging notes
  • “Score” cycles between graphical views of melodies (traditional score, note chart and none)
  • “Set” loads pre-set sets of source melodies for interpolating, as displayed under INTERPOLATION and SOURCES.
  • “More…” / “…Less” toggles between all and essential controls

• The TARGET section contains the current working melody, which can be played back, changed for another saved one and saved. If you save a melody, it can be used as a SOURCE for interpolation (via the drop down lists under SOURCES)

  • Press “Vary” to generate variations of the melody using Google magenta
  • Only two bar melodies are used at present
  • This section will turn green if the recording (below) matches the target melody.
    • It only matters to have the correct order of notes for this to match. Rhythm and timing do not matter.

• the RECORDER section contains a representation of the current recorded melody, and controls for recording to it and playing it. Recording is done via MIDI, either using the QWERTY midi keyboard included, or using an external instrument.

  • Try pressing “REC” and playing some notes on the keyboard below. Once you press “REC” = “STOP” again, the new recording should be displayed.
  • When you finish recording, a sound will be played to indicate whether the recording matches the TARGET.
  • There are buttons for transferring the recording to the TARGET section, as well as saving it and varying it (as under TARGET)
  • “Use click” will play a metronome when recording

• Next are some TRANSPORT CONTROLS, useful for controlling the imitation cycle which is central to the application:

  • “Review Play” will play the recording and then the target, allowing aural comparison.
    • Shortcut key: Alt
  • “Play Rec” will play the target, and then start recording so that you can try to play the same melody (on the QWERTY keyboard or another MIDI instrument). The recorder automatically stops after an appropriate time.
    • Shortcut key: Space
    • Alternatively, on a MIDI keyboard, a short press of the sostenuto pedal will work.
  • “Timed Rec” starts the recorder and stops it automatically after an approriate time.
    • Shortcut key: Alt Gr
  • You can press “ESC” to cancel all of these operations.
  • “Next” advances to the next melody, as shown on the INTERPOLATION grid below (current position highlighted in purple).
    • Shortcut key: Enter
    • Alternatively, on a MIDI keyboard, a longer press of the sostenuto pedal will work.
    • Instead of presing Next / Enter, you can use the arrow keys on your computer keyboard to specify which directly to move in.
  • “Tempo” sets the playback and recording tempo in beats per minute.

• KEYBOARD is a qwerty keyboard, for use it you don’t have an alternative instrument. INTERNAL PIANO below can be used to produce sound,

  • You can use alternative web instruments such as the excellent AQWERTYon.
  • Press CAPS LOCK to move up an octave
  • “First note” is used to change the relation between keys and pitch. By default, C below middle-C is on the C key.

• PLAYER controls the sending of MIDI instructions. You can change the MIDI out port to your own instrument here, and enable a MIDI metronome on playback.

• INTERNAL PIANO is a basic sound producing instrument in case you don’t have your own. Switch it off if you want to use another, or adjust its volume.

• INTERPOLATION shows the interpolation grid. Melodies are generated on an 8 by 8 grid, between sources A (top left corner of grid), B, C and D (bottom right corner), shown under SOURCES.

  • Click “interpolate” to re-run the interpolation (can take a few seconds)
  • The current location is shown in purple
  • Melodies which have been correctly reproduced on the recorder are shown in green
  • Click on each number on the grid to view the score, and to play, set to TARGET or save the melody, as described above.
  • Note that arrow keys can be used to move the current melody (purple). The new target will be played on moving.

• Lastly the SOURCES section allows specification of the four source melodies.

  • Use the drop down menus to change melodies used.
    • Press “interpolate” after changing them to generate new melodies
  • The melodies are preset, but you can save new ones (e.g. by recording, or varying existing ones)
  • The melodies here can be changed using the “set” control (top right)
  • The usual play, set to TARGET and vary controls are available.

Suggestions

• Try using the application with the scores (graphical representations of melodies) hidden (use the “score” button). This will force you to use your ear to hear differences.
  • Or try using scores without listening to develop sight reading skills.
• It should be possible to operate the application using only your computer keyboard, with or without a midi keyboard (or using the sostenuto pedal on your MIDI piano). Try to develop a flow.
• Try different means of navigating the grid (using arrow keys, or otherwise the set target button under INTERPOLATION).
• Trying using the application with an alternative instrument, such as aQWERTYon, an online accordian.

Troubleshooting

• There may be MIDI issues if you don’t have a MIDI driver installed. This will prevent recording from functioning.
• If you need to give permissions to Tone.js to play sound, click on the “More…” / “…Less” button once.