Music and artificial intelligence

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

Research in artificial intelligence (AI) is known to have impacted medical diagnosis, stock trading, robot control, and several other fields. Perhaps less popular is the contribution of AI in the field of music. Nevertheless, artificial intelligence and music (AIM) has, for a long time, been a common subject in several conferences and workshops, including the International Computer Music Conference, the Computing Society Conference [1] and the International Joint Conference on Artificial Intelligence. In fact, the first International Computer Music Conference was the ICMC 1974, Michigan State University, East Lansing, USA [2] Current research includes the application of AI in music composition, performance, theory and digital sound processing.[3]

Several music softwares have been developed that use AI to produce music[4]. Like its applications in other fields, the A.I. in this case also simulates mental task. A prominent feature is the capability of the A.I. algorithm to learn based on information obtained such as the computer accompaniment technology, which is capable of listening to and following a human performer so it can perform in synchrony.[5] Artificial intelligence also drives the so-called interactive composition technology, wherein a computer composes music in response to the performance of a live musician. There are several other A.I. applications to music that covers not only music composition, production, and performance but also the way it is marketed and consumed. Companies like Apple and Spotify rely on user data to augment their engagement metrics to power their ability to get consumers to listen more to music or push the right songs to users according to their preferences.[6]


In 1960, Russian researcher R.Kh.Zaripov published worldwide first paper on algorithmic music composing using the "Ural-1" computer.[7]

In 1965, inventor Ray Kurzweil premiered a piano piece created by a computer that was capable of pattern recognition in various compositions. The computer was then able to analyze and use these patterns to create novel melodies. The computer was debuted on Steve Allen's I've Got a Secret program, and stumped the hosts until film star Henry Morgan guessed Ray's secret.[8]

Software applications[edit]

Interactive scores[edit]

Multimedia Scenarios in interactive scores are represented by temporal objects, temporal relations and interactive objects. Examples of temporal objects are sounds, videos and light controls. Temporal objects can be triggered by interactive objects (usually launched by the user) and several temporal objects can be executed simultaneously. A temporal object may contain other temporal objects: this hierarchy allows us to control the start or end of a temporal object by controlling the start or end of its parent. Hierarchy is ever-present in all kinds of music: Music pieces are often hierarchized by movements, parts, motives, measures, among other segmentations. [9] [10]

Orb Composer[edit]

A program developed by Hexachords and directed by Richard Portelli, mainly focused on orchestral music.[11][12] The IA called "Orb" help the user in the composition process. The software is commercialized on their official website since march 2018.[13]


A program developed by David Cope which composes classical music. See Experiments in Musical Intelligence. Emily Howell is an interactive augmentation of EMI. (As a popular example, the background music of the viral video Humans Need Not Apply was created by "her", as revealed in the video to illustrate the likely fate of creative jobs.)


This program was designed to provide small-budget productions with instrumentation for all instruments usually present in the full-fledged orchestra. If there is a small orchestra playing, the program can play the part for missing instruments. High school and community theaters wanting to produce a musical can now benefit from the virtual orchestra and realize a full Broadway score. This software is able to follow the fluctuations in tempo and musical expression. Musicians enjoy the thrill of playing with a full orchestra, while the audience enjoys the rich sound that comes from the combination of the virtual orchestra with the musicians.[14][15]


Computer Accompaniment (Carnegie Mellon University)[edit]

The Computer Music Project at CMU develops computer music and interactive performance technology to enhance human musical experience and creativity. This interdisciplinary effort draws on Music Theory, Cognitive Science, Artificial Intelligence and Machine Learning, Human Computer Interaction, Real-Time Systems, Computer Graphics and Animation, Multimedia, Programming Languages, and Signal Processing.[17] One of their project is similar to SmartMusic. It provides accompaniment for the chosen piece follows the soloist (user) despite tempo changes and/or mistakes.



SmartMusic is an interactive, computer-based practice tool for musicians. It offers exercises, instant feedback tools, and accompaniments meant to aid musicians. The product is targeted at teachers and students alike and offers five categories of accompaniments: solo, skill development, method books, jazz, and ensemble. Teachers can give students pre-defined assignments via email and scan in sheet music that is not yet in the SmartMusic catalog. Students can choose the difficulty level they want to play at, slow down or speed up the tempo, or change the key in which to play the piece.[19] SmartMusic also compares students' playing with digital template, which allows it to detect mistakes and mark them on a score. It also simulates the rapport between musicians by sensing and reacting to tempo changes.


StarPlay is also a music education software that allows the user to practice by performing with professional musicians, bands and orchestras. They can choose their spot and watch the video from that spot. They can hear the other musicians playing. Again, the program listens to the user's performance and helps them improve their performance by providing constructive feedback as they rehearse. StarPlay was developed by StarPlayIt (formerly In The Chair), a music technology company that has won many awards for its platforms for online musical performance and participation.[20]


Developed at Princeton University by Ge Wang and Perry Cook, ChucK is a text-based, cross-platform language that allows real-time synthesis, composition, performance and analysis of music. .[21] It is used by SLOrk (Stanford Laptop Orchestra) [22] and PLOrk (Princeton Laptop Orchestra).



The Impromptu[24] media programming environment was developed by Andrew Sorensen for exploring 'intelligent' interactive music and visual systems. Impromptu is used for live coding performances and research including generative orchestral music[25] and computational models of music perception.[26]

REAPER's TabEditor[edit]

MIDI to string instrument (guitar, violin, dombra, etc.) tablature conversion is a nontrivial task, as the same note can reside on different strings of the instrument. And the creation of good fingering is sometimes a challenge even for real musicians, especially when translating a two handed piano composition on a string instrument. So in TabEditor[27] (the tiny plugin for REAPER DAW), an AI was used that solves this puzzle the same way as a musician would: trying to keep all the notes close to each other (to be possible to play) while trying to fit all the piano notes into a range that can be played simultaneously on the instrument. When direct translation is impossible (piano part has more notes than are possible on the guitar) the AI tries to find an acceptable solution, removing as few notes as possible from the original composition. The Prolog programming language was used to create this AI.


Ludwig[28] is an automated composition software based on tree search algorithms. Ludwig generates melodies according to principles of classical music theory. The software arranges its melodies with pop-automation patterns or in four-part choral writing. Ludwig can react in real-time on an eight-bar theme played on a keyboard. The theme will be analysed for key, harmonic content and rhythm while it is being performed by a human. The program then without delay repeats the theme arranged e.g. for orchestra. It subsequently varies the melody to create a little piece as interactive answer to the human input.


OMax[29] is a software environment which learns in real-time typical features of a musician's style and plays along with him interactively, giving the flavor of a machine co-improvisation. OMax uses OpenMusic and Max. It is based on researches on stylistic modeling carried out by Gerard Assayag and Shlomo Dubnov and on researches on improvisation with the computer by G. Assayag, M. Chemillier and G. Bloch (Aka the OMax Brothers) in the Ircam Music Representations group.


Melomics is a proprietary computational system for the automatic (without human intervention) composition of music, based on bioinspired methods and produced by Melomics Media. Composing a wide variety of genres, all music composed by Melomics algorithms are available in MP3, MIDI, MusicXML, and PDF (of sheet music), after purchase. Music composed by this algorithm was organized into an album named Iamus (album), which was hailed by New Scientist as "The first complete album to be composed solely by a computer and recorded by human musicians."[30]


MorpheuS[31] is a research project by Dorien Herremans and Elaine Chew at Queen Mary University of London, funded by a Marie Skłodowská-Curie EU project. The system uses an optimization approach based on a variable neighborhood search algorithm to morph existing template pieces into novel pieces with a set level of tonal tension that changes dynamically throughout the piece. This optimization approach allows for the integration of a pattern detection technique in order to enforce long term structure and recurring themes in the generated music. Pieces composed by MorpheuS have been performed at concerts in both Stanford and London.


Founded in 2015, MXX (Mashtraxx Ltd) is a creative AI technology company, which has developed the world’s first AI engine that atomises and edits music to video instantly.

The MXX technology is the only artificially intelligent music editor that can work on the stereo file alone, without the need for stems, and can adapt the narrative to fit a brief just like a music editor would.

With the MXX products, music can be adapted to perfectly wrap around certain experiences such as UGC of different lengths and forms; users workouts and runs; computer game dynamic narratives; and object-based media viewing of dynamic lengths. This presents a new future for music in itself, as music is one of the few remaining static artefacts in a world where all media needs to be more dynamic and flexible.

The first MXX product, Audition Pro, enables anyone to edit music against video with intensity rises, fades, dropouts and climax points, load any track, and immediately listen to the edited track, and makes responding to the demands of today’s media fast and simple.

Flow Machines[edit]

Flow Machines is a research project funded by the European Research Council (ERC) and led by François Pachet. Flow Machines aims at transforming musical style into a computational object to apply to AI-generated melodies and harmonies. Flow Machines has composed two fully-fledged pop songs, issued from a collaboration between the AI software and pop composer Benoît Carré: Daddy's Car [32] and Mister Shadow.[33] Flow Machines also produced DeepBach, a neural network system which produces harmonisation in Bach style indiscernible from original Bach's harmonisations.[34]


Created in February 2016, in Luxembourg, AIVA is an Artificial Intelligence that specializes in the composition of personalized soundtracks for any type of media. AIVA became the world’s first virtual composer to be recognized by a music society (SACEM). By reading 30,000 scores of history's greatest composers (such as Bach, Beethoven, Mozart) AIVA is capable of understanding concepts of music theory and compose on its own. The algorithms behind AIVA are based on deep learning architectures[35] AIVA has also been used to compose a Rock track called On the Edge[36], as well as a pop tune Love Sick[37] in collaboration with singer Taryn Southern[38], for the creation of her 2018 album "I am AI".


AlgoTunes is a music software company building apps that generate music. On, anyone can signup to generate music of their preferred mood and style with the push of a button. The music is composed instantly in a few seconds by AlgoTunes' web app, and available for download as WAV or MIDI files.[39]

Amper Music[edit]

Amper is a New York City-based music software company. Its core technology is an artificial intelligence music composer that powers their enterprise platform for creating custom music. Amper’s tools can make music across a range of genres and moods and is designed to give users creative control regardless of expertise.

Amper’s technology was used by singer Taryn Southern in her 2018 album “I Am AI.”

See also[edit]


  1. ^ INFORMS Computing Society Conference: Annapolis: Music, Computation and AI Archived 2012-06-30 at Retrieved on 2010-12-22.
  2. ^ International Computer Music Association - ICMC. (2010-11-15). Retrieved on 2010-12-22.
  3. ^ Research in Music and Artificial Intelligence - Ergonomics Abstracts. Retrieved on 2010-12-22.
  4. ^ D. Herremans, C.H., Chuan, E. Chew (2017). "A Functional Taxonomy of Music Generation Systems". ACM Computing Surveys. 50 (5): 69:1-30. doi:10.1109/TAFFC.2017.2737984.CS1 maint: Multiple names: authors list (link)
  5. ^ Dannenberg, Roger. "Artificial Intelligence, Machine Learning, and Music Understanding" (PDF). Semantic Scholar. Retrieved August 23, 2018.
  6. ^ Grant, Kristin Westcott. "How To Think About Artificial Intelligence In The Music Industry". Forbes. Retrieved 2018-08-23.
  7. ^ Zaripov, R.Kh. (1960). "Об алгоритмическом описании процесса сочинения музыки (On algorithmic description of process of music composition)". Proceedings of the USSR Academy of Sciences. 132 (6).
  8. ^
  9. ^ Mauricio Toro, Myriam Desainte-Catherine, Camilo Rueda. Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music 8 (1)
  10. ^ "Open Software System for Interactive Applications". Retrieved 23 January 2018.
  11. ^ Jallet, Fabrice (13 January 2017). "Hexachords a créé une IA pour les compositeurs de musique / Actualités / Irma : centre d'information et de ressources pour les musiques actuelles". Hexachords a créé une IA pour les compositeurs de musique / Actualités / Irma : centre d'information et de ressources pour les musiques actuelles (in French). Retrieved 23 January 2017.
  12. ^ Richard, Portelli. "Orb Composer Software - Music Artificial Intelligence Engine". Hexachords (in French). Retrieved 23 January 2017.
  13. ^
  14. ^ Realtime Music Solutions Archived 2006-09-10 at Retrieved on 2010-12-22.
  15. ^ AITopics / Music. Retrieved on 2010-12-22.
  16. ^ Realtime Music Solutions. Retrieved on 2010-12-22.
  17. ^ Computer Music Group. Retrieved on 2010-12-22.
  18. ^ Video demo of computer accompaniment
  19. ^ "Discover SmartMusic". MakeMusic, Inc. Retrieved September 12, 2015.
  20. ^ (BROKEN LINK) StarPlayit music learning and video game technology platform. Retrieved on 2010-12-22.
  21. ^ ChucK => Strongly-timed, On-the-fly Audio Programming Language. Retrieved on 2010-12-22.
  22. ^ Driver, Dustin. (1999-03-26) Pro - Profiles - Stanford Laptop Orchestra (SLOrk), pg. 1. Apple. Retrieved on 2010-12-22.
  23. ^ Pro - Profiles - Stanford Laptop Orchestra (SLOrk) - Video. Apple (1999-03-26). Retrieved on 2010-12-22.
  24. ^ impromptu. Retrieved on 2010-12-22.
  25. ^ A Computational Model For The Generation Of Orchestral Music In The Germanic Symphonic Tradition: A progress report | QUT ePrints. Retrieved on 2010-12-22.
  26. ^ Interrogating statistical models of music perception | QUT ePrints. Retrieved on 2010-12-22.
  27. ^ TabEditor
  28. ^ The automatic composition software Ludwig. Retrieved on 2011-10-27.
  29. ^ The OMax Project Page
  30. ^ "Computer composer honours Turing's centenary". New Scientist. 5 July 2012.
  31. ^ D. Herremans, E. Chew (2016). "MorpheuS: Automatic music generation with recurrent pattern constraints and tension profiles". IEEE Transactions on Affective Computing. PP(1). doi:10.1109/TAFFC.2017.2737984.
  32. ^ [1]. Daddy's Car
  33. ^ [2]. Mister Shadow
  34. ^ [3].
  35. ^ [4]. AIVA 2016
  36. ^ [5] AI-generated Rock Music: the Making Of
  37. ^ [6] Love Sick | Composed with Artificial Intelligence - Official Video with Lyrics | Taryn Southern
  38. ^ [7] Algo-Rhythms: the future of album collaboration
  39. ^ [8]. AlgoTunes

Further reading[edit]

External links[edit]