Above is the first working prototype of a true AI assistant for music production. An AI that puts back control into the hands of the creative ! Here’s some background.
My work on my BA(Hons) Creative Music Technology allowed me to verify where the tech industry has gone wrong with providing tools for DJ’s, musicians and artists. This led to the development of Phys DJ. An actual “physically modelled” DJ turntable.
Newsflash. Nobody has done this before !
Physical modelling is the same mathematical approach that physical modelling synths use that model a trumpet or a drum. In fact one of the earliest examples of a physical model was its influence on the film 2001. In fact I wrote about this in my final year dissertation on my BA(Hons) Creative Music Technology.
“The publics first exposure to physical modelling was when the film 2001: A space Odyssey was released (2001 1968). In the film HAL, the ship’s AI, is having his memory modules removed. He regresses to singing the song A Bicycle Built for Two. This is a reference to the first pioneering work on physically modelling of the human vocal tract in 1962. The software Max (by Cycling 74) is named after Max Mathews, a pioneer in computer music. In 1961 Max arranged a demonstration of technology that included a rendition of the song sung by a computer model of the vocal tract (Kelly 1962). The author Arthur C Clarke was present, and was so impressed that he included the song in the film 2001 to honour this achievement.”
FREEMAN, Michael. 2025. To what extent was the liveness of the DJ as instrumentalist discarded in deference to a technological “revolution”? Unpublished BA (Hons) dissertation. Falmouth University.
In fact the Phys DJ prototype was developed, proven, tested and physically modelled in the Max software !
As part of this journey, after the release of Chat GPT and the resulting meteoric rise of A.I., I quickly started using AI coding assistants in my university work. There are a lot of misconceptions about this use of AI, not helped by enormous marketing hype. The idea is still there that “AI can produce a full app“. This may be the case in some circumstances. But, anyway it kind of misses the creative potential of AI as an assistant. The misconception has carried over into music production that is held back by “one click solutions” to mastering tracks, and even worse spitting out entire tracks based on very little prompt interaction by the user. That’s OK. It has its place. What if you quickly want some background music for your video ? But it probably won’t sound very creative. AI can’t create. Only humans can. AI by definition is not conscious, sentient or self aware. But it is a very clever technology and is very good at assisting with coding.
This should be the same when a musician is putting together a creative music track in Ableton Live, Apple Logic or FL Studio. Yet there’s very little out there ! I started looking into developing an AI agent that can act as an assistant for the musician.
“Create a skeleton drum and bass drum track.”
“OK cut up this sample of a drum kit and apply it to the MIDI sequence.”
And so on.
The musician would then manually create a synth line over the beats using their own virtuosity.
In fact there’s already something out there that proves all this is possible. The Perplexity Comet browser is very effective at operating entire complex websites. It has fixed a number of very thorny problems on Microsoft Azure for me.
I developed a working prototype using The Gemini 2.5 Computer Use model and tool which is what is shown at the top of the page. Of course there’s a lot more work to do before it can help produce club shaking Progressive House. But I think the approach is a sound one. The app could also be educational and help revive many musical arts that are in danger of getting lost. I’ve noticed a reduction in the quality of dance music releases over the years. There are techniques in many of these types of music that are similar to techniques used in improvisational Jazz that aren’t often well defined and rely on often unspoken conventions between DJ, musician and dancers to do with subtle rhythm development and language. An AI assistant could help save these valuable musical languages.
Leave a Reply