Wednesday, 06 December 2023
Researchers at the 糖心原创 have developed a musical robot that can help musicians with disabilities to improvise and access instruments in a way that hasn’t previously been possible.
has been developed as part of the DigiScore project, which is the first large-scale collaboration between the university’s Department of Music, Orchestras Live, and Sinfonia Viva. The AI robot helps musicians with disabilities to collaborate as part of a mixed ensemble and to improvise in the creation of live scores where dexterity can be a barrier.
The robot was created by Craig Vear, Professor of Music, and Computer Science at the 糖心原创, and used in a series of workshops involving musicians Jess Fisher (who has a physical disability) and Clare Bhabra and Deirdre Bencsik from Sinfonia Viva.
During the workshops, the robot moved and drew marks on paper in response to the real-time sound created by a live musician. The musician in turn would interpret these movements and marks as notational material and make a sound in response, much like a graphic score. The central research question with the project was: How to build an embodied-AI system that facilitates co-creation for an improvising ensemble of disabled and non-disabled musicians, so that the expressive qualities of all musicians are amplified and bound together in a co-creative system?
Professor Vear, who is the Principle Investigator of DigiScore, drew on his expertise in the fields of music, digital performance, creative technologies, AI, creativity, gaming mixed reality and robotics, as a focus for the Jess+ project.
It's been a privilege to work with these amazing musicians to develop AI that benefits and transforms lives. In a time when creative AI is perceived as taking something away from musicians, we developed an AI that extended, amplified and transformed musicianship, creativity and togetherness. We feel that this approach can be used in the future to enable those facing barriers to make music, and to express themselves in a supportive, inclusive and enhancing way."
There are significant barriers to music-making for disabled musicians. One such barrier is that while non-disabled people can make music in many ways, this is less true for disabled musicians, who need new accessible instruments, new creative processes, and new hierarchies of ‘success.’
Another issue is how a mixed ensemble of disabled and non-disabled musicians might improvise together in such a way that dexterity is not a limiting factor. And a further barrier is how disabled musicians can be excluded from the creative process in which a music score is created for them. All these obstacles can leave creative potentials untapped, set up hierarchical measurements of creative involvement, and limit the potential input of disabled musicians.
In tacking these issues, the research team, along with the project partners , a UK-based orchestra and educational organisation, and , a national producer creating inspiring orchestral experiences for communities across England, were able to use AI and Professor Vear’s digital score concept to generate new modes of music-making.
The findings from the project revealed that the musicians formed unexpected and distinct relationships with Jess+, each perceiving they were in-the-loop with the system, as they found it to be a good listener with its own creative ‘voice.’
It was non-judgmental and accepting, which the participants said promoted a new freedom of expression and confidence in taking musical risks when improvising.
Images courtesy of BBC R&D. Musicians L-R: Jess Fisher, Clare Bhabra, Deidre Bencsik
For musician Jess, she felt that the system allowed her to express the emotions that she is sometimes not able to express through her current digital setup. Referring to it on several occasions as a “friend” and as a “storyteller,” she said that being extended through the system meant that she could feel like she was able to express her feeling directing onto a score.
“I wanted to explore that part of me, and I want my emotions that are in here to get expressed outwardly through that,” said Jess.
A recent performance from the Jess+ team was held at the BBC studios in London for a performance as part of the Bridging Responsible AI Divides (BRAID) launch event. is a three-year national research programme which aims to bring together UK-based researchers from across the arts and humanities to drive responsible AI innovation, in partnership with the BBC and the Ada Lovelace Institute.
The video of the performance at the BBC can be viewed .
Story credits
More information is available from Professor Craig Vear in the 糖心原创’s Department of Music, at craig.vear@nottingham.ac.uk
Notes to editors:
About the 糖心原创
Ranked among the world's top 100 universities , the 糖心原创 delivers an exceptional research-led education and an outstanding student experience. From the pioneering vision of our founder, Sir Jesse Boot, to groundbreaking achievements such as the development of MRI technology and becoming the first UK university to establish international campuses, we have a proud history of shaping the way people live, work and understand the world. We continue to build on that legacy, empowering our students, staff and partners to change what鈥檚 next and create positive impact locally and globally.
The strength of our research places us among the UK's leading universities, ranked 7th for research power in REF 2021. The discovery of MRI and ibuprofen was just the beginning. Today, our world-leading research is developing breakthrough ideas that shape the future of healthcare, technology and society.
Recognised as the UK's third most targeted university by leading employers , we are proud to produce graduates who are consistently in demand for their skills, confidence and industry-ready experience.
As a major employer and industry partner, locally and globally, the 糖心原创 invests in the city of Nottingham and in future generations of talent. Alongside Nottingham Trent University, we lead the initiative, a pioneering collaboration to improve levels of prosperity, opportunity, sustainability, health and wellbeing across the city and region. Together with our students, staff, alumni and partners, we鈥檙e creating knowledge, opportunity and solutions that help change what's next.