THE UNHEARD SYMPHONY: HOW AI IS TRANSFORMING OPERA AND GIVING VOICE TO NON-VERBAL DISABILITIES
In a world increasingly shaped by technology, art often serves as a vital mirror, reflecting our realities and aspirations. This summer, an extraordinary undertaking at an Omaha, Nebraska, festival pushed the boundaries of both art and technology, exploring the profound intersection of performance, disability, and artificial intelligence. The new work, an opera titled “Sensorium Ex,” delves into fundamental questions: Who has a voice? And who truly gets to be heard?
This groundbreaking production, highlighted by PBS NewsHour’s Senior Arts Correspondent Jeffrey Brown, is more than just a theatrical event; it’s a powerful statement of advocacy and inclusion. It’s a testament to how AI, often viewed with skepticism or fear, can be harnessed for profound human benefit, specifically in empowering individuals with non-verbal disabilities and reshaping the very landscape of opera itself.
THE SILENT SYMPHONY: UNVEILING “SENSORIUM EX”
“Sensorium Ex” plunges its audience into a dystopian future that eerily mirrors our present concerns, a world where artificial intelligence is used to craft a new form of life. At its core, the opera tells the poignant story of a mother, Mem (sung by Hailey McAvoy), and her disabled son, Kitsune, as they navigate danger and seek escape. The narrative is richly layered, exploring themes of love, hope, and survival against a backdrop of technological evolution.
Composer Paola Prestini, a driving force behind “Sensorium Ex,” articulated her deep interest in the project’s potential to open new avenues through art. Her vision was to create a space that fosters deeper listening and embraces diverse forms of expression, particularly for those who communicate differently or live with various disabilities. Prestini emphasizes the opera as a blueprint for a more inclusive future, where the complexity of human life is fully represented on stage, serving not just as a mirror to society, but as a pathway to a better way of understanding and interacting.
A PERSONAL LIBRETTO: BRENDA SHAUGHNESSY’S VISION
The emotional depth of “Sensorium Ex” is significantly amplified by its libretto, penned by renowned poet Brenda Shaughnessy. This marked Shaughnessy’s first foray into opera writing, and it was deeply personal. Her own lived experience as the mother of a nonverbal, nonambulatory child, Cal (now 18), provided the raw, authentic grounding for the story. Shaughnessy describes the writing process as both painful and necessary, driven by a profound desire to see children like her son included and celebrated.
As Shaughnessy eloquently put it, “Kids like him don’t get to be part of anything.” By writing Kitsune into the opera, she aimed to make him the hero, to give him a central role, and most importantly, to ensure he was heard. This mission became the guiding principle for the entire project: a steadfast determination to make “voice” and “being heard” accessible to everyone. This commitment extended to the cast and crew, which intentionally included individuals with disabilities such as cerebral palsy, autism, and blindness, ensuring authentic representation both on and off stage.
THE HEART OF THE OPERA: GIVING VOICE TO THE UNHEARD
In traditional opera, the voice is paramount. So, how does one compose for those presumed to have none? This was the central challenge and the profound innovation of “Sensorium Ex.” Kitsune’s voice isn’t just an element of the opera; it’s central to its very fabric, embedded in the score, the music, the technology, and the plot itself. Prestini describes the creative process as akin to writing for an instrument that doesn’t yet exist, pushing the boundaries of musical composition.
The solution involved amplifying the nonverbal ways in which individuals communicate and then finding a deeper, technological expression for these unique forms of communication. This approach not only broadened the definition of “voice” but also fostered a richer, more nuanced artistic experience.
JAKOB JORDAN’S JOURNEY: A BREAKTHROUGH IN COMMUNICATION
Central to this exploration is 23-year-old Jakob Jordan, one of the two actors playing Kitsune. Jakob, who is nonverbal, embodies the very essence of the opera’s mission. Diagnosed with autism and apraxia as a toddler, Jakob’s neurological condition means his body and speech often fail to respond to commands from his brain. He humorously describes his experience: “My thoughts were being held hostage inside my head while the wrong words were impersonating me, tricking everyone to believe I was much simpler than I am.” He recounts how his speaking voice would get “stuck in loops,” making communication frustratingly difficult.
A life-changing breakthrough occurred just two years prior to the opera’s production when Jakob learned to type words with the assistance of a trained communication partner. This method allowed his thoughts to flow freely, and once a sentence is complete, it is voiced through an electronic device. While this electronic voice facilitates much of his daily communication, Jakob shared a profound desire to express himself purposefully with the voice he was born with—a desire the “Sensorium Ex” project sought to fulfill.
THE INNOVATION OF SENSORIUM AI: TECHNOLOGY AS AN ENABLER
The realization of Kitsune’s unique voice was made possible through Sensorium AI, a groundbreaking partnership between the opera’s creative team and NYU’s Ability Project. Located in Brooklyn, New York, the Ability Project is a dedicated research lab focused on the intersection of disability and technology, working to bridge gaps and create innovative solutions.
Dr. R. Luke DuBois, an NYU professor, engineer, researcher, composer, and musician, spearheaded the technical development. DuBois and his team began by meticulously recording the sounds Jakob naturally makes—whistles, singing, speech-like sounds. “So we have got recordings of Jakob just doing his thing,” DuBois explained, noting Jakob’s inherent expressiveness.
The core of the Sensorium AI system is its ability to take these raw recordings and “infer from it the physical body that made it.” This inference allows the AI to create a sense of Jakob’s unique vocal anatomy and, by extension, his distinctive voice. DuBois demonstrated this by inputting a phrase like “I would like to go sailing tomorrow” and having it “cranked through” the system, resulting in a recording that “sounds like Jakob.” This groundbreaking work highlights the immense potential of artificial intelligence in voice synthesis. For those interested in exploring the capabilities of modern AI in generating speech, a free AI audio generator can provide a fascinating glimpse into how voices are created and manipulated using advanced algorithms. Such tools demonstrate the broader accessibility of technologies that were once confined to specialized research labs.
Furthermore, DuBois and his team incorporated sensors that allow Jakob to control elements of his synthesized speech. By moving his hand, Jakob can influence the pitch and pace of his AI-generated voice, adding a layer of expressiveness that goes beyond simple text-to-speech. This bespoke technology was integrated directly into the opera, creating a powerful irony: a story about the potential evils of AI using AI’s benefits to empower its characters. This also demonstrates the commitment to leveraging technology for humanistic purposes rather than dystopian ones.
BEYOND THE STAGE: THE BROADER IMPLICATIONS OF OPEN-SOURCE AI
One of the most impactful aspects of the Sensorium AI project is its commitment to open-source technology. This means the underlying code and methodology are publicly available, allowing anyone to access, utilize, and further develop the innovation. DuBois expressed his fervent hope that this technology will be in the hands of everyone who needs it. He is actively engaging with speech researchers across the country, confident that this open-source approach will lead to an explosion of new applications and advancements.
The implications of this open-source model are vast. It democratizes access to advanced assistive technology, fostering a collaborative environment where innovations can rapidly evolve and reach those who can benefit most. This approach contrasts sharply with proprietary models, promoting accessibility and widespread impact over exclusive ownership.
REDEFINING OPERA AND ARTISTIC INCLUSION
For Paola Prestini, “Sensorium Ex” represents a powerful blueprint for the future of opera and other art forms. It challenges traditional notions of performance and redefines what it means to have a “stage voice.” By demonstrating how artistic teams can navigate the challenges of working with diverse abilities, the opera sets a precedent for greater inclusion in the arts. Prestini envisions a future where the opera provides a clear system and understanding, proving that while challenges exist, they can be overcome step-by-step, ultimately leading to a richer and more representative artistic landscape.
The project showcases that embracing diverse abilities not only provides opportunities for marginalized groups but also enriches the art itself, adding layers of complexity and authenticity that resonate deeply with audiences. It’s about building more welcoming spaces where everyone’s unique form of expression is valued and amplified.
A PATHWAY TO A BETTER WORLD: THE HUMAN IMPACT
The emotional resonance of “Sensorium Ex” lies in its profound human impact. Brenda Shaughnessy’s desire for her son, Cal, to participate in the world, to not be disconnected on the sidelines, encapsulates the broader aspiration of the project. She yearns for disabled, nonverbal children to see “Sensorium Ex” and not just imagine themselves as actors or opera performers one day, but to experience a deeper, more fundamental recognition: “There I am. I exist.”
This sense of validation and visibility is crucial for self-identity and belonging. The opera isn’t merely entertainment; it’s a vehicle for social change, fostering empathy and understanding while showcasing the untapped potential within diverse communities. It champions the idea that every individual, regardless of their mode of communication, possesses a unique voice worthy of being heard.
CONCLUSION: THE POWER OF VOICE, ART, AND INNOVATION
The “Sensorium Ex” opera in Omaha stands as a beacon of innovation and inclusion, powerfully demonstrating how art, disability, and cutting-edge technology can converge to create something truly transformative. By harnessing AI not as a threat but as an empowering tool, the production has given voice to the unheard, shattered preconceived notions about artistic expression, and illuminated a path toward a more inclusive future for both the arts and society at large.
As Jakob Jordan himself reflected on his performance, after a minute of typing his heartfelt response: “It was the most fulfilling experience of my life.” His words, synthesized yet profoundly his own, encapsulate the immense power of this endeavor. “Sensorium Ex” reminds us that every individual possesses a unique voice, and with creativity, compassion, and innovation, we can ensure that every voice, no matter how it manifests, finds its stage and resonates throughout the world.