Big Sand

In 2023, the Data Arena supported two performances by the virtual band Big Sand as part of Powerhouse Late: Gaming at the Powerhouse Museum, and Real-Time Live! at SIGGRAPH Asia 2023 Sydney. Big Sand is the project of UTS PhD student Sally Coleman, a “science fiction virtual band” where performer and audience are geographically separated, but connected via streams of data, video and audio. For these shows, Sally performed from the Data Arena and streamed motion capture data to the Powerhouse Museum and Darling Harbour Theatre, where audiences were located. Big Sand’s virtual environments were rendered on-site in real time using Unreal Engine where Sally’s virtual character Taal comes to life in dance and audience interaction.

Live Data Streaming

The Data Arena’s production equipment includes an Optitrack Motion Capture system, including Motive 3 software for data streaming and recording. For Big Sand’s performances, Sally was dressed in a suit with over 40 individual tracking markers to identify limbs and joints. These constructed a virtual skeleton, which was streamed to Unreal Engine to animate Big Sand’s lead character Taal. As this data was very lightweight, there was very little latency in the stream, allowing a near-instant representation of Sally’s movements in the virtual environment.

In addition to body motion capture, facial expressions were also captured and streamed via an iPhone’s front-facing LiDAR and Unreal Engine’s Live Link Face application. Additional streams for hand and finger gestures, voice and timing/event triggers were all used by Sally to orchestrate the performance while kilometres away from the audience.

Audience interaction and engagement

During both shows, on-site camera and microphones provided an audience feed back to the Data Arena. This allowed Sally to engage with the crowds in real time via discussion, questions and call-and-response prompts during songs. Feedback from audiences described the nature of interaction as very close to an actual live show with present performers. All streams back and forth were designed for minimal delay to ensure a natural sense of engagement between Sally and the crowds.

Below are some saved Instagram stories from the audience of Big Sand’s performance at the Powerhouse Museum during Powerhouse Late: Gaming on 28 September 2023.