An area of interest for ExCITe researchers is exploration of novel techniques and applications that allow audiences to better engage with live performances. In 2012, ExCITe researchers hosted the Science of Jazz, a performance with world-renowned jazz musicians Marc Cary and Will Calhoun, where audience members were immersed in the science behind music through the use of large screen visualizations of frequency, harmony, and rhythm and a complementary iPhone app for individual exploration.. Building on the well-received interaction afforded by the iOS application, we continued to explore this idea of using mobile devices to enhance live performances. In 2014, in collaboration with the Philadelphia Orchestra, the Center released LiveNote. This mobile application, available for select performances in Philadelphia’s famed Verizon Hall, listens to the performance and allows users to view musically-relevant highlights for a piece in real-time on their mobile devices. The success of LiveNote, reinforced our understanding of the importance of minimally invasive technologies in performance spaces. We began rethinking our definition of performance and where else a low-distraction technology could enhance the experience.  With the launch of the Learning Innovation Conversations series, we began development of InterestPoint, an iOS application centered on quickly capturing audio and notes during a live presentation.

Once you begin a session in the app, InterestPoint starts listening, using the iPhone’s microphone. When you hear something of interest, you can tap the annotation button and save the previous 10 seconds of audio. Additionally, you can take a quick note, snap a photo, and add a simple indication of whether you agree or disagree with the presenter’s comments. Once you end your session, you can review your annotations and play back the recordings. These feature integrations stemmed from our work with two of our K-12 STEAM outreach programs, CyberSounds, an interactive concert for K-8 students and Remix Interactive, a program where Philadelphia high school students create and present a large-scale multimedia event with the guidance of graduate students from the Center. Both of these activities expanded our ideas of the potential for audience participation during a live performance, particularly ideas around incorporating audience contributed materials to the performance. 

In connection to the Learning Innovation Conversations series, we request permission for the user to share their data. By opting to share this data, only the timestamps from their annotations and the user’s agreement or disagreement with the content of the audio snippet are sent to a server housed at the ExCITe Center and we can use this data to understand what particular parts of the presentation resonated with the audience. 

The initial version of this application was built for our inaugural Learning Innovation Conversation with John Maeda in January 2017. A small group of ExCITe staff tested the app for feasibility and provided feedback for further iterations of the application. Their feedback indicated a need for a more user-friendly interface along with additional annotation capabilities (i.e., ability to type notes and add sentiment information to an annotation). There were also some technical issues with uploading the timestamps to the server that needed to be addressed. Incorporating these suggestions, and additional small group testing throughout the series, led to the current version of the application, which debuted publically at the  Learning Innovation Conversation with Melina Uncapher, January 2018. Although developed initially to support the Center’s event series, through testing both at other Center events and EduCon 2018, we’ve seen its utility well beyond formal recorded events. We hope you try out InterestPoint at the next presentation your attending, and let us know what you think.

Below, you can see an example of the data collected from our latest Learning Innovation Conversation with Melina Uncapher. This figure shows a timeline of the number of annotations collected throughout Dr. Uncapher’s talk. Overall 677 annotations were collected from 35 audience members for an average of 20 annotations per user. The audience seems to be particularly engaged with her presentation around the 17-minute and 52-minute marks.




With our collected data, we can also determine where audience members agreed with the presenter and where they disagreed as well. In the case of Dr. Uncapher’s audience, they mostly seemed to agree with her talk. 

Below are video clips of the four moments which generated the most engagement with InterestPoint during Dr. Uncapher's talk: