Deconstructing a live AR Setup

Deconstructing a live AR Setup: A Technical Breakdown Using a Backstage Photo

Hey there, it's nice to have you here with me for a behind-the-scenes tour through one of our smallest live AR show setups. I’ll give you a Birds Eye view on all the interesting tech and details that create an immersive show. So, let's take a tour through the world of running an augmented reality live show.

Content Area: Data Processing and Real-Time Rendering

The content area is all about creating and rendering the different scenes of the show. It makes sure data and network communication are managed efficiently and allows for creating and finalizing 3D scenes last minute.

Playout servers: Processing and Rendering Graphics

This is where the magic happens. Two machines are only for handling camera streams and adding the AR stuff. The other two are for the backdrop LED walls where most of the motion graphics happen. All of them get the camera tracking signal, so views can change depending on camera angle and position not only in the AR camera. Think about the LED wall as a window into another dimension. Stuff can go into it and come out of it onto the stage. (I'll do a separate post on this, I swear!)

Network Switches: Control, Clustering, and Content Delivery

All the real-time machines have two completely separated networks. One for cluster communication: the cluster sync hasn't got that high bandwidth, but it needs to be super precise in terms of timing. That's why a dedicated 1 Gigabit network hub handles nothing but cluster data. The other one (orange cables) is a 10 Gigabit for content delivery and show commands. Loads of data, sometimes 100s of GB of data need to get served in seconds when scenes are changed. Btw: our show command system is based on OSC, which is super lightweight, and our proprietary protocol structure can handle peak network usage, that’s why we don’t need a third network 🧐

Laptop: Editing 3D Scenes Onsite

Our graphics and real-time design department has its own machines in the show setup. We design 70% of the show off site. 30% of the show is produced on site. As we often get the latest data or 3D models only hours before the show, the content team adds animations and graphics through remote workspaces with the latest available data literally last minute.

Data Server: Secure Storage and Delivery of Scenes

This server has all the scenes, 3D models, and data securely stored in a protected storage. We serve scenes and 3D data from this server “directly” to GPU and CPU RAM of the playout machines without storing local data. Handling prototypes, engineering data, and other confidential stuff requires this extra level of security. And it’s no joke! (I’ll do a post on TISAX soon 😉)

Control Area: Real-Time System Monitoring

So, in the control area, we're checking all the real-time servers to make sure everything's running fine. We're also keeping an eye on clustering and the network to ensure proper communication and data flow. Plus, we're verifying that content delivery is on point. And, of course, we're monitoring the overall performance, because who runs a show without maxing out four RTX A6000, right?

Large Screen: Displaying Outputs for Monitoring and Redundancy

The large screen in the control (and command) area is used to display all the real-time rendering outputs. The top row presents the main outputs, and the bottom row shows the backup outputs. In this particular example, the left side is the camera feed with the AR elements on top, while the right side has the LED backdrop. (I'll explain in a separate post about TISAX and why I have to blur all the outputs)

Test Scene: Cluster Integrity and Performance Check

I could keep the left backup screen unblurred, it shows our test scene. (Btw: 90s kids will see similarities to the Windows XP Bezier screensaver) It helps with a quick check of cluster integrity. If the cluster is set up correctly, the bezier will behave absolutely the same across all screens even after long runtimes. The scene is also fully packed with effects and 3D stuff to check for GPU and CPU performance issues.

Laptop: Monitoring Software for System Insights

For a more detailed overview of the whole setup, we’ve got a monitoring system running. It shows network capacity, detailed cluster data (who’s the master, quality of sync, frame number), camera tracking data (view matrix, delay, frame) SDI input card temperature, GPU performance,...

Command Area: Steering the AR Show

In the Command area, we focus on steering the AR show. Multiple scenes with animations and graphics are combined with color correction and lens calibration here. We ensure technical components work together and make adjustments for a consistent viewing experience.

Main Show Software: Preselecting, Activating, and Unloading Scenes

The (blurred 😉 ) white screen is where the show happens. Each speaker's scene on the data server is available at a click of a button. Scenes can be pre-loaded, activated, and unloaded depending on who's next. It also shows each cue available in a scene. Cues are topics, or details speakers talk about. And, here is where these cues are called for virtual cars, animations, bar graphs, and all the visual eye candy you see in our shows.

Base Scene: Managing the AR Show

This is like the brain of the show. It runs on all real-time machines as the base scene. It has a scene loading dock where the actual scene gets loaded into. Stuff that stays the same and is not confidential is handled in here. It has layers for color correction, tracked cameras, lens calibration data, and our main communication stack. Btw: We often handle 20 to 30 different scenes on a show.

And, what about all the stuff around us?

Of course we’re not the only ones on such shows 😁There’s a lot more tech involved in getting the show out to the world. Camera operators, signal handling and distribution, the broadcast team, audio… but that’s up for another post 🙏

I hope you enjoyed this Birds Eye view at our tech playground. It's been a blast showing you the ins and outs of how we bring our shows to life.

Thanks for Joining me on this Backstage Tour!

I’m spending my time in all areas, but mostly in control and command. Where do you see yourself? More into control, content or command? Or are you in a totally different area? Let me know, I’m curious!

And feel free to reach out if you have any questions!

Stay tuned for more posts about our AR and broadcast endeavors!

Sebastian