A Look at Pluto’s Technology Stack

Pluto’s mission is to help humanity transcend physical location. In this post, I describe our technology’s architecture, focusing on our multi app framework, our communication service, and PlutoSphere. If you’re a developer, XR enthusiast or simply want to hear a technical perspective on how the industry is evolving, this article is for you.

Video 1. An overview of Pluto and the technologies we’re using to help humanity transcend physical location.

Background

Figure 1. A single application environment (DOS) vs a multi application environment (Windows).

In the early days of computing, applications took up the entire screen (Figure 1, left). Using a different application meant first closing the existing one. Today, we run several applications at the same time (Figure 1, right).

Figure 2. A single XR application environment (left) vs a multi XR application environment (right).

The XR industry is evolving from a single app environment to a multi app environment. Full scene XR applications, such as Rec Room (Figure 2, left), are made to be experienced one at a time. In contrast, multi apps (Figure 2, right), described in further detail here and here, are made to be experienced together.

Multi Apps

I like to think of multi apps as augmented reality (AR) applications. When run together, their impact is greater than the sum of the parts. To demonstrate that impact, the XR community developed a technique to run web-based multi apps in virtual reality (VR).

Figure 3. The tech stack we use to demonstrate the power of multi apps.

We use the tech stack in Figure 3 to render WebXR multi apps in Steam VR. Pluto’s Multi App front end uses XR Packages to run multiple WebXR applications in a single browser tab. Metachromium, an open-source fork of Chromium, renders the contents of the browser tab as an overlay in Steam VR. In effect, from the user’s perspective, it appears multiple AR applications are running in VR.

To learn more about Pluto’s Multi App framework, including how to build and run applications, click here.

Figure 4. An open standards approach to multi app.

In parallel to the web-based multi app development described above, we’re working towards platform agnostic multi apps. For example, popular tools like Unity and Unreal are expected to produce multi apps in the near future, and we’re helping develop open standards, like XR_EXTX_overlay, that make it easier to create and run multi apps using OpenXR — a royalty free open-standard for XR platforms and devices. Ultimately, we expect the resulting stack to look similar to Figure 4, where any application executing on top of an OpenXR runtime can run as a multi app.

Pluto’s Communication Service

Video 2. An overview of Pluto’s communication service running with other multi apps

Pluto’s communication service is a multi app that provides in-person interactions. When run with other multi apps, it enables two or more people to have shared experiences. To demonstrate the importance of shared experiences, we use the following architecture.

Figure 5. The tech stack we use to demonstrate the power of sharing multi apps with other people in XR.

Pluto‘s communication service runs on top of Steam VR and renders as an overlay alongside other multi apps (Figure 5). It receives user input, including head movements, hand movements and microphone audio, and transmits them to other people who can see the user move and hear them speak as if they were in person. In addition to representing people, Pluto maintains a shared coordinate system and uses it to place apps in the proper relative location for each user. The resulting experience is analogous to that which people have when they share physical objects in reality.

Figure 6. An open standards approach to sharing multi apps with other people.

Human representation, human-computer interaction, and common context are fundamental elements of shared experiences. Here are a couple examples of how these elements are being standardized. XR_EXT_hand_tracking, for example, is an OpenXR extension that defines how to track hands (human representation), and how hands interact with applications (human-computer interaction). In addition, XR_MSFT_spatial_anchor is an OpenXR extension that describes a method to maintain an application’s relative position in a coordinate system (common context). Because these extensions, and others like them, are written in OpenXR, we can leverage them to standardize the fundamental elements of shared experiences. We expect such standards to facilitate building and running these experiences in the future.

PlutoSphere

PlutoSphere is a service that helps people get access to powerful spatial computers. High-end PC XR, such as Pluto Communication Service, some multi apps, and graphics intensive games, require significant computer power, especially when run simultaneously. PlutoSphere offers high-end cloud computers that stream XR experiences to devices with less compute power, like the Oculus Quest 2 and HoloLens. This provides people with relatively low powered devices a way to access compute intensive experiences.

Figure 7. The tech stack PlutoSphere uses to make sharing multi apps with other people more accessible.

A PlutoSphere computer comes with all of the dependencies to run multi apps with other people — that is, it comes pre-installed with Steam VR, Metachromium and Pluto’s communication service. In addition, it comes with CloudXR server, which streams data between Steam VR and the PlutoSphere client running in the user’s headset. Because PlutoSphere computers are deployed to the edge (as close to the user as possible), when the user has a high-bandwidth connection they experience little to no perceptible latency.

Figure 8. An open standards, more accessible approach to sharing multi apps with other people.

We believe an open standards approach, such as that depicted in Figure 8, will help more people get access to some of the best experiences. In the figure, applications running on top of an OpenXR runtime are streamed in realtime to an OpenXR compatible client running on the user’s headset.

Wrap UP

The XR industry is evolving rapidly. An important part of that evolution is the transition from a single application environment to a multiple application environment where people can share experiences. We are taking a collaborative, open standards approach to help move the industry forward in that direction. If you are interested in working with us, hit us up!