Holographic lenses, as a kind of high-match augmented reality glasses, have been widely used in industry and even the military. But this time around, Gyroscope actually saw the emergence of a gaming app on HoloLens that lets wearers box up with Pikachu. This is the HoloLens version of Pokemon Go, which was shown by Microsoft in collaboration with Nintendo at the IT event Ignite 2021 last week. This version is different from the previous mobile version. Players can interact directly with pet sprites, and two players can meet in reality to see each other's pet sprites and engage in pet sprites battles. Compared to Pokemon Go, Microsoft's replication of the black technology "holographic transmission" is more striking, Microsoft technology researcher Kipman appeared at the conference through holographic projection.
What is Microsoft Mesh?
Mesh, the new Azure-powered mixed reality platform, was officially introduced by Microsoft at the conference. According to Microsoft, this will simplify the creation of multi-user XR applications. We can understand Mesh as Microsoft's development tool for developers to create multiplayer XR apps.The Mesh platform allows people in different physical locations to collaborate and share in the form of virtual images or even holographic projections through different types of devices, projecting their truest selves to communicate with people in the MR world as if they were interacting with people in the real world.
Microsoft's black-tech "holographic transfer" is available. How good is its MR platform Mesh
Kipman said:
"You can feel like you're sharing content with someone else, or transmitting content through different magnetic **** vibration devices, even if you're not in the same physical location." According to Microsoft, Mesh is the result of years of research and development, covering hand-eye motion tracking and the development of Hololens to the creation of persistent holograms and the creation of personalized virtual images of artificial intelligence models. Cloud computing is Microsoft's core business, and Mesh is underpinned by Azure, thanks to its vast computing resources, data, AI and MR services. As such, Mesh is a further extension of cloud computing.The holographic transfer and high-precision virtual character face and motion tracking supported by Mesh requires cloud computing to help solve many issues such as operations, hardware, rendering and storage.
Kipman said, "We are building more and more value in Azure, the intelligent cloud. In these collaborative experiences, content is stored in the cloud, not in devices or apps, and we need special lenses to see him." With grid-enabled apps, users have access to 3D physical models regardless of geographic location (from small furniture to jet engines, etc.). appear as virtual images of themselves or holographically projected images in *** enjoyable virtual spaces, and collaborate on holographic models.
So why is Microsoft launching Mesh, a multi-user XR application development platform? In its introduction, Microsoft said that MR is the fourth wave of computers after mainframes, personal computers, and smartphones, and is becoming mainstream across the consumer and business sectors.MR frees people from the screen experience, allowing them to interact directly with people or things in their own space. Hundreds of millions of users around the world have reportedly experienced Nintendo's MR through their cell phones, and more than 50% of Fortune 500 organizations counted by Fortune Magazine are using MR solutions such as HoloLens to improve return on investment and grow their business. Given this, the number of apps that can give an MR experience should be significant, but this is not the case, as a number of development conditions limit developers MR experiences, in particular the following four issues:
1. It takes a significant amount of time and resources to present a realistic person in an MR experience;
2. It takes a significant amount of time and resources to ***HEEDING MAGNETIC*** VIBRATION*** across time zones and device types space, keeping the hologram stable is not easy.
3. it is difficult to introduce high-fidelity 3D models to support customer-owned file formats in MR experiences;
4. synchronizing user movements and expressions in geographically dispersed magnet**** vibration conversations is a complex task.
To address these challenges, Microsoft introduced Mesh, which allows developers to design multi-user MR applications without worrying about complex technical challenges.
Mesh functionality demonstration and technical analysis
Microsoft's positioning for Mesh is primarily to create an experience that can be used across screens, interact naturally, and with real views and people. So the platform needed to be designed around Microsoft's existing HoloLens platform, MR headset platforms, and more common platforms. In fact, the Mesh platform defines some of the characteristics of future XR applications in terms of cross-platform support, cloud services, LBA AR support, and more. From a single perspective of each feature, there are corresponding companies and services that face a number of specific user groups that are booming.
1. Communicate with other Microsoft platforms and tools
Mesh is based on Azure and can communicate with other Microsoft platforms and tools. With identity services such as Azure Active Directory and Microsoft Accounts, users can enter secure and trusted sessions with proper authentication and authorization. Microsoft Graphics allows users to access information such as content and contacts in the business or consumer space. And as a developer, you don't need to worry about core infrastructure such as billing, audio and video delivery, and underlying real-time management features. Beyond the core platform, key Microsoft capabilities powered by AI enable Mesh to solve some complex technical challenges by implementing massive multi-user online (MMO) scenarios for MR.
2. Presence
Mesh's reference to presence refers to two main points. The first point is a representative avatar. In a multi-user MR experience, different types of devices use different representative avatars that can be customized by the user.Mesh provides users with representative 3D avatars that are rendered by the device's inside-out sensors.The Mesh platform has its own avatar and customization authoring tool that allows avatars to be used directly. Additionally, the platform can help existing avatars capture movements and expressions that are consistent with the user's actions through its AI-powered motion model.
The second point is to enable true 360-degree capture with an outward-facing sensor that allows customization of camera settings, such as saving captured 3D images in high fidelity with Mixed Reality Capture Studio or capturing depth-sensitive app images with Azure Kinect to help generate holograms. Once the hologram is generated, applications developed on the grid can be enabled on an XR head display or on a phone, PC or tablet to use the hologram and present it to the user.
3. Spatial Maps
Before the advent of Mesh, various devices had only their own local maps of the world, and with the help of Mesh, these local cached maps can be merged and optimized to form a global understanding of space and environment. This framework anchors content, shares vision, and collaborates in 3D.Mesh creates maps that are more accurate than GPS, even in spaces where GPS cannot locate. It helps deliver a "world-locked hologram" associated with a particular point of interest. In addition, mesh generates the same understanding of the precise layout and geometric alignment of a given object so that developers can easily build application objects that may need to be overlaid with visual information that exactly matches the object with visual information such as commands, service records, and other important data.
4. Holographic Rendering
Holographic rendering is a prime example of smart edge cloud computing and smart cloud architecture. With the support of Mesh, holographic rendering can be realized with high fidelity, without the need for device operation and power consumption. Mesh allows developers to choose between local standalone rendering and remote rendering for each scene and model via cloud connectivity, giving developers greater flexibility in designing applications that optimize latency and fidelity for different devices. And holographic rendering also supports most 3D file formats for local rendering in mesh-enabled applications, solving the problem of bringing in a user's existing 3D model for collaboration.
5. Multi-User Synchronization
Creating a ***same hologram view and communicating with each other in a virtual collaboration meeting is a big challenge for developers. In the grid, *** enjoying the content is achieved through multi-user synchronization, which will help to update the participants with changes in pose, movement, expression or spatial hologram. All of this happens within a delay of 100 milliseconds, either in the same physical space or in different time zones.Spatial audio in the Mesh enhances these features, thus creating the feeling of being in the same physical space in a multi-user scenario.
To take advantage of these features, Mesh will provide developers with a cross-platform SDK that will allow developers to create applications for different device types, according to Microsoft. Currently Mesh supports Unity along with C++ and C#, and in the coming months Mesh will support Unreal, Babylon, and React Native.In addition to accessing the features, the SDK provides a pre-built UX structure for developers to use in their apps. This will allow developers to deliver a magnetic **** vibration experience faster and more concisely. With the advent of the Mesh platform, this new set of features comes together as a characteristic presentation of XR applications. This convergence turns piecemeal technology directions into a whole, making the concept of XR concrete step by step, which is its greatest contribution.
The value of Azure Smart Cloud behind Mesh
XR apps developed based on Mesh can be used in a wide range of domains and contexts, thanks in large part to the value brought by the Azure Smart Cloud.Mesh's virtual collaboration capabilities enable users across time zones to digitally come together to discuss and understand them together, which helps to view faster data and make better decisions. Users can train together at any time. Thanks to Holoportation (a 3D video capture system developed by Microsoft), holographic ****enjoyment, and visualization capabilities, partners can learn face-to-face without having to spend time and effort on the road.
For example, in telemedicine, specialists can see other doctors' diagnoses and overlay data to ensure *** knowledge, better problem solving and *** same decision making. Designers and engineers can work together on 3D designs, and team members can share thoughts and ideas using different devices as they participate. Colleagues can collaborate on content in real time, either in person or via holographic transmission.The multi-application scenarios of Mesh and the nature of azure, a cloud-based computing platform, means the expansion of Microsoft Azure industry and MR business.
Alex Kipman, a Microsoft technology researcher, said:
"From the beginning, this has been the dream of mixed reality." You can actually feel like you're in one place with the person who's sharing the content, or you can transmit it from different mixed reality devices and be with the person even if you're not physically together."
He argues that as more and more people are active on the grid platform, all of us are involved in building the value of the Azure Intelligent Cloud. In these collaborative experiences, the content is not on the current user's device or application. Holographic content uploaded to the cloud means that the mixed reality device worn by the user is only responsible for the immersive presentation of visuals and interactions. For individuals with limited computing power, this is a leap forward." Most importantly, as XR applications evolve and mature on the Meah platform, more never-before-seen security risks and cybersecurity incidents are at the center of attention. Compared to previous experiences, the new AltspaceVR experience available on the Azure-based SmartCloud Meah platform allows companies to hold meetings and work parties in virtual reality using enterprise-grade security features, including secure login, session management and privacy compliance.
The Azure Intelligent Cloud provides Meah with the prerequisites to host a large number of modeled scenarios and characters, which is what sets the platform apart from a traditional app. As you can understand, in the XR environment, shuttling between apps is more like traveling between cities, during which there will be no download process or hardware computing power adaptation. In fact, since the end of 2020, all major manufacturers have begun to fully implement their cloud-based platform service strategies. Among them, Epic's digital human platform and HTC's VR social platform have embraced cloud technology. The power and resources of cloud computing will be distributed to every average consumer for a safer and more cost-effective XR experience, which is the dawn of the popularization of the XR industry.
Microsoft has spared no effort in the XR layout, just as technical director Alex Kipman suddenly "flashed" from the "depths of the sea" and then began with a poetic and romantic reading --One person's dream is just a dream, a dream where everyone ****s the same dream is called reality. Microsoft, which missed the opportunity of the mobile Internet, is combining the advantages of cloud computing in the field of MR to bring everyone closer to the future. Mesh is by no means the icing on the cake. If this advantage is maintained, Microsoft, which has the underlying XR development system and cloud computing support, will define some of the standards for the future of XR.