ARISE
Dec 2015 - Jan 2024
ARISE is an Augmented Reality Platform designed specifically for enterprise users. It completes with features including: Remote Collaboration, Digital SOP (Standard Operating Procedure), AI Recognition and Analysis, Digital Twin, etc.
Contributions
Project management, product definition, market analysis, user research, UX and UI design, system architecture design, marketing
Design&Developing Tools
Figma, Sketch, Unity3D, Spline, Adobe After Effects, Xmind
Enterprise AR Solution
Recognized By Industry Leaders
How One Airline Is Using AR to Improve Operations
This article describes an experiment under way at China Southern Airlines that is weaving augmented reality, artificial intelligence, cloud computing, the Internet of Things, and 5G connectivity into a single network that is being used to extend human capacities, enhance safety, and improve performance.
During our in-depth research on the front lines of various enterprises, we identified some common needs across different companies:
Ensuring that work processes are traceable and can be reviewed, thereby achieving better quality management.
Quickly organizing multi-person, remote collaborative work to carry out specific task command and dispatch or troubleshooting.
Enhancing the users' work capabilities and efficiency.
Accumulating work experience as high-value assets for the enterprise.
After analysis, we believe that these needs not only represent a considerable market scale but also can be translated into products, forming an iterative plan that aligns with the company's technology roadmap.
Market And User Research
Since 2015, we have been researching the applications of augmented reality technology across various industries. We have visited hundreds of clients in over a dozen sectors, including oil, power grids, logistics, manufacturing, nuclear power, and construction. Through this process, we have established our company's development philosophy of focusing on AR enterprise services and gradually defined the "ARISE" platform.
Define The Product
Based on our research across various industries, we defined the following functional modules for ARISE:
Remote Collaboration
Supports audio and video calls, along with features like whiteboards, remote screen sharing, annotation, instant messaging, and document sharing during calls.
AR Workspace
Supports importing 3D models and media files, allowing users to create and edit 3D scenes that can be displayed on AR glasses through drag-and-drop functionality, and controlling animation logic via scripting.
Operation Guides
Allows synchronization of work order data from the enterprise ERP to the AR glasses, displaying step-by-step instructions in card format to guide workers through complex tasks. Supports the collection of camera and sensor data related to current operations, syncing it with the AR glasses and backend. Enables the generation of detailed operation reports, accurately binding field images, operation time, location, and work order data.
Knowledge Graph
Aggregates business data collected from all the above functions, extracting content from videos, images, audio, and spatiotemporal information through cross-modal retrieval algorithms. It generates highly valuable operation records for creating enterprise knowledge base materials for training purposes.
Intelligent Recognition
Trains machine vision recognition algorithms for asset detection based on datasets provided by enterprise users. This can be used to check the status of equipment or to trigger the display of relevant operational references at specific locations.
Digital Twin Platform
Visualizes the current operational status, overall personnel efficiency analysis, and other platform operational data on the backend screen, enabling managers to quickly understand frontline situations. Also supports data permission management within this backend.
Product Management
In the early days of the XR industry, LLVISION, like many other XR startups, felt deeply that the industry chain was incomplete. Therefore, we decided to develop ARISE, bringing AR glasses and high-quality enterprise services to users. After solving the problem of whether there is software and hardware, we began to focus on optimizing product development and services. The main measures include:
Continuous Optimization: We continuously reviewed and evaluated each key task from requirement gathering and product design to final operation, ensuring continuous optimization of pre-sales/after-sales response speed, improving product quality, and not missing any potential business opportunities.
Agile Development: Balancing customer satisfaction and development/testing efficiency, we adopted an 8-10 week sprint cycle. This ensured that product, design, development, and testing teams could quickly iterate to the next version after completing the current version within 1-2 weeks, while also allocating some resources to support other departments on the previous version. Through this model, we ultimately achieved the release of a new software version every 2-4 weeks (depending on complexity), effectively supporting the delivery and rapid iteration of our sales and business teams.
Pilot Customers: To effectively increase the ROI at the commercial level, we annually select 8-10 "Pilot Customers" with large order volumes, good replicability, and industry influence. We focus on iterating our products based on their specific needs.
User-Centric Feedback Loop: By quickly collecting users' opinions and needs through online surveys, regular telephone interviews, and on-site research.
Compelling Demos for Increased Sales. Create high quality demos that sales and marketing teams can easily execute, designed to wow users and showcase the product's value in real-world scenarios. This will drive higher sales conversion rates.
Prototyping
From 2015 to 2023, ARISE iterated 5 major version to meet the ever-changing market demands. When designing ARISE, we faced these primary challenges:
Ease of use: The product needed to be highly user-friendly across different industries and use cases.
Compatibility: The product had to be compatible with major enterprise ERP/MES systems and be able to integrate with new technology modules to create more business value.
Architecture: The software architecture required security, reliability, efficiency, and minimal costs for customization and version updates.
Based on our analysis of user research and industry-standard enterprise software, we proposed the following solutions:
User Interface: The mobile interface and user experience were designed to be as similar as possible to mainstream ERP apps, while the AR glasses provided views adapted to different use cases.
Modular Development: Each functional module was developed and maintained independently using mobile mini-programs and backend microservices. Cross-calling between different modules was supported as long as user experience and technical feasibility allowed.
Remote Collaboration
The early remote collaboration module was based on video calls. As early as 2015, we identified the unique value of AR glasses' "first-person perspective + hands-free operation" in scenarios such as telemedicine, media broadcasting, and factory video conferencing. Therefore, we iterated on our initial RTMP/RTSP based video collaboration foundation and introduced a WebRTC-based remote audio and video collaboration solution.
ARISE Remote Collaboration leverages WebRTC to capture on-site audio and video data using AR glasses. The specific integration and data distribution may vary depending on the enterprise system.
In addition to leveraging WebRTC's advantages to minimize video latency and accurately defining video resolution, bitrate, and frame rate for different scenarios, we also added several auxiliary functions to enhance efficient communication in AR remote collaboration.
Smart Tagging
Allows other users to highlight items in the real-time video captured by the AR glasses, with the bounding box continuously tracking the target.
Whiteboard
Provides a shared whiteboard for all users participating in the collaborative meeting, supporting drawing and annotations.
Screen Sharing
Allows participants to share the content of their computer screens during the meeting.
Instant Messaging
Enables participants to send text messages, images, or documents into the meeting.
From ERP To AR sOP
Typically, enterprises have already set up work orders, organizational structures, and permissions within their own ERP systems. However, work order interfaces designed for smartphones and tablets are not suitable for direct display on AR glasses - especially given the smaller screen size and the emphasis on lightweight design of AR glasses. Moreover, there is a significant variation in the data structures of work orders across different industries; the number of information levels and the key content that needs to be highlighted can vary greatly.
Based on these characteristics, we redesigned the work order and task interfaces for AR glasses and their accompanying smart mobile applications.
The diagram below illustrates the architecture of a case study where LLVISION has integrated ARISE with the Field Service Management module of SAP ERP. By successfully connecting all data interfaces, we have ensured both the efficiency of information display on AR glasses and the completeness of data collected by the backend system.
Based on our research findings across various industries, we believe that AR glasses should primarily design work orders around the following three field service scenarios. Enterprises and users can modify the default view to suit their needs by long-pressing the function button on the AR glasses.
Instruction
This scenario is typically used to guide users through complex tasks. By breaking down work orders from the ERP system into individual steps and displaying them as cards in front of the user, combined with text-to-speech for reading out work order content, highlighting key information, and the ability to toggle attachments on and off at any time, users can complete advanced tasks with minimal training. This is the default mode.
Asset
In this scenario, users usually need to perform inspections, maintenance, or training around a fixed asset. Compared to the task content, users need to focus more on their relative position to the work points. AR glasses will highlight the user’s completed work path and the current and upcoming work points, and use text-to-speech to play the content of the current step.
Capture
In this scenario, users are familiar with the task content and need minimal or no reference information. However, users need to ensure that the captured footage is accurate and that the recorded data belongs to the correct steps. In this mode, AR glasses will display a recording preview and task time by default and use text-to-speech to play the content of the current step.
IoT Devices Integration
Although the AR glasses' camera adds valuable continuous data collection and first-person video capabilities to traditional field service, its limitations are still evident: AR glasses cannot capture additional critical IoT data. After researching some pilot customers, we decided to enhance the AR glasses with the ability to integrate other camera video streams and sensor data via Bluetooth, USB hubs, and other peripherals. This additional data will be bound to the current task steps and stored accordingly.
Paint Thickness Gauge
Youxin uses ARISE in second-hand car transactions to check the consistency of car paint thickness. When the inspector places the paint thickness gauge on the car's surface, the device's readings are synchronized with the display on the AR glasses worn by the user and recorded under the current work order step.
Industrial Endoscope
China Southern Airline's maintenance team uses an industrial endoscope to inspect jet engine fan blades. The endoscope's images can be displayed in real-time on the AR glasses screen, the smart mobile app, and the backend PC client screen.
Infrared thermometry
With the help of an infrared thermometry module attachment and its alignment calibration with the AR glasses camera, AR glasses can utilize facial or object detection and tracking algorithms to identify the temperature of a target.
AI Recognition
As AI technology advanced, by 2020, we began integrating AI-based recognition capabilities into ARISE. This brought value in two main aspects:
Automatically opening related work orders or manuals upon recognizing assets. Previously, users had to manually open these contents.
Enhancing the manual inspection of asset conditions (such as verifying asset placement, correct installation of parts, or panel faults) with an AI double-check.
Using few-shot learning technology, we typically only need a 5-way 5-shot configuration to achieve over 94% target detection accuracy. With further optimization, the few-shot learning algorithm based on ResNet18 can achieve over 80% accuracy on the Intel® Movidius™ Myriad™ X chip, eliminating the need for server computational power in most industrial scenarios.
AR Workspace
Sure, here is the translation of the Chinese text into English: With the maturity of large field-of-view, high-resolution waveguide display technology, customers have begun to try to develop augmented reality content that integrates with three-dimensional space on our hardware platform. However, due to the lack of relevant design and development experience and the relatively long development cycle of Unity3D and WebXR, most customers cannot quickly build 3D AR scenes that are suitable for themselves.T
o address this pain point, we have added the AR Workspace function to the ARISE platform. By dragging and dropping 3D files (in .glb or .gltf format), images, videos, and audio into a workspace with a three-axis coordinate system, users can complete the construction of 3D scenes. In addition, users can also set the properties of each component in the sidebar and write scripts to create corresponding animation logic.
To help novice users quickly convert existing 2D data assets into 3D AR assets, we have also added several commonly used templates. These templates provide a starting point for users who are new to 3D AR development. They can be used to create a variety of common AR experiences, such as product visualizations, virtual tours, and educational games.
2D editor
By adding text, images, videos, or 3D models to the content area of the canvas, AR Workspace displays these contents according to preset dimensions and positions, preventing issues such as inappropriate appearance, obstructing the user's view, or being difficult to observe. This editor is suitable for beginners.
3D editor
Drag and drop text, images, videos, or 3D models into the 3D scene editor, and users can rotate or scale the scene and the file using the mouse. While building the scene, users can also edit the interactive scripts of the model in the right-hand side bar.
Knowledge Graph
AR glasses generate a large amount of field service data through Remote Collaboration and SOP, and we quickly realized the value of this data. For example:
The work records generated by experienced senior employees when using AR glasses are often of great teaching value and can be summarized into many how-tos for other employees to learn.
AR glasses can accurately bind personnel, work orders, time, location, assets, and work record data. By analyzing work reports, they can help enterprise managers better understand the overall labor efficiency of the enterprise, improve work process design, and handle emergencies.
Therefore, in addition to the PC-side functions, organizational structure, and data permission management of the above modules, we have also added knowledge base and data visualization functions to the ARISE management platform.
The content of the knowledge base comes from cross-modal retrieval of field service data on the platform. For example, when a user searches for "jet engine maintenance" in the knowledge base, all relevant video clips, images, and text records will be displayed in the search results. The content with higher matching scores, views, and star ratings will be displayed at the top.
Digital Twin
When using AR glasses with 6DoF spatial interaction capabilities for inspections or training, the sparse point cloud generated by the AR glasses' SLAM capability allows the work instruction interface to be anchored around the asset being worked on, making the task at the work point more intuitive. Additionally, the user's head pose and movement trajectory are recorded in the backend, providing the necessary content for generating work reports and extracting knowledge graphs.
For enterprise managers, it is very important to be able to quickly understand the field service situation. Through visualization technology, ARISE can simulate and display the current location of field workers, the work orders they are executing, and the on-site images. The platform can also count the overall work progress and labor efficiency.