AR C4ISR
Mar 2020 - Jul 2021
In our military-oriented solution, we have explored the integration of augmented reality (AR) technology with C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance) to design a next-generation single-soldier situational awareness and data fusion solution that fully leverages the advantages of both technologies.
Contributions
Solution design, UX and UI design, development
Design&Developing Tools
Sketch, Cinema4D, After Effects, Unity3D, Xmind
situational awareness
Utilizing the positioning device of AR helmets, individual soldiers can report their location, supplies, and vital signs data to the C4ISR system in real time. Meanwhile, through reconnaissance means such as robots, drones, and satellites, the C4ISR system can also obtain the location information of enemy combat units and even acquire on-site imagery.
By fusing data from various sources and projecting it onto the user's field of view through affine transformation, the AR helmet can identify friendly and enemy combat units near the user with different color markings and display information such as unit type (personnel/vehicle) and distance.
Data Fusion
With the help of a military control module, we can display real-time video captured by high-definition cameras, low-light night vision devices, infrared thermal imaging cameras, and electronic gunsight on AR helmets. Users can switch between these modes through buttons, voice, or gesture recognition, selecting the appropriate display mode for different environments:
AR Situational Awareness (Default): The display module of the AR helmet maintains a high degree of transparency, providing minimal yet essential friendly/foe identification and other battlefield data to assist in combat operations.
High-Definition Camera: Used to record or transmit high-definition images of the scene to teammates, facilitating intelligence sharing and collaborative operations.
Low-Light Night Vision Device: Enables rapid perception of the battlefield environment in low-light conditions.
Infrared Thermal Imaging Camera: Quickly detects personnel or vehicles with temperatures higher than the environment in low-light and camouflaged conditions.
Electronic Gunsight: Displays ammunition and ballistic calculation information, allowing users to aim and shoot without exposing their bodies from cover.
In addition to displaying images from the soldier's own equipment, the AR helmet can also receive intelligence information shared by friendly forces from the C4ISR system, including reconnaissance images from drones and robots. Furthermore, to enable more natural and intuitive interaction, we read the gyroscope data from the AR helmet and, after affine transformation, use it as control data to operate the camera gimbal on the drone.
AR Night Vision
In low-light conditions, low-light night vision devices can provide users with a view similar to daylight, but they are not effective at detecting camouflage. While infrared thermal imaging cameras can quickly locate targets with different temperatures, thus revealing camouflage, their display characteristics make it difficult for users to understand the environment. We utilize image segmentation and augmented reality technology to extract images of human bodies and vehicles with higher temperatures from the infrared thermal imaging video stream and then composite them into the video stream of the low-light night vision device, which is finally displayed on AR glasses. This mode helps users to quickly find camouflaged targets with abnormal temperatures while understanding the environment.
