Autonomous Virtual Vision for AR Characters
The visual freedom of AR characters is a fascinating topic. Imagine your virtual girlfriend reading a book on the bed by herself, or walking around the room, and all you need to do is pick up your phone camera to see where she is during your leisure time. This is as exciting as Joi in Blade Runner 2049.
Existing physical robots are indeed one of the current hot topics, but robots are too expensive to build, have monotonous forms (e.g., mainstream ones are only humanoid robots, robotic dogs, drones, etc.), and their physical components are very fragile, falling apart with the slightest carelessness. They also have low scalability (because of too many dependencies). In contrast, various complex robots with different forms can be displayed in the AR space. Robot vision can also provide cameras with different resolutions and focal lengths by adjusting parameters. If it’s just about perceiving the space without making substantial impacts, such as chatting with a “ghost” (like an anime character or deceased people) in real space, all you need is a smartphone with AR. This is much cheaper and more universal than physical robots.
However, a major visual problem with AR characters is that they either have no vision and rely entirely on the phone user’s observation, or they can only see what the phone camera sees. Thus, I started researching how to let AR characters see things on their own. My idea is to generate a scene volume and load it into the Unity space where the character model is located, ensuring that the model’s position in the AR space completely matches reality. Then, by mounting a virtual camera on the character’s eyes, the world the character sees will be completely consistent with the world seen by people in reality. At this point, by invoking vision algorithms, the VLM can judge for itself what actions to take in this space.
However, the research did not go very smoothly. Initially, I wanted to use SLAM methods to generate precise scene volumes with 3D Gaussian splatting as primitives, since this is the most straightforward path. But existing precise SLAM-3dgs algorithms universally require CUDA support and devices that can measure depth in real-time, like depth cameras or LiDAR. Since I only have a Mac and an ordinary iPhone, implementing such algorithms is unrealistic. So I hoped to use photos or videos with only RGB information to build the 3D scene volume. I borrowed ideas from PhySIC (maybe a common method in the field), using DepthPro for depth extraction on the original image, then using MoGe to generate a relative scale point cloud for the original image, mapping it back to DepthPro’s depth, and using RANSAC to correct the depth shift. There was also the method of mapping DepthPro’s depth map onto a 3D surface for sculpting, and conducting inpainting reconstruction for objects segmented in the image. I also tried using the diffusion method Rodin for generation. However, the segmentation and inpainting method of DepthPro yielded mediocre results, and when predicting depth for stitching with low inpainting precision, the noise was amplified even more. The method of using DepthPro’s depth map for 3D surface sculpting actually obtained a relatively good result. The 3D surface method compensated for the sparsity of the point cloud, but it brought the problem of having too many unnecessary bridging surfaces between distant and near objects. Even if all bridging surfaces were cleaned up, some poor-quality but actually necessary objects, like walls, could easily be mistakenly deleted. Meanwhile, the diffusion-based Rodin method suffered from severe hallucinations. Once it involved relatively abstract objects, it could only generate scene volumes that looked similar but were actually completely different.
Ultimately, I chose to use Apple’s newly released open-source model SHARP as the generation tool for the scene volume. SHARP can achieve high-definition photo-realistic effects in generating scene volumes. A major advantage of SHARP is that it can run on Mac. SHARP is also modified based on DepthPro, but it enhances its depth estimation accuracy, making the scene structure more realistic. For example, SHARP can reflect the depth of distant buildings out the window, whereas DepthPro only recognizes it as a painting. SHARP uses 3DGS as primitives instead of traditional points or surfaces, which ensures its color authenticity and the fullness of the scene. At the same time, it makes some points at depth discontinuities transparent, getting rid of visual bridging effects (or the hollow effects after cropping noise points). These advantages make virtual vision feasible under existing conditions.
However, the scene volume generated by SHARP is not a standard 3dgs scene volume/model and cannot be directly displayed in the scene in Unity through plugins. Therefore, I specifically attached a splat-transform in the backend to convert SHARP’s scene volume into a standard 3dgs model. Then, it can be easily imported into Unity. With a simple addition of a camera and spatial arrangement work in the AR environment, the job is done!
In fact, there is still a lot of room for improvement in this project. Now that I have a phone that can capture image depth in real-time, I hope to achieve full-house scanning into a 3D scene volume in the future, and then automatically match it into the AR real space through calculations (rather than the current method of generating a scene volume from a single RGB image and then manually adjusting its position to correspond with reality). With my current knowledge, I think scanning into a scene volume using SLAM is relatively easy, while precisely matching it to the real AR environment is rather difficult. This might involve real-time adjustment of depth in real scenes and smooth optimization of other parts during the adjustment process.
