Introduction to ray trace unit and 3dgs algorithm

Way to 3D Gaussian Splatting

References

3D Representation Methods: A Survey

3D Representation

volume-based(voxel grid):

• extend from 2d pixels to 3d cubes space partition (VoxNet: 3D CNN for object recognition)

point-based(point cloud):

• storage format (x, y, z) (r, g, b, d), do not require connectivity information and volumetric data
• sparse, unstructured and contain noise
• efficiency optimization on GPU(TorchSparse++)

mesh: skip

NeRF:

• data representation as ground truth to train MLP (volume based or point based)
• view synthesis (base on pinhole imaging principle, need a camera position, each pixel in the synthesized image is generated on the sampling ray)
• sample data points along the ray, sampled data points are encoded to be fed to the MLP to generate the radience and density
• efficiency optimizatiion on GPU(Instant-NGP)

3D Gaussian Splatting

Spherical harmonic function

Ray Trace Unit on GPU