How to Render Point Cloud with LIPSedge AE400 3D Camera

If you're working with 3D cameras, you know how important point clouds are. They allow you to create 3D models of real-world objects and environments. In this article, we'll explore how to use LIPSedge AE400 to populate the point cloud.

Using AE400 SDK, OpenGL, and GLFW, we'll show you how to render the point cloud's vertices and image texture. We'll also provide helpful functions for OpenGL and GLFW. You can find more information about these tools on their respective websites (OpenGL, GLFW).

Code

* Download full code example from github

1. Let's create an empty project folder. Then create CMakeLists.txt with content below

cmake_minimum_required(VERSION 3.8)

project(ae400-pointcloud CXX)

add_executable( ${PROJECT_NAME} pointcloud.cpp)

set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 11)

find_package( OpenGL REQUIRED )
target_link_libraries( ${PROJECT_NAME} ${OPENGL_LIBRARIES} glfw realsense2)

2. Create an empty file pointcloud.cpp for all the code we need to write. Then Create file example.hpp, copy and paste all the helper script from here.

3. In the main function in pointcloud.cpp. First, we need to create the window and setup prepare a glfw_state variable for tracking the app states. Also, register a callback function for tracking the mouse movement to controller pointcloud's view projection.

window app(1280, 720, "Pointcloud Example");
glfw_state app_state;
register_glfw_callbacks(app, app_state);

4. Declare a video pipeline and start the camera. Also, declare pointcloud and points variables to store the pointcloud info.

rs2::pipeline pipe;
pipe.start();
rs2::pointcloud pc;
rs2::points points;

5. While the video is streaming, get the color and depth frame from AE400.

while (app) 
{
    auto frames = pipe.wait_for_frames();
    auto color = frames.get_color_frame();
    auto depth = frames.get_depth_frame();
 }

6. On each frame captured, we first make pointcloud to map the color frame. Then calculate the depth frame and texture. To see what texture should be used at each vertices in depth frame. Pointcloud class has map and calculate function for that.

while (app)
{
   ...

   pc.map_to(color);
   points = pc.calculate(depth);
 }

7. With the helper script, we update the texture for openGL to use.

while (app)
{
   ...

   app_state.tex.upload(color);
}

8. Finally we use the vertices and texture to draw the pointcloud.

while (app)
{
   ...

   draw_pointcloud(app.width(), app.height(), app_state, points);
}

Let's take a closer look at draw_pointcloud function. After setting some openGL draw attributes, use glVertex3fv() and glTexCoord2fv() to actually draw the points.

Compile and Run

cd pointcloud 
mkdir build 
cd build 
cmake .. 
make 
./ae400-pointcloud

Here's the pointcloud viewer. You can use mouse to control the rotation or zoom in/out. Zoom in for a closer view you can see all these tiny points in it.

Conclusion

In this article, we've shown you how to render point clouds using LIPSedge AE400, OpenGL, and GLFW. By using the LIPSedge AE400 SDK, we were able to convert 3D frames into point cloud vertices. We also provided helpful functions for OpenGL and GLFW to make the rendering process easier. If you have any questions or need more information, you can reference the full example in the LIPS Help Center or post to the LIPS Forum.