In the previous articles, the JET BI project, Structure Sensor SDK, and the problem of identifying the scanned object were described. Also, we considered the calculating algorithm of the found 3D model position for displaying in the scanned object coordinates. Today we perform the process of developing the JET BI FieldXR project.
1 Pic. LaunchScreen.storyboard for App
Introduction
Based on the code of the project Scanner.xcodeproj sample, let's create a project that meets the following requirements:
- application for iOS (iPad);
- getting 3D object data in real-time.
Challenges and solution
Let's review each requirement by itself.
1. iOS App for iPad
You can find how to connect the SDK Structure Sensor to an empty project in the article Setting up structure-sdk. I chose not to reinvent the wheel but modifying the Scanner.xcodeproj, where the SDK is already connected, the necessary project settings are made.
2. Getting 3D object’s data in real-time
Let’s modify the Scanner.xcodeproj for getting the necessary data in real-time.
The idea is that during the scanning process, the collected object’s 3D data, with the frequency selected by the user, is written to an OBJ file and sent to the server. Also, it will be useful to save the image taken on the iPad color camera during the scanning and send the final obtained JPEG file to the server in the same way.
2.1. Scan mode on/mode off button
The project already has a Scan button that starts the process of collecting 3D data by clicking on it. Let’s leave it unchanged. As for turning off the scan mode, two buttons are responsible for it: Done and Reset. The Done button is adapted to the function of launching the scanned data viewing window. There is no need for it in the FieldXR app, so we disable the Done button. Now only the Reset button will be responsible for disabling the scanning process (the Reset button resets the values of the current session).
2 Pic. The Scan turning on the button. You can start scanning now
3 Pic.The scanning process. To finish, press the Reset button
2.2. Saving 3D data to an OBJ file
The geometry of the scanned object is stored in STMesh. The task is to take the STMesh content and save it to an OBJ file with a certain frequency. The timer is implemented for these purposes. When a timer event occurs, the following code is executed:
4 Pic. Representation of saved data in the file system
5 Pic. Fragment.obj
2.3. Saving a 2D image from iPad as a JPEG file
The algorithm of saving a 2D image from the iPad camera consists of several methods. When the timer is triggered, you need to look into the buffer containing the image from the camera, convert the pixel buffer from YUV format to BRGA, and then create a JPEG file from the received data.
To begin with, while the scanning process, the color camera is doing the capture session. A function that returns the results of the capture: 
Data is obtained from the iPad color camera:
Creating a function converting the ColorFrame buffer data to UIImage format:
And then save the UIImage data in JPEG:
As a result, we get:
6 Pic. Fragment.JPEG
2.4. Sending ZIP and JPEG files to the server
The code for sending ZIP=YES, JPEG=NO files to the server is below:
2.5. Configuring the interface and adding features to select the saving and uploading session data periods
The implementation of this functionality consists of creating a Stepper and the related Label. The process will be displayed in the application window with the Scan button at the same time. The value in Label is measured in seconds, varies from 1 to 100, and resets to 1 when returning to the scan preparation window. Clicking on +/- increases and reduces the value, respectively. The data from the form is passed by the parameter to the timer that triggers the events 2.2-2.4.
2.6. Configuring the bounding cube
The Scanner.xcodeproj allows the user to control the scan area: increase or decrease the cube displayed on the screen before the scan starts.To minimize user interaction, you can make the bounding cube’s settings constant, and extend their borders to the application window. The Structure Sensor SDK documentation has some tips for this:
“kSTMapperVolumeBoundsKey:
The extents of the bounding volume in the number of voxels along each dimension.
NSArray of 3 NSNumber integral values.
Defaults to @[ @(128), @(128), @(128) ].
To maintain proper performance, we recommend keeping the borders under 290x290x290”.
Result
As a result of modifying the Scanner.xcodeproj code we have created a project that meets the following requirements:
- application for iOS (iPad);
- getting 3D object data in real-time.
Also, we added features configuring the data upload period and removing the bounding cube.
Collecting the 3D data about everyday objects in real-time by using the Structure Sensor SDK is not a difficult task. Project examples and developer forums greatly facilitated the process of investigation and development. As part of the current task, the revision of the existing Scanner.xcodeproj has reduced development time, unlike creating a project from scratch. Also, open-source projects like Viewer.xcodeproj and RoomCapture.xcodeproj helped with finding the solutions to narrower problems.
However, the issue of setting parameters constant values of the cube limiting the scanning area, the minimum value of voxels per inch of the screen while scanning remain under attention. The solution to this problem lies in using the approach described in the previous articles.
Also, the solution doesn’t include the 3D object materials obtained during the scanning process. The estimate of their necessity and cost will deal at the next stage of development.
If you are interested in the JET BI FieldXR project, get in touch and our experts will find the proper solution for your business.
References
How to start working with Structure Sensor. FieldXR project part 1
How to work with the identified 3D objects. FieldXR project part 2
About author
Volha Dzeranchuk - developer C++. Education: Bachelor degree (year of graduation: 2012), BSU, Faculty of Mechanics, and Mathematics. Experience in the IT field for more than 8 years. Development of GUI applications for Windows, creation of DLLs, project support (C++, UE4, SQL/QT, Visual Studio/VR). More than 3 years of experience in teaching IT disciplines.
