Skip to main content

Augmented Reality for Business. FieldXR Project part 3

About Us
Published by jetbi
29 March 2021
27

In the previous articles, the JET BI project, Structure Sensor SDK, and the problem of identifying the scanned object were described. Also, we considered the calculating algorithm of the found 3D model position for displaying in the scanned object coordinates. Today we perform the process of developing the JET BI FieldXR project.

3d-3-article-1

1 Pic. LaunchScreen.storyboard for App

 

Introduction

Based on the code of the project Scanner.xcodeproj sample, let's create a project that meets the following requirements:

- application for iOS (iPad);

- getting 3D object data in real-time.

 

Challenges and solution

Let's review each requirement by itself.

1. iOS App for iPad 

You can find how to connect the SDK Structure Sensor to an empty project in the article Setting up structure-sdk. I chose not to reinvent the wheel but modifying the Scanner.xcodeproj, where the SDK is already connected, the necessary project settings are made.

 

2. Getting 3D object’s data in real-time 

Let’s modify the Scanner.xcodeproj for getting the necessary data in real-time.

The idea is that during the scanning process, the collected object’s 3D data, with the frequency selected by the user, is written to an OBJ file and sent to the server. Also, it will be useful to save the image taken on the iPad color camera during the scanning and send the final obtained JPEG file to the server in the same way.

 

2.1. Scan mode on/mode off button

The project already has a Scan button that starts the process of collecting 3D data by clicking on it. Let’s leave it unchanged. As for turning off the scan mode, two buttons are responsible for it: Done and Reset. The Done button is adapted to the function of launching the scanned data viewing window. There is no need for it in the FieldXR app, so we disable the Done button. Now only the Reset button will be responsible for disabling the scanning process (the Reset button resets the values of the current session).

3d-3-article-2-0

2 Pic. The Scan turning on the button. You can start scanning now

 

3d-3-article-3-0

3 Pic.The scanning process. To finish, press the Reset button

 

2.2. Saving 3D data to an OBJ file

The geometry of the scanned object is stored in STMesh. The task is to take the STMesh content and save it to an OBJ file with a certain frequency. The timer is implemented for these purposes. When a timer event occurs, the following code is executed:

3d-3-article-1-code

 

3d-3-article-5-1

4 Pic. Representation of saved data in the file system

 

3d-3-article-6

5 Pic. Fragment.obj

 

2.3. Saving a 2D image from iPad as a JPEG file

The algorithm of saving a 2D image from the iPad camera consists of several methods. When the timer is triggered, you need to look into the buffer containing the image from the camera, convert the pixel buffer from YUV format to BRGA, and then create a JPEG file from the received data. 

To begin with, while the scanning process, the color camera is doing the capture session. A function that returns the results of the capture: 3d-3-article-7

Data is obtained from the iPad color camera:

3d-3-article-8

Creating a function converting the ColorFrame buffer data to UIImage format:

3d-3-article-3-code3d-3-article-4-code

And then save the UIImage data in JPEG:

3d-3-article-11

 

As a result, we get:

3d-3-article-12

6 Pic. Fragment.JPEG

 

2.4. Sending ZIP and JPEG files to the server

The code for sending ZIP=YES, JPEG=NO files to the server is below:

3d-3-article-5-code

 

2.5. Configuring the interface and adding features to select the saving and uploading session data periods

The implementation of this functionality consists of creating a Stepper and the related Label. The process will be displayed in the application window with the Scan button at the same time. The value in Label is measured in seconds, varies from 1 to 100, and resets to 1 when returning to the scan preparation window. Clicking on +/- increases and reduces the value, respectively. The data from the form is passed by the parameter to the timer that triggers the events 2.2-2.4.

 

2.6. Configuring the bounding cube

The Scanner.xcodeproj allows the user to control the scan area: increase or decrease the cube displayed on the screen before the scan starts.To minimize user interaction, you can make the bounding cube’s settings constant, and extend their borders to the application window. The Structure Sensor SDK documentation has some tips for this:
“kSTMapperVolumeBoundsKey:

The extents of the bounding volume in the number of voxels along each dimension.

NSArray of 3 NSNumber integral values.

Defaults to @[ @(128), @(128), @(128) ].

To maintain proper performance, we recommend keeping the borders under 290x290x290”.

3d-3-article-6-code

 

Result

As a result of modifying the Scanner.xcodeproj code we have created a project that meets the following requirements:

- application for iOS (iPad);

- getting 3D object data in real-time.

Also, we added features configuring the data upload period and removing the bounding cube.

Collecting the 3D data about everyday objects in real-time by using the Structure Sensor SDK is not a difficult task. Project examples and developer forums greatly facilitated the process of investigation and development. As part of the current task, the revision of the existing Scanner.xcodeproj has reduced development time, unlike creating a project from scratch. Also, open-source projects like Viewer.xcodeproj and RoomCapture.xcodeproj helped with finding the solutions to narrower problems.

However, the issue of setting parameters constant values of the cube limiting the scanning area, the minimum value of voxels per inch of the screen while scanning remain under attention. The solution to this problem lies in using the approach described in the previous articles.

Also, the solution doesn’t include the 3D object materials obtained during the scanning process. The estimate of their necessity and cost will deal at the next stage of development.

If you are interested in the JET BI FieldXR project, get in touch and our experts will find the proper solution for your business.

 

References

Setting up structure-sdk

How to start working with Structure Sensor. FieldXR project part 1 

How to work with the identified 3D objects. FieldXR project part 2

 

About author

Volha Dzeranchuk - developer C++. Education: Bachelor degree (year of graduation: 2012), BSU, Faculty of Mechanics, and Mathematics. Experience in the IT field for more than 8 years. Development of GUI applications for Windows, creation of DLLs, project support (C++, UE4, SQL/QT, Visual Studio/VR). More than 3 years of experience in teaching IT disciplines.


Volga Dzeranchuk
Question to the expert
Related Articles
All articles
Save the dates! Upcoming Salesforce events are here!
Training is an essential part of the person’s growth as a professional. Especially it’s important to learn something new for developers on an ongoing basis because they are the ones who meet unique business issues and create the best solutions that bring to life any client's wishes. Salesforce has always tried to provide all the partners with opportunities to learn and develop. We present you a useful list of upcoming events from Salesforce that can be an interesting and valuable pastime.
08 April 2021
How to work with the identified 3D objects. FieldXR project part 2
This post is the second article in the series that covers FieldXR app, an extension for Salesforce Field Service with deep visualization functionality.  We have already raised the issue of collecting 3D information about the real world required for the app’s correct operation. In this article, we will highlight the next two pain points: identifying the scanned object and calculating the position of the found 3D model for displaying in the scanned object’s coordinates.
25 March 2021
How to schedule Flow Monthly in Salesforce
Sometimes managers need to schedule Salesforce flows on monthly basis to review existing processes in the enterprise and create business strategies for the future. Unfortunately, for now, there is a limitation in Salesforce, you can schedule a Flow only once, daily or weekly. Often business needs are focused on the long-term perspective so the task to schedule Salesforce Flow monthly appears quite often. Have your admins ever wondered how to do it?
19 March 2021