Introduction of 360 views is a significant advancement of camera development. With this technology, people can capture and view 360 images. Question is

  • How we can represent a 360 image in a 2D space?

  • Is there really a camera sensor which can capture 360 frames at a time?

  • Is our perspective of observing a frame, need to be changed?

  • Or, even this change in perspective of view an image frame, has any good user experience?

The 360 videography has solved a very important security problem in CC camera. To cover all different blind spots, traditional CC camera needs to be placed in multiple locations where a 360 camera can cover all of it.

Not only that, but also experiencing 3D image in virtual reality a new technology trends. 360 cameras play an important role to experience 3D in real nature, remote occasion, live concert, live presentation housing industry etc.

Samsung is one of the market leaders to develop the hardware and software solutions of 360 camera ecosystem. We have leaded the market in terms of feature, cost and quality. We call it Gear360 camera. It is tiny mushroom lookalike device with no display. Then how will a user use this? Answer is very simple - by connecting the Gear360 device with Android, iPhone or MAC. It was a complete package with cutting edge imaging technologies and software solutions to process thousands of image frames to make 360 video and render it to mobile screen with new dimensions of image viewing technology.

There are amazing features in Gear360 camera which make this device as a no 1 in market. Like

  • Connectivity with All types of phones and Pcs using WiFi, USB and Bluetooth.

  • We have added live view to see the 360 image before capture.

  • Support multiple video codecs. MJPEG is for live view and H265 is for capture video. H265 reduces the data size and overload in network.

  • Recode 4K video with 24 fps. This is amazing. For the first time user have used 360 4K. H265 plays a very important role to compress the size almost half compare to H264.

  • We have introduced 5 different views to render 360 frames for the first time. The beauty is how geometry works in the perspective of user eye. Calculations of those projections are significant mathematical equations in modern graphics industry.

  • People can go live to Facebook and YouTube in 360 for the first time.

  • After capturing image or record video, people can share it different social media.

  • People can experience the 360 live broadcasts in VR as well. Aren’t those amazing?

It was not an easy job. Developing network protocol for transmitting data from camera to phone and sending the action command from phone are complicated as there is no display in camera. Moreover, we have 2 fisheye lenses which can capture two 195 degree images from camera sensor. We have to create one 360 frame from two 195 degree fisheye frames. After that, Video data will be encoded with H265 and with audio channel data will go through a mixer to create a complete 360 frame.

We have used UPnP protocol to communicate action data between camera and phone where camera will create a WiFi hotspot to accept connection from phone. All audio and video channel data will be fetched using http based streaming service in both camera and phone.

Now the challenges are to create 360 from 195 degree fisheye frames. We have developed a stitching engine which can process image on to GPU with our proprietary algorithm. To process it, we have used openGL for Android and Metal library for iPhone to get hardware acceleration. Two Fisheye lenses give 390 degree output. We have to remove the overlapping 30 degree region, apply geometric calibrations, depth calibration to align the matching feature points. Then, we have used alpha color blending with custom parameters to merge image frame according to the matching feature points to get 360 frames.

After that we have used HEVC hardware encoder to get enough frames per seconds with the help of accelerations. Not only that, but also HEVC encoder reduces the file size to almost half which help us to support 4K in Gear360. Then a muxer mixes both audio and video channel data and create new 360 video with 24 fps. For 3K video we have supported 30 FPS and for Full HD, we have supported 60 fps.

Look all the screenshots above.

Screenshot source and app source :