Google's developer demonstrates Augmented Reality-based 'Copy & Paste' feature using a smartphone camera

The future of copy-paste could be with the use of Augmented Reality (AR). A Google Arts and Culture Lab programmer has demoed how smartphone cameras coupled with AR can capture objects and paste objects from surroundings directly to Photoshop or other tools.

Cyril Diagne has uploaded a short video which shows images of objects dragged from the real world and dropped into Adobe Photoshop. Diagne says that he used a tool called BASNet which is designed for detecting edges of an object and removes the background. He also used another app called OpenCV SIFT, which uses image-tracking technology to detect where the phone is pointing at the screen.

While BASNet separates the foreground object from the background using Machine Learning (ML), the other component, OpenCV SIFT, detects where the phone is pointing at the screen. It takes about 2.5 seconds to copy the object and four seconds to paste it.

4/10 - Cut & paste your surroundings to Photoshop

Code: https://t.co/cVddH3u3ik Book: @HOLOmagazine Garment: SS17 by @thekarentopacio Type: Sainte Colombe by @MinetYoann@ProductionType Technical Insights: ↓#ML#AR#AI#AIUX#Adobe#Photoshoppic.twitter.com/LkTBe0t0rF — Cyril Diagne (@cyrildiagne) May 3, 2020

“It’s part of a series of experiments I’m doing every weekend to explore how machine learning and AI can help create more digital interactions that are more natural. Basically using the algorithm’s ‘intelligence’ to remove the layers of abstractions we had to build in digital interfaces up to now,” Diagne told Cult of Mac.

Does this mean that the tech is coming soon to smartphones that support AR? The answer to that question is not anytime soon. Although the demo was carried out using an Android phone and a MacBook Pro, Diagne stated that the underlying architecture was not created using Apple’s ARKit or Google’s ARCore.

The copying and pasting process using AR is currently a proof-of-concept. Diagne could soon develop a mobile app that offers such a feature. If the concept can be a reality, it could replace the conventional Ctrl + C and Ctrl + V method for copying and pasting text and image.