3-5 Evaggelikis Scholis, 14231 Nea Ionia, Greece

This Guy Figured Out How to ‘Cut and Paste’ the Real World Into Photoshop

Originally posted on petapixel.

Artist, designer and programmer Cyril Diagne recently created a bit of tech that looks more like science fiction that science fact. Using a combination of augmented reality and machine learning tech, he’s figured out a way to “copy and paste” objects from the real world into Photoshop, using just a smartphone.

Diagne unveiled his creation over on Twitter earlier this week, where even the Adobe Photoshop account was blown away by the results:

Embedded video

What might, at first blush, seem like special effects trickery or an April Fools prank is actually a piece of bona-fide tech that he’s actually published to GitHub. As the description on GitHub explains, this is “an AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software.” He’s using Photoshop, but it could, in theory, work with anything.

According to Diagne, the “secret sauce” is a piece of machine learning (AKA “Artificial Intelligence”) tech called BASNet, which automatically recognizes and cuts out real-world objects when taking a photograph. That takes care of the “copy” in “copy and paste.”

The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal.

The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image: https://github.com/cyrildiagne/basnet-http 

View image on Twitter

And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.

I also packaged it as a small python library: https://github.com/cyrildiagne/screenpoint 

Send a camera image + a screenshot and you get accurate x, y screen coordinates!

View image on Twitter
All of this comes together into an app that Diagne designed for his smartphone, and the results are nothing short of astounding. It takes about 2.5 seconds for the phone to cut an object out of the real world, and another 4 seconds to paste that object into a Photoshop document.

As we mentioned above, the code behind AR Cut and Paste is available on GitHub, but keep in mind that this is a “research prototype” and not a consumer-ready feature. That said, now that we know Adobe is watching, we sincerely hope they’re planning to integrate something like this into Photoshop ASAP.

Source: petapixel

Related Posts