This Guy Figured Out How to ‘Cut and Paste’ the Real World Into Photoshop

Originally posted on petapixel.

Artist, designer and programmer Cyril Diagne recently created a bit of tech that looks more like science fiction that science fact. Using a combination of augmented reality and machine learning tech, he’s figured out a way to “copy and paste” objects from the real world into Photoshop, using just a smartphone.

Diagne unveiled his creation over on Twitter earlier this week, where even the Adobe Photoshop account was blown away by the results:

According to Diagne, the “secret sauce” is a piece of machine learning (AKA “Artificial Intelligence”) tech called BASNet, which automatically recognizes and cuts out real-world objects when taking a photograph. That takes care of the “copy” in “copy and paste.”

BASNet is then combined with another piece of tech called OpenCV SIFT, which allows Diagne’s computer to sync up with his phone and detect where exactly he’s pointing the camera for the “paste” part of this trick.

The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal.

The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image: https://github.com/cyrildiagne/basnet-http …

And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.

I also packaged it as a small python library: https://github.com/cyrildiagne/screenpoint …

Send a camera image + a screenshot and you get accurate x, y screen coordinates!

All of this comes together into an app that Diagne designed for his smartphone, and the results are nothing short of astounding. It takes about 2.5 seconds for the phone to cut an object out of the real world, and another 4 seconds to paste that object into a Photoshop document.

As we mentioned above, the code behind AR Cut and Paste is available on GitHub, but keep in mind that this is a “research prototype” and not a consumer-ready feature. That said, now that we know Adobe is watching, we sincerely hope they’re planning to integrate something like this into Photoshop ASAP.

Source: petapixel

This Guy Figured Out How to ‘Cut and Paste’ the Real World Into Photoshop

Related Posts