Originally posted on petapixel.
Artist, designer and programmer Cyril Diagne recently created a bit of tech that looks more like science fiction that science fact. Using a combination of augmented reality and machine learning tech, he’s figured out a way to “copy and paste” objects from the real world into Photoshop, using just a smartphone.
Diagne unveiled his creation over on Twitter earlier this week, where even the Adobe Photoshop account was blown away by the results:
The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal.
The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image: https://github.com/cyrildiagne/basnet-http …
And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.
I also packaged it as a small python library: https://github.com/cyrildiagne/screenpoint …
Send a camera image + a screenshot and you get accurate x, y screen coordinates!