Take a selfie anywhere in the world.
Selfie Anywhere is an augmented reality app out now for Android 6.0+
Short demo video:
More info and how it works:
Selfie Anywhere puts you inside a photosphere by changing the background of a selfie in realtime. It's like a green screen putting you on a relaxing beach, in front of a famous monument, or on the surface of Mars, but without requiring you to have a green screen in your living room.
The process of automaticlly finding background of a selfie is a surprisingly complex task. People can take many poses while taking the selfie, have various numbers of people in the selfie, and be in a large variety of places and clothing. While some modern smart phones have front facing depth sensors which makes this much easier, SelfieAnywhere was created before this was common (late 2016, early 2017. (Before IPhone X)), and had the challenge of working with a single camera, in near realtime, and on slower hardware than is available now.
After experiments with clasical computer vision appraoches did not meet desired performance, machine learning based appraoches were explored. State-of-the-art “off the shelf” pretrained DL models at the time were too large to run quickly on a phone and were not tuned specifically for selfies. Instead, I experimented with dozens of different models in Keras and Tensorflow (#Selfienet) and created a custom dataset of thousands of labeled selfies (#Selfieset) using Amazon Mechanical Turk. There are two models that ship with the app, one for realtime segmentation, and one larger one for refining the picture after you take it.
The backend for the app uses AWS Lambda and S3 to store locations you can take the selfie at. Selfie Anywhere supports photospheres and panoramas that pan and tilt in sync with gyro and accelerometer readings from your phone, which was also quite a challenge to get working (though is helped greatly by the awesome Pannelum library). There is also basic functionality for a gallery of pictures you have taken in the app and sharing your selfies to other apps.
This started as a hackathon project in October 2016 and was released May 2017. Selfie Anywhere was also part of Spring 2017 class of Longhorn Startup Lab and was selected as one of the about top 6 teams to present at South By Southwest 2017.
Note: As of July 2017, SelfieAnywhere isn't being actively updated. However, it still works pretty well, and you can definitely give it a try.
See dgros.io for information about my other projects or contact info