Hi everyone!
For the last few months I have been building an app (Smart AR Home) which lets you look at the things in your home and control them from your mobile app using Augmented Reality.
So far app supports only switches and dimmers for lights and I am planning to add more devices based on your feedback. I am curious what type of devices you are mostly using at home and in what scenarios you think the app could be useful and when it doesn’t make sense.
From my observations, simple turn on/off for the light is much easier done with devices like Alexa but whenever we need a better control over the device (like dimming), being able to pull the phone, look at thing and complete the action, might be a perfect case for the Smart AR Home.
You need a phone that supports AR Core or AR Kit to use the app as it’s all based on Augmented Reality.
After installing, you will need to publish the smart app to let Smart AR Home integrate with your home. Step by step instruction is part of the app. Later, using any image or family picture or cover of the book, you will be asked to create synchronization point. From that point, you will place the devices in your home and app will use that point after each launch.
I appreciate any feedback, especially critical one that I could use to improve the app. I have some ideas for the next steps and improvements but I will share it later so it won’t influence your creative thinking.
App is of course free and at this moment I don’t plan to monetize it.
I’m game to test the iOS version once it’s up.
A couple of things that immediately spring to mind: audio control (volume for starters) and radiator thermostat up/down, as well as on/off switches for virtual DH control of whatever.
Very neat idea. I’m not sure how practical it is as a phone app (by the time you load and look at something, you could have loaded ST and tapped a button). But I don’t think that’s the point of the app.
Imagine if this was on HoloLens or other AR glasses product. Now, looking at something is natural, and your overlay handles the rest.
Couple of questions:
does this allow you to walk around the house and know where you are in context to recognize those switches?
Is it mainly recognizing wall switches, or could I look at a table lamp itself and have that be the active element to turn on?
How hard is the setup/training to assign all the switches from ST to their AR locations?
Let us know when iOS is out. Sounds like many of us are on iPhones I would be happy to try this out
Thanks. I have over 30 switches on just one floor and as I am planning to add more, list is getting longer and harder to browse. This inspired me to look for a better way to manage all of them
To answer your questions: does this allow you to walk around the house and know where you are in context to recognize those switches?
Yes, you need to do initial setup for all of the devices (place them) but later you launch the app, synchronize with the synchronization point and can walk around the house. AR Kit/AR Core will lost tracking over time so things might start drifting a little once you are far away but as technology is getting better, I think it will be resolved. Also, I thought to support more synchronization points around the house in the future so you don’t need to start always in one place.
Is it mainly recognizing wall switches, or could I look at a table lamp itself and have that be the active element to turn on?
You are free to use any element in your home. As you are placing the smart devices with augmented reality, you can put the virtual switch on any object in your home. I was trying for some time an idea where I would build my virtual dashboard by placing all smart switches in one location and it also worked.
How hard is the setup/training to assign all the switches from ST to their AR locations?
Not hard but it’s manual, you are walking around the house and you are placing them on top of the physical objects. I will try get a video showing how it’s done.
Very cool concept! I tried it out and it seems like something that would make for a really cool demo. As others have mentioned, I’m not sure of the practicality of it but it’s very cool!
Really interesting and cool, kudos to you for the original / innovative idea, I’ll definately give this a try!
Having a single entry point seems like a big constrain though, specially for houses with 2+ floors. Putting ugliness aside, would it be possible to put stickers next or on each switch with QR codes so you can just point to any device and it gets recognized?
This reminds me of this awesome video of a scary future that does not seem quite so distant:
Thank you. I am super happy that you like the idea! The app is going through review for a beta program on iOS and as I understand, they don’t have a concept of “open beta” (like on Android) so I would need to have some kind of email address to send you an invite. So let’s try that and I will also try to publish it in the store ASAP.
Apple will send you then an email with the invite and instructions
From what I saw, during initial setup, on iOS I couldn’t copy/paste client secret id for Smart app from SmartThigns and so I had to type the whole guid. It would be great to have the smart app published in SmartThigns or have some kind of API to access devices. Could be something like generic app released by SmartThings that we could just use for integration.
and to answer @Lumute question, app actually contains QR scanning logic but I disabled it as I thought no one would put codes around the house. At least that was my wife’s feedback, telling me that this “thing” will not hang on the wall . I would love to hear more feedback for this approach, scanning codes (I used ArUco instead of QR codes) is usually also more precise than pattern recognition that I am doing now. The advantage of pattern recognition is that you can use family picture or poster which is usually much bigger than QR/ArUco code on your wall and so it’s easier to scan it from the distance.
@pmjoen Have you created a synchronization point? If you did and there is a problem with finding it, go to settings -> Edit synchronization point and create a new one again. Choose a book cover, poster… something with a lot of color contrasts which will give you feature points. The more features points (shown as small circles in the app), the better image recognition.
Thanks again for trying the app. Please keep me up to date. I am making notes for every feedback.
I am stuck at the creating the synchronization point. It says looking for synchronization point and hangs there, no access to settings. I choose a colorful painting centered the camera took the picture. Do I have to stand there after the picture is taken and wait till it finds the points?
I have redone it many times, now I am at the screen with the picture with hundreds of yellow dots on areas of the picture, but it sits there looking for the point. How long should this take.?
I have created the synchronization point selecting a magazine. and no matter what I select it brings me back to the “Looking for synchronization point”
@pmjoen@joelw135 thanks again for reporting! Let’s find out if it’s a bug on some iPhones or just my not intuitive UI.
Here is the video of how it works on my iPhone. I am using iPhone 8. On the video I am creating a new synchronization point and base don its location, I am adding a device (switch) to control hallway lights:
Can you let me know what iPhone version you are using AND if after you create a synchronization point, do you see the screen hang or is it updating very slow? The last one is important as image processing is heavy on CPU and I am wondering if you are on older iPhones version, if you are getting performance issues. It would be really good findings for me as I can try few things to improve it.
I am on a 64GB iPhone 6s with the latest OS. I take the shot and it shows the picture and hangs. Here is a screen shot. But as soon as I move the phone I get a camera view. I am almost sure it is a screen hang. As I can’t get back to settings to create another. i have tried many different subjects none work all hang. I can’t even get back to settings now.
OK, I should get some crash logs from Apple at some point so I will take a look at these. In the meantime, I want to try performance fix. Please expect update build soon for that. My third approach will be to get iPhone 6s and try it myself but it will take few days.
I would be happy to try this out
It would be great to have the smart app published in SmartThigns or have some kind of API to access devices. Could be something like generic app released by SmartThings that we could just use for integration.
. I would love to hear more feedback for this approach, scanning codes (I used ArUco instead of QR codes) is usually also more precise than pattern recognition that I am doing now. The advantage of pattern recognition is that you can use family picture or poster which is usually much bigger than QR/ArUco code on your wall and so it’s easier to scan it from the distance.