Automated testing on online IDE emulator using Selenium

Recently, I want to make some random testing on my smartthings apps relying on the online IDE simulator. As other people did, I decided to use the Selenium to perform the automated Web UI testing.

However, I encountered a problem. I found that the online simulator may be hardened to prevent automated testing. At least, it makes hard to automatically login in.

Have someone also encounter this or know the solutions?

Thanks for any advice.

I’m not sure what would prevent selenium from working in the IDE - is it the redirects or something?

One possibility is to expose a rest api in your smartapp and test w/ something like postman or jmeter. Then the endpoints would just proxy into your methods (if you’re trying to unit test) and return the result to your testing tool. You could also have certain endpoints to create device events for simulated devices which your app is subscribed to, etc. I would personally go this route as I have had trouble with the simulator in the past and I believe there are some known issues with it which would make me hesitant to rely on it for anything automated.

1 Like

Thanks for your answer firstly. I tried to login in to the online IDE simulator through my Selenium scripts. However, after logining in, another page is directed to and need to enter the username, password, verification code again, as below:

If I login in manually, I won’t encounter this page. I think that there may be some kind of mechanism to prevent the automated login in.

I need to test a hundreds of smartthings apps. Thus, I may still need to rely on the IDE simulator.

Thanks

The simulator is not trustworthy. I’ve typed here many times: Never trust the simulator.

I recommend using an Android based testing tool. There must be plenty of choices, since Android App developers also run automated regression testing.

This could possibly be accomplished using an Android emulator within Windows combined with a Windows application automated testing too.

Or… Just possibly, a browser based Android screen mirror.

2 Likes

Thanks for your advice.
In addition to SmartThings IDE online simulator, if I want to run a SmartThings app dynamically without physical devices, do you know what other simulators can be used for this?

You can manually pre-create as many different persistent Virtual Devices using the SmartThings IDE.

You may need to write simple SmartApps that monitor and control these Virtual Devices to set up your test scenarios (e.g., reset a Virtual Temperature sensor to 70*, wait 1 minute, change temperature to 75* before running your real SmartApp that is meant to trigger when the thermostat goes above 74*).

Virtual Devices are also more reliable and flexible than the simulator. You can code a Virtual Device to do anything you want, including errors and edge cases.

OK, I will have a try.
Thanks a lot.

Hi,
Thanks for your previous advice. I have set several Virtual Devices and test some smartthings apps successfully.

    Currently, I have a question.  How can I run two or more smartthings apps at the same time with these virtual devices? 

     Thanks a lot.
1 Like