Debugging Setup Test Automation
I spent about 3 hours today with a couple of other guys trying to figure out what was messing things up with our test automation. There were no visible problems that we could see but after setup was done a bunch of files were missing from the computer. We're not sure we totally solved the problem but one thing we did find was an example of the things that could go wrong with UI test automation.
The component that we use to set up our program drives the UI of our setup program. The setup program itself installs a bunch of components, each one of which is provided by a different product or technology team. In order to determine whether setup is done it looks for the words "Finish Page" in the caption of the setup program's top-level window. Well, what we saw tonight just before heading home was that one of the component in the middle of setup did something to change the top-level windows caption to something that contained "Finish Page" but the setup wasn't done yet. The setup driver component then clicked on the 2nd component on the setup program's page. Instead of "OK," that button at the time the driver saw "Finish Page" in the caption was the "Cancel" button. This brought up an unexpected confirmation dialog and things went rapidly downhill from there.
Ugh. There must be a better way...