What is Automated Testing?

Automated testing has been around for over 20 years in different forms and is essential in DevOps and Agile cultures now commonplace in software development. A way to guarantee continuous and sustainable timely delivery of software is by automating lifecycle processes including the testing stages. Software is a critical tool in most research projects, and the incremental testing of software as it is developed makes bugs easier to spot, ensures they are spotted early and are fixed in the minimal amount of time.

In the days before automated testing, software systems were manually tested using check lists, test plans and feedback given to the (often separate) software development teams. In large companies, a QA department would be dedicated to quality and testing and the whole process was repetitive, laborious, time-consuming, expensive and error prone.

Automated testing software today is totally separate software from the system being tested. There are tools and frameworks widely available on the market to help speed up the development of automated tests written by developers as they develop new features. Automated testing functionality enables quick, repetitive execution of system testing and provides comparison of actual outcomes with predicted outcomes to indicate whether tests are successful. Using test automation as part of a formalised testing process speeds up repetitive but necessary tasks and enables additional forms of testing which would be difficult to do manually.

The benefits of using test automation are consistency, efficiency, repeatability, early bug identification, ease of maintenance, and an increase in the volume and scope of test devices. Automated testing reduces the time it takes to validate and release new software updates and helps deliver faster, more reliable updates to software.

Why use Automated Testing for Mobile Apps?

Automated mobile app testing does not replace manual testing within the MDS but is done in addition and compliments manual tests. During the development of an app, manual testing will provide accurate user interface (UI) testing, replicating real user experiences. Complicated use-case scenarios can be tested quickly using manual tests and generally it is more cost-effective for small projects where writing automated tests would use a disproportionate amount of the available budget.

Planning and writing automated tests are an additional overhead which should be considered when initially planning costs for a mobile app project. However, over an extended period, on longer term or complex projects, automated UI testing becomes a cost-effective addition to manual testing. For cases where an app is being developed for use with a very small cohort of participants or will be installed only on one type of mobile phone, then the overhead of developing automated tests may not be justified. However, where the app is released for wider public use and will be installed on numerous different devices then the app must be comprehensively tested, and automated testing becomes essential.

To make devices unique and increase sales, many device manufacturers add additional UI features on top of the standard Android distribution to provide extra features and more distinctive designs. This makes the job of mobile app testing extremely onerous on Android as versions of the OS can vary between device manufacturers – not a problem with iOS since all Apple devices run the standard distribution. Before public release on the app stores, MDS must ensure apps can successfully run bug free on numerous operating systems, and different screen resolutions, including custom User Interfaces (UI’s).

It is simply impossible for MDS to test apps manually on the hundreds of different devices available on the market. The only practical way MDS can do this is by using an automated testing service. MDS currently uses Visual Studio App Center (VSAC) connected to Microsoft Azure. This service provides remote access to a wide collection of physical phones in a Microsoft data centre originally termed the Xamarin Test Cloud.[GS2]

Automated Testing with Xamarin (.NET)

MDS use the following software to develop then automate the testing of our cross-platform mobile apps :

• Xamarin Forms, a Microsoft’s Xamarin.Forms framework
• Visual Studio, an integrated development environment from Microsoft
• NUnit , an open-source unit testing framework for the .NET Framework and Mono. Xamarin.
• UITest, a C# testing framework using NUnit for UI Acceptance Tests on iOS and Android apps.
• Xamarin.UITest, the Automation Library that allows the NUnit tests to execute on Android and iOS devices.

To use automated testing in a project, developers add a separate UITest project to the existing Xamarin.Forms solution. UITest automates the actions of a user by activating controls on the screen and providing the input via variables where a user would interact with the mobile app. To enable tests to interact with controls within the code like pressing buttons or entering text into boxes the test code needs to identify the controls on the screen. To reference controls, each control needs a unique identifier and the recommended way in Xamarin.Forms is to use an AutomationId property. In the above Visual Studio image there is an example of code where the app is automatically scrolled down to the position of the UI element tagged as ‘Top_cancel’ using a UITest command ‘ScrollDownTo’.

Tests can be executed both locally or remotely; UITests can be run on emulators or connected devices through Visual Studio’s Test Explorer or posted to the Xamarin Test Cloud. Currently, on Windows, only Android apps can be run locally using Visual Studio; iOS apps can only run locally using Visual Studio for Mac running on MacOS.

Microsoft Azure

Microsoft Azure is a public cloud computing platform and MDS has access to a paid Azure subscription so can make use of all the App Center services:

• Automated Build and Deploy
• Automated Testing using the Xamarin Test Cloud and Xamarin UITest
• Analytics
• Crashlytics

Before testing an app via the test cloud, automated testing of the app is checked locally by generating an Android Package (APK, the file format that Android uses for distribution of apps) or iOS App Store Package (IPA) and running the tests locally. Subsequent tests can then be submitted for running on the cloud.

To run both Android and iOS remotely on the cloud, tests can be deployed from the command line to the Xamarin Test Cloud via the Visual Studio App Center (VSAC) command line API on both Windows and Mac. To be able to use the test service you must have an active App Center account connected to an Azure subscription or sign up for a trial.

Once an automated test has been completed on the cloud, the App Center provides diagnostic information collected about the different test runs. For example, a summary percentage graph is displayed at the top of the screen and below is a list of test runs with date and version details.

Further details about a particular test run on the console can be viewed by clicking on an individual test. The screen will show the test and device success percentages and a list of the name of UITest method names.

Clicking on the test name, a further screen will be displayed with the make and model of the devices used and any screen shots generated during the run.

GitHub Integration

App Center integrates with GitHub and can be configured to automatically build and run tests on a particular branch as part of a Continuous Integration pipeline. We grant App Center read access to the repository and it is able to pull the code, build and run tests without developer involvement, even posting results back to Microsoft Teams or Slack.

Feedback

If you have an idea for a mobile app and want more information or have any suggestions or questions about our service, please feel free to email us and let us know. You can also contact us through the Support Portal via the MDS request form under Research Computing.