Best Practices for Test Automation: Framework Selection, Test Coverage, and more

When was the last time you were tasked with test automation but ended up spending more time and effort on maintenance?

If you’re reading this article, chances are you’re familiar with test automation and may have even conducted it yourself. In case you’re not, this article on The What, When, and Why of Test Automation will give you a quick refresher on what it is and why/when you should use it.

However, if you’re like other newbies, you may have managed to design a framework, but then you’d spend a lot of time, effort, and resources on simply maintaining it.

That right there is what happens when you do test automation without knowledge of the best test automation tools and practices.

And if you read further, you’ll find precisely what the best practices for test automation are and how they can get any beginner started with test automation in the most efficient way.

Framework Selection

Test automation is heavily dependent on tools, low-code or no-code.

Therefore, selecting the right tool is essential for high ROI with test automation. Here are a few points you should consider while choosing a test automation tool:

• Scope of testing: Will you be testing web-based applications or mobile-based applications? Or both. In order to test only web-based applications, there are several tools like Selenium or Cypress that are pretty powerful. In contrast, for mobile-based applications, Appium is still considered the go-to tool.
• Available support: Whenever opting for a test automation tool, one should always consider the level of documentation and the degree of support you can get from the respective community, if you ever need it. Since Selenium and Cypress have been the favorite tools among the testing community for quite some time, the level of support you can get from the community is incredible.
• Tech stack: Another factor to consider while selecting the automation tool is the tech stack your dev team uses to develop the application under test. Since you’d want to keep and execute your E2E tests in the same repo and environment, respectively, it only makes sense to use a tool that uses the same stack. For example, if your application is being developed using Angular, you might want to use Protractor, which was developed specifically for testing Angular applications.
• Open Source or not: Depending on budget constraints, one may choose to use open-source tools such as Selenium or Appium for automation purposes. However, it is important to remember that all open-source tools are not inferior to their commercially available counterparts.
• Available Features: While choosing the tool, you should check if the automated testing tool supports record-and-playback test creation as well as manual creation of automated tests. Does it include features for implementing checkpoints to verify values, databases, or key functionality of your application? Does your tool integrate with your CI/CD pipeline, such as Jenkins or Azure? Or a source control such as Git? Does the tool have the ability to test enterprise applications? Does your tool offer out-of-the-box support to test packaged applications like SAP, Oracle, and Salesforce?

Selecting the Right Selectors

Using the right kind of selectors to get your elements for your automated tests is the only solution for flaky tests. If you select un-reliable or generic selectors, your tests are destined to fail. For example, using Xpaths or generic class names as selectors will result in flaky tests and a lot of maintenance for the test suite, as they’ll be invalid whenever there is some change in the UI of the page under test. In my personal experience, I’ve found the CSS & JQuery selectors more reliable and effective.

Let’s assume we want to get the following element:

<a class=”tile” aria-label=”YouTube” title=”YouTube” href=”https://youtube.com/“></a>

Which of the following selectors do you think is more reliable?

Xpath: //*[@id=”container”]/a[1]

CSS: a[href=”https://youtube.com/]

Which Tests Should You Automate?

It is impractical to automate all testing, so it is vital to determine what test cases should be automated first. Good test cases for automation are ones that are run frequently and require large amounts of data to perform the same action.

You can get the most benefit out of your automated testing efforts by automating the following:

• Repetitive tests that run for multiple builds
• Tests that tend to cause human error
• Tests that require multiple data sets
• Frequently used functionality that introduces high-risk conditions
• Tests that run on several different hardware or software platforms and configurations
• Tests that take a lot of effort and time when manual testing
• Test Cases that touch a key piece of functionality and should be run before release

Test Coverage

Here we’ll not be talking about test coverage in terms of unit testing but the different types of testing you should cover as part of your automation test suite even if it weren’t required. Here are a few that, in my personal opinion, should be covered as part of best practices implementation:

• Accessibility Testing: Without going into too many details about accessibility testing, as in what it is or why it is important, the borderline is your application will perform better if it follows the WCAG guidelines, and while it should be part of development itself, nonetheless it should be part of your automation testing. With tools like the AXE engine, it is very easy to integrate the accessibility test cases within your existing test suite.
• API Calls: Even though APIs are tested separately by the QA and the development teams, it is always better to check the API calls on runtime while they’re being made from the application. For example: If you need to test a login page, sometimes the errors on the FE are very generic, and they’ll not tell you more than ‘Something went wrong.’ Still, as a tester, while you need to observe if something goes wrong, you also need to identify why it did. In this scenario, if you’re inspecting that API call made on clicking the login button, you’ll get the exact response and status code back and can quickly identify what exactly happened based on it.
• Client-side Performance: While server-side performance is important and is covered extensively during testing, client-side performance is equally essential and, most of the time, is left untested. A general client-side performance audit will tell you things like the FCP and LCP, which is the time your application takes to load the first item versus the page’s main content. The easiest way to implement this is by using Google’s lighthouse integration.
• Visual Testing: I’ve covered this topic in detail in my other article on Visual AI and Autonomous Testing, but just for introductory purposes, with modern AI/ML tools available like Applitools and Percy, it doesn’t make sense to keep validating your application’s look and feel via coding assertions. Implementing Visual AI testing in your automation suite will save much of your time and resources, especially with maintenance.

Test Structure & Strategy

While designing your test automation suite, you must take into account the importance of a good test structure and strategy that can benefit not only the live development but also its maintenance. Following are a few points that have helped us in the past:

• Test suites: Structure your tests in the form of test suites that can be independently configured and executed. If each test case represents a piece of a scenario, such as the elements that simulate completing a transaction, use a test suite.
• Self-contained /Independent: While writing your tests, make sure they’re self-contained, meaning that they should not depend on their successor tests to pass in order for them to pass. We’ve seen many examples where teams build their tests to run in a sequential order while it’s opposite to what the automated test suite is supposed to be: flexible, agile, etc.
• Data-driven: Always make sure your tests are data-driven in order to ensure the reusability of the work that has already been done. By using a data-driven approach, you can generate test cases just by changing the data stored in external files.
• Page Object Model: In some cases, when your application under test is prone to frequent UI changes, it is a good idea to implement POM for your automated tests. It not only ensures the reusability of the component but also helps in the readability and maintenance of the test code.

Dedicated Automation Environment

This might not be possible for each team or project. Still, since we’re talking about the best practices, I have to mention Dedicated Automation Environment, as having one will only increase the productivity and overall efficiency of your automation framework and hence the crucial time and resources of your team.

It has been observed many times before, more than I hate to admit, that tests frequently fail due to unannounced changes deployed to the environment in which the tests were executed, especially when multiple teams are working with the same dev or test environment. Therefore, having a dedicated environment that is up to date with the latest changes that are supposed to go live in the next release is essential for non-flaky tests.

Testing Cross-platform and On Real Devices

More often than not, it is observed that testers design their tests to execute only on a single platform. It could be a platform like a desktop against mobile or in terms of a browser like Chrome vs. Firefox, while as a best practice, your tests should be designed in a way that they can be executed against multiple platforms, and they should.

To have ample test coverage for a web application, you should ensure you execute your tests against all the popular browsers and on all the popular devices such as desktops, tablets, and mobile. Moreover, if you require more realistic results, your tests should run on real devices instead of emulators or simulators. This task can be humongous and expensive if done manually or even using local resources hence making use of cloud platforms such as Browserstack or SauceLabs can be beneficial.

Execution

This is where all your preparation and work come to life. This is also the most critical step to automate, which most of the testers fail to do. It is often understood that integrating the E2E tests into the existing CI/CD pipelines is a DevOps job that is somewhat true but not always. I like to do it myself, as it gives you more control and ownership of your tests. Anyhow, whoever does it, ultimately, it should be done, and the following are a few points to remember while making doing so:

• Parallelize the automated test cases that are independent. For example, if a test suite contains 5 tests that are self-contained, none of them should wait for the previous test to finish before starting. Use multiple machines and servers to run your tests in parallel. It can decrease your execution time from ~25 to ~5 minutes.
• To increase the speed of your release while maintaining the quality, configure your tests to run on each pull request from a child branch to the main branch. This will ensure that none of the new changes are breaking any existing functionality and that the code is safe to merge.
• If your team doesn’t push out frequent releases, it might be a good idea to execute your test cases based on a cron-job to ensure nothing unexpected breaks the application. For example: Run the tests every other day at noon or every first Monday of the month.

Reporting

Finally, in the last step, if there is no thought-out plan for effectively collecting and analyzing the test results, the automation effort can be all in vain. Therefore, a well-defined process will save teams from many conflicts and resources in refining the application.

Automation should serve to reduce the amount of time QA teams have to spend verifying test results. Set up adequate reporting infrastructure using the tools that will generate high-quality test reports after each cycle. If possible, assign your tests different but relevant tags to group and filter them out easily in the reports. A good test summary report should be created after each cycle and shared with all the stakeholders via multiple mediums, such as Email & Slack.

Takeaway

Each application of any complexity will likely have its own combination of testing requirements, and none of the discussed points together will ever be in play.

It is still essential to know the best practices for making the right decisions early on. Study these practices, and implement them in a way that best suits your software, business, and users.

If you want to take your application of test automation a step further to practice better, faster QA, this 30-minute video will give you all you need!