Nothing is forever except change.
Be aware of what is changing: Take the test context out of the equation and make sure it can be changed without extra effort.
Test Context
"Test context" refers to the combination of data, configuration settings, platform, language, and other factors that define the conditions under which a test case is run.
According to the ISTQB (International Software Testing Qualifications Board), one of the software testing principles states that “Testing is context dependent.”
In contrast to test data, test context describes the conditions that are not expected to impact the test result. However, this conditions are essential for the function under test to be operational.
Non-Functional Test Context
The non-functional test context combines the aspects associated with non-functional requirements, like platforms, browsers, and languages. It is usually evident enough to be handled explicitly. Test analysts should know what aspects apply to the specific software product and ensure the tests are either context-friendly or context-agnostic.
Functional Test Context
Functional test context is not that obvious to extract. It references the functional preconditions, which are essential for test execution but are not specific to the nature of the test. A typical example is user authorization. In most applications, the majority of the tests require the user to be registered and logged in to exercise the function under the test. As such, it is better to design tests in a user-agnostic way rather than providing registration and login instructions at the beginning of each test. The number of preconditions that make up the test context can be enormous in complex software. Controlling them requires skills and dedication when creating test design specifications or developing a test automation system.
Test Context Extraction
There are two approaches to handling the test context:
- Listing all the test context aspects explicitly in the test setup;
- Moving test context out of the test and implementing context management means controlling the context so it can be substituted without modifying the test case itself.
The first approach bloats the test with context details and is not recommended. The second approach makes the tests flexible, which is vital for testing the software in different contexts, like environments, platforms, browsers, languages, user credentials, etc.
Context Management
Configuration Management
Use configuration files or settings to define the test context. Changing the configuration file allows you to alter the environment variables, URLs, or other settings relevant to the test's execution.
Environment Switching
Implement a mechanism to switch between different environments (e.g., development, staging, and production). This can be done through command-line arguments, environment variables, or custom settings in the testing framework.
Context Injection
Separate context data like current user credentials or permissions to make it possible to iterate the same test against different contexts without modifying the test case itself.
Context-Agnostic Tests
For a test to support context substitution, it should be designed in a context-independent way:
- Avoid using UI labels to make the tests localization-friendly.
- Avoid using hard-coded conditional statements (e.g., if-else) based on the current environment settings;
Reasons to apply Context Substitution Principle
Robustness
Many flaky tests are fragile because they either encapsulate a context setup that has changed or require some environment or infrastructure conditions they do not control. The test becomes much more stable once such dependencies are identified and extracted.
Speed
Once extracted, a test context can be built by following a smart approach to increase tests performance:- Automating test setup for manually executed tests,
- Automating UI test pre-setup via API to make it faster,
- Preload application state from a backup DB image, etc.
Reusability
An ability to substitute test context will increase the value of a test as it will allow the reuse of the same test in a different context with minimal overheads.