The data management test approach is based on the following principles:
- The tests shall be easy to write and maintain
- The tests can be executed multiple times, depending on the scheduled milestones and needs
- The test conditions shall be well identified in order to: i) make the test repeatable ii) make the behaviour and the results as close as possible to operations
The documents involved are:
- The Test Specification: it contains the list of test cases written on a specific DM component, plus some descriptive parts on the test activity involved for that component.
- The Test Report: it contains some descriptive part on the test execution, a test case runs summary report and a detailed report on the test done.
DM consider the possibility to use the Test Management for Jira (by Adaptivist). The main advantages is to have an easy way to manage test cases and test report and to related them to requirements.
The documentation can be extracted from Jira:
- The test specification will contain the test cases defined to a specific DM component.
- The Test Report: will be divided in 2 parts:
- the first part (the plan) contains the planned test activity information (first list of test to be executed), environments, datasets, personnel.
- the second part (the report) contains the actual test runs results, their assessment and additional information derived from the the execution of the test
Test Specification
The test specification will include all test cases defined for a specific component in the Document Tree. Obsolete test case will be included also.
The test cases have to be written in JIRA, in the LVV project, under the corresponding folder.
Sections 1 and 2 of the test specification need to be written directly using Latex and changes submitted in the corresponding github repository. Section 3, 4 and appendix will be generated from JIRA.
How to write a test case
A test case need to be formulated in a general way, in order that it can be executed multiple times in different conditions, like for example, different software versions or different dataset versions.
Follows a table with some recommendations for each test case field:
| Field Name | Old Test Spec Name | How to fill it | Additional Comments |
|---|---|---|---|
| Name | Test case subsection title | Has to be short string indicating the purpose of the test. | It is recommended to avoid using the requirement name. LDM-639 test cases have been named following the verification elements, and therefore the requirements names, mainly for lack of time. |
| Objective | Test Items | Describe what the scope of the test. | This field will appear in the Test Specification as "Test Items". Avoid including the requirements text. |
| Precondition | Input Specification | This field should include all inputs required to run the test. | For example the input data required (do not specify versions). The field is provided by the test management plugin in JIRA and can't be renamed. There is ongoing a request to SE to have a specific Input Specification custom field. |
| Folder | A Test Specification document will be produced for each folder | ||
| Status | When you create a new test case the status is set to "Draft". Once the Test Specification is approved (via RFC) the test draft test cases will be moved in status "Approved". Once a test case is not valid anymore, its status has to be set to "Deprecated" When you are going to modify an approved test cases:
| Do not delete test cases, this may remove important information from the system. The plugin is not going to enforce a new version when an approved test case is modified. | |
| Priority | Set the priority that you think is more appropriate. By default is set to "Normal". | In the future this information can be used to prioritize test cases. At the moment is not used. | |
| Component | DM (Data Management) | This field may be useful when filtering across test cases. | |
| Owner | Who is in charge to write and maintain the test case. | In the future the test case may be executed by a different person, but the owner will not change. | |
| Estimated Time | Fill it in case you have an idea how much time it can take to run the test case, otherwise let it blanc | ||
| Verification Type | It can be "Test", "Inspection", "Demonstration" or "Analysis". Use the most appropriate type. In general this should be "Test". | This information should be derived by the requirement and verification element definition. | |
| Verification Configuration | This field should not by used in DM. | It can be used in case a test case shall test a specific configuration of a software product, but this is against the principle that a Test Case should be generally formulated. Specific configurations ban be specified during the test run definition and are related to the environment. | |
| Predecessor | Intercase Dependecies | This is the list of test cases that need to be completed (successfully) before the actual test case can be executed. | This do not imply that those test cases are part of the test script |
| Critical Event | This field is mandatory but should not be relevant for DM. Therefore set it to "False" | ||
| Associated Risks | This field should not be relevant for DM. Leave it blanc. | ||
| Unit under test | This field should not be relevant for DM. Leave it blanc. | In the future we may use the components defined in the model in Magic Draw to group test cases, instead of using the folder. | |
| Required Software | Environment Needs - Software | List here the required software packages that is need to be installed in the system in order to run the test | In case the test case is to verify the functionalities of a specific DM software package, for example science_pipeline, this shall NOT be listed here, but in the Objective field, Test items section. |
| Test Equipment | Environment Needs - Hardware | List here the required hardware that is need to be installed in the system in order to run the test. This usually imply a server with a specific CPU power, RAM and available disk space. | The exact hardware used for the test will be specified in the Test Plan or in the Test Run. This information can be different each time a test case is executed. |
| Test Personel | This field has not been used until now by DM during the definition of the test. Leave it blanc or list here external people that may need to be involved during the verification | These people are not the owner nor the test engineer that is going to run the test. They may be for exampke stakeholder, that need to assess the scientific results in order to ensure that the test has passed. | |
| Safety Hazards | This field should not be relevant for DM. Leave it blanc. | ||
| Required PPE | This field should not be relevant for DM. Leave it blanc. | ||
| Postconditions | Output Specifications | Specify here what output is expected from the test. | For example the output data expected. This field has been called "Postcondition" due to the duality with the "Precondition" field provided by default. There is ongoing a request to SE to have a specific Output Specification custom field. |
Follows the rules to use to write steps in the Test Script section
- Write the step in the most reproducible way,
- Add in the test data field the input data to use as input (if any)
- Specify the expected result when not clear from the step description
- When a different test case is used in a step:
- use only test cases that have been created for that purpose
- avoid recursive use of test case (only one level)