XRAY Test Management System Implementation on Health Cloud Project

Introduction

The company got a new Health Cloud-related project. The product was already implemented successfully, JET BI was only supposed to improve and maintain it. Several clients successfully use the product, and the testing team was supposed to ensure the quality of the product with every release.

Unfortunately, testing processes were complicated for the following reasons:

• A large number of unrelated tickets in Jira, including bugs, epics, stories, tasks, and subtasks make that difficult to identify what they are related to. Things were not put in order in Jira since several people worked on a project previously and they didn't adhere to the same process of creating tickets in Jira. At some point, all of that became quite messy, it was difficult for the QA team and the stakeholders to find any information in Jira regarding existing bugs and the requirements that are supposed to cover them
• The requirements were unclear, incomplete, or stored in various places, which also led to the fact that it always took a long time to find a bug in Jira and figure out what test case and what requirement it refers to and if it’s even relevant
• Test coverage was not visible in one place, we were not sure we aren’t missing anything in testing

The quality of the product was at risk, and the test team decided to put things to save time in searching for information and to provide the best test quality for the product by implementing clear test traceability in Jira. It was decided to adopt a test management solution as a Jira extension. XRAY test management system was chosen as a solution. 

XRAY configuration

To configure it, a new custom type of Jira ticket was created first - the Requirement. And that's because XRAY can use any of Jira tickets as requirements to be covered by XRAY tests, for example, stories, but sometimes it's not easy to change the overall processes on a project to force anyone in the team to create stories if only they will be covered by tests. And if at least one story will be created by mistake and will not be covered by tests, it results in the breach appearing in test traceability (not all the requirements are covered by tests).

The new ticket type was selected as the coverable one:

So, this ticket type was supposed to be used only by QA to clarify the requirements and get confirmation from stakeholders. And it is always good to keep information regarding requirements just in one place.

Then all the test cases were exported from a place where test documentation was stored before (it can be anything actually), and matched to appropriate requirements. This means that each test case is a separate ticket in Jira with the XRAY test type issue, and appropriate test tickets have been linked to each requirement ticket. 

For example, we have 5 test cases covering Requirement 1:

Now the challenge was how to distribute test cases by functionality and also make test sets from them that may include different features testing at the same test set level. XRAY provides a test repository where folders can be created, the test team decided to use that to divide test cases by the different features they cover. A particular test case belongs to a single folder only.

At the same time, test sets can include test cases from different folders. It can be used to make smoke, and regression test sets for any feature and variations of different features as well. For example, some code changes can cause regression tests in the feature that was changed and smoke tests in adjacent parts. All that can be included in a separate test set and then easily put into a test execution.

Test execution stage

Now we can create test executions from the test sets. By the way, adding test cases to both test sets and test executions requires just a couple of clicks in the test repository.

Once we've got a test execution, we can execute it. XRAY provides a separate window for test execution where each test step can be tracked, commented on, linked to a bug, provided with evidence, etc. The same for the whole test case.

As for the test execution, all its measures are visible from only one page which gives the whole picture to the tester. For example, the statuses of each test run and what bug is associated with the failed ones. It also allows generating test result reports at any point (PDF or Word formatted). This document is configurable and can contain any information the stakeholders need to see. 

It has a lot of benefits but let's get back to the main one we wanted to describe - test traceability.

Test reports: Requirements Traceability Matrix and Test Coverage Report

Once the test execution is completed and assuming that all the test cases were executed at least once, we have enough information to finally see test traceability:

This page displays the overall information starting from the requirements and the tests that they cover and finishing to the bugs created based on the test cases. It’s a sequence of the following instances, from left to right: requirements -> test cases that cover each requirement -> list of test runs for each test case within different test executions the test case was included to, with test runs statuses -> bugs that are linked to each test run. 

All of that is just on one page!

For example, we can see that the TRAC-36 test case failed within the TRAC-50 test execution causing the TRAC-59 bug, and passed within the latest TRAC-61 test execution which means the TRAC-59 bug has been fixed. It means that the TRAC-7 requirement is OK now. 

We can also see that the TRAC-5 requirement is NOT OK because the TRAC-33 test case failed twice within both TRAC-61 and TRAC-50 test executions with the same TRAC-51 bug:

Of course, the test traceability report can be filtered by whatever information we need.

Apart from this report, we can generate the test coverage report which shows the percentage of OK and NOT OK requirements with the number of tests passed/failed within them: 

It is also possible to manage test environments and execute test cases in different sorts of them, manage preconditions for tests as separate tickets, add specific types for automation-related tickets, and so on, but all of that is worth one more article. 

To track test traceability on an existing project, it takes time to link everything together and this time depends on the project size. But once it has been done, it doesn't require a lot of time to maintain test documentation and perform test activities. The project size can be huge, but if it’s structured well in terms of test documentation based on the examples above, there will be a lot of benefits while managing it.

Conclusion

So, eventually, we achieved the following: 

• Increased product quality by putting the requirements in order and strictly clarifying them;
• Increased product quality by finding additional issues and fixing them;
• Increased project work quality by making sure all the tickets can't get lost or go unnoticed and unassociated with anything;
• Reduced time of searching for existing information in Jira, overall simplifying of processes on the project.