TestOps (or test operations) refers to the discipline of managing the operational aspects of testing within the software delivery lifecycle.
Software testing is an evolving discipline that includes both functional and non-functional testing. The first mention of this is regarded in March 2019 where, Ditmir Hasani, CEO of QA Tech consultancy is thought to have coined the term.[1] Increasingly, software testing, especially in agile development processes is shifting to become more of a continuous testing process where software developers, quality engineers, manual testers, product owners, and more are involved in the quality process.[2] As more people have become involved in the testing process and testing projects have grown, so too has the need to advance the discipline of software testing management and to manage the software quality processes, programmers, people, systems, and tests.[3]
TestOps helps teams scale their teams, tests, and quality processes in and effectively.
Elements of test operations (TestOps)
editTestOps involves several important disciplines that can be broken down into:
Planning — Helps identify how the software is going to be tested. What are the priorities for testing? How will it be tested? Who will do the testing? In addition, the planning phase should identify the environment for the tests. Will they be run in a test environment, or in production? Production data can be valuable to identify real user flows that help prioritize test cases.[4][5] The outputs include identifying the type of tests to use, the test automation tools, the timing of the testing, the ownership of the testing at different phases, and the design and outputs of the tests.
Management — Test management includes the organization and governance of the team, the tools, the test environment, and the tests themselves. Tests follow a lifecycle pattern and must be managed whether they are in stages such as draft, active, or quarantine.[6] TestOps helps ensure that the testing processes are efficient and scalable.
Control — As teams and tests grow in number and diversity, they naturally increase complexity. Change control processes such as pull requests, approvals on merges, collaboration tools, and ownership labeling can help ensure that changes to tests are properly approved.[7]
Insights — The data and information you derive from your test automation systems should help inform operational as well as process transformation decisions. Operational information about your testing activities includes the test results (pass/fail), release readiness criteria, failure diagnostics, team testing productivity, test stability, and more.[8] Information that can inform process improvements includes team and individual performance data, failure type trend analysis, test coverage mapping, and risk coverage analysis.[9][10]
TestOps Features
editDevOps integration — TestOps exists to ensure that the product development pipeline has all the testing frameworks and tools needed. It is common for QA engineers to rely on the pipelines that IT puts together without much input. TestOps changes this by owning test activities related to DevOps, allowing QA engineers and developers to have full ownership and visibility of the development pipeline, so they can tailor it to meet their needs.
Cloud integration — Integrating tests and test runs with cloud providers
Enhanced test planning — Automation is not effective if the entire codebase has to get tested every time a line of code is changed. TestOps provides a centralized platform that makes it easier for testers and developers to plan what tests to write, as well as identify what tests to run and when.
Test lifecycle management — Automated tests follow a lifecycle including creation, evaluation, active, and quarantine or removal. The status of the test should impact how it is treated in build automation systems like CI/CD.
Test version control — Processes that help ensure that changes to tests are properly reviewed and approved through capabilities like pull requests in code.
Real-time dashboards — Real-time results and status help the test teams understand the state of software releases and the work that needs to be done to create, approve, or run more tests.
DevOps is a broader, more inclusive concept that includes software feature planning, code development, testing, deployment, configuration, monitoring, and feedback. It was an attempt to integrate some of the disconnected toolchains. Testing is included in the broader DevOps methodology.
TestOps[12] is not simply providing additional emphasis on testing. It is focused on operational aspects of testing that are necessary to ensure that testing, whether performed in development, production, or in its own testing phase, is well planned, managed, controlled, and provides insights to enable continuous improvement.
The siloed working mindset is aimed to be removed from activities such as continuous delivery, software testing (manual and automated testing), environment setups, infrastructure and log management, and built-in security enforcement.
References
edit- ^ Hasani, Ditmir (2 September 2024). "DevOps and the emergence of TestOps!". DevOps Online. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ "Who Should Test Software?". Istqb Exams at Home Worldwide - Astqb - Istqb in the U.S.
- ^ "What is Test Management?". ProfessionalQA. 13 April 2020. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Ying, PuLe; Fan, LingZhi (June 2017). Methods For Test Case Prioritization Based On Test Case Execution History (Masters thesis). Blekinge Institute of Technology. Retrieved 2 September 2024.
{{cite thesis}}: CS1 maint: multiple names: authors list (link) - ^ "What is UX Research?". Interaction Design Foundation. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Vijay (9 March 2024). "What is Software Testing Life Cycle (STLC)?". Software Testing Help. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Kaitano, Francis (March 1, 2007). "Change Control Audits - A Must for Critical System Functionality". Archived from the original on 2021-04-30. Retrieved 2021-04-30.
- ^ Alam, S. M. Didar Al; Pfahl, Dietmar; Ruhe, Günther (2016), "Monitoring and Controlling Release Readiness by Learning Across Projects", Managing Software Process Evolution, Cham: Springer International Publishing, pp. 281–303, doi:10.1007/978-3-319-31545-4_14, ISBN 978-3-319-31543-0, retrieved 2022-11-04
- ^ "11 Test Automation Metrics and their Pros & Cons". SeaLights. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Garrett, Thom. "Useful Automated Software Testing Metrics" (PDF). ResearchGate. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ "DevOps and the emergence of TestOps!". DevOps Online. April 2019. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link) - ^ Colantonio, Joe (15 July 2021). "What is TestOps (How Microsoft Does it)". Test Guild. Retrieved 2 September 2024.
{{cite web}}: CS1 maint: url-status (link)