{"id":733,"date":"2016-06-20T15:15:28","date_gmt":"2016-06-20T05:15:28","guid":{"rendered":"http:\/\/www.software-testing.com.au\/blog\/?p=733"},"modified":"2016-06-20T15:15:28","modified_gmt":"2016-06-20T05:15:28","slug":"how-to-write-a-test-strategy","status":"publish","type":"post","link":"http:\/\/www.software-testing.com.au\/blog\/2016\/06\/20\/how-to-write-a-test-strategy\/","title":{"rendered":"How to write a test strategy"},"content":{"rendered":"<p>I&#8217;ve documented my overall <a href=\"http:\/\/www.software-testing.com.au\/blog\/2009\/07\/21\/thinking-about-test-strategy-a-mnemonic-device\/\">approach to rapid, lightweight test strategy <\/a>before but thought it might be helpful to post an example. \u00a0If you haven&#8217;t read the original post above, see that first.<\/p>\n<p>This is the a sanitised version of the first I ever did, and while there are some concessions to enterprise concerns, it mostly holds up as a useful example of how a strategy might look. \u00a0I&#8217;m not defending this as a good <em>strategy<\/em>, but I think this worked as a good document of an agreed approach.<\/p>\n<h1>Project X Test Strategy<\/h1>\n<h2>Purpose<\/h2>\n<p>The purpose of this document is \u2013<\/p>\n<ul>\n<li>To ensure that testing is in line with the business objectives of Company.<\/li>\n<li>To ensure that testing is addressing critical business risks.<\/li>\n<li>To ensure that tradeoffs made during testing accurately reflect business priorities<\/li>\n<li>To provide a framework that allows testers to report test issues in a pertinent and timely manner.<\/li>\n<li>To provide guidance for test-related decisions.<\/li>\n<li>To define the overall test strategy for testing in accordance with business priorities and agreed business risks.<\/li>\n<li>To define test responsibilities and scope.<\/li>\n<li>To communicate understanding of the above to the project team and business.<\/li>\n<\/ul>\n<h2>Background<\/h2>\n<p>Project X release 2 adds reporting for several new products, and a new report format.<\/p>\n<p>The current plan is to add metric capture for the new products, but not generate reports from this data until the new reports are ready.\u00a0 Irrespective of whether the complete Project X implementation is put into production, the affected products must still be capturing Usage information.<\/p>\n<h2>Key Features<\/h2>\n<ul>\n<li>Generation of event messages by the new products (Adamantium, DBC, Product C, Product A and Product B.<\/li>\n<li>New database schema for event capture and reporting<\/li>\n<li>New format reports with new fields for added products<\/li>\n<li>Change of existing events to work with the new database schema<\/li>\n<\/ul>\n<h2>Key Dates<\/h2>\n<ul>\n<li>UAT &#8211; <ins datetime=\"2007-03-06T11:18\">Mon 26\/03\/07<\/ins> to Tue <ins datetime=\"2007-03-06T11:18\">10\/04\/07<\/ins><\/li>\n<li>Performance\/Load \u2013 <ins datetime=\"2007-03-06T11:18\">Wed 28\/03\/0<\/ins>7 to <ins datetime=\"2007-03-06T11:18\">Wed 11\/04\/07<\/ins><\/li>\n<li>Production &#8211; <ins datetime=\"2007-03-06T11:18\">Thu 12\/04\/07<\/ins><\/li>\n<\/ul>\n<h2>Key Risks<\/h2>\n<table border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"550\">\n<tbody>\n<tr>\n<td width=\"150\" valign=\"top\">Risk<\/td>\n<td width=\"150\" valign=\"top\">Impact<\/td>\n<td width=\"150\" valign=\"top\">Mitigation   Strategy<\/td>\n<td width=\"100\" valign=\"top\">Risk area<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">The team\u2019s domain knowledge of applications being modified is weak or   incomplete for key products.<\/td>\n<td width=\"152\" valign=\"top\">Impact of changes to existing products may be misjudged by the   development team and products adversely affected.<\/td>\n<td width=\"206\" valign=\"top\">\n<ol>\n<li>Product C team to modify their application   to create the required messages and perform regression testing.<\/li>\n<li>Product B application will not be   modified.\u00a0 Only the logs will be   processed<\/li>\n<li>Product A has good selenium and unit test   coverage, and domain skills exist in the team<\/li>\n<li>Greater focus will be placed on creating   regression tests for PRODUCT D and leveraging automated QTP scripts from   other teams.<\/li>\n<\/ol>\n<\/td>\n<td width=\"98\" valign=\"top\">Project<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Data architect being replaced.<\/td>\n<td width=\"152\" valign=\"top\">Supporting information that is necessary for generating reports may   not be captured.\u00a0 There may be some   churn in technical details.<\/td>\n<td width=\"206\" valign=\"top\">\n<ol>\n<li>Development team has improved domain   knowledge from first release<\/li>\n<li>Intention is to provide improved technical   specifications and mapping documents<\/li>\n<li>Early involvement of reporting testers to   inspect output and provide up-front test cases<\/li>\n<\/ol>\n<\/td>\n<td width=\"98\" valign=\"top\">Project<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Strategy for maintaining version 1 and version 2 of the ZING database   in production has not been defined.<\/td>\n<td width=\"152\" valign=\"top\">Test strategy may not be appropriate.<\/td>\n<td width=\"206\" valign=\"top\">None.<\/td>\n<td width=\"98\" valign=\"top\">Project<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Insufficient time to test all events and perform regression testing of   existing products.<\/td>\n<td width=\"152\" valign=\"top\">Events not captured, not captured correctly, or applications degraded   in functionality or performance<\/td>\n<td width=\"206\" valign=\"top\">\n<ol>\n<li>Early involvement of reporting testers to   inspect output and provide up-front test cases (defect prevention)<\/li>\n<li>Additional responsibility of developers to   write dbUnit tests.<\/li>\n<\/ol>\n<p>These two   activities will free testers to focus on QTP regression scripts.<\/td>\n<td width=\"98\" valign=\"top\">Project<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Technical specifications and mapping documents not ready prior to   story development<\/td>\n<td width=\"152\" valign=\"top\">Mappings may be incorrect.\u00a0 Test   to requirement traceability difficult to retrofit.<\/td>\n<td width=\"206\" valign=\"top\">Retrofit   where time permits.\u00a0 Business to   determine value of this activity.<\/td>\n<td width=\"98\" valign=\"top\">Project<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">XML may not be correctly transformed<\/td>\n<td width=\"152\" valign=\"top\">Incorrect data will be collected<\/td>\n<td width=\"206\" valign=\"top\">Developers   will use dBUnit to perform integration testing of XML to database   mapping.\u00a0 This will minimise error in   human inspection.<\/td>\n<td width=\"98\" valign=\"top\">Product<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Usage information may be lost.\u00a0   It is critical that enough information be captured to relate event   information to customers, and that the information is correct.<\/td>\n<td width=\"152\" valign=\"top\">No Usage information available.<\/td>\n<td width=\"206\" valign=\"top\">Alternate   mechanisms exist for capturing information for Product A and Product B.\u00a0 Product B needs to implement a   solution.\u00a0 Regression testing needs to   ensure that existing Product D events are unaffected.<\/td>\n<td width=\"98\" valign=\"top\">Product<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">No robust and comprehensive automated regression test suite for PRODUCT   D components.\u00a0 May not be time to   develop a full suite of QTP tests for all events and field mappings.<\/td>\n<td width=\"152\" valign=\"top\">PRODUCT D regressions introduced, or regression testing of Product D   requires extra resourcing.<\/td>\n<td width=\"206\" valign=\"top\">Will   attempt to leverage PRODUCT D scripts from other projects and existing   scripts, while extending the QTP suite.<\/td>\n<td width=\"98\" valign=\"top\"><\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Project X changes affect performance of existing products<\/td>\n<td width=\"152\" valign=\"top\">Downtime of Product D products and\/or loss of business.<\/td>\n<td width=\"206\" valign=\"top\">Performance   testing needs to cover combined product tests and individual products   compared to previous benchmark performance results<\/td>\n<td width=\"98\" valign=\"top\">Parafunctional<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Project X may affect products when under stress<\/td>\n<td width=\"152\" valign=\"top\">Downtime of Product D products and\/or loss of business.<\/td>\n<td width=\"206\" valign=\"top\">Volume   tests should simulate large tables, full disks and overloaded queues to see   impact to application performance.<\/td>\n<td width=\"98\" valign=\"top\">Parafunctional<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Reliability tests may not have been performed previously.\u00a0 That is,<\/p>\n<p>tests that all events were captured under load. (I need to confirm   this)<\/td>\n<td width=\"152\" valign=\"top\">Usage information may be going missing<\/td>\n<td width=\"206\" valign=\"top\">Performance   testing should include some database checks to ensure all messages are being   stored<\/td>\n<td width=\"98\" valign=\"top\">Historical<\/td>\n<\/tr>\n<tr>\n<td width=\"150\" valign=\"top\">Unable to integrate with the new reports prior to release of new event   capturing.<\/td>\n<td width=\"152\" valign=\"top\">Important data may not be collected or data may not be suitable for   use in reports.<\/td>\n<td width=\"206\" valign=\"top\">\n<ol>\n<li>Production   data being collected after deployment needs to be monitored.<\/li>\n<li>Output of   transformations to be inspected by reports testers.<\/li>\n<li>Domain   knowledge of developers is improved.<\/li>\n<\/ol>\n<\/td>\n<td width=\"98\" valign=\"top\">Product<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Project X Strategy Model<\/h2>\n<p><a href=\"http:\/\/www.software-testing.com.au\/blog\/wp-content\/uploads\/2016\/06\/architecture.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-734\" title=\"High-level architecture\" src=\"http:\/\/www.software-testing.com.au\/blog\/wp-content\/uploads\/2016\/06\/architecture.png\" alt=\"High-level architecture of application under test showing key interfaces and flows\" width=\"667\" height=\"333\" srcset=\"http:\/\/www.software-testing.com.au\/blog\/wp-content\/uploads\/2016\/06\/architecture.png 667w, http:\/\/www.software-testing.com.au\/blog\/wp-content\/uploads\/2016\/06\/architecture-300x149.png 300w\" sizes=\"auto, (max-width: 667px) 100vw, 667px\" \/><\/a><\/p>\n<p>The diagram above defines the conceptual view of the components for testing.\u00a0 From this model, we understand the key interfaces that pertain to the test effort, and the responsibilities of different subsystems.<\/p>\n<h3>Products (Product D, Product A, Product C, Product B)<\/h3>\n<h4>Capabilities<\/h4>\n<ul>\n<li>Generate events<\/li>\n<\/ul>\n<h4>Responsibilities<\/h4>\n<ul>\n<li>Event messages should be generated in response to the correct user actions.<\/li>\n<li>Event messages should contain the correct information<\/li>\n<li>Event message should generate well-formed XML<\/li>\n<li>Error handling?<\/li>\n<\/ul>\n<h3>SCE<\/h3>\n<h4>Capabilities<\/h4>\n<ul>\n<li>Receive events<\/li>\n<li>Pass events to the ZING database<\/li>\n<\/ul>\n<p><strong> <span style=\"font-size: 1em;\">Responsibilities<\/span><\/strong><\/p>\n<ul>\n<li>Transform event XML to correct fields in ZING database for each event type<\/li>\n<li>Error handling?<\/li>\n<\/ul>\n<h3>Reports<\/h3>\n<h4>Capabilities<\/h4>\n<ul>\n<li>Transform raw event information into aggregate metrics<\/li>\n<li>Re-submit rejected events to ZING<\/li>\n<li>Generate reports<\/li>\n<\/ul>\n<h4>Responsibilities<\/h4>\n<ul>\n<li>Correctly generate reports for event data which meets specifications<\/li>\n<li>Correct data and re-load into ZING.<\/li>\n<\/ul>\n<h3>Interfaces<\/h3>\n<h4>Product to SCE<\/h4>\n<p>This interface will not be tested in isolation.<\/p>\n<h4>SCE to ZING<\/h4>\n<p>Developers will be writing dBUnit integration tests, which will take XML messages and verify that the values in the XML are mapped to the correct place in the ZING database.<\/p>\n<h4>ZING to Reports<\/h4>\n<p>The reporting component will not be available to test against, and domain expertise may not be as strong as for previously releases with the departure of senior personnel.\u00a0 Available domain experts will be involved as early as possible to validate the contents of the ZING database.<\/p>\n<h4>Product to ZING<\/h4>\n<p>System testing will primarily focus on driving the applications and ensuring that \u2013<\/p>\n<ul>\n<li>Application\u2019s function is unaffected<\/li>\n<li>Product generates events in response to correct user actions<\/li>\n<li>XML can be received by SCE<\/li>\n<li>Products send the correct data through<\/li>\n<\/ul>\n<h2>Key testing focus<\/h2>\n<ul>\n<li>Ensuring existing event capture is unaffected (PRODUCT D).<\/li>\n<li>Ensuring event details correctly captured for systems.\u00a0 This is more critical for systems in which there is currently no alternative capture mechanism (Product C, Adamantium, DBC).\u00a0 Alternative event capture mechanisms exist for Product B and Product A.<\/li>\n<li>Ensuring existing system functionality is not affected.\u00a0 Responsibility for Product C\u2019s regression testing will lie with Product C\u2019s team.\u00a0 There is no change to the Product B application, but sociability testing may be required for log processing.\u00a0 Product A has an effective regression suite (selenium), so the critical focus is on testing of PRODUCT D functionality.<\/li>\n<\/ul>\n<h2>Test prioritisation strategy<\/h2>\n<p>These factors guide prioritisation of testing effort:<\/p>\n<ul>\n<li>What is the application\u2019s visibility?\u00a0 (ie. Cost of failure)<\/li>\n<li>What is the application\u2019s value? (ie. Revenue)<\/li>\n<\/ul>\n<p>For the products in scope, cost of failure and application value are proportional.<\/p>\n<p>There may be other strategic factors as presented by the business as we go, but the above are the primary drivers.<\/p>\n<p>Priority of products \u2013<\/p>\n<ol>\n<li>Adamantium\/Product D<\/li>\n<li>Product C<\/li>\n<li>Product A<\/li>\n<li>Product B<\/li>\n<li>DBC<\/li>\n<\/ol>\n<p>Within Product D, the monthly Usage statistics show the following \u2013<\/p>\n<ul>\n<li>97% of searches are business type or business name searches.<\/li>\n<li>3% of searches are browse category searches<\/li>\n<li>Map based searches are less than 0.2% of searches<\/li>\n<\/ul>\n<h2>Test design strategy<\/h2>\n<h3>Customer (Acceptance) Tests<\/h3>\n<p>For each event, test cases should address:<\/p>\n<ul>\n<li>Ensuring modified applications generate messages in all expected situations.<\/li>\n<li>Ensuring modified applications generate messages correctly (correct data and correct XML).<\/li>\n<li>Ensuring valid messages can be processed by SCE.<\/li>\n<li>Ensuring valid messages are transformed correctly go to the specified database fields.<\/li>\n<li>Ensuring data in the database is acceptable for reporting needs.<\/li>\n<\/ul>\n<h3>Regression Tests<\/h3>\n<p>For each product where event sending functionality is added:<\/p>\n<ul>\n<li>All other application functionality should be unchanged<\/li>\n<\/ul>\n<p>Additionally, the performance test phase will measure the impact of modifications to each product.<\/p>\n<h3>Risk Factors<\/h3>\n<p>These tests correspond to the following failure modes \u2013<\/p>\n<ul>\n<li>Events are not captured at all.<\/li>\n<li>Events are captured in a way which renders them unusable.<\/li>\n<li>Systems whose code is instrumented to allow sending of events to SCE are adversely affected in their functionality.<\/li>\n<li>Event data is mapped to field(s) incorrectly<\/li>\n<li>Performance is degraded<\/li>\n<li>Data is unsuitable for reporting purposes<\/li>\n<\/ul>\n<h2>Team Process<\/h2>\n<p>The development phase will consist of multiple iterations.<\/p>\n<ol>\n<li>At the beginning of each iteration, the planning meeting will schedule stories to be undertaken by the development team.<\/li>\n<li>The planning meeting will include representatives from the business, test and development teams.<\/li>\n<li>The goal of the planning meeting is to arrive at a shared understanding of scope for each story and acceptance criteria and record that understanding via acceptance tests in JIRA.<\/li>\n<li>Collaboration through the iteration to ensure that stories are tested to address the business needs (as defined by business representatives and specifications) and risks (as defined by business representatives and agreed to in this document).\u00a0 This may include testing by business representatives, system testers and developers.<\/li>\n<li>The status of each story will be recorded in JIRA.<\/li>\n<\/ol>\n<p>When development iterations have delivered the functionality agreed to by the business, deployment to environments for UAT and Performance and Load testing will take place.<\/p>\n<h2>Deliverables<\/h2>\n<h3>High priority<\/h3>\n<ul>\n<li>QTP regression suite for PROJECT X events (including Adamantium, DBC) related to business type and business name searches<\/li>\n<li>Test summary report prior to go\/no go meeting<\/li>\n<\/ul>\n<h3>Secondary priority<\/h3>\n<ul>\n<li>QTP regression suite for Product A (Lower volume, fewer events and the application already collects metrics).\u00a0 Manual scripts and database queries will be provided in lieu of this.<\/li>\n<\/ul>\n<h3>Other<\/h3>\n<ul>\n<li>Product C should create PROJECT X QTP regression tests as part of their development work<\/li>\n<li>Product B test suite will likely not be a QTP script as log files are being parsed as a batch process.\u00a0 GUI regression scripts will be suitable when Product B code is instrumented to add event generation.\u00a0 If time permits, we will attempt to develop a tool to parse a log file and confirm that the correct events were generated.<\/li>\n<\/ul>\n<h2>To do<\/h2>\n<ul>\n<li>Confirm strategy with stakeholders<\/li>\n<li>Confirm test scope with Product C testers<\/li>\n<li>Confirm events that are in scope for this release<\/li>\n<li>Define scope of Product D testing and obtain Product D App. Sustain team testers for regression testing.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve documented my overall approach to rapid, lightweight test strategy before but thought it might be helpful to post an example. \u00a0If you haven&#8217;t read the original post above, see that first. This is the a sanitised version of the first I ever did, and while there are some concessions to enterprise concerns, it mostly [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,4,5,9,31,35],"tags":[],"class_list":["post-733","post","type-post","status-publish","format-standard","hentry","category-agile","category-agile-testing","category-analysis-skills","category-communication","category-strategy","category-test-management"],"_links":{"self":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/733","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/comments?post=733"}],"version-history":[{"count":17,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/733\/revisions"}],"predecessor-version":[{"id":751,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/posts\/733\/revisions\/751"}],"wp:attachment":[{"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/media?parent=733"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/categories?post=733"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.software-testing.com.au\/blog\/wp-json\/wp\/v2\/tags?post=733"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}