Types & Levels of testing classification. Where's the source of truth?

Having another discussion around different classification approaches for types and levels of testing recently, I've decided to outline my understanding of the subject here.

Currently the most popular and world-wide recognized professional certification system for QAs is based on ISTQB, having starting point at Foundation Level. So let’s jump from here.

Further.
Last year the version 2018 of ISTQB Foundation Level (hereinafter “the Syllabus 2018”) has been introduced and put into action.

Going on..
ISTQB 2018 uses characteristics of software quality from ISO/IEC 25010 standard (source: page 40 of the Syllabus 2018)
And deeper, with respect to ISO 25010 the classification of software product quality characteristics and, consequently, testing types refers to this:

Product quality model defined by 8 quality characteristics with respect to ISO/IEC 25010

Source is ISO/IEC 25010 standard.

 

Here let me step out and clarify one thing: ISTQB declared correspondence to ISO-25010 standard only in 2018, putting in action new version of syllabus. Interesting here is that by itself 
ISO-25010 has replaced its predecessor (ISO-9126 standard), it actually means that for some 7 years (2011-2018) ISTQB have being stayed based on an outdated quality model. Not a big deal (or it is..), just you'd better know about it now.

 

Coming back to ISTQB, you can find types of testing on page 39+ of the Syllabus 2018.

In their turn, levels of testing are outlined on page 30+ of the Syllabus 2018.

Another frequently discussed point is deeper classification of integration testing level.
Sub-levels (being more specific: strategies) of Integration testing are outlined in “Specific approaches and responsibilities” on p. 34 the Syllabus 2018.

That's easy.

Definitely, there are alternative approaches. One of my favorite is spotlighted in SWEBOK v.3.0.
Though - again - so far only ISTQB is the only well recognized certification for testers all over the world and I believe trustworthy enough to be the single source of truth.

How to clear tokens in jMeter

Flushing all cookies / tokens in jMeter

When launching a chain of experiments or jobs in jMeter project you might require to flush all the tokens from the previous step. To make it exactly when you need it, I suggest using simple Java script withing JSR223 processors.
Use PreProcessor if you want to remove tokens at the beginning of a job,
or PostProcessor for flushing all tokens in the end of some process.

1. Add JSR223 Pre- or PostProcessor:

2. Set language to what you like (in my case - java);

3. Provide your code into Script field. E.g. let's use CookieManager in java:

import org.apache.jmeter.protocol.http.control.CookieManager;

CookieManager cManager = sampler.getCookieManager();
    int count = cManager.getCookieCount();
    for (int index = 0; index < count; index++) {
        cManager.remove(0);
        }

Here it is. That will remove all the tokens including, for example, JSESSIONID token.

jMeter. Gettings variables and their values

This piece of advice came from one of discussions on StackOverflow. Seems to be helpful for some tasks, so putting it here.

Sometimes its extremely valuable to have a chance to pull from the jMeter project all the pre-defined variables and their values. These cases are - but not limited to - exploration existent big jMeter project, debugging issues, intermediate checks of changes made to a project and so on. 

I used to get Set of vars right through the code (variant with Java code in JSR223 PostProcessor):

1. Add "JSR223 PostProcessor" by right click wherever you need to check jMeter variables in your project:

2. Set Language, in my case - to java;

3. Add following code to Script window:

import java.util.Map;
String jMeterVars;
jMeterVars = "Quantity of vars: " + vars.entrySet().size() + ".\n";
jMeterVars += "[VARIABLE NAME] ==>> [VARIABLE VALUE]\n";
for (Map.Entry entry : vars.entrySet()) {
    jMeterVars += entry.getKey() + " ==>> " + entry.getValue().toString() + "\n";
    }
    try {
        FileWriter fw = new FileWriter("D:\\jMeterVarsForStackOverflow.txt",true);
        fw.write(jMeterVars);
        fw.close();
        } catch(IOException ioe) {
            System.err.println("IOException: " + ioe.getMessage());
            }

5. Check that everything in the JSR223 PostProcessor looks like that:

6. Start your project in jMeter. The code above will create jMeterVarsForStackOverflow.txt file at root of D: and put all variables there:

Charles setup and root certificate installation

Charles Setup

Installation

1. Download an up-to-date version for your system on charlesproxy.com/download/
    
2. Launch the installation process with respect to your operational system;
    
3. Follow the instructions till completion of the installation process.

First launch

NOTE:in order to use Charles, user’s PC or Laptop should have WiFi network module (in other words – should be compatible to connect to internet via WiFi). If you use a PC without WiFi network module, you should upgrade your hardware respectively. E.g. install WiFi PCI card or WiFi USB adapter.

1. If you are connected to internet via WiFi go to the next step.
If not (e.g. you’re connected via Ethernet), please connect to WiFi either. You may keep both Ethernet and WiFi. When properly connected on Windows you see similar situation:

         

2. Launch the Charles.

NOTE: right after the launching Charles will try to catch interactions between operational system and external network. User will not be able to see any data in Charles because a certificate is not installed yet. At the same time user will not be able to use internet.
To avoid such a situation, turn off windows proxy for a while. To do so use combination Ctrl+Shif+P or click on Proxy in toolbar and untick Windows Proxy:

          

3. In order to log and see any exchange thrown via Charles, we need to proxy all possible locations.
Go to Proxy / SSL Proxying Settings, click on Add, type Wildcard * for both Host and Port fields. Click OK in Edit Location popup and OK again in SSL Proxying Settings popup.

         

Setup root certificate

1. Go to Help / SSL Proxying / Install Charles Root Certificate:
   
2. Click on Install Certificate…, keep Current User as a Store Location, click on Next:
   
3. Keep Automatically select the certificate store… as a location and click on Next;
4. Click on Finish in Completing the Certificate Import Wizard;
5. Click on OK in The import was successful popup.

Charles debugging proxy application. Introduction

This article is a starting point for thread devoted to great tool - Charles Debugging Proxy Application.
Please note: all links herein are direct and safe, not hide any promo-keys, referrals or ads.I just really enjoy using this tool and happy to share information about it.

General information

What is the purpose of Charles

Charles is a web debugging application designed for intercepting and logging the traffic between a client and a server. In general, such an information helps to investigate:

• where (to which hosts) does the client send the requests;
• what is an actual header and content of the requests;
• what is the server’s response;
• what content does the response have;
• when does all this happen (what is a sequence of requests-responses in time).

What does Charles consist from

Charles is a single desktop application. It can be used standalone (w/o network connection) e.g. for reading and analyzing logs, which were saved previously. Application is available in versions for MS Windows (32 and 64 bit), MacOS and Linux (32 and 64 bit).
Add-on for Mozilla Firefox browser is also available. It helps to configure Firefox automatically for been used with Charles.
Charles also has a web-based interface. Literally it means that user may observe Charles as a web page in browser. That may be extremely useful for automation test purposes via Selenium Driver.

How does it work

Briefly – Charles comes an intermediate chain between a client (desktop browser, iOS native application, mobile browser in Android) and a remote server.

Charles uses trust certificates, which allows not only “listen” the exchange between client and server, but also “understand” most of the transmitted information and show it to a user in multiple friendly views.

How can I get and use it

Charles has a full functional trial version. The limitation of trial version is session length not longer than 30 minutes. Also during work in trial a user faces warnings several times per session which lasts several second.
And one more thing, the trial version has a warning that trial will expire in 30 days. Actually for all Charles versions up to 4.1.2 it never happens.
There are no other limitations in using non-registered version of Charles.
Download an up-to-date version for your system on www.charlesproxy.com/download/
Check with your local network administrator or department manager if you have a corporate license key.
To buy a license key, please refer to www.charlesproxy.com/buy/
Charles authors are constantly improving their product and they are doing great! So, it’s really appreciated to buy a license:

Add-on for Firefox is available here: addons.mozilla.org/en-US/firefox/addon/charles-proxy/
Charles is developed by XK72. In case of any questions the developers and founders of Charles may be reached at www.xk72.com/ or via support section of www.charlesproxy.com .
Instructions on installation of the program and setting up a root certificate are available in the Next articles.

Severity, or Priority, or both?

Hi dear readers -

Following thoughts came directly from discussion with @asavin started from this article about necessity or redundancy of both - Severity and Priority - attributes of tasks / defects / bug reports.

My humble opinion stayed with classics - both attributes are required. Let me try to substantiate why.

1. Dynamic world

Severity and Priority are in different planes by nature - with time one attribute may change it's value, when another - stay stable. Or both might change. In this case combination of Severity and Priority (S x P) can better reflect dynamically changed environment around us.
E.g. some financial system has a critical bug related to switching a calendar year.
Severity is S1, sure.
But what about priority? It's P4 in January 1, but gradually raises till P1 till the end of the year coming a show-stopper on December 31.
Thus we can track this ticket and easily prioritize it among others in a backlog:
It was S1 x P4 - not so urgent to spend time on that those days, but when it's S1 x P1 - let's fix it finally.
Of course it's a rough example, but representative, I hope :)

2. Sorting the mass

This side is both - not applicable for a very small backlogs but should be considered for large projects with a lot of tickets in To Do lists.
Simple: assuming that we have only one attribute, let's say Priority, and this attribute is of 4 levels (P1 - to - P4) we are bucketing all the tickets in only 4 sets. Thus we should always think about - what to take first among all these tasks with P1. And then the same about tasks with P2 and so on.
But having two attributes with 4 levels each (P1 - to - P4 for Priority and S1 - to - S4 for Severity) we are sorting all the tickets in 16 (!) buckets.
And than it might be considerably simpler to make a decision: what should be done earlier and what may be put aside.
 

I would suggest using the following prioritization for S x P combinations:

• P1 & S1 - to be done ASAP;
• P1 & S2;
• P1 & S3;
• P1 & S4;

---

• P2 & S1;
• P2 & S2;
• P2 & S3;
• P2 & S4;

---

• P3 & S1;
• P3 & S2;
• P3 & S3;
• P3 & S4;

---

• P4 & S1;
• P4 & S2;
• P4 & S3;
• P4 & S4 - to be done last or never...

Or maybe it is more comprehensible with such an image:

Severity x Priority decision matrix

Sorry, I'm not a paint-guru at all :) But hope you've got my point.
I would appreciate any suggestions, advice or critics in comments or via personal e-mails.

And thank you for reading till this very point :)

Lots of testing without defects found. Good? Bad? To be improved?

I've wrote this after having one more (who knows what's the count) talk with colleagues and managers about testers' purpose. Here's what I think:

Testers and - even worse - non-testers around, e.g. managers, tend to forget that testing is not only looking for bugs, but also its a verification that some parts of a system have no defects.
 

Case to discuss: tester spent 90% of time and found nothing but the least 10% gave her/him showstoppers and multiple defects. What does it mean?

Sometimes in this case I hear that testing was wrong and those 90% of time was redundant, just wasting of time..
Especially these talks can be escalated if something (maybe really severe or absolutely not) was found by somebody in production.
Or by managers who just had a look and caught an interesting issue. "Oh guys! Are you testing something there at all or what?" 

Sure we do :)
 

One of the best managers I used to work with usually asks his testers: "what is the level of your confidence that the system is good to go?"
Ha!? That is the clue for 90% of time spent on testing when no bugs are found! These 90% raise the level of confidence in a product. That what it means.
But lets come back to the case to discuss in the beginning. Sure we should be fair here and go through an analysis. There is always place to improve the situation:

1. Were all of those 90% really necessary or the next time something can be skipped?
2. Consider better using of already used test design technics or involving of those which are not used for now;
3. Extend covering of testing scope by automation tests. If you have no automation on the project - maybe it's the best time to start it.
4. Unit test go here too, they could help you to avoid really amazing sets of end-to-end tests.
   
5. How can we shift finding those bugs from 10% of time to the earliest possible point of time in testing?
   
6. More and better communication with developers usually helps to reveal where defects could be;
7. Ask your architects and developers about the level of risk of particular changes or bug-fixes;
8. Always test new features or improvements first. Even if they seam to be very simple;
9. Try to check and improve overall development ==>> testing process. Maybe something can be split into pieces and thus go faster to Ready for QA. Or maybe some parts of processes are just a legacy bureaucracy and are to be excluded.
10. Refresh 7 basic principles of testing in your memory. Try to use all of them.
    
11. Haven't we lost something really important because of those 90%?
    
12. Even if you went to production and everything seems to be working fine, constantly gather a feedback from a team, users and Customers if something was missed or not. In a good communication atmosphere you should receive a fair and emotionless feedback constantly and immediately. But who know if it's true in you situation. Simply ask more.
13. P - stands for Prioritization of the project parts here. Try to slice your project into pieces by a risk level:
    --- aha changes in this part are very risky and can have a very big impact (e.g. pieces with legacy code on old architecture, or some things which were not properly tested by some reason);
    --- and if something is changed here nothing extraordinary should happen (that could be true for something recently developed and well known, as well as for some parts which were properly tested, well documented and got a great feedback from all members of process).

Well, that's it from the top of my head regarding correlation of time spent on no-defect-testing and full-strike tiny moments :)