The Truth about Testing?

It takes a lot for me to get riled up, but here I am.

Stuart Reid is doing a keynote at EuroSTAR titled “When Passion Obscures The Facts: The Case for Evidence-Based Testing.”

Here are three things he intends to show:

• How testing ‘evangelists’ use their apparent passion to conceal a lack of evidence supporting their claims
• Which claims are supported by evidence, which are just plain wrong, and which lack real evidence.
• How we should collect metrics to provide evidence to support testing improvements.

To me, these are not articles of scientific inquiry for an honest presentation about the origins and intricacies of controversies in our craft, they are weak opening arguments in a frivolous lawsuit he is bringing against it. 

His argument is that there are rival philosophies of testing (called “schools”) that are misleading you about testing. (Though for what purpose, he does not say).  This talk seems to be about how he will drag these rival, passionate evangelist ne’er-do-wells before the High Council so that he can show how they are obscuring the truth as represented by what he calls “facts” & “evidence”.
First, I identify myself as one of the “passionate evangelists” from one of the schools he is taking to task (the Context-Driven School). Second, I consider myself an advocate for the craft and science called “software testing” and that questions like “is exploratory testing more effective than scripted testing?” need to take a lot of context into account before they can be answered to someone’s satisfaction.  But to say I have the “facts” about controversial testing topics like this framed as “evidence” that can transcend years of controversy would not only be ridiculous, but arrogant and insulting.

But he goes on…

“This presentation will identify which claims are supported by valid evidence, which claims disagree with the available evidence, and those claims where there is currently insufficient evidence to reasonably support a claim one way or the other.”

Did you notice what words he chose to accompany the word “evidence?” — “real”, “valid”, “available”, and “insufficient”. 

According to whom?  You, dear reader? 

Of course not.  You can’t use these words because you don’t know any better.  You’ve been manipulated.  He hasn’t, thank goodness.

His case depends on convincing you how his evidence — obscured to you by people like me [see his title] — finally allows you to sort out six specific software testing controversies that have persisted for years.  How else other than showing you his briefcase full of facts will me and the other svengali evangelists from rival testing schools be exposed for misleading you about these issues? How else, other than seeing his evidence, will you be free once and for all from the polarizing debates we svengalis perpetuate?

I see Reid as a misguided politician-lawyer who needs a big case to get noticed.  He’s hoping you will not be smart enough to see that any premises (and promises) of “evidence” are subjective.  In other words, they need context — the theme of one of the very schools he says is swaying you. 

Is he really the crime-fighting hero, armed with a briefcase that once opened, would settle these testing debates bewteen the rival schools that have been misleading and plaguing gentle, innocent, unsuspecting tester-folk for years?

I think it’s more likely that you’re the jury in this case, knowing that software testing is a challenging intellectual process, not a set of absolute truths held in someone’s briefcase waiting to be laid out for you — especially by someone who doesn’t think it is. 

At least, that’s what the “evidence” of his title and abstract show to me.  The main difference between me and Reid is that my School has taught me that evidence, like in court of law, can be circumstantial.


18 Responses to “The Truth about Testing?”

  1. Joe Strazzere Says:

    Wow! You got bothered this much by an abstract? Perhaps you should wait until he actually gives his presentation before getting so riled up?

    JB: What I saw in the abstract was demeaning to the need in our industry to discuss and debate controversial topics like he listed. We need passionate, educated arguments for and against things like certification to help people know where they stand. Only after a lot of that can we hope to find common ground like the “passionless” want us to. But when I hear “Because I am not burdened by passion, I have the ‘real evidence’ for you about how you should think”, that bugs me, and I reacted.

    Perhaps after the actual presentation, we can hear more about his argument and your counter-argument…

    I strongly suspect I would agree more with your views than with Stuart’s, but I’m afraid that jumping the gun like this may simply add credence to an argument about “passion without evidence”.

    JB: I thought of that. If he uses my blog as ammo, he’d be proving my point that evidence does not speak for itself — it needs interpretation (context). I’m not saying boycott the talk, I’m saying there is evidence (for me, based on what I read, and based on my knowledge about Stuart Reid) that a talk like he described is an attack on our craft because it minimizes testing. To have that come from someone inside our industry is unacceptable to me. The alarm I sounded is for people to scrutnize talks like this when they happen because in my opinion, they do all of us a disservice by saying that testing can be distilled to an objective set of conclusions, perpetuating a belief that it is NOT a craft.


  2. ELizaF Says:

    Usually when I write, it is about the trivial side of software testing like how to explain context in IT using naughty sounding words and a seemingly unrelated list of random characters.

    I think one of the reasons for this is I feel people like Reid get so bogged down in macro-defining what we do and the best ways to do it that I want nothing to do with them.

    In a way, this process obsessed approach detracts me from the very fundemental function of a software tester i.e. to test software. I like working at a rate of 75% ‘sitting down actually doing the testing’ and 25% ‘process improvement’.

    How many defects are actually found by:
    promoting a particuliar testing mindset,
    arguing the case against a particuliar school of testing,
    collecting metrics against any one type of testing approach (or many)
    reading about any of the above

    …. bugger all, that’s how many.

    Right, I am finished being serious.

  3. cowboytesting Says:

    Wow, it sounds like the speaker is attempting to put different philosophies into his context in order to show how they are not appropriate when compared against his views. Eyeglasses don’t work because the room is dark when I turn off the lights, see? This proves it. Throw away your eyeglasses.

  4. Cem Kaner Says:

    Jon, I wrote a follow-up to your post, here

    Joe, you make a good point. Unfortunately, as I’ve seen this in other contexts, anything we say, before or after this presentation, that is critical of this approach can be used to lend credence to an argument about “passion without evidence.” That’s the beauty of adopting “evidence-based” as your debating position.

  5. Joe Strazzere Says:

    Jon, Cem, I understand what you are saying, and pretty much agree with your distaste for the type of approach outlined in the abstract.

    Your opinions on the overall theme have a ton of merit, and deserve more discussion. But attacking a presentation that hasn’t yet happened may actually prevent such a discussion.

    You risk getting thrown into the hopper of “kooks who demand that a book be banned, but admit that they have never read it”. (And yes – I know you haven’t asked that his presentation be banned. Hopefully, you catch my drift here. I don’t want to see you come off as kooks.)

    • banovotz Says:

      The problem with the overview of “schools” begins if the overview is actualy a manifest for one of the schools.

    • banovotz Says:

      Sorry Joe, I meant “manifesto”. English is not my mother tongue. Even if the text starts with intent to discuss different approaches it is clearly taking side. To me it looks like it wants to put an end on the debates mentioned in the first sentence.

  6. Joe Strazzere Says:

    Jon, Cem – I should have asked before – will you be attending EuroSTAR, and have the chance to tell us how this keynote actually comes out live?

  7. Dave Whalen Says:

    What happened to our right to have a different opinion or idea? Why can’t we just say we respectfully disagree and leave it at that. There is no one perfect methodology that applies to every possible application in software testing. I don’t want to over step here but I believe this is one of the basic tenets of the context-driven school is it not?

    JB: He does have a right to a different opinion, but an opinion is not what he’s claiming to present here. That’s my point. His abstract implies that he will prove that people with strong opinions (what he calls “passion”) are obscuring and misleading others. That’s a direct attack. Rhetoric, I know, but it’s foolhardy and diminuitive rhetoric which I think needs to be challenged. I’m encouraging others to do so, too.

  8. Cem Kaner Says:

    Joe, I rarely get to the STAR conferences. The meetings allow only a few minutes for post-talk questions — no time for real discussion or debate. I contrast this with CAST where there is unlimited time for discussion of each session and critical debate is encouraged. Conferences are expensive, in travel cost, meeting cost, and time. I just can’t afford an event like EuroSTAR.

    So, what I get to see is the abstract. It has its own assertions and it targets my work. It is published to all the world, now, unlike the talk which will be seen by relatively few people. So, it might be an advertisement for a talk, but it is also a document in its own right and a promise in its own right of things to come. That’s what I responded to.

    — cem

  9. A proof and a challenge « Markus Gärtner's blog Says:

    […] believe the evidence you get presented, or start to challenge it, or to challenge it, or maybe to challenge it, or just to challenge it, or even stick with challenging it. Take your […]

  10. Ruth Nofchissey Says:

    The tru test of software testing is only applicable in regards to thesatisfaction of the audience of users

  11. Let passion be our guide | thoughts from the test eye Says:

    […] fervid, fiery, flaming, glowing, heated, hot-blooded, hotheaded, impassioned, perfervid, red-hot, scorching and torrid. Dispassionate can be seen as not showing, and not affected by emotion, bias, or […]

  12. Jon Bach Says:

    I think my main problem with what he is describing above is an assumption that facts and evidence speak for themselves.

    I believe facts and evidence need interpretive rules before conclusions can be drawn.

    Actually, in a court of law “facts entered into evidence” DO have rules:

  13. A new brand of snake oil for software testing « Cem Kaner, J.D., Ph.D. Says:

    […] Jon Bach addresses the tone directly. You’ll have to form your own personal assessments of the speaker. But I agree with Jon that this does not sound merely like advocacy of applying empirical research methods to help us improve the practice of testing, an idea that I rather like. Instead, the wording  suggests a power play that seems to me to have less to do with research and more to do with the next generation of ISTQB marketing. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: