Testing the testing talks

I’ve attended a lot of testing talks over the years and there are three things I’ve heard that really bug me.  If any of these statements are in your slides for your next talk, let me try to compel you to get rid of them because they may distract testers like me enough to cause us to tune out:

1) “Insanity is doing the same thing over and over and expecting a different result.”

Einstein allegedly said this and it’s used a lot by speakers who want to decry what appear to be silly pathologies of project management. 

But wait a second.  Let me summon you testers out there to help me examine this.

Have you ever done the same thing over and over, expected a different result and felt *more* sane because of it?

Have you ever done a series of key presses (like a “slot machine” test) and expected to uncover a possible buffer overflow any second?  Press Enter <no change>, Press Enter <no change>, Press Enter <no change>, Press Enter <BANG!>.  You’re expecting that BANG after many trials of doing the same thing.

This is insanity?  No more insane than people who exercise the same way every day and expect to get thinner than they were the day before.  No more insane than a fisherman’s wife looking hopefully out to the ocean every day to see the mast of her husband’s ship on the horizon.

Lesson:  Things change despite our routines.  In fact, it is our routines that catch new behavior from a system that changes underneath and around us.  Like a Build Verification Test is meant to be the same thing over and over, hoping to catch the new code doing something it shouldn’t today, routine can help us find bugs. 

Let me flip it around… it is a SANE thing to do something over and over and expect a different result. 

2) “The longer you wait to fix a bug, the more costly it is.”

I’ve had three experiences in my career where the opposite has been true.  Here’s one of them:

I worked on a project whose developer had a bug assigned and open to him for 183 days without any status other than “Yeah, I know about it, but will not be an easy fix …”  The pressure mounted week after week, and a few days after his latest one-line status report, the bug was fixed.

There was celebration in the triage committee.  After 190 days, it was closed at last! But people were dying to know, how did he do it?  Well it just so happened that the best outlook of a fix came in an recently released SDK.  He used an API that fixed the problem and the implementation took 30 minutes.  If he had jumped right on the bug the day it was opened and spent weeks fixing it, all of that time would have gone to waste. 

Lesson: Sometimes procrastination pays off.  Projects get cancelled, schedules get flipped, priorities get rearranged — all of this can result in re-work and wasted time.  Not always, but neither is it always the case where the sooner you find and fix a bug, the cheaper it is to fix. 

3) “When you “assume”, you make an “ass” out of “u” and “me”.

Do you know how many assumptions we make in one hour?  It has to be thousands!  I assume the sun will go down today, I assume I will still have a job in the next 60 seconds, I assume there will be livable oxygen to breathe by the time this sentence is written… 

I do not question these and I am not an ass because of it — and neither are “u”.

If we questioned every assumption, we’d go insane — or at least sound insane.

Like this:

“I assume that you are reading this sentence.  I assume you know that I am a human and that you do not think this sentence was generated by an electronic algorithm. Assuming that, let me now assume that you will allow me to make this next remark in these words that you are choosing to read of your own free will and volition and let me assume that I may have to define “volition” for some of you which I will do now, assuming that you care about such things…”

Lesson: I think the spirit behind the “ass out of u and me” remark can be restated like this:

“If we don’t examine the consequences of some assumptions, we run the risk of looking foolish and losing credibility in our community.” 

But which assumptions should be questioned?  

I don’t know, but I do have a triggering heuristic that helps me.  Whenever I’ve been stuck or frustrated or bored or confused or mad or off-center in some way, I’ve learned that those are good times to question assumptions.

I’ll try to list a few assumptions I have as I test and question one them before my next test and see what happens. 

I ran into a bug in a tool my brother and I wrote — a crazy bug since the tool had worked for years the same way and one day it stopped working.  I spent 3 hours investigating the problem.  No luck.  Somehow the tool just got corrupted.  I took a break and came back.  A heavy sigh later, I started at square one, questioning every step.  And there it was. 

I assumed I had Perl installed. 

No, I could have SWORN I had Perl installed.  I remembered installing it the night before.  But it turns out I had just downloaded it and never ran the EXE.  I never thought to question my memory.  Maybe that’s why a few times over the years when testing, I see the error dialog “This bug should never happen.” — a note put in by programmers who perhaps have made those kinds of simple assumptions and have felt the consequences of not questioning some of them.

So, if you’re going to use aphorisms like these in a testing talk, test them first.  I’m not annoyed at people who use these in their slides as much as I am annoyed at the people who nod and even laugh when the speaker says them.

Ugh.  C’mon, everyone, we’re testers!  I’m not suggesting we test *every little thing* that’s said at a talk, but whenever the speaker is trying to provoke or persuade you, it should be a trigger for you to learn why it is you are provoked or persuaded and to test the premise of the remark. 

Maybe all it takes to help with this is awareness — which they say is “half the battle.”




Are you thinking “Well, Jon, what’s the *other* half of the battle?” and “Who is the *they* you’re referring to?” and “What does ‘half of a battle’ look like anyway?”

Cool.  Then I don’t have to say another word…


19 Responses to “Testing the testing talks”

  1. Sharath Byregowda Says:

    Nice Post Jon,

    I like the “triggering heuristic”. I used it in one of the tech talk at my organization and I had everyone in the room starring at me as if I committed a murder.

    The presenter had a slide – which defined a process to increase loads (5-10%, 20%, 40%, 60%, 80%, 100%) for performance testing. When I questioned why this cannot also be a bottom-up approach, I had heads looking at me like I committed some crime. I justified my statement that bottom-up approach might help us save time and conform whether we meet the expected load. What if the management wants to decide on investing in development effort if only program y captures x packets/min?

    The only valid argument for a top-down approach by increasing the loads exactly as per the process was to be easy on the “hardware”. My question to them was how many work on projects which risk even thousand dollar hardware?

    But then, I was out-numbered and had to sit down. I was extremely angry and annoyed about myself and the audience.

    Do you have any ideas/tricks that might work in these scenarios?

    – Sharath B

  2. Corissa Says:


    I thought ‘knowing’ was half the battle…

    GI Joe!!!

  3. solutionographer Says:

    I assume, it’s okay to forward your blog link, to my test team?

    • jbtestpilot Says:

      Absolutely. Assuming you still have the same team — it’s been so long since your reply to my post, I hope you went ahead and forwarded it anyway.

  4. jbtestpilot Says:

    Sharath: Assuming that by “top-down” you mean starting at 100% and working down to lower loads, I have something that might help.

    You said “The only valid argument for a top-down approach by increasing the loads exactly as per the process was to be easy on the “hardware”.”

    It might be wise to practice some safety language here. Say instead, “One valid argument for a top-down approach is…” or “The only valid I argument I can think of right now is…”

    I’ll bet there are several valid arguments for a top-down approach. As testers, our job is to imagine ways that statements can be either true or false. Otherwise, our biases get us into traps that might rob us of credibility and respect.

    For example, another valid argument I can think of for a top-down approach is that if it fails at 5 – 10% load, why go any farther? If you started bottom-up and it failed at 100% as your first test, you’d have to back track (perhaps using a binary search approach) to the point at which it was an “acceptable” load.

    Anyway, that’s a bit beside the point. It sounds like you felt outnumbered and discounted, and it may be useful to realize and reflect upon that. You sound intelligent and passionate, and good ideas come from people with those qualities.

    If you’re looking for a way to be heard in a crowd who has seemed to make their decision, make your point (which you did), and follow it up offline with someone you respect.

    Sam Kaner just did a great keynote at PNSQC about consensus-making. There were a lot of tips for me in his book “The Facilitator’s Guide to Participatory Decision-Making” where he talks about meetings with people who have diverging views and how compromise and consensus can be reached, even with those who have strong, diverging opinions.

  5. Sharath Byregowda Says:

    It might be wise to practice some safety language here. Say instead, “One valid argument for a top-down approach is…” or “The only valid I argument I can think of right now is…”

    Point noted Jon, Thanks I shall remember this for all my further conversations. A lesson learnt.

    I shall look for a copy of “The Facilitator’s Guide to Participatory Decision-Making” in Bangalore. I wish I get it in a local book store (rupee is pretty low per dollar :-().

    Also, if you don’t mind could you please share your email ID, so that I could discuss more on the topic offline? My mail ID is sharu.b@gmail.com


  6. Cosmo Says:

    The statement “The longer you wait to fix a bug, the more costly it is.” is giving me real problems right now fighting to ease the defect fixing on a website where releases are just 3-4 weeks apart. The receiving party lives after this word, I don’t.

    Sometimes this rapid development means that it is much more costly to halt development and fix a defect before launch than to just acknowledge the defect and fix it with next, or coming, release.

  7. Marta Says:

    Hi Jon!

    I found your blog almost by chance, it’s good to hear from you!

    When Iread point one of this blog, it got me thinking, and I do have a comment about it. I would argue that the sanity of doing something over and over and expect a different result… depends on the context :). In certain cases, like the example you describe, not only it’s sane to do the same thing over and over and expect a different result, but it actually would be pretty dum not to do it (you would miss a potentially catastrophic issue). But in other cases, it is indeed madness to set out in the same conditions, do the same actions and expect a different result, and it has been the downfall of many a project.

    Say that I have a set of identical glasses, and for some misterious reason I decide to drop them one by one from my first floor window, thinking that maybe they are so strong they might make it. Pretty risky, you’d say, but hey, they might. The first one smashes to smithereens, and so does the second. Now, it wouldn’t be particularly smart of me to assume the remaining glasses will not break. They are all glass, same shape and thickness, even manufacturer, and as far as I’m aware, the laws of phisics haven’t changed in the last 5 minutes. It would be even less smart to complain afterwards that all my glasses are broken. A similar thing can be observed in many projects: if you have a set of tests and you know how long those tests will take you to run in the best possible conditions, because you’ve run them before and have those results, it’s not particularly smart to think that next time you need to run that same set of tests with the same resources you will be able to do it in half the time. And still I keep seeing that happen in project after project.

    I think the difference between that scenario and the scenario you describe is that, in your case, you have additional information that tells you that things indeed might change. Your experience, previous tests, a look at the code or a chat with the developpers, precognition… you name it. But something really tells you that not only things have a reasonable chance of changing, but also that it’s important that you validate if that change actually happens and what effect it ewill have. If I were dropping the glasses from shoulder height on a carpet, there is a good chance that some of them will survive the fall without smashing, and I might want to find out which ones those are if I have small kids that can get cut if there are bits of glass on the floor (alegedly what I should really do is to buy plastic cups, but hey, it’s just an example). So, for me, the difference is all in the context.

    Have a good weekend!


    • jbtestpilot Says:

      I like that. Drop a glass onto a plush carpet from a height of one inch again and again. You may look insane as you execute 1000 trials, expecting a different result (a broken glass). It’s all about risk. What’s the likelihood of something bad happening? Change the context to one with a lot of risk (not a glass, but a fragile bubble you just blew from dish soap and dropped from the height of one inch), and yes, doing identical trials might reveal useful information about the integrity and reliability of the viscosity of that brand of dish soap.

  8. Maaret Pyhäjärvi Says:

    Just a comment on the cost of fixing a bug late. I’ve also seen plenty of cases in which postponing a fix actually saves effort/money overall, but what these cases seem to have in common is that the issue has been accepted for the time being and the group monitoring the issue has been minimized.

    I would however not be sure about this particular case on the saved effort, with a change control board being this interested in the issue. A typical change control board can easily use 10 people and couple of minutes on a regular interval to just see what is happening with the bug report, adding significantly to the costs. The poor developer would also use significant time on commenting on the issue. Could be quite easily that the cost of monitoring goes over the cost of fixing it early.

    • jbtestpilot Says:

      Good point, Maaret. Your comment made me think of the most obvious case: triage. Most of the time, it’s cheaper to defer issues to the next release than to try to fix everything in the current one. The point of triage (or a change control board) is to see what bugs we can live with. If you tried to fix every bug (no matter how trivial) as soon as you could, it would get very expensive indeed. In my experience, Business usually wants to take a wait-and-see approach to bugs, just like many companies seemed to do during Y2K. They adopted a fix-it-when-they-report-it mindset, and in many contexts, that was good business sense.

  9. Eddie Correia Says:

    This post resonated with me. As a conference chair, I look at a lot of slide decks, and I’d say at least half contain one or more of these annoying aphorisms (especially #2, I just HATE that one and the tired bar chart that typically comes with it).

    If you have no objection, I’d like to occasionally refer prospective speakers to this post. And if you’d ever consider speaking at STPCon, I’ll “assume” not to find such expressions among your materials!

    Best to you,
    Eddie Correia

    • jbtestpilot Says:

      Thanks, Eddie. And sorry for the late reply… I’d love to speak at STPCon. James told me you might want him and I to do a duo and I’m always up for that.

  10. Tom Says:

    Nice post. I hope there are more to come.

  11. Shmuel Gershon Says:

    Jon, this post was excelent, I specially enjoyed the first two topics.
    Your writings are always great 🙂 ;

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: