Archive for October, 2008

Testing the testing talks

October 10, 2008

I’ve attended a lot of testing talks over the years and there are three things I’ve heard that really bug me.  If any of these statements are in your slides for your next talk, let me try to compel you to get rid of them because they may distract testers like me enough to cause us to tune out:

1) “Insanity is doing the same thing over and over and expecting a different result.”

Einstein allegedly said this and it’s used a lot by speakers who want to decry what appear to be silly pathologies of project management. 

But wait a second.  Let me summon you testers out there to help me examine this.

Have you ever done the same thing over and over, expected a different result and felt *more* sane because of it?

Have you ever done a series of key presses (like a “slot machine” test) and expected to uncover a possible buffer overflow any second?  Press Enter <no change>, Press Enter <no change>, Press Enter <no change>, Press Enter <BANG!>.  You’re expecting that BANG after many trials of doing the same thing.

This is insanity?  No more insane than people who exercise the same way every day and expect to get thinner than they were the day before.  No more insane than a fisherman’s wife looking hopefully out to the ocean every day to see the mast of her husband’s ship on the horizon.

Lesson:  Things change despite our routines.  In fact, it is our routines that catch new behavior from a system that changes underneath and around us.  Like a Build Verification Test is meant to be the same thing over and over, hoping to catch the new code doing something it shouldn’t today, routine can help us find bugs. 

Let me flip it around… it is a SANE thing to do something over and over and expect a different result. 

2) “The longer you wait to fix a bug, the more costly it is.”

I’ve had three experiences in my career where the opposite has been true.  Here’s one of them:

I worked on a project whose developer had a bug assigned and open to him for 183 days without any status other than “Yeah, I know about it, but will not be an easy fix …”  The pressure mounted week after week, and a few days after his latest one-line status report, the bug was fixed.

There was celebration in the triage committee.  After 190 days, it was closed at last! But people were dying to know, how did he do it?  Well it just so happened that the best outlook of a fix came in an recently released SDK.  He used an API that fixed the problem and the implementation took 30 minutes.  If he had jumped right on the bug the day it was opened and spent weeks fixing it, all of that time would have gone to waste. 

Lesson: Sometimes procrastination pays off.  Projects get cancelled, schedules get flipped, priorities get rearranged — all of this can result in re-work and wasted time.  Not always, but neither is it always the case where the sooner you find and fix a bug, the cheaper it is to fix. 

3) “When you “assume”, you make an “ass” out of “u” and “me”.

Do you know how many assumptions we make in one hour?  It has to be thousands!  I assume the sun will go down today, I assume I will still have a job in the next 60 seconds, I assume there will be livable oxygen to breathe by the time this sentence is written… 

I do not question these and I am not an ass because of it — and neither are “u”.

If we questioned every assumption, we’d go insane — or at least sound insane.

Like this:

“I assume that you are reading this sentence.  I assume you know that I am a human and that you do not think this sentence was generated by an electronic algorithm. Assuming that, let me now assume that you will allow me to make this next remark in these words that you are choosing to read of your own free will and volition and let me assume that I may have to define “volition” for some of you which I will do now, assuming that you care about such things…”

Lesson: I think the spirit behind the “ass out of u and me” remark can be restated like this:

“If we don’t examine the consequences of some assumptions, we run the risk of looking foolish and losing credibility in our community.” 

But which assumptions should be questioned?  

I don’t know, but I do have a triggering heuristic that helps me.  Whenever I’ve been stuck or frustrated or bored or confused or mad or off-center in some way, I’ve learned that those are good times to question assumptions.

I’ll try to list a few assumptions I have as I test and question one them before my next test and see what happens. 

I ran into a bug in a tool my brother and I wrote — a crazy bug since the tool had worked for years the same way and one day it stopped working.  I spent 3 hours investigating the problem.  No luck.  Somehow the tool just got corrupted.  I took a break and came back.  A heavy sigh later, I started at square one, questioning every step.  And there it was. 

I assumed I had Perl installed. 

No, I could have SWORN I had Perl installed.  I remembered installing it the night before.  But it turns out I had just downloaded it and never ran the EXE.  I never thought to question my memory.  Maybe that’s why a few times over the years when testing, I see the error dialog “This bug should never happen.” — a note put in by programmers who perhaps have made those kinds of simple assumptions and have felt the consequences of not questioning some of them.

So, if you’re going to use aphorisms like these in a testing talk, test them first.  I’m not annoyed at people who use these in their slides as much as I am annoyed at the people who nod and even laugh when the speaker says them.

Ugh.  C’mon, everyone, we’re testers!  I’m not suggesting we test *every little thing* that’s said at a talk, but whenever the speaker is trying to provoke or persuade you, it should be a trigger for you to learn why it is you are provoked or persuaded and to test the premise of the remark. 

Maybe all it takes to help with this is awareness — which they say is “half the battle.”




Are you thinking “Well, Jon, what’s the *other* half of the battle?” and “Who is the *they* you’re referring to?” and “What does ‘half of a battle’ look like anyway?”

Cool.  Then I don’t have to say another word…

Channel 47

October 8, 2008

Just got back from the STAR West conference, where I was lucky enough to have been chosen to do a keynote.  As often happens when I do testing talks, I got an epiphany the night before — an anchoring idea to frame my talk and make it more memorable.

The conference was at the Disneyland Hotel.  They have their own series of TV channels there, one of which is the Fireworks Music Channel (channel 47 if you stay at the hotel).  I noticed this when flipping around the channels trying to find one of the many Disney stations so that my 2-year-old daughter Charlotte could get in some Mickey Mouse time before she met him in person.

I happened upon channel 47, which was a static image of the Magic Kingdom with a black background, with Disney soundtrack music playing softly.  That was it.  All day, all night, music playing softly with an image of the Magic Kingdom — perhaps for weary parents or children who need to wind down from being over-stimulated by rides and sugar.

Handy to know, but we had just arrived, so little Charlotte was ready to wind up, not wind down.

We found a channel with the Mickey Mouse Club and she was happy.

The next morning, Sunday morning, was quiet, and with Charlotte still asleep in the bed next to mine with my wife, I wanted to start my day quietly — no news, no Mickey Mouse Club, just quiet music.   I remembered Channel 47.

I flipped to it and saw this:

Day 1

Nice!  Finding bugs is my business, but sometimes they find *me*.

I took a picture of the error to use in a testing class one day.

The next day, I was flipping through the channels, and saw this:

The story had unfolded a bit — two error dialogs which appear to say the same thing: “<unintelligble path” is not a valid win32 application” and I noticed the menu bar at the bottom with the system tray.  It was PowerPoint that was failing.

The next day:

That’s right… 3 error dialogs.  One per day?  Clearly, no one at the hotel is looking at this.

The next day was my keynote, titled “Telling Your Exploratory Story” and I knew I had my hook — a way to anchor my talk about how to describe the flow of thinking when there’s no test script to follow.  I would use this as an example that sometimes details slowly reveal themselves, and it’s the thinking about the new, emerging context (and how you react to it) that really underscores the art and craft of exploratory testing — telling your story of the dynamic things that happened in your testing and what you did about it.

Thinking that it was a date-driven bug — perhaps midnight being the trigger — I checked channel 47 one more time before going to bed just after midnight on Day 4.

I saw this:

Cool.  Four dialogs, four days.

The day of my keynote, I told the front desk.  After some trouble explaining that it was not my TV or my laptop, (and no, a technician does not need to be sent to my room) I felt that I had done my civic duty as a tester — reporting a problem in such a way that it had a likelihood of getting fixed.

I added the pictures to my keynote slides and kicked off my talk with them, saying that sometimes a bug story unfolds without us having to do anything but collect context.  It enhanced my talk, I think — got some good laughs and made my point.

A good keynote sets the tone for the conference — grounds the attendees to a meaningful social meme.  And sure enough, for the rest of the week, I had evidence that my talk did exactly that.  People came up to me the next day and asked me what was happening with Channel 47.  I told them it was fixed because I did not see any dialogs that next day.

But someone came up to me the day after that and said they saw the error return.  I checked and confirmed it.  One error dialog. But later that evening, well before midnight, there were two dialogs, blowing my theory that midnight was the driving event.

I mentioned this at a separate talk I was doing the next day.  Someone in the audience pointed out there is also a Disney site in Florida, not just California, and if the channel was hosted at Disney World in Orlando, it would be three hours ahead, meaning that it still could be driven by midnight!

But it was the final day of the conference that was the critical incident for me.  I was in the front row of Rob Sabourin‘s talk titled “Toward an Exploratory Testing Culture.”  He talked about ways testers could find things in common like how to add value to a project, how to be a bug advocate, how to represent their work in credible ways.  He invited discussion from the audience of about 250 people.  And then it hit me.  I had a two-word comment that to me, was an iconic example of an exploratory testing culture — something that grounded us that week, bonded the attendees into a common story, that got people out of their boxes and shells and compartments for just a little while to think about one common, curious, critical problem.

Channel 47.

“What’s going on with Channel 47 today, Jon?”

“I didn’t see the error today, Jon.  Did you?”

“I can’t get Channel 47 at all here at the hotel, I called the front desk to see what the deal was.”

“What do you think the invalid win32 application is, Jon?”

“I saw something similar in the hotel elevator — it appeared to be a digital test pattern underneath the floor indicator.”

“I like that theory that the server is based in Florida.”

“Why do you think the title bar doesn’t show until day two of the problem?”

It was these comments that made me feel connected to everyone else at the conference.  I was the just vehicle for the culture, which, like the bugs that exist in the software that’s delivered to us, was already there, waiting to be discovered.