Archive for June, 2010

The Right Combination

June 8, 2010

Once upon a time, I was in a meeting in a conference room with a projector on the table, chained to the desk with a combination lock — the kind that has four dials with numbers from 0 to 9.  The bulb on the projector was burned out, so it was a big interfering brick.  Word was, the admin had forgotten the combination, so there it stood.

It occurred to me that I had nothing to lose by trying to crack it.  I did some math in my head and determined that it would take just four uninterrupted hours to try every combination from 0000 to 9999.

This meeting was a two-hour requirements meeting, so there was half of it right there…

Now I wouldn’t be worth my reputation if I just started at 0000 and tried every combination until I hit 9999.  No, no, no.  I am a rapid tester, schooled in the use of risk-based methods and heuristics – especially when stakeholders want to ship as soon as possible.

So before the meeting started, I asked a few questions of the people around me.

“Does anyone know the combination to this thing?” I asked. (Always good to check some major assumptions that can haunt you later)

Nobody knew.

The conference room was 1215.

No luck.

Our address started with 2445.

No love there either.

“What’s Tracy’s birthday?” I asked, referring to our admin.

No one knew.

I had already taken stock of the spinners — four spinners, 0 – 9.  I spun them to see if any were sticky, maybe that would give me a clue as to how the tumblers were seated. A sticky number may be a clue.

I quickly tried 0000, 1111, 2222, and up.

Nope.  Just like a good password, this one wasn’t immediately obvious.

Some people offered ideas (welcome to paired testing).

“Try the phone extension — #16787.”

I tried 1678 and 6787.  Nope.

“2468,” someone else offered.

Why that?!?

“Random testing.”

Didn’t sound too random to me, but it was easy and quick to try anyway.

I blindly spun the spinners to get something more random.



Other ideas from people around me:

0101 — binary — it was the room closest to the developers. I tried that and 1010, 1011, 1101, 1001, 0001, etc.

Then 2525 (that was how old someone thought Tracy was).

Then I remembered to write my ideas.  This was a testing problem after all — just like a bug repro.

I wrote one combination and tried to change the last number 0 – 9 as I pulled on the lock each time to see if I could feel it getting looser.  Then I tried the next spinner over, and spun it 0 – 9 (the heuristic “one factor at a time”, OFAT as opposed to MFAT for “many factors”).

Then it opened.

Just like a good math exam, it doesn’t matter that I got the answer right, it matters how I GOT the answer. Show your work. And thank goodness I had a little diagram to show my last few tests.

For a few happy seconds, I was a hero, a magician, a svengali, as a few people laughed at their astonishment of what they had just seen.  It kind of sucked that it opened right there in the meeting.  It was a bit distracting, and hard not to celebrate my accomplishment even a little bit.

Since then, I have always carried with me a combination lock to present to testers and take notes on what I observe as they try to discern the combination.  The exercise is very much like reproducing a bug. You try things on your own and you try to elicit data from sources.

A day came when I was in Walgreens and found a combination lock with letters instead of numbers.  You could also set the lock to a custom word to make it easier to remember.  I thought that was a great idea.

I have tried this on professional colleagues like James, Rob Sabourin, Shmuel Gershon, and Justin Hunter from Hexawise – a company that makes a combinatorial testing tool! I wrote their questions, recorded their tries, watched their techniques, and asked them questions as they tested. I also invited them to turn the tables on me by setting their own combination and giving it to me to discern.

Then, about two weeks ago, I met a 12-year-old at a peer conference about testing. It was the son of one of the attendees.  His name was Steven and he seemed bored, of course.  It wasn’t even a testing conference, after all, it was a *writing* about testing conference. Snoozefest for him, for sure.

Never one to shy away from new perspectives on testing no matter what age, I asked if he would be interested in helping me with a problem.

I handed him the letter lock.

“I forgot the combination to this lock,” I said.  “What if I said ‘If you find it, there’s a half million dollar prize…? What would you do to open it?”

I said he could ask me anything he wanted.  I took out my notebook ready to observe what he tried.

The following is a report from Steven about what happened next.

I present it free and without commercial interruption…

“I went to a conference in Colorado with my mom.  It was a group of software testers.  This guy, Jon, gave me a lock and told me to try to open it.  He said I would get a half million dollar prize.  So, I tried to lift the metal part of the lock because when it got stuck, that’s probably where the combination was.  That’s what I have done on some other combination locks.

I tried doing that but it didn’t work.

Then I started asking him questions, like “Was it something you forgot in the office?”

I tried the combination “DOOR “, but that didn’t work.

Then I asked him “Were you trying to remember something?” and he said “Yes.” So then I tried “BUGS ” and that didn’t work either.

Finally, he gave me a hint and he said the first letter was T and the last letter was S.    When I heard that I tried TESTS because it was the first thing that came to my mind when he said that and it popped open!

Then I asked him “Where is my half million bucks?”

After that, we went back inside, and I asked him to give me another problem.  I kept trying and trying and trying and this time he gave me a few hints and then I finally got it — the combination was “WRITE”.

Then he said I could set the lock, so I wanted to try to give him one.  I gave him one where I put the letters into the default code of WORDS and then I picked another set of letters from another side of the lock.  They didn’t make a real word.  But eventually he got that.

Then I tried giving him a combination of “DALLAS”, which was too long, so I used “DALAS”.  But as I was setting it to that combination, after I turned the key to set it, I tried to open it with that combination and it wouldn’t open.  I mixed it up and tried “DALAS” again.  But that didn’t work.  I realized that as I was setting it to that combination, I think I mixed up some letters.

I tried doing the letters just before and after the ones in DALAS, but that still didn’t work.  Then I didn’t know what to do, and then I told Jon and he tried to open it but he couldn’t either.  I tried working on this for many hours, but I couldn’t get it open.  L

I remembered that you can open a lock with a soda can.  I’ve seen on the internet where you cut it in like little hill shapes, and then it’s like a rectangle on the bottom and you fold it in half and then slide the hill part where the button is inside the combination lock and then when you slide the hill part over you can push on the lock and then it should open.  But I cut a whole can apart and I did it wrong and I messed it up.

I was really upset because I really wanted to get this lock opened.  I got so frustrated that I went downstairs to play video games on the computer.

Jon came down and said that I actually had given him a new challenge. He told me that this was like when testers couldn’t reproduce a new bug.  So he told me that this would be a really great challenge for him and next time he brings this to some students, they could try to get it open.


Steven working on the lock as I record his ideas (photo by Lisa Crispin)

Here’s what I noticed in his report:

1) Authenticity: he was upset and frustrated at his mistake of forgetting the combination once he set it.

2) Obsession and drive: he said he worked on the lock for many hours. He never asked for a hint, but I felt bad and gave him one, which he accepted and improvised ideas around.

3) Cognition and recall: he remembered seeing a video about hacking a lock (be careful, Mom).

4) Integrity: he admits he messed up the soda can hack.

5) Curiosity: he tried things.

6) Inquiry: he asked questions.

7) Humor: “where is my half million?”

He also gave me a web reference to where he had seen the soda can lock hack, but for security and ethical reasons, I’m choosing not to disclose it.

This is a wonderful combination of skills and traits that make me think this kid has a bright future.

Yes, I had rolled my eyes when Steven mentioned something about needing a coke can to open the lock.  He explained, but I could not visualize what he was talking about.  It was no different than working with someone with a thick accent trying overseas. I needed him to show me.

The next day, he did.  He found a can, cut it up into little pieces and used one of those little pieces to insert around the hasp in an attempt to make a shunt for where the tumbler met the lock.  I thought that was pretty cool.  I obsessed about it with him for several minutes.

I think he came up with this idea because of his guilt for forgetting the combination he had set. Call it peer respect, but it seemed important for Steven to make sure I could get this thing open before I left.

That blew me away.  Unlike my teenage nephews who could not care less what I do for a living, this young man was engaged and engaging.  They call these kinds of kids “problem children” in school, but give them a real problem to solve that interests them and the “problem” goes away.

His hack didn’t work, but that wasn’t the point.  When his mother said he felt awful about forgetting the combination, I knew it would take a mere 3 seconds to think of what I would say to him. I told him I saw more heart and energy in his idea than I see in professional testers sometimes, and even though it didn’t open, I would have a challenge for the plane ride home *and* a story to tell about it at the next conference.  At the very least, I said, I’d have a blog entry about it.  And here it is.

I’ll have to take his Mom’s word for it that I made a difference to him somehow, but even then that’s not my aim.  Steven confirmed for me that the spirit behind the lock exercise is a good one, no matter the age.  Maybe we’re all just grown-up 12-year-olds looking to apply ourselves to something that needs our skill and insight.

Well done, Steven.  Given more time, you would have gotten that lock open, but that’s beside the point. There’s a future in this business if you want one, and all you have to do is show up and try different combinations. I can’t think of a better life metaphor than that, so thanks for the life lesson.

NOTE: Steven tried some four-letter words above despite it being a 5-letter lock, but he forgot to mention (and so did I) that there was a blank space on the 5th spinner, so the ideas were legitimate tests.

Context: male, female, or N/A

June 6, 2010

If you’re reading this, it’s a safe bet that you are either male or female.

But maybe you’re reading this at work right now and feel gender neutral.  After all, gender is irrelevant in testing.

Or is it?

You could be testing a website tailored to women or acting out a male persona as you test an e-survey about your last prostate exam.

But let’s say you’re testing a password login for an e-commerce site with the standard bag of tricks for test ideas: cross-site scripting, SQL injection, embedded HTML, super long passwords to exploit a buffer overflow.  In that case, you may be asexual, gender neutral, and think that ideas are ideas regardless of sex.

But let’s say you do this testing thing very well and have garnered a bit of a name for yourself.  You get an award for it, public recognition, accolades, blog mentions, a testimonial dinner in your honor.  Oh, and to qualify for this great honor, it was required that you be female.

Now how does it feel? Your ideas were great, but better that you’re a woman!

I doubt a condescending tone was the intent of the organizers of “Women In Agile“.  I’m sure they felt there are not enough women in testing — though it’s unclear how they calculate such a thing — and this is their way of promoting diversity, or in their words: “give a voice to this group and promote the empowerment of women in agile teams.”

I hadn’t realized women were “underpowered” and voiceless, but maybe I’m nieve (wouldn’t be the first time).  Regardless, they’re going to find ways to empower women. I’m not included in that just because of my gender.  Apparently, my gender already makes me empowered enough not to need outside help.  In fact, if there was a Men in Agile org, there would be an outcry, right? 

I would be mad about this female bias if I felt I needed empowerment from an outside source.  Maybe I have it made because I’m a man, but I prefer to believe it’s because I have found ways to develop my own power. Just because I’m a man doesn’t mean the way has been paved for me.  If it was, I must have missed the secret meeting.

But what really struck me about the Women in Agile program was this: “[Women’s] stories will describe how embracing the diverse opinions, experiences and special perspectives of women can and does make agile teams and projects better.”

I felt that not only was sexist but would even be condescending to some women testers I know.  I asked some and they confirmed that.  And that’s why I felt justified in reacting strongly on Twitter. The WIA says just because you are female, you have a “special perspective”, but special in what context? Any context? I suppose women would have a special perspective about prostate cancer, but wouldn’t I have a special perspective about uterine cancer?

With this, I tweeted about the WIA Friday night and it kicked off a conversation between Lanette Creamer and James. (Lanette has since posted about this topic).  Marlena Compton joined in and it escalated. After a few tweets, she suddenly (and oddly) condemned the Context-Driven philosophy.

A follower supported her, tweeting “Context-Driven School implodes”, referencing the debate between Marlena and James, tagging Marlena’s tweet that the Context-Driven School was “sexist bullshit.”

Not sure how she made that leap. I’ve met Marlena, I’ve read her smart and thoughtful posts about data visualization and other technical topics. She’s never been one to like Twitter debates, but I was disappointed about how much anger she had so fast, deciding to condemn an entire testing philosophy after a few tweets with one of its founders — especially on a subject that was all about context — in this case, the context of gender in testing.

Though Marlena might have imploded that night, the Context-Driven School did not.  It was stronger and more affirming to me because gender may indeed have an important context in testing.

Marlena has since written a blog saying: “So if you are among those who think we all ought to be wearing badges announcing how great it is that we fit some cultural stereotype/straightjacket, I hope you take some time to rethink that stance.”

I was going to agree, but then I remembered Louann Brizendine’s two books: “The Male Brain” and “The Female Brain.”  In the latter, she writes: “scientists have documented an astonishing array of structural, chemical, genetic, hormonal, and functional brain differences between women and men.  We’ve learned that men and women have different brain sensitivities to stress and conflict. They use different brain areas and circuits to solve problems, process language, experience and store the same strong emotion.  Women may remember the smallest details of their first dates, and their biggest fights, while their husbands barely remember that these things happened.  Brain structure and chemistry have everything to do why this is so.”

With that, maybe we do need a “Women in Agile” organization.  Maybe women do have “special perspectives” by virtue of having something Brizendine calls a “female brain.” Should they be rewarded for that perspective, though? I still don’t think so.

Just when I was confused on which side of the issue I was on, Context came in to clarify it.  Actually, Context and Maura van der Linden, to be exact.  I’ve known Maura for years and I forgot how much I respect her judgment.  Forget my gender-neutral password security testing example above — Maura happens to *be* a security testing expert (author of the extremely useful “Testing Code Security“)!  But it was her most recent blog that clarified it for me:

“When I think of any group called “Women in X”, I immediately try to figure out what the purpose of the group is. I am never a fan of any type of diversity quotas or rules. But I consider that there are HUGE numbers of ways to be different from another person. Things like skillsets, experience, interest, hobbies, etc. Being a female is a part of my makeup but it’s only a small part of the puzzle. I’m more likely to consider myself an Agile tester or a security tester than I am a female tester because I don’t think being female is a major point I bring to the table.”

I don’t think being male is a major piece I bring to the table, but in the right context, it could be meaningful.  I just don’t want that meaning to qualify me in any way for rewards or recognition.

Automated Baseball?

June 3, 2010

Something big happened in a baseball game last night that is causing a buzz in the sports world today.  I think it’s related to a buzz in the world of software testing.

Armando Galarraga, a pitcher for the Detroit Tigers, was on the verge of pitching a “perfect game” — a game not only in which no batter of the opposing team gets a hit (a “no-hitter”), but in which no batter even makes it to first base. That means pitcher Galarraga would have had to outlast 27 batters trying to smack the ball into play. That’s some great pitching on his part along with some exceptional defensive support from his teammates.

Perfect games are rare. In the 134-year history of Major League Baseball, there have only been 20 perfect games.  Two of them, amazingly, happened last month, which has never happened in one season.

And last night at 6 pm Pacific Standard Time, Armando Galarraga was set to be the 21st.

In the 9th and last inning, Galarraga faced one last batter: Jason Donald. Galarraga delivered a pitch and Donald connected.  The ball was covered by Tiger first baseman Miguel Cabrera who was way off the base to field the ball, so pitcher Galarraga ran to cover the base that Donald was running for. Cabrera threw the ball to Galarraga in time to beat Donald by a full step before hitting  first base in mid-stride.

But to everyone’s astonishment, first base umpire Jim Joyce called Donald safe!   Being safe means Donald had made it to first base before the ball reached Galarraga’s glove, spoiling his perfect game.

As the crowd booed, Tiger manager Jim Leyland came out and argued with Joyce, but the call stood.  The crowd then watched the instant replay which showed the Indians batter Donald out by a full step. Donald had not beaten the throw. He should have been out. Jim Joyce got the call wrong and everybody saw it.

But in baseball, even though umpire judgment calls can be argued, those calls rarely get reversed unless by another umpire who saw the play.  It was hopeless.  Furthermore, it was time to move on to the next batter, which Galarraga did — and subsequently got him out to end the game.

It didn’t matter that the Tigers won the game.  The “perfect game”, a game in which Galarraga technically allowed no batters to reach first base — was spoiled even though the objective truth (according to the camera footage) showed that Galarraga did not allow Donald to safely reach first base.

Unlike other sports, the camera has no say in how baseball games are decided.  In baseball, it’s the umpires that decide.  It’s purely human judgment in the moment. Other sports allow appeals to officials if the camera shows a different story than what their ruling indicated. Not baseball.  At least, not *yet*.  After last night, that might change because this particular game had a bearing on some historical statistics that make baseball much more interesting for a lot of people to follow.

That judgment call by umpire Jim Joyce is now the topic of sports radio call-in shows, newspaper sports sections, and online blogs and articles all across the country today – how he got the call wrong, what the camera showed, if baseball should allow instant replay to influence the game, even how the call was handled by the pitcher, the umpire, the manager, and soon, the Commissioner of Baseball, who oversees everything in the sport.

How is this important to software testing?

There is a balance in baseball between what the camera sees and what the umpire sees.  In testing, there is a balance between what the tester can test and what the computer can test.

In software, testers use their judgment.  Machines have no judgment other than what they are programmed to do.  They are programmed to execute and record, to render and calculate.

As it happened, about an hour before that game, I was talking with Michael Bolton and Ben Simo online about the term “exploratory test automation.”  I had retweeted Elisabeth Hendrickson‘s post about a class she was hosting at Agilistry (called “Exploring Automated Exploratory Testing“).

Bolton, Simo, and I were discussing that title, trying to see if we could come up with something more accurate, because Elisabeth’s title seemed to be a contradiction-in-terms. How do you automate exploration when exploration is inherently human judgment and skill as we react to what we learn in the moment and automation is not?  We were pretty sure we knew what she meant by the class, but how best to describe the interaction between machine and human?

It’s important to know that me, Michael, Ben, and my brother are people who believe in the power of language to convey ideas and meaning.  We argue over precision and semantics because they communicate more than just words.  We believe it is important to debate these kinds of things, openly, publicly, because it propels and provokes conversation about meaningful ideas that are meant to help all testers everywhere win more credibility and respect, much in the same way arguing baseball calls can evolve the sport.

So we traded ideas of how to describe the computer’s role in exploration.  Since it was a public discussion on Twitter, people following that thread could chime in:

Michael Bolton’s idea was to call it “Tool-Supported Exploratory Testing” (proving to be a humorous, dyslexic TSET)

James wanted to flip the words and call it “Exploratory Automated Testing”

Oliver Erlewein liked “ETX” (and so did I) but doesn’t yet know what the X could be — it’s just cool.

Zeger van Hese suggested “Hybrid Exploratory Testing”

I offered the playful “Bionic Testing” after the Six-Million-Dollar Man.

Alan Page said it could simply be called “exploratory testing” and leave it at that because no matter whether your exploration was computer-assisted, it’s still exploration.  James liked that and so did I.

But isn’t there a term or a phrase or a word that can more accurately and precisely describe the computer’s role in assisting testing?

Is it automation when you use a tool to help reveal a bug?

Is it automation when a machine executes a programmed test procedure?

Is it automation when you use Task Manager to see the list of processes in memory?

Is it automation when you execute Cucumber or Fitnesse (keyword-driven) tests?

What do you call it when you click a button on a test harness and it clicks on the objects on the screen for you and delivers a report at the end of the script?

If it’s all “automation”, doesn’t that imply that it needs no human intervention?

I think we can find a better term.

Everyone can agree that computers help exploration.  Call them “probe droids” or “bots” or “tools” — they inform a human about things that are notoriously hard for humans to know on our own.  They do things that are hard or slow or tedious or expensive or impossible for a human to do.

But we also know that it’s also impossible for software to test itself in all the ways we can test it — just like it’s impossible for a camera to replace umpires at baseball games.  Computers and humans enhance each other.

Today in baseball, there’s a lot of energy and debate because of that game last night.  Galarraga’s near-perfect game may lead to a major change in using replay in baseball games.  The Commissioner of Baseball may even overturn Joyce’s ruling, meaning that the official record books would reflect a perfect game last night in Detroit.

Today in software testing, there’s energy and debate around the word “automation”, especially with more classes like Elisabeth’s and the more we talk about Test-Driven Design and tools on projects.

While baseball debates whether to use instant replay in helping to decide close plays , I’ll bet you if they decide to use it, they will not call it “automated baseball.”  We testers *know* we use technology to help us with testing, I just think we can do better than “automated testing”.