CAST-ed

Well, CAST 2011 has shipped.

A few minor bugs, but it looks like the value far outweighed the problems.

James and I wanted it to be THE context-driven event, and it was — speakers from around the world, a tester-themed movie, tester games, a competition, lightning talks, emerging topics, half-day and full-day tutorials, an EdSig with Cem Kaner present on Skype, a live (3D) Weekend Testers Americas session, professional webcasts to the world, debates, panels, open season, facilitation, music, a real-time Skype coaching demo, real-time tweets, live blogs, live LIVE blogs posted as it happened (via http://typewith.me) and red cards for those who needed to speak NOW.

Notice the pattern? Just enough structure to allow play and exploration.  Function and facilities for people to listen and trade ideas.  We provided the place to play and a rough agenda of what could be discussed, but self-organizing professionals did the rest.

And after the conference, they are still organizing.  Sure, I bet a lot of business cards were exchanged, but there’s still traction on the #CAST2011 Twitter hashtag, there are new blogs, there’s talk about EuroCAST and a Brazil CAST.

I may have been presiding officer, but when the show started, I turned it over to the crowd and rode along. Sure, I spoke a few times to the assembled guests, made announcements and reminders, kicked us off in the morning and kicked us out at night, but it was more like being a maitre d’ — I just seated the guests and they made their own dinner conversation.

I’ll help the AST find a place to stage 2012 (likely here in the Bay Area), but I’ll advise next year’s president to make sure there is the same open space to adapt, improvise, and learn so that professionals can decide for themselves if it was worthy of their time.  Lessons here for software development, for sure. 

Until next time, thanks to my core staff — James (theme, tracks and speakers), Ben Yaroch (marketing and media: website, printing, signage broadcasting), and Paul Holland (facilitation, finances, online registration) — to the amazing Danielle Guenther with Lynnwood Convention Center’s extraordinary staff — to Dawn Haynes for turning the reg desk into a full-fledged command center… to the facilitators who kept conversation moving (Paul, Tim Coulter, Sherry Heinze, Nancy Kelln, Mike Goempel).. to Matt Heusser and Peter Walen who encouraged topics to emerge all day, to Michael Larsen for hosting WTA3D, to all of the speakers who sacrified their time, money and sleep… to Doug, Tim, Dee Ann, Cem, Becky, and Lynn for keeping a presence for AST programs and elections, to the sponsors, board members, and people who wanted to help but were turned down because things were going smoothly…

….but last and best, my thanks to the 200 people who showed up in person (and over 800 unique visitors online) who gave it all a reason.

Putting Things in Context

In the movie “The Right Stuff” there’s a scene where character Chuck Yeager is barreling up to the sound barrier in the Bell X-1. 

He’s being shaken to bits from turbulence (transonic speed before 800 mph).  Just when you think the plane will break apart, it breaks the sound barrier (goes supersonic) and all is smooth.

That’s what it’s like to reach the zen of “crazy busy.”  You reach a point where one day you look at your to-do list and everything is Priority 1.  You buffet and shake and are about to implode when you get yet another to-do item to handle.  Then you get another one.  Then, just as you start to fear email, someone walks up to your desk and asks you a question, or worse, reminds you of a commitment you made.  A minute later, you remember you are late for a meeting, and then the phone rings, and then a text message buzzes on your iPhone just as Skype pops up with a request… then, out of nowhere, there is calm. 

It’s a panic-calm as if someone has told you “go to the ocean and put it all into this little cup.”  You can’t do it and you fear failure, but you go to the beach anyway and you see the impossibility of it all and the voice says: ”Yes, this was your test. You found your limit. Nice job.”

I am 7 months after my last post here and I’ve been tested and tested and tested and tested and tested.  Sometimes failing, sometimes passing, sometimes the context changes and what used to pass in me now fails, and vice versa.  It’s like being crazy-busy.  You reach a point where you know you are doing the best you can, even though you may forget it 5 minutes later when a new email comes in that tests your abilities.

The tests don’t stop.  And rightly so.  I am made up of a code base to which I am constantly making changes, and with the code that doesn’t change, the Universe seems to provide a test suite of cases with a taxonomy as diverse as James’ Heuristic Test Strategy Model.

The name of Pradeep Soundararajan’s blog is “Tester Tested”.  I’m not sure what he means by it, but I’m assuming he means that to be worth our salt as testers, we have to agree to be scrutinized.

There’s an ocean of things to do as a Quality Director at eBay — as there is with most e-commerce companies.  The site is open 24 x 7, worldwide. At any time there could be a live site issue that affects any type of user Right Now.  I get 200 emails a day.  I am often triple booked and have to choose which meeting I’m going to grace with my presence.  I walk 2.5 miles a day going from building to building talking to people and solving problems.  I come home, spend a few hours with Charlotte, put her to bed, and get right back online — my quiet time to do homework for the next day of classes.

But always always always, there’s me, evaluating myself.  Am I good enough (in this person’s judgment), to accomplish (this task), (at this time)? 

“Good enough” depends on (at least) those three contexts, and I’m embarassed to admit that I have forgotten (more than once) my own advice in Playing the Expert Game.

If you’re looking for those kinds of answers in your testing of software or in yourself, let me say that I’m hoping CAST 2011 starting Monday in Lynnwood, Washington has some answers. It’s a conference dedicated to talking about Context-Driven Testing whether the topic is software or your good-enoughness as a tester.  The speakers there will be tremendous.  They’ll be available and they’ll be willing to be scrutinized unlike most conference speakers.  There will be discussion and debate, competition and collaboration, food and fun.

I’m ready for a bit of a “spring break” from a long semester at eBay to reacquaint me with colleagues I care about and meeting friends I haven’t met yet — those who might have answers I’m looking for.

I tweeted tonight that I have given up my career in testing to learn what I think I *know* about testing. So far, so good.

A colleague tweeted back “Intense is good, otherwise anyone could/would do it.”

That’s a context I was hoping to be reminded of, and I hope not the last for me before it’s time to get back my place by the ocean of all-there-is-to-do.

A man of auction

Wow, what a New Year already…

Opportunities are sometimes fleeting, but this one wouldn’t let me go…

In two weeks, I’ll be in California to serve as a QA Director at eBay San Jose.

Effective January 18, I’ll be leading testers and managers on the ”Buyer Experience” team — they’re responsible for all of the tools (mainly search) that help users make  a decision on whether to take part in an auction or to “Buy It Now”.  The quality of the experience of the actual auctions is another department, but if I succeed, I may be able to help those guys, too.

Regardless, in two weeks, consider me an eBay ombudsman.  If you have a problem with an eBay experience, I want to know about it.  It may not be my department, but I’ll funnel it to where it needs to go.

They don’t know yet that I  consider the “A” in my title to be “Assistance”, not  ”Assurance”, but they’ll just have to forgive me.

You know me for my ideas, and you know I’ll delight in hiring people smarter than me down there.  Julian Harty is already there and I’ll get to work with him.  So is John Belbute, formerly of Intuit and WebMD. He’s my boss. Above him is VP Todd Paul, ex-senior-Microsoftie and former boss of Harry Robinson on the Bing team.

That’s my new crew. Hard to turn down that kind of team and that kind of brand.  I also have a chance to try some new material and enhance a testing culture.

I thank Quardev for 6 years of opportunities to try my half-baked ideas.  They’re the best testing force in Seattle and they’ve survived ten years with no venture capital, only revenue-funded, privately owned. They’ll do fine without me because the way they approach projects is authentic and honest, and they understand that I need this challenge next in my career.

So, yes, you got that right — one of two uncertified, context-driven Bach brothers will be leading senior test managers at a major brand name company very soon. I’m rarin’ to go, hopeful and optimistic, but also busy repeating the astronaut’s prayer.

I’ll still be shepherding CAST in Seattle, still be going to STAR East and West, STP, QUEST, and PNSQC.  If you’re there, say hi and tell me what you did to find a bug on the eBay site. Hopefully my response will be “I’m on it” or “It’s already in the queue.”

What happens at a real con-FER-ence

Last week, the Software Association of Oregon hosted a one-day Open Space conference.

Accompanied by Andrew Richards, Stephanie El-Hajj, and a cadre of SAO and Portland QASIG / DevSIG volunteers, Diana Larsen did a masterful job facilitating 160 people gathered (at first) in a huge circle of chairs.

An Open Space conference is one where the agenda and speakers are unknown. No one knows what will happen until they get there and start thinking about what they want to talk about. The theme was “Blurring the Line Between QA & Dev in an Agile Environment”, but that was the only structured idea to start with.  People were about to shortly discover where their brain met their heart to figure out what to do next.

Diana asked anyone who wanted to solve a problem to write up a short title on a piece of paper and talk about it for 20 seconds or so in front of the huge circle.  Then they’d tack it up to a huge bulletin board that had time slots and spaces to talk more deeply about it. Then people would decide what most resonated with them.

It just worked.  It met a variety of explicit and implicit requirements to many degrees.

The energy was not too light or too strong. Discussions happened in parallel, allowing people to “vote with their feet” if a topic strayed into malaise for them. At the end, Diana hosted a short retrospective where everyone got a chance to talk (very briefly) about what happened for them.  And everyone did, and it was useful.

It was hard to jump from that wonderful con-FER-ence to one where I thought it would be the opposite.

The next day I was scheduled to speak at the Application Lifecycle Management (ALM) Summit at Microsoft.

ALM Summit Organizer Keith Pleas had approached me back in July saying ”We’re targeting the ALM practitioner / lead more than the journeyman developer, and we’re going to be doing sessions more on ‘best practices’ than product features.”

Uh oh.

“Best practices” around what Wikipedia labels “a continuous process of managing the life of an application through governance, development and maintenance… the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.”

ZZZZZzzzzzzzzzz…..

Knowing that it was going to be held at Microsoft, and that staff from Visual Studio Team System were major players, I did not have high hopes that it was going to be very interesting for me.  It looked like a Microsoft tool-driven conference with preachers preaching to those who already had the tools?  I wasn’t sure.  Just an impression.

I assumed it was going to be as opposite from that Open Space as I could imagine — a dry conference with lots of Information Technology or Enterprise Process Engineering executives wearing three-piece suits.

True to my assumptions, when I walked into building 33 on the Microsoft campus last Thursday, it wasn’t long before I heard many platitudes from speakers:

* “The sooner you fix a bug, the cheaper it is to fix”…

* “Get testing involved earlier in the process”…

* “Communication and management buy-in are very important”…

Attendees (about the same number as that Open Space in Portland) had no choice but to listen.  There was one room, one track, all day… one speaker after another.

But to my surprise, the Twittersphere natives at the ALM conference — the ones using the #almsummit hashtag — were growing restless with this kind of talk.

cacharbe: Ok, we get it, everyone is using Agile. Can we move on to the content?

caffeinatedgeek: @erwilleke I was hoping we would be moving on to the”real world”; HOW and challenges, and not rehashing the WHAT and WHY!

arcs001: After three days of #almsummit we are getting a little bit repetitive…

bsktcase: It’s 2010 and this is ALM Summit. WE don’t need to be sold on agility! We are ready for NEXT steps! Is this what we are saying?

shai_rai: All sessions on Agile Testing at #ALMSummit are using the same Demo and Scenario – Where is The Transparency???

learntfs:  Few are showing what they actually DO on their team. Maybe fearing it looks too much like waterfall? #mindreading

ChadGreen: I love gated checkins and its saved my teams, but how many times do we need to hear about it at

AlexandreLobao: I believe #ALMSummit should have (at least) 2 tracks: 1 for starters, with basics, other focused on real-life problems and advanced topics.

@caffeinatedgeek: feeling like groundhog day at #almsummit. If you only have ONE track, the sessions should be unique.

chrislowndes: #almsummit now turning into TechEd, Season 2, Episode 5 [repeat]

As I monitored the stream of tweets, I saw that the guy next to me had Tweetdeck open, too.  His name was Eric Willeke and not only was he tweeting, he was scheduled to be the next speaker (after lunch).

He tweeted the seed of an idea:

@erwilleke: I’m willing to facilitate roundtables rather than speak today w/ event organizer permission… #almsummit

When he leaned over to tell me his idea, I was sold as soon he had said “What if…?”

All I needed to know was that this guy was reacting to emerging information and was *doing* something novel about it. He was like a developer reacting to results from an exploratory test.  How could I NOT like this guy?!?

Conference organizer Keith Pleas sat to his left.  Eric leaned the other way and whispered to him the idea.

With this, Keith said, “Let me take the temperature of the room about it before lunch.”

And when the current speaker finished (to polite applause), Keith did exactly that.  He went to the front of the auditorium and told everyone Eric’s idea.  It got a rousing ovation.

And with that, the attendees and organizers of the 3-day ALM Summit were about to self-organize. They were responding to change over following a plan — not just words in slides or in a popular Manifesto, but actions.  It was a beautiful, simple, organic thing — no politics, no hesitation, no fear.  Just smart colleagues freeing themselves with a simple idea.

Eric was the right guy to lead this insurrection.  Without dogma or drama, he took the lectern in the large ballroom where lunch was winding down an hour later. He got everyone’s attention and introduced the idea:

“Seekers”, anyone in the audience  who had questions or problems to which they wanted answers, would seek counsel from “consultants”, any volunteer who had something constructive to say about it.  They’d meet at tables and anyone else could sit in and listen. Simple as that.

He timed-boxed it and facilitated the 3 or 4 rules that went along with it.  It was focused and energetic, and it ended right on time — just in time for me to speak next.

D’oh!

I had to follow *that*?!?

I had to follow the rave (and inevitable) success of a self-organizing team with a strong mission and a strong leader?

Uh oh…

Good thing I’m speaking about exploratory testing.  There was no better segue to talk about a testing approach built on the idea of adapting to emerging information, coupled with notions of Agile Development.

I like to think I brought my A-game to my presentation.  I did adapt my talk a bit — punched it up to emphasize (and honor) what had just happened in Eric’s impromptu dojo, but really… what speaker can rank against a group of smart, passionate people who have just replaced a rigid structure that was not meeting expectations?

That, dear friends, is what makes me excited to stage CAST 2011 in Seattle next year.

As James and I plan CAST, we promise it will be built to embrace change. That is our theme, after all — Context-Driven Testing — where Q&A will be facilitated and great ideas will be honored (and tried) if there’s energy around them. If we need to, we will respond to change over following our plan, especially if it means people get to really CONFER at our conference like they did at the SAO’s Open Space and those few adaptive hours at the ALM Summit.

In the meantime, my three heroes this week are Keith Pleas, Eric Willeke, and Diana Larsen.

These three folks embody what it really means to agile and self-organizing, but with grace and simplicity.  All three helped to remind me that the word *confer* is the root of the word “conference”, but also proved why *commune* is the root of the word “community”.

Courting Coincidence (aka “networking”)

I spoke at a panel at Edmonds Community College today.  It was designed to help students get ideas to increase their chances of finding a job.  The other panelists tended to treat it like a resume clinic or job interview seminar.  They missed the point.

The point was *networking* — the ways people connect and relate to each other.

It occurred to me that even if someone in the audience was successful in landing a job, there was no guarantee they were going to keep it.  I was hired once, and laid off 7 months later. The anxiety of the economy might not stop just because you got hired.  Sure, some bills might get paid, but now you have the problem of wondering when they won’t.

It’s tough out there, but networking has saved me. My reputation has been the battle jitney that has not only protected me from some turmoil, but has been conspicuous enough to attract opportunities.  In that spirit, I give you my ideas to increase the likelihood you will be at the right place at the right time or meet the right person with the right opportunity.

* Twitter: tweet something that really resonates with you. Sure, sometimes saying what bagel you had for lunch might help (that’s actually how I met a worthy speaker for QASIG), but give others a chance to know what you find meaningful. Find a hashtag that you’re interested in (like #testing) and reply to someone who said something that resonated with you.

* Facebook: comment on someone’s status with a link you think is relevant and cool. Post your own status about an test idea you have or a thought about a test tool.

* LinkedIn: There are lots of forums in which to participate, so don’t just post a profile and status and leave it there.

*  Blog: start one or comment on one with your full name and email. Give others a chance to say “Where has *this* guy been?!? He’s got game!”

* Seattle Job Social: A regular meetup of employers and seekers in an informal setting — a popular bar in Belltown. I’ve been to two of them — one as an employer, one as a seeker.  Both were useful, but became less so the longer people drank.

* QASIG: Quality Assistance Special Interest Group — free — meets every second Wednesday of every odd month. Sponsored by Quardev — hosts a speaker and free pizza. (I’m its speaker chairman.)

* SASQAG: The Seattle Area Software Quality Assurance Group — free — meets every third Thursday at Compucom (Excell) – hosts a speaker and free snackies. Keith Stobie and Tracy Monteith from Microsoft are its speaker coordinators.

* SEASPIN: The Seattle Eastside Area Software Process Improvement Network — free — meets every month at Construx — – hosts a speaker and free pizza.  Jeff Smith and Steven Smith (no relation) are its speaker coordinators.

* Seattle Lean Coffee: meets every Wednesday morning at Uptown Espresso — free — hosted by Jim Benson and Jeremy Lightsmith.  Devoted to talking about software development problems in the context of  principles and notions of Lean, Agile, XP, Scrum, kanban.

* Open Coffee: meets every Tuesday morning at Louisa’s Cafe on Eastlake. Devoted to entrepreneurs who want to meet with others, as well as venture capitalists.

* OWASP: The Open Web Application Security Project — a meetup dedicated to discussing software  security issues.  This is a worldwide org.

* Volunteer to be a guest speaker at a community college.

* Your local Chamber of Commerce has an events calendar or newsletter.

* Local Business Journals.  In Seattle, we have the PSBJ with an events calendar.

* Technology Associations. In Washington, we have the WTIA with an events calendar.

* Industry Associations. Like the Association for Software Testing. You can volunteer to be a facilitator or help with the website, or the yearly conference.

* Colleges and tech schools. Volunteer to be on their industry (“real world experienced professional”) curriculum advisory board.

I’ll think of more and post them here.

Comment to add your ideas, too.

Or, do one of these above and tell me how it went.

Questions from the future

I did a guest lecture today at University of Washington’s Bothell campus.

It was for CSS 490 (“Software Testing”), taught by a colleague of mine named David Socha.

He handed out 3 x 5 cards to the students and asked them to come up with questions for me about testing.

In the interest of giving all of you fellow software professionals a glimpse into the mind of our future workforce, here is what they asked:

* I’m looking for a career in the gaming industry as a game programmer. I know software testing can help me as a game programmer but I want to learn more. Where do I start?

* Is it possible to work both as a tester and a developer?

* What certifications are industry standards?

* How do I get started as a tester, and also more interview questions / techniques?

* How much do testers make in comparison to devs?

* Is there a different approach to take when testing critical systems?

* What’s it really like to be a tester?

* Where do you get started?

* Can you tell us any interesting stories?

* What is the salary range for entry level, 5 years, 10 years?

* What are the working hours?

* Is the software testing job secure and stable?

* What is the major challenge?

* How do you know you’re a tester, not a developer?

* How much time a day does a tester spend documenting, coding, and testing?

* What are some testing career paths?

* What are the popular testing tools?

* During an interview how is a tester tested?

* What are the internship options around here?

* How do I get a tester job at entry level with no experience?

* What role does the PM have in testing?

* Is it a very stressful job?

* Can you give us some interview tips?

* What one thing can a dev do in coding to make things easier on a tester?

* Why should I be an SDET over an STE?

* What is the growth path for a tester at your company?

* What is the difference between a tester and an SDET?

* How do I write code in a way that is testable?

* How do I test my own code?

* How do I work with testers efficiently and in a friendly way to keep a positive relationship?

* What should I know to be a successful tester?

* How do you define a successful tester?

* What other college courses can help me become a tester?

* What’s it like if you have to work as a tester for a defense contractor in terms of ethics?

* Do testers work longer hours compared to other team members?

* Does paired testing result in better tests than individual testing?

* What’s a good answer to the classic “How would you test a soda machine?”

My next-to-last favorite:

* As a PM, how do I learn to like testers?

And my favorite… :

* Is testing depressing?

These questions would make Utest proud given their skill in asking questions for their “Testing the Limits” series.  In fact, I mentioned them today as an answer to “How do I get started with no experience?”  [ David Socha (the instructor) also mentioned mifos.org as a good way to build skill by volunteering. ]

Anyway, I’m sure we can build a whole conference around answering these, and I plan to do them service next time I’m asked to guest lecture (on Wednesday, actually). If you want to drop a dime on some of these, feel free to comment below and I’ll say to them: “This is what my colleagues had to say…”

Class, class, and class and STARWest

Just back from STARWest in San Diego.

The hotel was great and so were the attendees I ran into. Well, it was easier to run into some mainly because James was in fine form drawing a mob around his tester games in the main walkway.

Class was in session, even when it wasn’t. To me, that’s the mark of a good conference.

I had emerged from a session to find about 30 people crowded around him.  He was like a chess master, playing his dice game with 3 people at once.  He always carries three bags of about 30 dice each with him wherever he goes, and he seemed to be using every one of the dice in those bags.

Knowing the operating premise of the game, I decided to help him out.

He looked over and saw me, and told me he was glad to see me and could use my help because he had to go meet someone.  I’ve never had to follow James after he finishes a talk, but this was as close as it got — I inherited his crowd and did my best to keep the momentum.

He got back a few minutes later, glad to see the party was still happening.  But he noticed that the bags of dice were all laid out on the table — all 100 or so of them.  He seemed annoyed.

“Hey, man! My dice are all mixed up!” he said.

I had no idea that the dice had an organization to them. I felt bad, but I told him there was no way for me to have known that.

Still, this was not an impossible problem to solve. The “dice game” is about recognizing a pattern in  James’ head.  You do this by rolling dice.  After each time you roll, he says a number.  You have to figure out why he’s saying what he’s saying. It’s fast feedback from testing, and you have to form conjectures just as fast — then you roll the dice to either confirm or refute your conjectures.  You win when you can describe the operating pattern to a high degree of certainty.

Welcome to testing. Fast, fun, and authentic skill-building.

But back to James’ annoyance.  I replied, “ok, then, don’t tell me how they were organized in the bags.  Let me figure it out with you giving me feedback as I do it.”

He seemed to lighten with the perspective of that challenge.  But before we could make this into a game, he said “Each bag uses the principle of requisite variety.”

I knew what James meant by that. It’s one of the many terms to describe tests.  In this context, it meant the bags were equivalently and sufficiently diverse.

The first thing I wanted to do was  make sure the crowd knew this was spontaneous.  I wanted them to know this was something new between James and I, so they could pitch in to help if they wanted. Like everyone else at conferences, he and I follow threads of ideas that pique our interest. I had no idea what was going to happen, but I knew it would be fun, interesting, and meaningful somehow, and I hoped the crowd would play along as James tested me.

So I took all the dice in one big (laid-out) pile and scanned for affinity. It quickly emerged that there are several ways to make affinities. The ISTQB folks want you call these “equivalence classes”, and that’s fine, but they don’t tell you what to do when you suddenly realize that many tests might belong to more than one equivalence class.

For example, a red translucent die with black pips could go in 5 categories: red dice, black pipped dice, lightly opaque dice, dice that were identical to it, and approximate size to other dice like it.

My aim was to quickly take stock of the primary dimensions that stood out to me, then group them by those properties. There was background color, size, shape, pip color, pip type (dots, numerals, or symbols), opacity, albedo, elasticity, number of dice that were identical, and weight — and many fit in more than one category as I sorted them, so the notion of “equivalence partitioning” was difficult.

Good thing I remembered the context.  These dice are for games. The point is to have three bags that are equally diverse such that the odds are greater of having a similar experience no matter what bag James reached for. If one bag had too many of one kind, or not enough of a different kind, that may stall the exercise or the lesson he wanted to teach.

So I grouped mostly by “similarity to each other.”  I grouped the white dice together, the red translucents, the small standard blues, the more-than-six sided, the “few huge” (relative to most of the others), the ones with numerals instead of pips.

Then someone noticed that in the red dice pile, some had black pips and some white.

I took the suggestion to separate those into two piles.

In the white dice pile, someone noticed a few felt like they could be “eraser” dice — they had a rubbery consistency to them — which brought another dimension to dice: “the tendency of the pips to wear off with heavy use.”

I separated those.

Then I just did a straight count of each affinity and divided by 3.  If it was not perfectly divisible, someone suggested I make a separate pile — a kind of Holding Pattern /  Issues List pile to check with James about his preference for which bag they were most appropriate for.

The exercise took about 5 minutes, and with James as my “onsite product owner”, the mission was done when he grabbed the last 5 “issues list” dice and threw them into two bags, as if to say “good enough” (translation: further sorting into deeper affinities would be a waste of time and resources).

Too cool. Mission accomplished with a potential blog idea (mission accomplished there, too).

That was my best class that day, and a classy way to have an impromptu experience about class.

A New Thread

(Mirrored from http://www.quardev.com/blog/2010-08-26-551545950)

Some of you know me and my brother James as exploratory testing practitioners.

You also know us as firmly aligned with the principles of the Context-Driven School. But sometimes we’re more known for being the guys who created Session-Based Test Management to solve the problem of how to manage and report effort from exploratory testing.

Ten years ago, SBTM was considered a pilot project with promise. Dreamt up on a napkin at a Denny’s in Boise, Idaho in March 2000, it was showing a lot of promise. We had done 150 sessions worth of experimentation by this day (August 26) of that year and liked where it was going.

I was struck by how powerful and efficient it was. For the first time I had a method for creating sheet music for jazz – a way to describe creative, improvisational work, and make it understandable and measurable. We didn’t invent exploratory testing, we had designed words and structures for it to give testers and managers a new way to describe their work to stakeholders other than to say they were “just playing around.”

Well, James and I have a new context-driven idea to experiment with. Like SBTM, it is meant to help you describe activities you’re already doing.

I picked James up in Seattle yesterday after he returned from a difficult project. A lot was going on in a small amount of time – a project with typical daily chaos. We went to my favorite sushi place and talked about it. He couldn’t disclose much because of NDA, so he talked in patterns and generalities, but he said something simple that stuck:

“Seemed like I did a lot of my work in threads.”

“Threads?” I asked.

He explained there are situations where we might leave a task and come back to it many times.

I agreed. That was a typical day for me at Quardev and as a tester on most projects I’ve worked.

He said that since the nature of the task could change over time, you wind up pursuing it like you pursue a thread in a conversation.

I agreed there, too.

“When our work is so tentative and exploratory,” he said, “the word ‘task’ seems too narrow a word to capture it. It’s really a thread.”

We talked about what it means to follow a thread, work in threads, drop a thread.

An artifact-based approach to test management usually aims to create documents like test cases. But the concept of working in threads de-emphasizes artifacts. It focuses on what you actually do: it’s an activity-based approach, not an artifact-based one. It’s meant to emphasize the learning that happens along the way toward solving an authentic problem.

James and I agreed that a session charter is an example of a thread. But where a charter seemed to differ from a thread is that a charter is a commitment, an agreement to accomplish a task (or set of tasks) in a session. A thread, on the other hand, has no such agreement or commitment, and no timebox as sessions do. It’s more general.

In short order, we had created a new sibling for SBTM, and we called it Thread-Based Test Management.

TTM is a generalized form of Session-Based Test Management.

While SBTM seeks to manage exploratory testing in timeboxes toward a commitment or agreement to execute a charter, TTM is more general – with threads focused on emerging problems and objectives that you need to solve.

You choose an activity that needs to get done, and you follow that thread. You follow it hither and yon, up and down, back and forth, round and round, over hill and dale until you decide it’s over – that further time spent on it is not worth it (for now).

Think of threads as the Notes you take in a session report. It’s the list of activities that tell the testing story of that session.

With threads, James and I are suggesting a new pattern of testing activity we think all knowledge workers already do, and we offer a way to think about managing activities you might do while on a thread.

The premise of TTM is this:

There are kinds of projects where we cope with a great many interruptions as we pursue objectives. How could we manage testing if we embraced those interruptions?

TTM is a test management approach that organizes testing around patterns of activity (“threads”). By identifying and organizing threads we’d might be able to keep testing on track and provide a credible report of the ongoing story of the test project.

James and I left the sushi place after flushing this out a bit more and continued our “Thread Theory” discussion back at Quardev. We set up shop in a conference room where we were quickly interrupted by Quardev’s Enterprise Manager who needed status from me on a project proposal. I excused myself from the room and let James hash some stuff out on the whiteboard about what TTM could be, knowing I’d return soon.

15 minutes later, I returned, picking up that thread with him – the one about exploring what Thread Theory could mean. We resumed a brainstorm about the mechanics and philosophy of why it could have value to label the interruptible courses of action we take when we test.

We discussed what might happen in the course of a day of following threads.

You could:

- focus on one thread or many;

- drop threads;

- create new threads;

- pick up dropped threads;

- “comb” threads by creating a structure to organize them;

- “knot” threads by declaring a meaningful checkpoint in your exploration;

- “untangle” threads by uncovering new context or seeing a pattern that’s valuable to know;

- spawn child threads;

- realize an overarching parent thread…

Fifteen minutes into these bullet points on the whiteboard, the conference room phone rang again. It was Scott, apologizing for the interruption but wanting clarity about insurance for a subcontractor as part of the proposal we were working on earlier.

Ah, the “Scott/proposal” thread I dropped earlier had resumed.

We let it interrupt us because it was important (and a self-referential example of embracing emerging context on another thread), and after a few minutes, James and I resumed (again) our other thread of talking about Thread Theory on the whiteboard.

Another 15 minutes later, Scott came into the conference room, reporting that the idea we had given him earlier about the subcontractor insurance had some merit in a call he had just made in the other room.

When you’re following threads, it doesn’t matter how many times they get interrupted, it’s about being aware that the number and types of threads you’re following might help you tell the story about your day of testing and solving problems.

Some people report their day around the stories they develop or test.

Some people report sessions.

Some people report the test cases they ran.

James and I suggest a thread paradigm because it may be more useful on chaotic projects to get to the heart of what we actually do – follow the flow of thinking and activities around solving ambiguous problems.

So, yes, he and I had just two threads going on for a few hours yesterday. If we were taking notes about our day or making a report to some stakeholder that mattered, these two threads would be included in what we learned or what value we created. If there were an Agile-style standup the next day, I may decide to talk not in stories or session charters, but in threads-followed.

Whenever you think, “I need to get something done” or “I need to solve a problem”, TTM is a way to manage that.

Since it’s about maximizing opportunities and following the flow of problems, there are two main questions that are useful to ask:

1) Primary: “What thread is most important right now?”

2) Secondary: “On what thread can we make the most progress right now?”

Whether or not you call them threads, these patterns of activity are already there in our day. Knowledge workers (like testers) already know this.

When we find a bug and start our investigation into how bad it is, that’s a thread.

When the Triage Team has a question and we test because we think we can give them more more context to make a better decision, that’s a thread.

When we need to know if testing in the cloud is a solution for us, and start researching VMs, Hyper-V, Azure, and different notions of virtualization, that’s a thread.

And instead of lamenting the interruptions to those missions, we hope you take them in stride, asking the two questions above. Maybe it makes sense to cut your current thread for now because of something more urgent. Maybe make a knot or two along the way, comb through some tangles, discover a child thread lurking, or drop a thread entirely and pick it up later. Maybe devote a thread to tie up loose threads. That’s the spirit of TTM.

Either way, try it. When someone asks you how you day went, talk in terms of threads-followed or threads-in-progress, not artifacts-produced.

As you’re reading this, James has written a simultaneous blog on this topic that goes into more detail.

The “testing moment” with Jon Bach

Recently, I met a so-called “testing expert”, and it was a profound experience. 

I was on a project at a client site in the Bay Area.  I was doing an exploratory session, modeling a product, but sloppily so.  I was not using the session template.  My notes were rough and unpolished. I was not using James’ HTSM or GFS or even SFDPOT.  I was not using any of the skills I teach others, it seemed. 

I was winging it because I was overwhelmed.  With all there was to know about a new product to test and only two days to learn it and deliver a test plan for the crew back in the lab, I let myself take on everything at once, get scattered, lose my focus, disoriented, and the pressure fed on itself the more aware I became.

There were 20 documents open on my screen, there were interruptions from the client as they gave me new pieces of info, there was email coming in to answer, there were things installing on my laptop that were crucial, there were IT issues for me to attend to as I got set up. Everything was getting done at once, but I was making no progress.

A thought came to me right away:  “Jon Bach would not be proud.”

I felt he was watching me — this expert famous tester Jon Bach guy — co-inventor of SBTM, conference keynote speaker, article author, blogger, blah blah blah… He was watching a person I’ll just call “me” — a struggling student tester for 15 years, not as technical as he should be, certainly no programmer, scatter-brained, impatient, too self-critical, easily overwhelmed at times as he tried to live up to Jon Bach’s reputation.

I was in the “testing moment” and I was not doing well. My habits were awful. I started things and left them unfinished and moved to something else. It was like I forgot everything I had learned in 15 years of testing.

As I got up-to-speed on the product given to me to test, I found that habits were driving me more than my skill. I felt compelled to dive in and know it all right away – the readme, the build notes, the FAQ, the software, the test plan, the spec, the strategy, the existing bug database.  I had no compass. I had lost track of my charter. I read things and didn’t know what they meant, and tried to cure that by reading it again, slowly. I read things 5 times and it bounced right off. Nothing got in.  I asked the same question to a stakeholder 3 times that day. It was not good. I felt out of shape and not good enough.

Recently, I traded Tweets with Pradeep Soundararajan, one of India’s most famous testers.  We have never met, but his stock has been rising over the past few years because of his soulful dedication, practice, leadership, and writing about our craft. More and more, he is someone I want to meet. When I commented on his notoriety lately, he said: “I think I am doing what every other tester is actually supposed to do. [Others] are just offering an opportunity to me to make me look good.”

I agreed.  Others have made me look good, and I, too, am always trying to do what I think every tester is supposed to do.  But why did I feel like I was failing (and flailing)?

It occurs to me now that we can take all the training in the world, and have all the experience in the world, but what it often comes down to that drives us in that “testing moment” is our own self-worth.  We are only as good as our last great test idea, our last bug found, our last oral report to stakeholders; and if we have none of those to immediately recall, we can feel pretty lousy.  Worse, that attitude tends feeds on itself and we expect more from ourselves, adding to the pressure and compounding the problem.

This blog has always been about the humanity in software testing, and I never felt more human than in those recent overwhelming moments at that client site. I should have stopped testing and taken a break.  I should have written an issues list with every concern and question I had. I should have not dismissed my confusion so easily.  I should have remembered what I tell people — “confusion is a powerful tool — use it.”

I did not do any of that until the plane ride home the next day.  That’s when I had a coaching session with Jon Bach, the expert. He schooled me there on the plane.  He gave me a robust critique and reminded me of some key points to practice.  Outside of the pressure cooker, he systematically called out my mistakes. Better, he helped me point them out for myself, and he was fair about it. He also called attention to the things I did well (even though I disagreed with him on some of those). 

More importantly, he and I decided to write this blog together, and we wanted to say that we agree on one major point: “We should find ways to rehearse our testing so that when we’re in the ”testing moment”, we have a better chance of feeling worthy.

If testing is a performance (as some say), then when do we get a chance to play badly before the curtain goes up? When do we practice scales and read new sheet music, and study other musicians to see how they play?

One answer is why Pradeep is so notorious — he inspired and mentored the “Weekend Tester” phenomenon into what seems to be a reliable and powerful culture of learning and testing practice that offers everyone a chance to try new things and fail safely.  I’ve been a part of just three of them, but there have been over 30 as of this writing. It is a culture of “learning moments” borne from many participants’ “testing moments” on projects.

So with this blog, I dedicate myself (and challenge so-called “testing expert” Jon Bach) to do more in his training to create more “learning moments” — epiphanies and discoveries of one’s skills through hands-on exercises that give testers a chance to rehearse.

If we both do that well enough, we may hear an inner voice in those testing moments that says things like “slow down, remember your charter, ask for help, one thing at a time, remember this tool, remember this technique, remember that this is normal, and I know just what to do…”

Else, we may feel lost when those testing moments come, and beat ourselves up about it.

But Jon Bach reminds me in this last sentence to say there is another choice — we can realize that the failure to be brilliant and proud of ourselves in any given “testing moment” could be considered an important “learning moment”.  (With that, I agree and hopes he reminds me of that from time to time.)

The Right Combination

Once upon a time, I was in a meeting in a conference room with a projector on the table, chained to the desk with a combination lock — the kind that has four dials with numbers from 0 to 9.  The bulb on the projector was burned out, so it was a big interfering brick.  Word was, the admin had forgotten the combination, so there it stood.

It occurred to me that I had nothing to lose by trying to crack it.  I did some math in my head and determined that it would take just four uninterrupted hours to try every combination from 0000 to 9999.

This meeting was a two-hour requirements meeting, so there was half of it right there…

Now I wouldn’t be worth my reputation if I just started at 0000 and tried every combination until I hit 9999.  No, no, no.  I am a rapid tester, schooled in the use of risk-based methods and heuristics – especially when stakeholders want to ship as soon as possible.

So before the meeting started, I asked a few questions of the people around me.

“Does anyone know the combination to this thing?” I asked. (Always good to check some major assumptions that can haunt you later)

Nobody knew.

The conference room was 1215.

No luck.

Our address started with 2445.

No love there either.

“What’s Tracy’s birthday?” I asked, referring to our admin.

No one knew.

I had already taken stock of the spinners — four spinners, 0 – 9.  I spun them to see if any were sticky, maybe that would give me a clue as to how the tumblers were seated. A sticky number may be a clue.

I quickly tried 0000, 1111, 2222, and up.

Nope.  Just like a good password, this one wasn’t immediately obvious.

Some people offered ideas (welcome to paired testing).

“Try the phone extension — #16787.”

I tried 1678 and 6787.  Nope.

“2468,” someone else offered.

Why that?!?

“Random testing.”

Didn’t sound too random to me, but it was easy and quick to try anyway.

I blindly spun the spinners to get something more random.

3374.

Nope.

Other ideas from people around me:

0101 — binary — it was the room closest to the developers. I tried that and 1010, 1011, 1101, 1001, 0001, etc.

Then 2525 (that was how old someone thought Tracy was).

Then I remembered to write my ideas.  This was a testing problem after all — just like a bug repro.

I wrote one combination and tried to change the last number 0 – 9 as I pulled on the lock each time to see if I could feel it getting looser.  Then I tried the next spinner over, and spun it 0 – 9 (the heuristic “one factor at a time”, OFAT as opposed to MFAT for “many factors”).

Then it opened.

Just like a good math exam, it doesn’t matter that I got the answer right, it matters how I GOT the answer. Show your work. And thank goodness I had a little diagram to show my last few tests.

For a few happy seconds, I was a hero, a magician, a svengali, as a few people laughed at their astonishment of what they had just seen.  It kind of sucked that it opened right there in the meeting.  It was a bit distracting, and hard not to celebrate my accomplishment even a little bit.

Since then, I have always carried with me a combination lock to present to testers and take notes on what I observe as they try to discern the combination.  The exercise is very much like reproducing a bug. You try things on your own and you try to elicit data from sources.

A day came when I was in Walgreens and found a combination lock with letters instead of numbers.  You could also set the lock to a custom word to make it easier to remember.  I thought that was a great idea.

I have tried this on professional colleagues like James, Rob Sabourin, Shmuel Gershon, and Justin Hunter from Hexawise – a company that makes a combinatorial testing tool! I wrote their questions, recorded their tries, watched their techniques, and asked them questions as they tested. I also invited them to turn the tables on me by setting their own combination and giving it to me to discern.

Then, about two weeks ago, I met a 12-year-old at a peer conference about testing. It was the son of one of the attendees.  His name was Steven and he seemed bored, of course.  It wasn’t even a testing conference, after all, it was a *writing* about testing conference. Snoozefest for him, for sure.

Never one to shy away from new perspectives on testing no matter what age, I asked if he would be interested in helping me with a problem.

I handed him the letter lock.

“I forgot the combination to this lock,” I said.  “What if I said ‘If you find it, there’s a half million dollar prize…? What would you do to open it?”

I said he could ask me anything he wanted.  I took out my notebook ready to observe what he tried.

The following is a report from Steven about what happened next.

I present it free and without commercial interruption…

“I went to a conference in Colorado with my mom.  It was a group of software testers.  This guy, Jon, gave me a lock and told me to try to open it.  He said I would get a half million dollar prize.  So, I tried to lift the metal part of the lock because when it got stuck, that’s probably where the combination was.  That’s what I have done on some other combination locks.

I tried doing that but it didn’t work.

Then I started asking him questions, like “Was it something you forgot in the office?”

I tried the combination “DOOR “, but that didn’t work.

Then I asked him “Were you trying to remember something?” and he said “Yes.” So then I tried “BUGS ” and that didn’t work either.

Finally, he gave me a hint and he said the first letter was T and the last letter was S.    When I heard that I tried TESTS because it was the first thing that came to my mind when he said that and it popped open!

Then I asked him “Where is my half million bucks?”

After that, we went back inside, and I asked him to give me another problem.  I kept trying and trying and trying and this time he gave me a few hints and then I finally got it — the combination was “WRITE”.

Then he said I could set the lock, so I wanted to try to give him one.  I gave him one where I put the letters into the default code of WORDS and then I picked another set of letters from another side of the lock.  They didn’t make a real word.  But eventually he got that.

Then I tried giving him a combination of “DALLAS”, which was too long, so I used “DALAS”.  But as I was setting it to that combination, after I turned the key to set it, I tried to open it with that combination and it wouldn’t open.  I mixed it up and tried “DALAS” again.  But that didn’t work.  I realized that as I was setting it to that combination, I think I mixed up some letters.

I tried doing the letters just before and after the ones in DALAS, but that still didn’t work.  Then I didn’t know what to do, and then I told Jon and he tried to open it but he couldn’t either.  I tried working on this for many hours, but I couldn’t get it open.  L

I remembered that you can open a lock with a soda can.  I’ve seen on the internet where you cut it in like little hill shapes, and then it’s like a rectangle on the bottom and you fold it in half and then slide the hill part where the button is inside the combination lock and then when you slide the hill part over you can push on the lock and then it should open.  But I cut a whole can apart and I did it wrong and I messed it up.

I was really upset because I really wanted to get this lock opened.  I got so frustrated that I went downstairs to play video games on the computer.

Jon came down and said that I actually had given him a new challenge. He told me that this was like when testers couldn’t reproduce a new bug.  So he told me that this would be a really great challenge for him and next time he brings this to some students, they could try to get it open.

================================

Steven working on the lock as I record his ideas (photo by Lisa Crispin)

Here’s what I noticed in his report:

1) Authenticity: he was upset and frustrated at his mistake of forgetting the combination once he set it.

2) Obsession and drive: he said he worked on the lock for many hours. He never asked for a hint, but I felt bad and gave him one, which he accepted and improvised ideas around.

3) Cognition and recall: he remembered seeing a video about hacking a lock (be careful, Mom).

4) Integrity: he admits he messed up the soda can hack.

5) Curiosity: he tried things.

6) Inquiry: he asked questions.

7) Humor: “where is my half million?”

He also gave me a web reference to where he had seen the soda can lock hack, but for security and ethical reasons, I’m choosing not to disclose it.

This is a wonderful combination of skills and traits that make me think this kid has a bright future.

Yes, I had rolled my eyes when Steven mentioned something about needing a coke can to open the lock.  He explained, but I could not visualize what he was talking about.  It was no different than working with someone with a thick accent trying overseas. I needed him to show me.

The next day, he did.  He found a can, cut it up into little pieces and used one of those little pieces to insert around the hasp in an attempt to make a shunt for where the tumbler met the lock.  I thought that was pretty cool.  I obsessed about it with him for several minutes.

I think he came up with this idea because of his guilt for forgetting the combination he had set. Call it peer respect, but it seemed important for Steven to make sure I could get this thing open before I left.

That blew me away.  Unlike my teenage nephews who could not care less what I do for a living, this young man was engaged and engaging.  They call these kinds of kids “problem children” in school, but give them a real problem to solve that interests them and the “problem” goes away.

His hack didn’t work, but that wasn’t the point.  When his mother said he felt awful about forgetting the combination, I knew it would take a mere 3 seconds to think of what I would say to him. I told him I saw more heart and energy in his idea than I see in professional testers sometimes, and even though it didn’t open, I would have a challenge for the plane ride home *and* a story to tell about it at the next conference.  At the very least, I said, I’d have a blog entry about it.  And here it is.

I’ll have to take his Mom’s word for it that I made a difference to him somehow, but even then that’s not my aim.  Steven confirmed for me that the spirit behind the lock exercise is a good one, no matter the age.  Maybe we’re all just grown-up 12-year-olds looking to apply ourselves to something that needs our skill and insight.

Well done, Steven.  Given more time, you would have gotten that lock open, but that’s beside the point. There’s a future in this business if you want one, and all you have to do is show up and try different combinations. I can’t think of a better life metaphor than that, so thanks for the life lesson.

NOTE: Steven tried some four-letter words above despite it being a 5-letter lock, but he forgot to mention (and so did I) that there was a blank space on the 5th spinner, so the ideas were legitimate tests.