lightweight QA
You’re a PM at a startup, you’ve got more engineers that you can shake a finger at. The backlog is hangry. It needs more calories story points. Your deploys are continuous and—if you’re lucky—you might have time to test that what’s being shipped meets its acceptance criteria. You receive word that your feature doesn’t work on IE [insert incredibly old version here]. There are questions as to how it got past QA. QA? What QA? You are QA.
Take a breath. Product is a TEAM SPORT. No, really… it is. You might be acting QA right now, but the truth is your whole product engineering team should be acting QA. The good news is, with a little planning, you can have a pretty lightweight QA process that will cover your biggest needs.
Oh god do I need a test plan
Anyone who’s ever tried to develop an extensive test plan knows the following:
It’s exhausting
It’s not for the faint of heart
It’s nearly impossible
You only ever get 80% completed constructing a test plan (Seriously, it’s like infinite divisibility. Zeno’s paradox. That last 20% just keeps going on and on and on and on and…)
If you’re asked to make a test plan, run away. RUN AWAY
Good news, fam. YOU DON’T NEED an exhaustive test plan. As it turns out, exhaustive test plans pair with waterfall software development rather well (as well as a test plan can go), but just don’t make any sense in our agile development world. Besides, that’s not how Google does it.
How does Google do it?
To be honest I have no idea… I’ve never worked at Google. But according to the interwebs—that are definitely always right—they use the ACC Framework. Attributes, components, and capabilities. I was first introduced to this by Heather Wells during my time at Zendesk, and it really resonated with me.
Here are a bunch of people who talk about it:
https://testing.googleblog.com/2011/09/10-minute-test-plan.html (the blog post written by the OG)
https://www.amazon.com/gp/product/B007MQLMF2/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 (the book written by the OG)
https://99tests.com/blog/log-bugs-google-way-acc-methodology/
Directed testing using the ACC Framework
Here’s the tl;dr of ACCs.
Attributes (think: adjectives & adverbs)
Attributes are the descriptors essential to your product’s brand and value promise. Attributes for payment software like Venmo could include accurate, secure, and timely, whereas attributes for social products like Instagram might include easy, engaging, and enticing. Pardon the alliteration.
Different parts of a single product can absolutely have different attributes. If I think back to the Zendesk Support product, I can easily assume that workflow tools and the agent interface will have pretty different attributes, but both might include Zendesk’s mantra of “beautifully simple.”
Components (think: nouns)
Components are the basic building blocks of your product or feature. The amino acids to our product proteins, if you will. (Who would though. Honestly, Erin.) Components are not as small as something that could be described as a single user story, but rather the size of a group of user stories or even a lower-level epic.
Let’s look at this blog for an example. Yes—the very one you’re reading right now! Here are the components I spot:
Post body
Share options
Comments area
Comment compose box
Paginator
Menu bar [which would itself be broken into its own set of ACCs]
Capabilities (think: verbs)
Finally, capabilities are the things that components should do. What are the things this product or feature must be able to do to fulfill its promise to users? My blog must provide these capabilities:
A viewer can read a post
A viewer can react to a post
A viewer can share a post
A viewer can navigate to the previous entry
A viewer can navigate to the next entry
A viewer can comment on a post
A viewer can view other comments
A viewer can like other comments
The ACC matrix
The final step is mapping which capabilities belong to which component and to be tested against which attributes. It makes more sense visually:
On directed vs exploratory testing
The ACC framework is a perfect example of directed testing. Directed testing is simply making sure products work in their expected manner in the most standard configurations. Basically, does it do the thing it’s supposed to do when it’s expected to do it?
But there’s a whole other piece of QA that is super easy to miss when we’re feeding the backlog and testing the acceptance criteria and going through the ACCs and pitching the vision to the company and sitting in on a sales call and talking to customers and… you get the point. Exploratory testing is literally undirected testing. Put your product in front of a 3-year old and watch them get it twisted into a knot of unexpected states. The entire point of exploratory testing is to find the things that we didn’t even realize people might do because it’s so not what we meant to happen.
When logging bugs or “wait what’s supposed to happen here”s found during exploratory testing, do make sure you log them against the components you identified in the ACCs.
None of this excuses engineers from writing good unit tests
Say it with me: “QA is not a substitute for unit and integration tests.” That’s right. QA is necessary but not sufficient, just as unit tests alone are necessary but not sufficient. Unit tests answer the question, “Does the code do what it’s supposed to do?” whereas QA testing answers the question, “Does the product work as expected and handle unexpected things gracefully?”
Other things you should document and use during QA
Common user states
All products have weird quirks that result from the built it fast culture of an earlier startup. Often there are weird dual states from “legacy” features, and there can also a lot of intentionally built-in complexities from various plans and pricing models. Document the ones that come up over and over!
At Zendesk, user states would include each base plan, as well as (potentially) any add-on combinations. For example, an admin on the Enterprise plan with the Collaboration add-on may see some different states than an admin on the Enterprise plan, who might see a different state that an admin on the Team plan.
At Patreon, user states would include the creator’s chosen plan as well as their payment model. For example, a creator on the Lite plan with a per post payment model will see some different states than a creator on the Pro plan with a monthly payment model.
Supported browsers
Yep, it’s true. If you’re dealing with web development in any way, this is a fact of life. Find out what browsers you support, and document those in your test plan as well. You’ll want to test on the major versions.
If you don’t know what browsers you support, perhaps because no one’s ever asked the question and no one’s ever decided it, find a nearby engineering manager and lovingly hand that problem off to them. (Thanks, Derek!)
That’s all I’m gonna say about that.
Real talk
I am not an expert on QA, and I’ve never played a QA engineer on tv. So if you’re serious about wanting to go down this path, I’d highly recommend reading some of the resources linked above, going down a Google rabbit hole yourself, or hiring an expert (like Heather). What I do know is it’s damned hard to get it all done when you’re a PM at a startup. You may not have the power to hire a QA engineer, and that’s ok. Do yourself a favor and empower your team to develop ACCs and run QA alongside you.
