throw out your spec—meet the problem page

If not a spec, what?

We all know—I don’t love a spec. I don’t like “requirements doc,” and I certainly do not like a waterfall process. That said, we product managers have to do, well, something, right? Probably. One of the key foundational activities of a product manager should be exploring and defending what should be prioritized and why. We can’t just say “no” and expect people to go along with it. That would be much too easy. Instead, we have to justify why certain things are more important than others.

A quick tangential aside on prioritization

One of my pet peeves with some leadership teams is their inability to effectively prioritize. You’ve probably seen this before—a list of 20 company priorities are presented, and 10 of them are P1s, 8 P2s, and 2 P3s. I just want to be clear here… this is not prioritization. This is basically saying “EVERYTHING IS EQUALLY IMPORTANT except for these other things that we will never have enough time to do”

This IS NOT helpful. By definition, you should have more P2s than P1s. P1 is the top of the pyramid. If the top of the pyramid is larger than the bottom, well, first of all, it isn’t much of a pyramid is it? And more importantly, the damned thing is going to topple over. Please don’t crush us with your inability to prioritize, dear leaders.

Enter Problem Pages

Since we know people won’t take our word for it, we need to put pen to proverbial paper and make the case. To do this, I find it very effective to create an internally-facing document for each general set of problems—a problem page. This document should articulate what problem set you’d like to address, for what audience, and why.

While this page should serve up data and facts, its purpose is also to persuade your internal audience that this is, in fact, the thing you should be working on. This is the time to get on your soapbox, to tug at heartstrings. You want a rallying cry that is as persuasive as Daenerys on her best day. You want for someone to read your words and jump up from their desk to CHARGE!

Note: This has never happened to me, but a girl can dream.

Anyway, I love to see this in a wiki format—whether that’s Confluence, Notion, or something similar. It should live in a place that is easy for engineers to find, product people to collaborate on, and internal stakeholders to engage with through comments. This is why I call it a “problem page” and not “problem document that will never change.”

I also want to note that what is covered below only pertains to the first part of the process. This same page will eventually be used to communicate possible solutions, eventual designs, and progress updates when actually in development. But, for the sake of brevity (who, me?) we’ll leave that for a future post.

Before you dig in

By the time you’re writing a problem page, you should already have done a ton of information gathering that led you to the point of creating the page. You shouldn’t need to do explicit research on the what, as that was likely the impetus for prioritizing the problem to begin with.

Before you even start, you should already roughly know the following:

• Who within your customer base is facing this challenge or would benefit from addressing this area

• What the problem is

• How the problem fits into your vision for the product

• How the opportunity fits into your team’s goals

A Word to Newbies

This is one of those things that gets way easier with practice and familiarity with your product area. If you’re new to being a PM or new to this concept, this is gonna suck a little bit. That’s ok. If you’re new to a particularly complex product, this is gonna suck a little bit. I promise you, though, that with time and experience this will be incredibly straight-forward.

If you are new to this concept, I’d highly recommend reading examples that already exist in the world. It might also help to start with a straw man of one of your problem areas, and then pair up with a more experienced PM to flesh it out. Engineers are always pairing up, but it’s just as useful for product managers. Talking these things out have a way of solidifying the reasoning in our heads.

---

So what’s in a Problem Page?

There is a basic, recommended structure to a problem page. The exact headings and sections aren’t worth losing sleep over, but you should make sure the content is there.

---

Overview

Description of Problem or Opportunity

This is an overview of the problem you’d like your team to solve.

It could be:

• A challenge current users are facing

• A capability deficit preventing new customers from adopting your product

• A usability issue preventing customers from finding / using / valuing something in your product

• An opportunity for some tangentially related problem space that isn’t represented in your current product

It should be:

• Phrased in terms of the challenge(s) you hope to tackle

It should NOT be:

• A specific feature

• A description of a solution to the problem

Target Audience

This might seem relatively self-explanatory, but it’s important to make this as broad or as scoped as is needed to really describe who you’re going to solve these problems on behalf of.

Things to consider

• If you have a multi-tiered product, is the problem space most important for a subset of tiers?

• Is there a certain size of customer? A certain captured revenue?

• Is this for existing customers, is it to encourage existing customers to choose a higher tier? Is this to attract new customers?

• Is this meant to impact/help an internal audience? (Very often this might be Support, Fraud, Trust & Safety, Billing, etc) 

Problem Prioritization

To really make a compelling argument about WHY you’re assigning priority to an issue, you may need to go back and gather more detailed information than you  needed when working on the overview section. I highly recommend you have as much explicit data as possible—both qualitative and quantitative.

Customer Impact (Qualitative!)

Get yourself some customer quotes. I’m talking first names of specific people. I want you to be able to write about Andrea who is trying to accomplish this task for her company. It’s a brutally manual process, and she dreads it every month. In the meantime, Wes from this other company is having nearly the exact same challenge.

You’ll want to discuss the FUD (fear, uncertainty, and doubt) of users in this area, and really tug on the heartstrings of your audience.

Supporting Data (Quantitative!)

Collect a variety of quantitative data, which could include any of the following:

• How much this impacts your customer

  • Hours spent

  • Money spent

  • Satisfaction levels

• How this issue impacts growth

  • Blocking deals?

  • Would unlock a market?

  • Contributing to churn?

• How this issue is placing burden on an internal team

  • Billing Team has to do x things manually each month

  • Customer Support is receiving x tickets per week

  • Trust & Safety spends x hours per week addressing this issue

• Don’t forget to quantify qualitative results if you have them!

  • NPS

  • Customer satisfaction

  • Task success

Company Goal Relevance

Time to bring it home. You’ve already tugged on heart strings to show why this is important to your target audience. You’ve queried the data warehouse to prove this is a meaningful pursuit. Now it’s time to relate it back to your team’s goals (which... hopefully roll up to your company’s goals…). How will this move your team toward its goals? How will this help achieve your company’s goals?

Now, this does not have to be a hard company metric type goal. A lot of us get caught up thinking this has to be a number like cohort retention rates or customer acquisition numbers or churn rates, but those are just a subset of the type of goals your team might be pursuing. Your goal might be to increase customer satisfaction, or to increase task success on a particular feature, or even to simply half an answer to a problem area that your competitors already address. (Though, never implement something just because your competitor has it. Perhaps this is a good topic for another post.)

The key here is to make a compelling argument that not only is this problem important to your customers, but it also helps the team move toward its goals.

---

Go forth and write

Let me know if you’ve ever tried something like this, and how it went for you. Do you have an example you’d be willing to share? Any helpful tips to share with your fellow PMs? Comment below! I may pull together some examples for a future post (with permission).

lightweight QA

You’re a PM at a startup, you’ve got more engineers that you can shake a finger at. The backlog is hangry. It needs more calories story points. Your deploys are continuous and—if you’re lucky—you might have time to test that what’s being shipped meets its acceptance criteria. You receive word that your feature doesn’t work on IE [insert incredibly old version here]. There are questions as to how it got past QA. QA? What QA? You are QA.

Take a breath. Product is a TEAM SPORT. No, really… it is. You might be acting QA right now, but the truth is your whole product engineering team should be acting QA. The good news is, with a little planning, you can have a pretty lightweight QA process that will cover your biggest needs.

---

Oh god do I need a test plan

Anyone who’s ever tried to develop an extensive test plan knows the following:

• It’s exhausting

• It’s not for the faint of heart

• It’s nearly impossible

• You only ever get 80% completed constructing a test plan (Seriously, it’s like infinite divisibility. Zeno’s paradox. That last 20% just keeps going on and on and on and on and…)

• If you’re asked to make a test plan, run away. RUN AWAY

Good news, fam. YOU DON’T NEED an exhaustive test plan. As it turns out, exhaustive test plans pair with waterfall software development rather well (as well as a test plan can go), but just don’t make any sense in our agile development world. Besides, that’s not how Google does it.

---

How does Google do it?

To be honest I have no idea… I’ve never worked at Google. But according to the interwebs—that are definitely always right—they use the ACC Framework. Attributes, components, and capabilities. I was first introduced to this by Heather Wells during my time at Zendesk, and it really resonated with me.

Here are a bunch of people who talk about it:

• https://testing.googleblog.com/2011/09/10-minute-test-plan.html (the blog post written by the OG)

• https://www.amazon.com/gp/product/B007MQLMF2/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 (the book written by the OG)

• https://99tests.com/blog/log-bugs-google-way-acc-methodology/ 

---

Directed testing using the ACC Framework

Here’s the tl;dr of ACCs.

Attributes (think: adjectives & adverbs)

Attributes are the descriptors  essential to your product’s brand and value promise. Attributes for payment software like Venmo could include accurate, secure, and timely, whereas attributes for social products like Instagram might include easy, engaging, and enticing. Pardon the alliteration.

Different parts of a single product can absolutely have different attributes. If I think back to the Zendesk Support product, I can easily assume that workflow tools and the agent interface will have pretty different attributes, but both might include Zendesk’s mantra of “beautifully simple.”

Components (think: nouns)

Components are the basic building blocks of your product or feature. The amino acids to our product proteins, if you will. (Who would though. Honestly, Erin.) Components are not as small as something that could be described as a single user story, but rather the size of  a group of user stories or even a lower-level epic.

Let’s look at this blog for an example. Yes—the very one you’re reading right now! Here are the components I spot:

• Post body

• Share options

• Comments area

• Comment compose box

• Paginator

• Menu bar [which would itself be broken into its own set of ACCs]

Capabilities (think: verbs)

Finally, capabilities are the things that components should do. What are the things this product or feature must be able to do to fulfill its promise to users? My blog must provide these capabilities:

• A viewer can read a post

• A viewer can react to a post

• A viewer can share a post

• A viewer can navigate to the previous entry

• A viewer can navigate to the next entry

• A viewer can comment on a post

• A viewer can view other comments

• A viewer can like other comments

The ACC matrix

The final step is mapping which capabilities belong to which component and to be tested against which attributes. It makes more sense visually:

---

On directed vs exploratory testing

The ACC framework is a perfect example of directed testing. Directed testing is simply making sure products work in their expected manner in the most standard configurations. Basically, does it do the thing it’s supposed to do when it’s expected to do it?

But there’s a whole other piece of QA that is super easy to miss when we’re feeding the backlog and testing the acceptance criteria and going through the ACCs and pitching the vision to the company and sitting in on a sales call and talking to customers and… you get the point. Exploratory testing is literally undirected testing. Put your product in front of a 3-year old and watch them get it twisted into a knot of unexpected states. The entire point of exploratory testing is to find the things that we didn’t even realize people might do because it’s so not what we meant to happen.

When logging bugs or “wait what’s supposed to happen here”s found during exploratory testing, do make sure you log them against the components you identified in the ACCs.

---

None of this excuses engineers from writing good unit tests

Say it with me: “QA is not a substitute for unit and integration tests.” That’s right. QA is necessary but not sufficient, just as unit tests alone are necessary but not sufficient. Unit tests answer the question, “Does the code do what it’s supposed to do?” whereas QA testing answers the question, “Does the product work as expected and handle unexpected things gracefully?”

---

Other things you should document and use during QA

Common user states

All products have weird quirks that result from the built it fast culture of an earlier startup. Often there are weird dual states from “legacy” features, and there can also a lot of intentionally built-in complexities from various plans and pricing models. Document the ones that come up over and over!

At Zendesk, user states would include each base plan, as well as (potentially) any add-on combinations. For example, an admin on the Enterprise plan with the Collaboration add-on may see some different states than an admin on the Enterprise plan, who might see a different state that an admin on the Team plan.

At Patreon, user states would include the creator’s chosen plan as well as their payment model. For example, a creator on the Lite plan with a per post payment model will see some different states than a creator on the Pro plan with a monthly payment model.

Supported browsers

Yep, it’s true. If you’re dealing with web development in any way, this is a fact of life. Find out what browsers you support, and document those in your test plan as well. You’ll want to test on the major versions.

If you don’t know what browsers you support, perhaps because no one’s ever asked the question and no one’s ever decided it, find a nearby engineering manager and lovingly hand that problem off to them. (Thanks, Derek!)

That’s all I’m gonna say about that.

---

Real talk

I am not an expert on QA, and I’ve never played a QA engineer on tv. So if you’re serious about wanting to go down this path, I’d highly recommend reading some of the resources linked above, going down a Google rabbit hole yourself, or hiring an expert (like Heather). What I do know is it’s damned hard to get it all done when you’re a PM at a startup. You may not have the power to hire a QA engineer, and that’s ok. Do yourself a favor and empower your team to develop ACCs and run QA alongside you.

eboyle's guide to writing user stories

Rules:

• Thou shalt have a shared backlog with all engineers on your pod

• Thou shalt have all stories, chores, and bugs related to a product initiative in a single backlog

  • Exception: Tech debt or tech-only projects may have a separate technical backlog

---

STORIES

What does a story look like?

Name

As a [persona] doing [x], I can [y]

Description

Give a bit of context of why this story is important to the persona. If this is a part of a large set of stories, reference where it fits into the larger picture

UX/UI

[link to invision]

embedded image when possible

Acceptance Criteria

1. When x is clicked, y happens

2. Edge case expected behavior

What belongs in Acceptance Criteria?

I mean. This isn’t gospel… but this is the law according to eboyle: Any action or interface element that needs to be present, conditionally present, or change through the course of the story should have at least one acceptance criteria. Basically if an engineer is going to have to know the answer in order to build it, it should probably be in there.

Let’s take a look at some examples:

EXAMPLE 1:

As a Creator viewing PRM, I can filter by charge status (because this helps me do x)

Description

We want to add a new drop-down filter! The filter's label should be "Charge status," and it should list the "friendly" charge status labels (paid, pending, refunded, declined, deleted, fraud, other).

UX/UI

[link to invision]

Acceptance Criteria

1. A new single-choice drop-down filter is present to the right of the existing filters

2. The default option is "All charge statuses"

3. When a charge status is selected, the list should refresh automatically

4. If "Reset" is clicked, the charge status filter should clear

EXAMPLE 2:

As a Creator viewing PRM, I can sort the list by pledge value (because this helps me do x)

Description

By default, the list of Patrons is sorted in alphabetical order. We want to add the option to filter by a few other columns as well. In this story, we'll let creators sort by the current pledge value of patrons.

UX/UI

[link to invision]

Acceptance Criteria

1. When a creator mouses over the "Pledge" column header, the cursor becomes the hand (shows it's clickable)

2. When a creator clicks on the "Pledge" column header, and no previous sort was set on Pledge, the list is sorted by pledge value ascending

3. When a creator clicks on the "Pledge" column header, and the list was already sorted by Pledge asc, the list becomes sorted by pledge value desc

4. When a creator clicks on the "Pledge" column header, and the list was already sorted by Pledge desc, the list becomes sorted by pledge value asc

5. When the list is being sorted by Pledge asc, an upward arrow is displayed to the right of the column header

6. When the list is being sorted by Pledge desc, a downward arrow is displayed to the right of the column header

7. When the list is no longer sorted by Pledge, the arrow is removed

8. Upon sort, you go back to page 1

What does an API story look like?

Pretty similar, but with a few tweaks tailored to the real need. If you don’t feel comfortable writing this as a PM, that’s cool - partner with your Scrum Conductor!

Name

As an API Consumer, I can [x]

Description

Give a bit of context of why this story is important to the persona. If this is a part of a large set of stories, reference where it fits into the larger picture

Permissions

Who should have access to this endpoint? Just the creator? Patrons and creators? Patron only?

Inputs

What parameters should this endpoint accept? An ID? an array of IDs? Which of the inputs are required? Will you allow any side loading?

Outputs

What data should be returned when this endpoint is hit?

Acceptance Criteria

1.

---

EPICS

What’s an epic

Text book definition: 

(https://www.atlassian.com/agile/delivery-vehicles)

An epic is a large body of work that can be broken down into a number of smaller stories. For example, performance-related work in a release. An epic can span more than one project, if multiple projects are included in the board to which the epic belongs.

Erin’s definition:

A way for a PM to keep their stories organized so as to stay relatively sane. Usually they align to some chunk of work that delivers value to a customer. I always prioritize within the epics, rather than try to prioritize the whole damned backlog. It’s just too much.

Do I have to use epics?

Absolutely not. If you’re working with only 2-3 engineers and you really only have a small scope of work, there’s no need for epics. Epics are more useful with larger teams who are working on multiple projects at once.

---

BREAKING IT DOWN

User stories should be granular enough that an engineer can complete the story within a day, generally speaking. Each team probably has their own point system, and each team should also have a threshold for pointing a story. Above that threshold, and you’ll need to split the story apart.

Most full-stack features will likely need some backend stories (database schemas, etc), some API stories (CRUD endpoints, for example), and frontend stories. Generally speaking, it’s great to keep these things separate.

Backend Stories

I’ll sometimes stub out the real technical work here (like creating a schema, etc), but mostly… I leave this up to the backend engineers to write.

API Stories

I like to write these out after getting a better understanding of what’s needed from the front end. What information will the page or state of your page need to be provided? In the case of a table or list feature, this comes in the form of which columns I need to display, or what information I need to be able to create or modify within the table.

Frontend Stories

To me, these are the easiest stories to break down (because I’m a visual person). I like to start with the largest, and least specific detail possible, and work my way down to various elements on a page.

For a typical “list” or “table” feature, I’d usually break it down like this:

• The page exists, with the title

• Data is displaying in an unformatted table (and here are the columns)

• Data is displaying in a formatted table

• Pagination controls are present

• Hovering over a row does x

• Button y is present, and when clicked does z

---

ESTIMATING

Are you an engineer? Great. You should be estimating. Are you a PM? Stop. Go get an engineer.

How should you estimate?

It’s cool for all engineering teams to do pointing in a slightly different way. However, there are definitely some best practices in how to think about estimates. Estimates should not reflect the amount of time it will take you to complete a task, but rather the relatively complexity of one task as compared to another. Fibonacci, as it turns out, is great for this.

Let’s say we have the following 5 stories in the backlog:

1. Change the description on Page x

2. Create a new modal that contains a list of data

3. Add a button on Page x that pops a modal

4. Create a new component to manage sorting for a table

A string change is probably… 1 point. If even. Is creating a new modal that hits and endpoint and displays data more or less complex than the string change? Sounds more complex. Great. Now, when you thinking about adding the button in comparison to the modal, is it less complex, or more complex? Probably a little less. And finally, how does the complexity of a new sort component compare?

At the end, you might say that the tasks, in order from least complex to most complex, are task 1, task 3, task 2, task 4.

To assign actual points, though, you have to go a step further. Is a button TWICE as complex as a string? Then 2 points is fine. Is it actually more than twice as complex? Maybe 3 points is a better estimate. The other thing to keep in mind is unknowns. The less certain you are about what it will take to fulfill any given task, the higher the point value you should give it.

After some time, the team should coalesce around roughly what points makes sense for your team, and have anecdotes like, “well a 3 point story feels like x, but a 5 point story feels like y.”

Why should you estimate?

A couple reasons. First, it should help you understand how much you might be able to accomplish in a week.

As engineers, estimates will inform how you think about milestones over time. At some point, you’ll have to say, “we think we might be done with x chunk of work by z date.” Understanding (over time) your point-based velocity will give you a more accurate prediction of this.

As a PM, estimates will help you understand the relative complexity of what may seem like small features, giving you the ability to prioritize more effectively. Is this 5 point story actually providing much more value than this other 2 point story? If not, maybe it’s time to scrap or de-prioritize that 5-pointer.

What even is an estimate?

It’s just that, an estimate. It is not a deadline, it is not a promise or guarantee, and it will often be wrong. It should also more often be directionally correct, and help us get better at things like milestones over time, because it’s a way for us to start quantifying and grasping something that is largely, let’s face it, unknown.

How much time will it take you to solve this puzzle?

Really hard question to answer, if you’ve never solved the puzzle.

How much time will it take you to solve this puzzle, given that it took you x minutes to solve this other puzzle that was similarly complex?

Easier to answer. But still very inexact.

Also, keep in mind that as a product organization we are AGILE. That means our priorities might change, or we might learn something new, or some unexpected thing will come up that will throw our milestone dates RIGHT out the window, even if we because wicked good at estimating with Fibonacci points to begin with.

tl;dr estimates are a tool, and nothing more

---

BACKLOGS

How much backlog is the right amount?

In general, the hand wavey rule on this (no, not just my rule, it’s a real thing) is about 2 weeks. In other words, you should always have 2 weeks of granular, fleshed-out stories for your team.

Can I have more?

I mean, of course. But, I’d encourage you to leave stories that won’t be addressed for >2 weeks at a higher level, like an epic.

What about all the ideas I don’t want to lose track of?

First of all, skeptical face. Second of all, keep an ideas backlog in some other place for you, as a PM, personally.

Also, I break these rules all the time, and should probably go blow really old stories away, because real talk we’re never going to get to them. And if/when we do we won’t remember there’s already a story.