Best Tool for Designing Journeys

Pretty regularly, I see the question "What's the best tool for designing marketing journeys?" The common suggestions are LucidChart or Visio, but I think it's much simpler: The humble whiteboard.  (In a video call, if you don't have a whiteboard, a large piece of paper taped to the wall will work.)

Journey Design is best done as a group exercise.  A lot of business requirements will surface and a lot of detail that would otherwise be forgotten will organically come up in conversation.  It may only be a first round prototype, but it'll go much quicker than if you just try to gather requirements and then go off on your own and try to draw it in some visual computer program.

Instead, gather people together (in person or virtually) around the whiteboard.  You want people to start talking about the "user experience" - getting business requirements for a journey can be one of the more fun collaborative efforts and is really helpful for uncovering expectations people forgot about.  I think I landed a job once because I did this with the CEO and CTO of a company during an interview. Here are some of the highlights to get people talking aloud:

• How do people become eligible for this journey/flow? (What is the criteria or logic? Do we have the data? Where is it? Do we need to do anything to the data before we can use it? Are there any suppressions? (i.e., groups of people who would be eligible but we shouldn't send, such as someone with an open Customer Service case or someone who is a known hostile customer or an identified cohort which will perform better without the experience, such as whales?)
   
• What happens to a user during the journey/flow? What marketing (emails, texts, ads, direct mail, phone calls, etc.) do they get? What's the timing or cadence? Are there any splits or testing?
   
• How do they leave? In addition to just exiting at the end, can they be ejected early for any reason?  A status change on their subscription, a new purchase, a return/refund. Do we have the data we need to evaluate for that? Are there Customer Service reasons to exit or pause the journey?
   
• What happens to other marketing efforts while someone is in this journey? Is there anything they should be suppressed from? Anything that should be delayed? Anything that needs to appear within the journey as optional content if it happens to the user mid-journey?

Get Started with Testing (A Work-Related Post)

from armydre2008

In Marketing, testing is a great way to hone in on what resonates with a customer.

This is not meant to make you an expert in testing, but hopefully get you started with simple testing or help you create better tests if you feel like your testing isn't yielding the results to expected.  If you pursue testing, don't let this be the only article you read on the subject.

Always Be Testing

When you're not testing, you're not learning or improving. There is a dangerous catch though - do not test just to say that you're testing.  Tests should build upon earlier learnings and should push you towards valuable corporate objectives (sales, newsletter signups, etc.) and not towards less quantifiable results (like "time spent on site" if you're not an online game or video platform).

Your testing should be coordinated with others who are testing and you should be documenting all of your testing so that others can learn from it and leverage it as the basis for other tests.

Also, be ready to re-test. When I worked at a video game company, if we tested on one video game's website and had good results, we'd test again on another video game website to see if it was a universal truth or if it was only valid for the audience of the one kind of website. When I worked at a non-profit, we repeated tests against different products because we knew that what worked for recurring giving might not work for one-time-gift appeals.

Vocabulary

Let's start with a few basic terms:

Hypothesis - this is your theory, created in advance, that describes what you are testing and what you expect the outcome to be. This should be detailed and precise.

Test - an attempt to find the relative performance of two or more distinct items over a set period of time.

---

Experiences - this is your control plus all challengers.  I like to label these A for the Control and B, C, D, etc. for the Challengers - this is where the A/B/N comes from.  (The labels help if you've got a table of results and little space.)

Control - this is the as-is experience.  This is your current website or email or advertisement or store layout or whatever.

Challenger - this is one an entity that has a chance that you want to test. On a website, it might be using blue buttons instead of green.  You can have multiple challengers in a single test.

---

Metrics / Success Metrics - this is the one or more things being tested.  I like to label these 1, 2, 3, etc. The labeling also helps if you've got a table of results but little space. If your tests are self-optimizing, metric 1 should be the one that's the trigger.  Often you're measuring rates (click rate, sign-up rate, purchase rate), not a raw number.

Conversion - this is my catch-all term for a tally mark counted towards a success metric. So it may be a click on a button, a view of a video, an actual sale. This is the term most likely to get me angry comments suggesting I'm a hack and that I should be calling them something else.  Comment away.

Results - this should prove or disprove your hypothesis.  The only failed test is one that is inconclusive. Otherwise, even a test that performs contrary to your hypothesis is still a chance to learn.  If you find that you are consistently wrong in your hypothesis, you may need to let someone else start coming up with future hypotheses.

Statistical Significance - this tells you that you can trust your results.  This is the likelihood that if you repeated the test you'd get the same results.  This is a really important topic as failing to achieve statistical significance makes your tests worthless and leaves you defenseless if someone challenges your results.  This can't be stressed enough.

Lift/Drag - typically you report on the performance of the Challenger over the Control, this tells the direction.  If it's not statistically significant, substitute the terms "Trend Lift" or "Trend Drag"

Winner/Loser - the experience that performs best towards your desired success metric, with statistical significance

Added Learnings - this is something you've observed, but not something you were specifically testing for.  If these seem valuable, it is important to write a hypothesis and give them their own test.

---

Multivariate Testing - a whole different topic. This is where you're testing multiple items at the same time. It might be button color, wording and photos. The math is far more complex and it's harder to know which item or items caused the change.  Best done with a website testing tool.

Self-Optimizing - some website testing tools will start to start to show the trending winner more often. If you're testing to make money, this helps you start to make more money quickly.

Ground Rules

Having some ground rules that apply to all testing will be beneficial when results are hard to understand, when someone's pressuring you for results, if you are wondering if you need to invalidate a test and start over or if you're trying to decide if a test is even worth performing.  Create your own to meet your specific needs.

Here are some of the types I've used in the past.

1. A test must run for at least one month and no more than three months.

Why: You need a reasonable timeframe. If you try to read the results too quickly, it could be a fluke or be influenced by things beyond your control - though it is possible that you've hit on some magic and found something so in need of optimizing that it was an instant success.  But, if you drag things out too long, again, things beyond your control can influence your test.  If you can't get to statistical significance in three months, it would also suggest that you're (a) testing something that has no impact on your customers; (b) you don't have enough traffic or (c) you're not being bold enough in your testing.

2. The test must receive at least 10,000 impressions before any results can be assessed.  

Why: Similar to #1, this helps make sure that you've got a big enough audience.

3. A test must receive at least 5,000 impressions a week to remain viable.

Why: If the volume is too low, it may not be worth it to test.  Either you're running too many concurrent tests or you're optimizing for something that may be of low value.

4. An experience must have 300 conversions.

Why: Similar to #3, tries to make sure you're not spending time on low value tests.  Of course, if each conversion is the sale of an Aston Martin, that's different. And an example of why you should establish your own ground rules.

5. Anything less than 3% lift or drag is considered "no change."

Why: You want big changes.  You want to take the kinds of risks that give you solid learnings.  If you can only push the needle a small amount, you're being too conservative in your testing.

6. Results over a year old should be re-tested.

Why: Things can change so much in the course of a year. Technology, world events, consumer tastes are ever evolving. What worked a year ago may make no sense now.

7. A test cannot have more than 4 experiences (control + 3 challengers).  

Why: If you can't pick the best four experiences, you may need to think about what you're really trying to measure for. It might be that instead of questioning whether your buttons should be red, blue, green, purple, orange or yellow you should first be testing whether you should have cold or hot colors.

8. Tests cannot cross seasonal events or major calendar events relevant to your company.

Why: This is especially important if you're in a different group from your advertising team.

9. Control/Baseline/As-Is has to be measured during the test

Why: As the old saying goes, past performance is no guarantee of future results.  You need to know how your control is performing at the same time as the challengers. If you can't do this, your test results will be suspect.

Let's Test!

Step 1 - Form a Hypothesis

"I believe that green buttons on the product page will result in more clicks to put items in the shopping cart than our current blue buttons.  A previous test (link here) showed that when we changed the button size from 20px tall to 40px resulted in more clicks (and more sales). This further tests the notion that our buttons are often overlooked."

Dissecting the hypothesis:

• Control - blue buttons on the product page
• Challenger - green buttons on the product page
• Theory - blue buttons are overlooked by our audience ultimately causing fewer sales
• Builds on - past test to increase button size
• Expected outcome - higher click rate for challenger 
• Measurable metrics - clicks, visitors per experience
• Baseline/As-Is - click rate (clicks over visitors) for the blue button control during the test timeframe

Plan carefully to avoid creating a reporting nightmare (see below).

Step 2 - Build the test

You'll need to make sure you can serve up the different experiences and measure the outcome.  If it's a website, can the website display the different experiences and track the results independently?  You need to know how many people saw the blue button and how many people saw the green button and how many clicked on each button.

And then you need an easy way to get to the results.  If your dev team is building this for you, the work's not done until you have a way to see the results and a way to turn the test on and off.  There are a number of tools that can make easy work out of this so that you don't need a dev effort to do a website test.

Of course, if it's an email, it's just a matter of splitting the audience and sending different versions to each audience. If you're testing store layouts, you might have to have multiple stores in each design to test.

Clear away any other factors that could influence your test.  If some of your acquisition funnel traffic performs differently, steer them clear of the test and do not include them in the test results.

Step 3 - Check the results

Based on the traffic, I like to present results every 1 to 2 weeks during a test.  Sometimes other stakeholders may decide to end a test, either because the results aren't telling us, or because we've reached statistical significance/confidence and it's time to move on to the next test.

You are always measuring in pairs.  If you have two experiences and one success metric, it's one measurement.

Click Rate of Blue Buttons vs Green Buttons =
A vs B (blue vs green)

But, if you get more complex, your assessments need to get more complex.  Four experiences and one success metric is six measurements:

Click Rate of Blue Buttons vs Green Buttons vs Red Buttons vs Yellow Buttons =
A vs B (blue vs green)
A vs C (blue vs red)
A vs D (blue vs yellow)
B vs C (green vs red)
B vs D (green vs yellow)
C vs D (red vs yellow)

Two experiences and two success metrics is two measurements:

Click Rate of Blue Buttons vs Green Buttons
and Sales Rate of Blue Buttons vs Green Buttons =

Clicks: A vs B (blue vs green)
Sales: A vs B (blue vs green)

And of course four experiences and two success metrics is twelve measurements.

The Super Very Good Absolutely Important Notion of Statistical Significance

In short, Statistical Significance asks "If we run this test again and again and again, will we get the same results?" It involves a lot of heavy lifting of your pairs of data.  You may hear talk of "p-values" but for our purposes, we simplify with lift/drag or winnner/loser and "confidence."  The best site on the internet to make this measurement is kissmetrics.com/growth-tools/ab-significance-test/

How it works: You punch in your impressions and clicks and it will give you conversion, lift and confidence.  Experience (B) is performing better, just from the conversion rate, which you can get yourself with Excel. Even though it has a lift of 2%, it's not yet statistically significant.

But, compare with this one - the lift is greater, the confidence is greater and now we've reached statistical significance.  (You should feel good when confidence exceeds 95%.)

Enter these (or other numbers) into the tool to see how the results change.

If you work with someone who regularly likes to question facts or "go with their gut," statistical significance is a powerful argument.

Happy Testing!

Your first tests are like the biggest chisels and hammers applied to a block of granite - they can get you to a rough outline for focus. But you need to build upon those tests with new tests with greater precision and focus. Refine with more specific tests or to more specific audiences and soon a picture will emerge.  Because if you don't refine, all you'll end up with are a bunch of rocks and dust.

One last thought... what if you just have data?

If you had no hand in deciding what was captured, how it was captured, when it was captured or who the participants being measured were, then all you can do is attempt to make forensic guesses about what happened -- which you then need to validate with new tests.

(Cross-posted to LinkedIn)

Tru Love / @trutv @trubyhilton

Do the TVs at tru get truTV?





Hilton's Twitter handle is http://www.twitter.com/TrubyHilton. Who is Truby? Shouldn't this be truByHilton?

Celebrating New Customers

(cross-posted on LinkedIn)

Much has been said about enticing prospective new customers with deals you wouldn't give your existing customers. (The courtship is over, no need to keep wooing.) The unfairness of it all. Fine. No need to revisit that, then.

But I got this in the mail the other day and it made me think about the other side of the deal. The courtship is over. I've said yes. Now what?

Recently jumped to the other side of the local telecommunications duopoly. To hear them tell it, they've changed their ways. They may be one of the most hated companies in one of the most hated fields, but according to them, they've recognized the error of their ways and have been trying to be better.

To their credit, the installation window was short, the installer early (calling ahead first to make sure early was OK), they were friendly, personable and showed no outward signs of annoyance when they realized they had to run a new line from the street when we discovered I had severed the existing line. Same of the different telephone support I worked with in the U.S. and Philippines, personable, friendly, knowledgeable and accommodating. Upsell attempts for additional projects weren't pushing or overbearing.

So I was actually quite pleased to receive an envelope marked "Inside: Something just for you from [brand]." (Yes, it's clear what brand this is, but this isn't a post about this brand, it's a post about what happens after the deal is done.)

What could it be? A thank you post-card showing a bunch of happy employees smiling and waving? Maybe a surprise $10 gift certificate to a local restaurant*? Maybe a discount off of a future bill? A free limited-time upgrade? A little 2016 wall calendar with some nice nature pictures and their logo**? A special customer service number? A survey?

Sadly, it was none of these. Instead, it was a list of additional products I could buy. We had just completed a transaction in which both parties had satisfactorally agreed on the terms: I want this product you offer and am willing to pay the price you have asked. I have endured your upsells during sign-up because you've been pleasant enough and I know that you have to ask. I get it. That's fair. But I was quite clear on what I wanted, and was even willing to give you reasons why your other offerings weren't for me.

But the ink is barely dry and now you're coming at me for more money? I'm not even sure how it was "just for me" unless it didn't include the one item I already purchased, but I didn't get that far, dropping it instead in the recycle bin thinking "same old, same old." Because if you had listened while you were courting, then you would have known not to send that.

What are you doing with your new customers? Your customer always has a choice. Today, they made a great choice and chose you. Affirm that decision and continue the courtship to make sure that they continue to choose you. Surprise, delight, be excellent. Be the company your customers and employees can't stop gushing about.

So, let's think specifically about Xfinity or Comcast (seriously guys, what are you called? I'm so confused.) - can you imagine someone saying "You have Comcast, too? Aren't they awesome!" and high-fiving?

Or Tweeting about being excited to have an appointment scheduled for new service?

Or running out and putting a Comcast bumper sticker on their car?

Or standing in line outside the store to buy the latest router?

You might be thinking "It could never happen in a million years." I don't believe that's true. I believe that if they wanted to, Comcast could be adored by its customers, loved by its employees, admired by its suppliers, contractors and competitors. I believe books could be written about the transformation.

You probably snorted derisively, but if you think about it, you'd probably begrudgingly admit that it is possible, but that you don't believe that the leadership of Comcast would ever bother.

But if it's possible for Comcast, then it's possible for your business. And undoubtedly much easier. So... what are you doing to affirm the great choice your customer has made?

*Pro-tip: Getting your customers to do business with each other is a great retention and word-of-mouth strategy.

** Pro-tip: For our house and home-office, there will always be a use for a small calendar that we can pin to a bulletin board or attach to a magnetic surface.

Feed Sift (06/12/2015)

Happy Friday, my friends! Here's a few more articles I wanted to share. Have a great weekend!

-1-

CRINGELY.COM -- Your PBX has been hacked!

-2-

DJC.COM -- Seattle might keep going

-3-

CHROME.BLOGSPOT.COM -- New Research: The Ad Injection Economy

-4-

LIFEHACKER.COM -- Outdoor Home Repairs You Can Totally Handle Yourself

-5-

BLOG.LOSEIT.COM -- Snacks on a Plane: 8 Ways to Lose Weight on Vacation

Kudos, @NavyFederal

Managing a brand in today's interconnected world is difficult - what is social media? Marketing? Branding? Advertising? Customer Service? Crisis Management? All of the above.

I got an email this week from Navy Federal Credit Union. I have not served in our nation's armed forces, but my dad served for a number of years in the Navy. The story, as I remember it, is that my parents brought me into the branch when I was a few weeks old and a teller my parents had a regular ongoing customer relationship with opened an account in my name with $20 of her own money (you can have an account as a dependent of a serviceman).  The two biggest pieces of financial advice my dad has consistently delivered over the years are "Don't end up with a rebate on your taxes - that's a tax-free loan to the government" and "Whatever you do, keep your Navy Federal Credit Union account."

I haven't used the account much over the years - every so often I add a little to it and it quietly earns interest.  They've also come in handy with great rates on car loans and debt consolidation.

A few days ago I got an email promoting boat loans. I thought that was kind of funny, considering most of its customers have probably served on a sea-going vessel much larger than they'd ever be able to get a loan for.  I didn't know if it was a good idea (you can get your own boat and finally be captain), or silly (why would I buy boat which will always feel small compared the ones I served on - plus I remember the constant maintenance).  I decided to tweet about it, but to take a more absurd train of thought, because, well, that's how I think.

Navy Federal originally misunderstood and their security mindset jumped into action.  I had to explain that, no, in fact it was a joke that just didn't work as well as I'd hoped.  (Another one of my followers got it - they retweeted it.)  

Once NFCU understood, a more playful side came out. So I have to give props to them for having that balance - security-minded, but also empowered/allowed to have a more playful, informal interchange with their customers. So, kudos, @NavyFederal - it's not easy being a bank in a social era.

Feed Sift (05/16/2014)

Five things I wanted to pass along...

ENGADGET -- Microsoft tests Live Tiles you can use without leaving the Start screen

SETH GODIN -- You are not the lowest common denominator

BASIC INSTRUCTIONS -- How to Utilize and Counteract Sarcasm

HUFFINGTON POST -- How to Get the Most Out of Your Netflix Subscription (h/t Lori)

I, CRINGELY -- Big Data is the new Artificial Intelligence

Feed Sift (05/09/2014)

Five things from around the web that I wanted to share...

TWISTED SIFTER -- This Pharmacy Prescribes Monster Spray to Kids that are Scared of the Dark

GIZMONDO -- Why Roku Matters More Than Ever

EXCELJET -- How to build a search box with conditional formatting

ENGADGET -- Watch filmmakers render realistic CG on the fly using $14k of graphics cards

SETH GODIN -- Thinking lifetime (don't break the chain)

Feed Sift (04/18/2014)

Here's a few things I've come across recently that I thought was worth sharing...

-1-

LIFEHACKER -- Eight Myths About Jury Duty Debunked

-2-

XKCD -- Before the Internet

-3-

Missed Opportunity @LibertyMutual

While I can appreciate the clean aesthetic devoid of all distractions, as a customer, I would accept (if not expect) to see cross-promotional marketing here for other products that might be relevant to me.

I was recently shopping for car insurance and their website wanted all kinds of information.  Several other sources had just creepily pulled this from somewhere but Liberty Mutual wanted me to type it all in by hand.

So I went and figured out my sign-in information, figuring that maybe they'd have more of it filled in if I signed in. I was surprised to find out that there was no way, once I was signed-in to even explore car insurance.

You'd think that you'd want to know who was thinking about buying additional products from you.

Big missed opportunity.

Feed Sift (04/11/2014)

A few things I've read recently that I wanted to share...

-1-

LIFEHACKER -- I'm a Telemarketer. Here's How to Get Rid of Me

-2-

I, CRINGELY -- An impending black swan for electric cars

-3-

Take 2 (A Work-Related Post)

I complained last March about this local business' flyer - 50% off for first-time visitors or 10% off for repeat customers (view the 2013 postcard). I suggested that instead both people should get 50% if the repeat customer brings the new customer.

I won't claim credit for this, but I was quite pleased to see this year's postcards.

Soylent Snack (A Work-Related Post)

I would have thought this placeholder copy, except for the registration mark at the end. But am I the only one who looks at "Real fruit from real people" and thinks "Eeeuww.... what kind of fruit comes from people?" (Yes, real fruit.)

Perhaps I've watched too many zombie movies or I'm just a little bit twisted, but good marketing is clear and leaves no room for misinterpretation. Any time you permit ambiguity, your intended audience may trip over meaning, be confused without context or form incorrect impressions.

This is a great case for a Devil's Advocate - someone who can look at your work critically, seeking to find potential problems before something goes to market.

Feed Sift

Some interesting things I wanted to pass along...

-1-

Thousands support special needs employee berated by customer - A feel-good story. (Today)

-2-

State the Price to Give B2B Sites a Competitive Advantage - It may not be "show pricing" but there's probably one thing that you don't do on your website and it's nagging you. This is why you should do it. (Nielsen Group)

-3-

Who's the Customer? (A Work-Related Post)

The label says "Joe Fresh" but the shirt says 1974.

I don't understand this advertisement from earlier this year. A t-shirt for a city known today for its utter collapse and failure.

Celebrating that team's success. From the year I was born.

On a child that might be the age of my children.

Who's buying this? The nostalgic-sports- fan-back-to-school-clothes-buying-Midwest-transplant-grandfather?

Call the lawyers, @Subway (A Work-Related Post)

Branding is hard work. It takes time and effort, once you've settled on a good look and feel (or better yet, experience). So when someone tries to take advantage of your work, it can be a minor annoyance, or worse yet, be damaging to your brand. And so goes this smoke shop that uses the Subway green and yellow in their signage. This isn't the greatest of pictures, taken from my dash cam, but the greens and yellows of the two signs match to make it appear as if this were the "Subway Tobacco Station."

This is sad (as in pathetic, disappointing, troubling) on several levels:

popcon (A Work-Related Post)

The package or Orville Redenbacher says "with 50% less fat and natural ingredients."

Now why on earth would you be promoting 50% less natural ingredients?

Keep Me (@DirecTV)

So, canceled DirecTV late last week. They tried, but not very hard, to keep us by offering to knock $30 off the bill.  I'm guessing it would be in exchange for another 24-month contract.  We declined and they let us go easily.  Within two days, this email:

This tells me several things:

(1) DirecTV can afford to charge you far less than they are. (DirecTV's just trying to get as much as they can as quickly as possible because they know they'll lose a lot of people quickly.)

(2) A product (Genie) which would have cost me a lot to purchase while still a customer, I can now get for free. (DirecTV doesn't care about its existing customers.)

(3) DirecTV doesn't use the information it knows about its customers.  (We never watched any sports in two years.)

(4) DirecTV is just playing a game. (Notice the fine-print underneath the price where it says football's not included at that price.  Also, I doubt anyone can get the $29 price - there will be taxes, fees for additional services, etc., etc., etc.)

So, yeah, if you like sports, DirecTV is probably a good choice for you.  If you want the latest CBS shows, probably also a great choice.  But if you just want to be entertained, Netflix has a really deep catalog and is a far better and cheaper choice.

Companies who have fans, those companies know retention.  But if your goal is only the next sale, or to lock someone into a contract, there's a problem with your product.  You haven't figured out how to differentiate or you haven't figured out the value proposition.

Or you're chasing after the wrong audience, allowing them to buy your product, be dissatisfied with it and then be disgruntled about it.

My experience with DirecTV was not good. There was a promotional discount period at the beginning which was complicated to actually enroll in, there was a bunch of add-ons that were designed to try to get more money out of me and were difficult to cancel, the promotional marketing was designed to always to try to extract more money from me.  And for the longest time, they couldn't even be read on an iPhone.  (The black line through the hero graphic as rendered above is my fault for shrinking the email, it rendered perfectly.)

Never did DirecTV do anything to affirm me as a customer or partner, I was just someone to shake down on a regular basis, and then come back around and see if I had more money in my pockets.

Better?

I try stay away from too much that's political - for the most part, politics frustrate me - I don't think we're getting these days what was intended when this country was founded.

But I can't resist this.  Jim Ferrell has been a fixture in Federal Way politics for some time now, either as a council member or as the deputy mayor.  I don't know much about him, but if this is expressing dissatisfaction with the status quo, then he needs to be counting himself in that.

So when I see signs like this, it's embarrassing and frustrating. First off, it's a weak slogan. Just about as weak as "It's All Within Reach" (which implies "None of it is here.")

Colorblind (@BestFoods)

When you think of mustard, what color comes to mind? Most people will say yellow. A few, brown and one smart aleck will say "Grey" and you'll glare at them and they'll cheerfully say "Pardon me, do you have" and then trail off as you continue to glare at them and then gloomily say "Yellow." (not from personal experience)

And when you think of mayonnaise, white is probably the only color you think of.  (Bonus if you also think in your head "I order all of my sandwiches with mayonnaise, I'm a whiz at minesweeper I can play for days, once you see my sweet moves you're gonna stay amazed.")

So what was Best Foods thinking?