The product review is a time consuming, risky endeavor that – if done properly, for the right reasons – can payoff big (no, HUGE!) for a startup. The key to running an effective product reviews program is to know when not to do it. Marketing gets pressured into doing product reviews. The pressure is mostly self-imposed but it is also frequently externally imposed. The people who do not understand the intricacies of a product review program will be the ones who inevitably blurt out, “why aren’t we in these product reviews” or “we need to do product reviews!” Not quite folks … not quite.
Product reviews are a two-edged sword. It’s not something that marketing does or doesn’t do in a silo. It’s something that the entire startup organization has to stand behind not merely with rhetoric or cheerleading but with concrete sweat, tears, and time resources. I’ve been a hero as well as goat in the past as it relates to product reviews. Through it all, I’ve picked up a few things and I’d like to share it with our Marketing Joint family of fans and readers.
A great product review is powerful for the following reasons or benefits – whether real or perceived:
1. They are conducted and published by an independent (and, presumably unbiased) third party (i.e., press) and thus, great at establishing a credible reputation for quality.
2. They are much more in-depth than other forms of editorial and thus, highly informative.
3. They can validate product claims either explicitly or implicitly.
4. They can endorse products either explicitly or implicitly.
5. They create exposure or awareness for a product.
6. They support all other marketing efforts (i.e., help win awards, increase chances of getting speaker slots, serve as sales collateral, etc.)
7. They can improve product launches.
8. They are an avenue for feedback.
9. They can improve overall relations with a publication and its editorial staff.
10. They can turn reviewers into active or passive evangelists for your product.
“Know What You Want, Know What They Need”
The most important stage of any product review program is the planning stage. This is the time for constructing an argument either for or against doing product reviews – or the pros and cons. In the pros camp, the list above is reason enough to want to do product reviews. But, remember that only a great product review outcome will get you any/all of the above benefits. If you have a mediocre review, many of the items in the list above are neutralized. If you have a bad review, almost all of the listed benefits are gone (except maybe #5 which, frankly, is a bad thing).
It’s easy to know what you want – a great review. To get a great review, you have to understand what kinds of reviews are available and what each reviewer needs – and, why. Let’s take a step back here and look at the various forms of product reviews. Each of these forms needs to be understood in the context of your product offering. Don’t just think about product suitability here. Think also about the stage of the product lifecycle (stability), customer acceptance (maturity), and market environment (competitiveness).
Editorial-driven standalone product reviews
These reviews generally have the lowest risks and lowest rewards. The publication will write about your product without actually touching the guts of the product or testing it against a benchmark or staging any form of a standardized test. Consumer products are sent to the reviewer but infrastructure products are rarely shipped. Most of the risks pertain to reviewers themselves. How you prepare and position the product, support it with proper documents, and treat the reviewer throughout the process will weigh just as importantly as anything else.
Test-driven standalone product reviews
These reviews have moderately low risk and high rewards. Typically, you’ll work with the reviewer to define or refine a test criteria and test bed. In most cases, you’ll also be allowed to directly participate throughout the testing itself. These reviews tend to take a long time as well as enormous amounts of staff resources so you’ll only want to do it for new products or line extensions – not minor releases. These reviews are very similar to customer trials or tests. Product is always shipped. It takes a lot of technical support and intense engagement. A lot of unexpected things go wrong in any trial so you have to really mitigate these risks by putting your best people on these reviews. Unfortunately, your best people are also juggling your key accounts and chasing critical deals. Thus, the main risks are really on the potential disruptions and the ramifications of those disruptions on the ultimately outcome of the review.
Editorial-driven roundup reviews
These reviews have high risks and moderate rewards. The very fact that the review is not testing the guts of the products (i.e, cracking it open and dissecting it) but containing opinions means that these reviews are inherently ticking time bombs. Without hands-on facts, the reviewer is inevitably going to base a lot of the editorial on what is suggested and claimed by vendors. More reviews fall into this category than people think. I’d almost go out on a limb and say that most reviews fall into this category. A reviewer may claim to test products but all that will really be done is to compare spec sheets and run a few specific tests that yield incomplete results, or worse, misleading results. You shouldn’t ignore these reviews entirely. If you can be the one who initiates the roundup review, it could be worth doing. But, if another vendor got the ball rolling, think it through really, really hard. Most often, the deck is already stacked against you. You can overcome it but the decision making now moves into a “likelihood of overcoming risks” assessment instead of weighing the “risk vs. rewards.”
Test-driven roundup product reviews (“bakeoffs”)
These reviews have moderate risks and extremely high rewards. But, the high rewards come with a disclaimer: only one vendor will win and even winning may not be beneficial if the “field” is not stellar. A lot of startups overlook the other participants in the roundup. You actually want to involve all of the major players in your category. The quality of the field is very important. The technical risks are easier to assess since these reviews are very specific as to review methodology.
Reviewers for these tests are also pretty transparent when it comes to the flexibility of their test criteria, vendor support expectations, process for handling situations that may come up, and so on. There are some test-driven roundups that are not designed to anoint a winner. In fact, a lot of bakeoffs actually evolve into “dead heat” type of reviews because they go through so many variations of testing that the only fair thing to do is to accent each vendors’ strengths. These reviews are great for vendors and the marketplace-at-large but they are few and far between. It takes enormous costs on the part of both vendors and the publication to conduct such tests. But, if a category of technology is nearing its adoption plateau, these tests should be out there.
Preparing To “Go The Mile”
Every product review is going to be time consuming. If it’s not, you’re doing something drastically wrong. Product reviews are probably 3x to 10x more resource draining than any other form of media placement. If you’re a product marketer or product manager, you can think of a typical review as being similar to an internal test project (whether in-house or outsourced) except with a lot less control, anxiety, and uncertainty since you’re not the one commissioning and running the test.
I’ve lost count of the numbers of days I went without adequate sleep just worrying about a product review. It’s a full commitment. People get this worried when something is worth everything or worth nothing. In the former, you pull out all the stops and throw the kitchen sink at the matter. In the latter, you just pull out. In either case, product reviews should only be pursued if the black-and-white full commitment of the entire startup organization exists.
The first step to preparation is NOT getting buy-in from management. You’re not looking for buy-in. You’re really looking for full commitment and ongoing support. Before you get commitment, you have to do the homework. This homework takes shape in the way of a full, goal-driven plan which clearly lays out the goal of the review, risk/reward assessment, resource needs, cross-departmental commitment and accountability, task force formation, and clarity around existing dependencies and their impact value on end results.
Getting a “great product review” is not a goal. That’s just a statement. A product review could be “establishing a new priority for evaluating content switches by moving customer attention to usability and manageability over raw performance.” Or, the goal could be to “crystallize our position as the mid-market player in our space.”You get the idea. Having clear goals will help you figure out which form(s) of product review is best suited as well as assist in gauging the right level of cross-functional involvement and commitment.
Product Review Axioms
Here are some laws of conducting product reviews that apply to most situations.
Product reviews directly involve many parts of your organization
The reviewer will initiate discussions with marketing. They will then need to engage a product manager to understand the product deeply, iron out kinks in the test plan, stage or acquire test bed, etc. Reviewers then may need to call for help during the course of actual testing. Upon completing the tests, reviewers will need supporting material (maybe even analyst or customer references) oftentimes from marketing or PR to add some editorial muscle. Some reviewers even check out a vendor’s customer support organization by calling them. This is why every product review program must be a collective, startup-wide effort. And, you need to have a single point person to run it. Make sure that it’s a very senior person who can be fully accountable for the outcome.
Product reviews involve products and people. The people factor is critical.
Tests are rarely ever standardized and fixed in stone. They’re usually very flexible and controllable. Hence, the relationship with the reviewer is critical to a successful product review. The level of trust you build or establish with a reviewer coupled with the proof of commitment you express through being accessible, resourceful, and helpful will be the difference between great vs. bad. I never try to manipulate what reviewers think or do. That’s a recipe for disaster. I always try to get the reviewer to be the most knowledgeable person outside the company as it relates to our products. This accomplishes two things. First, it’s harder for a reviewer to outright dislike something he knows extremely well. Even when he dislikes something, he’s likely to temper the criticism because I’m genuinely willing to listen to the feedback and consider it for future products. Second, I never assume that trust can be established on my own. I always try to get customers, analysts, and other journalists (especially from the same publication) to help build that trust between myself and the reviewer. It could be as simple as sending press clips or favorable quotes. I’ve gone so far as to get on a plane and fly out to a far off location to meet the reviewer over dinner. Anything you can do to establish real trust is a necessary thing – not simply a bonus.
Product reviews can be hidden time bombs. Be paranoid at times.
You really have to pay attention to some very biased reviews. These are designed from the beginning to be highly biased. I’ve seen some of these up close and learned through back door sourcing that they were paid-for assignments. The lesson I learned is that you have to allow some paranoia to exist upfront so you can thoroughly investigate the situation before it’s too late. In one specific case, a major trade publication contracted a firm to do a roundup product review. This firm received a sweetheart deal from a particular vendor through which the web hosting firm could resell the vendor gear to its clients at a steep discount not typically offered to traditional channel partners. In exchange, the firm would essentially co-opt the vendor’s marketing through a review commissioned by a publication.
You can guess who “won” the review. Yup. The publication had no idea of the backroom deal. All the vendors had no clue as to what hit them. I only found out because the marketing VP who actually cut the deal told me a couple years later when he left his company! Is this unethical? Absolutely. Does it happen? More than you think. How do you avoid it? There’s no clear cut way to identify it but there are some signs you should look out for. Does the roundup have a strong field of participants? If no, be weary. Does the reviewer have a long track-record of doing reviews? Does the review represent a core source of revenue for the reviewer or is it just an additional project? Does the reviewer have a long track record of working with the marketing head of a particular vendor? You get the idea. Investigate the situation down to the roots when you suspect a loaded review. Don’t be naïve. Business is not always about fair play. Avoid traps.
Product reviews take a long time and begin pre-launch.
Duration estimates for reviews vary wildly but in my experience every review takes anywhere from 6-12 weeks from initial discussions to when copy is submitted to editors. I’ve even seen one take half a year! If the review is intended to support a launch, you have to start the review weeks in advance of all other outbound activity. The review actually needs to start between the alpha and beta stage of release. There are advantages to shipping beta products to reviewers. From their perspective, the sooner the better so that their review doesn’t become stale by the time it is published (again, due to the lengthy process of producing a review). From your perspective, you get a bit more breathing room when what they’re reviewing is a beta product. It’s kind of a trade off you and the reviewer makes.
Beyond those reasons, it’s simply hard to orchestrate a review to get published as your product officially launches. You always want a product launch not to be an event in and of itself but rather the instigation or start of a sustained campaign to drive awareness and revenue. So, you want to spread out a few reviews from a resource standpoint as well as publication date or appearance standpoint (to keep the momentum going). Just keep the time horizon in mind when you’re planning reviews.
Product reviewer guides are good to have but not mandatory.
People overestimate the importance of Reviewer’s Guides. I get pretty amused when people panic over not having a top notch Reviewer’s Guide or handbook or whatever. For product reviews, great people can overcome bad or non-existent Reviewer’s Guides. But, a great, earth shattering Reviewer’s Guide will never compensate for mediocre people. If a review is important, get your best people across the board actively engaged in the review. Some people overly rely on a Reviewer’s Guide and put down their guards – and get punished for it.
Product reviewers are not customers.
While it’s true that reviewers will try to simulate a ‘live’ or real-world environment, they are going to go far beyond the customer use case. Customers are trying to do something specific. Reviewers are really working off a blank slate. They will want to try a bunch of things. A customer will buy something that does 10 things and spend 99 percent of their time using only 3 things and asking for 2 additional things you don’t even have! A reviewer will actually tinker with all 10 things! The big risk is that some of these 10 things are very hard to test. It’s the vendor’s job to anticipate this situation and mitigate it by making those complex parts easier to understand and test. In a competitive roundup, you must also figure out what part(s) of the competitor’s products are easiest to test then mitigate it by preemptively countering it as either a tradeoff or downplaying its importance in the scheme of things. Again, make sure you mitigate it only if you fully believe it. Credibility is everything.
Ok, that’s it for now! Hope you enjoyed this post. It’s a lot of what is naturally a simple but worthwhile reminder on how you should deploy your product review program.
- John
Comments