Recommendation Eval Framework V2!09!22 2011

Patricia Seybold Group / Evaluation Framework
Recommendation Evaluation Framework, Version 2

Evaluating Solutions for Personalization and Recommendations By Susan E. Aldrich, Sr. VP and Sr. Consultant, Patricia Seybold Group September 22, 2011
NETTING IT OUT Recommendations are hitting their stride. A decade ago merchants and publishers saw the recommendations on Amazons site and wanted it. A decade ago they had to spend a fortune to get it. Today, recommendation engines can be had for some thousands of dollars a month and can be implemented in a few weeks. My guess is that recommendations will be ubiquitous within the next two to three years. Im always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift. SaaS offerings greatly simplify the technology aspect of implementation and virtually eliminate up-front costs, making it relatively easy for customers to sign up with a vendor. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I offer a set of requirements and evaluation criteria set forth in this evaluation framework. This framework updates and replaces the recommendation evaluation framework published in January, 2010. I will be using the framework to evaluate a number of the leading products using the framework during 2011-12, leading to a detailed comparison. WHATS INTERESTING ABOUT RECOMMENDATIONS What Are Targeting and Personalization Targeted marketing selects content (including products and offers) for consumers based on traits such as consumer context or behavior. The consumer is likely anonymous. Personalization requires a consumer profile that includes traits such as demographics, purchase history, and behavior, and this profile is used by the algorithms that select the content to be presented. While registering and logging in increases the customer profile data, the consumer is most often identified by web browser cookies and an associated identification number. Without registration, it is likely the profile does not contain user information such as name and address, so consumer anonymity is retained. What Is Recommendation Technology Web site owners worldwide have yearned for recommendations ever since Amazon started telling us that people who bought this also bought that and today tell us 52 percent of people who looked at this bought that; 26 percent bought this other thing. A decade ago marketers had to spend a great deal to implement recommendations on their sites. Today, recommendation engines can be had for some hundreds of dollars a month and can be implemented in a few days. Dont you just love SaaS?
Direct link: http://dx.doi.org/10.1571/fw09-22-11cc

Customer Scenario and Customers.com are registered trademarks and Customer Flight Deck and Quality of Customer Experience (QCE) are service marks of the Patricia Seybold Group Inc. P.O. Box 783, Needham, MA 02494 USA www.customers.com Unauthorized redistribution of this report is a violation of copyright law.
2 Evaluation Framework
For the user, recommendations on a site can mean pretty darn good search results, much of the time; emails with interesting offers; banner ads that catch your eye; useful guidance when you are selecting content, whether its a digital camera, a news article, a problem resolution, or a research document; clicks that take you from Google right to the content you need. For the content ownerwhether merchant selling products, publisher presenting articles, or marketer presenting offersrecommendation technology means delivering the most attractive item in those few seconds before you lose your audiences attention. A recommendation engine can double or quadruple the click-through rate as compared with the recommendations selected by the expertsthe merchandisers or researchers or support specialists. It typically has significant impact on revenue, time on site, employee productivity, and customer satisfaction. Where Are Recommendations Used Because recommendations had their most visible debut in ecommerce, people tend to associate recommendations with shopping. In the ecommerce arena, recommendations are used most often on product pages, shopping cart pages, category pages, and order confirmation pages. Recommendations are also used to tailor the content of those emails that entice you to stop work for a moment and shop. As a marketing team becomes more familiar and more confident with using recommendations, they expand recommendations to cover more of the interactions across the customer lifecycle. But I think product recommendations are the tip of the iceberg for recommendations. They belong everywhere content must be winnowed for a user, or everywhere that personalization improves a users productivity or experience. For example: A news site that knows I love football and dont care about rugby, and always shows me the most interesting world news A personalized view of the corporate intranet that highlights my department, my division, and my projects My view of corporate research knowledge bases, weighted to what Im working on Web sites that deliver coaching, e.g., for runners, dieters, investors, tailored to the my style and goals My personalized support portal to the corporate help desk My companys portal to a key supplier, e.g., Cisco, personalized by role or person My dashboard with my KPIs and corporate reports, with the hottest items on the first page A personalized investment site, e.g., Fidelity, tailored to the kind of investor I am A personalized commerce site tailored to my relationship, e.g., the car I drive, the sports I play Billing inquiry that always shows the disputed bills first Order history inquiry that always shows the most referenced order first
Customers.com Research Service 2011 Patricia Seybold Group
Unauthorized redistribution of this report is a violation of copyright law. For Reprints/Redistribution rights, contact: sales@customers.com
Evaluation Framework for Personalization/Recommendations Solutions 3
My guess is that recommendations will be ubiquitous within the next two to three years. Im always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift. The technology effort, and the up-front costs, for deploying recommendations are relatively small, which makes it reasonably easy for customers to sign up for a recommendation service. The greatest effort for recommendation customers seems to be in learning how to use them, where to use them, and how to use them effectivelyskills which will be portable to any vendors solution. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I have compiled a set of requirements and evaluation criteria set forth in this evaluation framework. And I will be evaluating a number of the leading products using the framework during 2011 and 2012, leading to a detailed comparison. REQUIREMENTS The requirements for recommendation services are derived from customer requirements. Customers for recommendation services cover several roles, including the end-consumers of recommendations, the business people managing recommendations, and technical staff. Their requirements generate the evaluation criteria which are listed in the Table. Recommendation Consumer Requirements People consuming recommendationsthe visitors, shoppers, readers, researchers to whom content is being recommendedof course, need relevant and enticing recommendations, thats the whole point. But they also need privacy and may wish they had some control over what personal information is used and how it is used. As recommendations are increasingly used to personalize experiences, consumers may also want a mechanism to indicate the persona they represent. Dont recommend for me, recommend for my [boss, niece, colleague]. Recommendation Manager Requirements The marketers, merchandisers, editors, business analysts and other business people who are using recommendations as a tool to improve user experience have a broad range of requirements, starting from the initial deployment. The business people who are responsible for recommendations need help making great ones, including: Guidance on how to deploy recommendations effectively Advice on how to increase recommendation effectiveness Training and tools to track and analyze recommendation effectiveness Business people need tools to deploy, test, analyze, and optimize recommendations: GUI or wizards for specifying rules for how recommendations are selected and presented Consistent interfaces designed for their business processes, not for the structure of the recommendation product Granularity in controlling the content selection process, e.g., using customer history to select sports content but crowd wisdom in selecting fashion
Aggregation in controlling the content selection process, e.g., global rules that apply to multiple sites, countries, and pages Automation of deployment, analysis, and optimization Testing and reporting that will compare different recommendation deployments Integration with other marketing tools
Business people need to manage the recommendation service, and therefore need capabilities that include: Reporting on recommendation results, e.g., click through, conversion, revenue Reporting on recommendation engine service level, e.g., response time and availability of service Security of their content and user information; privacy of consumer information Reliable, high performance Controlled, role-based access to recommendation management functionality
IT Requirements IT personnel are potentially involved in deploying and managing recommendation technology, either on premise or as part of the SaaS implementation. They need: APIs or Web Services for requesting recommendations from other applications Ease of adding and testing recommendation data gathering to Web pages Ease of providing a content data feed to recommendation service provider or enabling a crawl Ease of creating a zone on Web pages, emails, phones, or other venues, where the recommendations will be displayed Capability to incorporate management of the recommendation engine or service into enterprise management Security for data provided to the recommendation service provider Integration of recommendation inputs and outputs with other segmentation, behavioral analytics, Web statistics, and data modeling applications
Importance, Visibility, and Differentiation As I prepare my requirements and evaluation criteria, I confess to taking certain shortcuts. If all products offer a capability, I will drop it from my criteria regardless of its importance in order to simplify the evaluation process. For example, it is unarguably critical that you be able to start and
stop the recommendation service. All products offer this capability, so its not a feature cluttering up my evaluation matrix. More problematical are the capabilities that are very important but cant be observed or measured. At the very top of this list is, for a specific web site, do Vendor As algorithms deliver better recommendations than Vendor Bs? Algorithm quality will be very important to the recommendation manager, but perhaps not in the first months or even years of deployment, when she is developing her recommendation expertise. During that time, vendor expertise and guidance are far more critical.
Recommendation Platform Requirements

Category
Solution Positioning
Recommendation Platform Evaluation Criteria

What solutions are offered by the vendor? o If the vendor has multiple product lines, list the major categories of products. o Briefly describe the solutions offered in the personalization, recommendations, and targeted marketing arena. Include the unique value proposition for each solution.
Guidance and Advice
What are the vendors target markets, by industry and company size? o What is the vendors distribution of clients across industries: retail, B2B ecommerce, media, travel, online services? See Table B. o What is the vendors distribution of clients across retail categories: apparel + accessories, books/film/music, computers/electronics, flowers/gifts/jewelry, food/drug, hardware/home improvement, housewares/home furnishings, mass merchants/wholesale clubs/department stores, specialty/non-apparel, sporting goods? See Table B. o What geographies are supported? What languages do the vendors customers work in? In what languages are they deploying recommendations? What is the distribution of the vendors clients, by continent: North America, South America, Europe, Asia, Africa? See Table C. o What percentage of clients are multi-site? Multi-country? Multi-currency? o Does the vendor leverage its client base by offering collaborative or syndicated recommendations? Syndicated recommendations include product catalogs of other merchants in a set of recommendations; collaborative recommendations recommend products at other retailers that match the customers interest, based on cross-matching activities on both sites. How successful are this vendors clients? o For ecommerce clients, what percent of lift (e.g., revenue increase for ecommerce sites, ad impressions for media sites) is provided by direct action on recommendations (shopper clicks on a recommendation and immediately buys the item), and what percent by delayed action on recommendations (shopper buys an item previously recommended)? o What is the average annual growth rate in number of recommendations consumed per client? o What is the retention rate of clients? o What is the average number of recommendations touchpoints deployed per client? o What is the distribution of usage of key solution capabilities per client: e.g., web recommendations, email recommendations, mobile recommendations, ads, APIs? See Table D.
2011 Patricia Seybold Group Inc.
Recommendation Platform Requirements (continued) Category

Guidance and Advice (continued)

What types of coaching and training are provided and at what points in the relationship? E.g., Implementation Classes for IT, 1:1 coaching for merchandisers new to recommendations, self-training for advanced merchandisers, user conferences, videos, blogs. What services are provided on site? What type of account management and relationship management services are provided (business guidance, best practices, integration support)? What is the frequency of business, planning or strategy review? What is the experience level of the people performing the reviews?
Recommendation Structure
What are the sources of content that can be presented in recommendations? For example, catalog/databases, content management repositories, CRM systems. What types of recommendations are supported? For example, people who bought/viewed this bought that; most popular; your-friends-liked; a dynamic bundle with a dynamic price; questions that collect and analyze answers; search results, etc. See Table E. If algorithms can be customized, what factors can be included, and who performs the customization? If behavioral recommendations are supported, what behaviors are observed and used? E.g., incoming site, search terms, views, time spent viewing, navigation selections, cart adds, cart abandons. What behavior-based recommendation types are provided? Personalization: If customer profiles are supported, what data is collected? How are customers recognized across sessions and touchpoints? What customer profilebased recommendations are provided? How does the solution use social media? For Facebook and for LinkedIn, what data is used, and what algorithms (strategies) are enabled? For ratings and reviews, what data is used, from what sources, for what algorithms? Describe the scientific principles, algorithms and data models used for each recommendation pattern. What associations do the vendors algorithms support? E.g., item-item; item(s)-item(s), person-item, person-person, other. What recommendation types are supported for each touchpoint: web, email, kiosk, mobile, business application, iOs app? When are the recommendations generated for each touchpoint: batch, at send time (for email), at open time? See Table E. Can the recommendation systems insights/analysis be used by other systems, such as search, customer segmentation, data warehouse? Describe.

Managing Recommendations Testing Describe the approach to testing recommendations: Is it A/B or multivariate; what is the limit for simultaneous recommendation tests? How does the solution measure success of the recommendations? Can it measure based on client-specified KPIs? In what ways can the solution automatically optimize recommendation approaches to reach client-specified business goals? Interface for managing recommendations: Describe the interface. How is it organized? What is the set of symbols, tabs, and other interface elements that is consistent (in appearance and meaning) from task to task? Does it include a preview of changes being made, and provide a workflow for managing review and approval of changes? Is access controlled based on roles? What controls are in place that enable groups of people to manage recommendations without getting in each others way? What tasks can be handled by business people via the interface without calling on technical staff for help? Or which tasks require technical (IT) staff? Describe the principles by which business people can control recommendations. For example, do they create (or use) templates, what does the template control. How can rules or filters be applied, and to what objects (e.g., pages, templates, segments, time periods)? What data can be specified in rules and filters, e.g., inventory, margin, segment, advertiser, combinations of factors? What support is provided for managing multi-site, multi-country, and multi-currency clients? For example, global and local rules; copy and edit templates; copy and edit configuration. Reporting: Does reporting and analysis provide suggestions on how to improve recommendations to achieve KPIs? Does the vendors staff? How does the reporting include targets, make forecasts, provide alerts, and analyze contribution to KPIs? What is the most recent period that can be viewed in reports? I.e., how near real time? What is the mechanism for scheduling automatic report creation and distribution?

Integration and Ecosystem

How does the solution fit into the marketing ecosystem? What information is made available to other marketing tools? What information from other marketing tools is used by the recommendation solution? How would recommendations contribute to a marketing campaign? How would recommendations that are part of campaign be monitored and managed? How would the results of a campaign be reflected in recommendations? How does the recommendation solution fit into the web site ecosystem, e.g., site search, content management, SEO, SEM, advertising, and other applications? Describe what data is required by the recommendation engine, and how it is collected, including the format of feeds, language of collectors, and testing process. Does the solution have auto-discovery capability for content and items? Describe the mechanisms for ensuring data privacy and security, for customer and user data. Describe APIs provided. What languages/methods are provided? What services are available via API? Describe partner programs. What support is provided partners who seek to provide more services to clients? What is the metric for partner value to the ecosystem? Describe the typical implementation cycle, including activities, milestones, timeframes, and project management tools. What role/skill typically manages the project (for vendor and for client)?
Operations
For on-premise software managed by the customer: What is the mechanism for availability and scalability? If multiple machines are involved, what is the mechanism for managing multiple machines, both day-to-day and during software upgrades? Are SNMP alerts provided for integration with enterprise management? What are the scalability limits, in terms of content and users? For Hosted Software as a Service (SaaS): Describe your SLA, including guarantees, penalties, and limits; include a summary of SLA performance for past 12 months. What is the typical response time for the recommendation service? How many recommendations were served in the past three months? Please define what you mean by recommendation: is it a query, or the number of items returned, or something else? How many data centers support the recommendation engine? Where are they located?

Vendors Development and Maintenance

Do you package named releases? If so, describe your release frequency for new functionality and for bug fixes. What methodology or standards are followed in support, bug correction, and testing? Describe the escalation path for performance or quality issues.
Company and Product Viability
Product background and release history: month/year of major releases, beginning with the initial release Product plans: frequency of major/minor releases; enhancements planned for the next three months Partner and OEM strategy for recommendation solution Number of clients and sites Pricing for recommendation solutions: o What is the basis for pricing? If recommendation success is a factor, please describe how recommendation success is defined. o What is the average low end and high end? Company history: date founded, founders, investors, # employees Financial performance as available
Table A. The evaluation criteria for recommendation engines or platforms are based on customer requirements. We will use the criteria to analyze leading recommendation solutions.
Industry Focus
Industry Retail Sub category Percent of clients
Total for all retail Apparel and accessories Books/film/music Computers/electronics Flowers/gifts/jewelry Hardware/home improvement Housewares/home furnishings Mass merchants/wholesale clubs/department stores Specialty/non-apparel Sporting goods Other
B2B Media Travel Telecom Online Services Advertising

Table B. Each vendor targets specific industries with its marketing, sales, technology, and customer care.
Geographic Focus
Percent of Clients North America South America Europe Asia + Pacific Africa
List of Clients Countries
Languages Supported for Console
Table C. Each solution is deployed on multiple continents and on Web sites in many languages. The console provides the interfaces that marketers and merchandisers use to manage and optimize recommendations.
Use of Services
Services provided by this vendor (add or remove services as appropriate) Web recommendations Email recommendations Mobile recommendations Kiosk recommendations Display Ads APIs Percent of clients using the service
Table D. Successful clients expand their use of vendors services. This table reflects the level of adoption of key services.
Recommendation Evaluation Framework, Version 2 13
Recommendation Types and Structures

Recommendation Structures Supported Item-item(s) associations Many items to item(s) Person-person associations Item(s)-Person(s) associations Recommendation Types (list here) Behaviorbased Profilebased SocialMedia Web Email Mobile Kiosk Business Appl. Generated as batch; at send; or at open Rules driven (e.g., top sellers, lists such as staff picks) Rules modified (e.g., white and black list) People who viewed this, viewed that People like you who viewed this, viewed that People who viewed this, bought that People like you who viewed this, bought that People who bought this, bought that People like you who bought this, bought that
Open
Send
Batch
Open
Recommendation Types and Structures (continued)

Recommendation Types (list here) Behaviorbased Profilebased SocialMedia Web Email Mobile Kiosk Business Appl. Generated as batch; at send; or at open Search results selected and ranked Landing page items selected and ranked ..etc Syndicated recommendations Collaborative recommendations
Table E. Each solution supports a variety of recommendation types. This is due in part to the recommendation structures that are supported. Each recommendation type may be available to multiple touchpoints. Some touchpoints allow for the most dynamic recommendations, that is, recommendations that are selected at the moment the page is accessed (such as opening an email). Other touchpoints call for the least dynamic recommendations, produced in a batch. A kiosk in offline mode would require batch recommendations.
CONCLUSION This evaluation framework will be applied to the leading recommendation solutions during the coming months. After we have evaluated a number of products, we will compare the leaders and analyze which solutions best suit which applications and markets. You are welcome to use our framework for your own evaluation. We welcome your comments.
About Susan E. Aldrich and Patricia Seybold Group
ABOUT THE AUTHOR
SUSAN E. ALDRICH is a Senior Vice President and Senior Consultant at the Patricia Seybold Group. Aldrich is a senior analyst for the firms Advisory Service. As leading authority on worldwide technologies, custom er requirements, practices, and governance for finda bility, she manages the Sear ch, Navigation, and Discovery Research Practice. Her research foc uses on c ustomer self-service, information management an d technologies and practi ces for mo nitoring, measuring, and managing the Quality of Customer ExperienceSM (QCE). Aldrichs experience includes commercial a pplications development, deployment, and implementation and operating systems deve lopment. She has provided information management, customer relationship, and distributed systems management consulting worldwide.
Patricia Seybold Group

Trusted Advisors to Customer-Centric Executives
If you're a visionary customer-focused executive, the Patricia Seybold Group should be your first choice for ongoing strategic advice, business and technology guidance, customer experience best practices, and help with customer-centric initiatives. Founded in 1978 a nd based in B oston, we provide consulting, research and advisory services, peer groups, and in teractive workshops. We h elp clients to design an d continuously improve their customer-focused business strategies and processes using our pro ven consulting methodology, Customer Scenario Design. The CEO and founder, Patricia Seybold, is t he New York Times best-selling author of Customers.com and The Customer Revolution. Patty's latest book, Outside Innovation, is available now. Patricia Seybold Group P.O. Box 783 Needham, MA 02494 Phone: (617) 742-5200 Fax: (617) 742-1028 Email: feedback@customers.com Web: http://www.customers.com

Recommendation Eval Framework V2!09!22 2011

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Recommendation Eval Framework V2!09!22 2011

Uploaded by

Copyright:

Available Formats

Patricia Seybold Group / Evaluation Framework

Recommendation Evaluation Framework, Version 2

Direct link: http://dx.doi.org/10.1571/fw09-22-11cc

Customers.com Research Service 2011 Patricia Seybold Group

Evaluation Framework for Personalization/Recommendations Solutions 3

Customers.com Research Service 2011 Patricia Seybold Group

Customers.com Research Service 2011 Patricia Seybold Group

Evaluation Framework for Personalization/Recommendations Solutions 5

Customers.com Research Service 2011 Patricia Seybold Group

Recommendation Platform Requirements

Recommendation Platform Evaluation Criteria

Guidance and Advice

Customers.com Research Service 2011 Patricia Seybold Group

Evaluation Framework for Personalization/Recommendations Solutions 7

Recommendation Platform Requirements (continued) Category

Recommendation Platform Evaluation Criteria

Customers.com Research Service 2011 Patricia Seybold Group

Recommendation Platform Requirements (continued) Category

Recommendation Platform Evaluation Criteria

Customers.com Research Service 2011 Patricia Seybold Group

Evaluation Framework for Personalization/Recommendations Solutions 9

Recommendation Platform Requirements (continued) Category

Recommendation Platform Evaluation Criteria

Customers.com Research Service 2011 Patricia Seybold Group

Recommendation Platform Requirements (continued) Category

Recommendation Platform Evaluation Criteria

Company and Product Viability

Customers.com Research Service 2011 Patricia Seybold Group

Evaluation Framework for Personalization/Recommendations Solutions 11

B2B Media Travel Telecom Online Services Advertising

Customers.com Research Service 2011 Patricia Seybold Group

List of Clients Countries

Languages Supported for Console

2011 Patricia Seybold Group Inc.

Customers.com Research Service 2011 Patricia Seybold Group

Recommendation Evaluation Framework, Version 2 13

Recommendation Types and Structures

Customers.com Research Service 2011 Patricia Seybold Group

Recommendation Types and Structures (continued)

Customers.com Research Service 2011 Patricia Seybold Group

About Susan E. Aldrich and Patricia Seybold Group

ABOUT THE AUTHOR

Patricia Seybold Group

Customers.com Research Service 2011 Patricia Seybold Group

You might also like