A pragmatist's guide to digital customer experience experimentation
Experimentation is essential to test whether new digital CX innovations will be worth the effort. But how can you make sure that the experiment itself is valuable and not too costly or time-consuming?
Customer experience can be challenging unless you have the right resonation with customers, users and employees. Experience design and business transformation initiatives can require time, effort and finances. Those initiatives also require the right change management and a communication campaign to obtain acceptance, adoption and the right tuning with stakeholders. When this is achieved, your customer engagement, retention, adoption and renewals will grow – along with your bottom line.
It’s not always easy to transform an organisation that’s not yet ‘’customer-centric’’ but has an initial desire to transform. Often, the organisation’s understanding of what’s required can differ from the amount of time, planning, tools and execution effort that’s actually needed. Sometimes it’s not straightforward or easy to change behaviours in your company, but it is possible.
How could we use experimentation to test the right way to develop an experience prior to creating one that won’t resonate with customers? Is it possible to test all our initial approaches for products, services and cross-silos with employees, partners and customers? Here is where agile, design thinking has a potential role.
For digital experiences and tech products in general, experimentation can be one way of making things work, since it’s all about validating hypotheses prior to advancing down the wrong path, to ensure things are worth the effort. There are several ways to pursue experimentation. The way we use has worked well for large and mid-sized organisations and could be applied to smaller companies.
There are many practical forms for digital experience experimentation. Here are seven of the principles:
1. Don’t emulate the best
What works for me, won’t necessarily work for you, and vice versa. Every company has a different DNA. What worked for Amazon, Samsung, SAP or Microsoft worked for them due to their culture, strategies, leadership, design, talent and their mindset – which, in many cases, puts employees first and customers at their core.
Of course, you can study and analyse their approaches to certain things, such as how to develop a digital experience that is so well-defined it ensures that customers always return. However, emulation will lead your company to frustration and, potentially, failure.
Is your company engaging with customers at all touch points and on all social platforms? Many "thought leaders" advise you to be where your customer is. However, in my opinion, you should focus on interacting with customers on the platforms and channels where you have the ability to be outstanding. Amazon, Apple and Samsung, are not in all channels, but they deliver a decent experience in those few they choose to use. From the customer’s standpoint, it’s better to have an efficient experience. There are ways to experiment across channels using customer journeys. One experimentation method is to test your team’s ability to deliver a great experience while analysing and designing your customer’s journey.
Experimentation is often used across channels for the development of apps, several technologies, enterprise software, and websites – to ensure something will generate adoption. It’s highly recommended to experiment prior to launching anything.
During experimentation and testing phases, these are 10 key focal points for customer adoption. I’ve extensively elaborated about them in the past:
- Design to generate adoption.
- Customer perception.
- Simplification works.
- Embrace necessary complexity.
- Analyse using data insights.
- Design for an intuitive experience.
- Unify data through journey mapping.
- Experimentation isn’t a one-time project.
- Measure experimentation results.
- Community and customers can help.
Brands are in control of designing their customer experiences. Go the extra mile via deep experimentation to ensure successful outcomes.
So experiment to ensure your business and customers achieve their goals faster and more effectively.
2. Expand your experimental approach
Retain a long-term approach of continuous adaptation. Using data analytics, text analysis, and adaptive design, you’ll be able to recognise what will resonate with the customers you are using (participating) for experimentation with your product, services, or CX programme. Prior to launch, you can measure whether your digital experience responds to customer needs, and this can serve as either a positive or negative parameter for your experimentation.
One approach is to test a variety of options. Another approach is A/B testing or experimentation modelling. Just because something works best in experimentation mode doesn’t necessarily mean you have it spot on. Adoption is the name of the game. Use data to identify aspects of your digital experience – such as whether it is easy to adopt or is intuitive – and ask questions, such as why it was easy to adopt.
3. Cloud solutions are a blessing
Nowadays, we have several advantages in the world of digital transformation and experimentation that we didn’t prior to cloud solutions. Today, you can conduct your experimentation in any enterprise software or multidisciplinary solution, and often, customers and users will not perceive the small changes being tested. For example, Facebook and Microsoft do this on an almost weekly basis through tests in different countries.
Unlike old solutions that needed down time in order to be updated, the cloud allows us to seamlessly implement updates for experimentation, without impacting performance or user experience. You can test the adoption of a design, a functionality, or a feature over a pre-determined period of time, and you can keep progressing with phases of experimentation until you find what really resonates with your customers, employees, and partners (as we discussed in this article).
4. For the initial tests, give it away or make it available
When experimenting with a new CX programme, product, or service that will involve people from many disciplines and departments with customers and across your company, you cannot judge the data and test outcomes without understanding the subject and all the KPIs, metrics and measurements.
One company that excels on this point is Microsoft, since Satya Nadella became the CEO. To ensure that the new Windows 10 solution would be far better than previous solutions, the company initially gave the product away in order to fine tune it and spot issues prior to a final launch with customers and partners.
When you already have a vast community of customers, test your solution directly with them. For example, Samsung tests new launches in similar way. It checks how its prototypes are perceived and how they performed versus expectations, even when it’s a totally new product or solution. Testing potential solutions for issues is much broader and more detailed than an A/B experiment.
Of course, there are pros and cons to this option since it does risk exposure, and you may not want to lose a competitive market advantage.
5. How to make an experimentation worthwhile
Experimentation is all about focusing in on very specific objectives and activities. If you define too many objectives and milestones – especially for your enterprise technology – with too many fluid variables, your core team and the initiative will not obtain clear, specific data. Therefore, I advise clients to be specific about variables and objectives. The goal is to drill down to very specific individual. For example, when checking your cloud ERP solution, the goal would be to analyse how the different modules in your solution contribute to the generation of adoption, as well as how to get customers to use them. Then you analyse the time expended, how a customer used it, what potentially turned them off. You can also check if there were points where the user seemed to be doing nothing because their time was expended trying to understand the software.
Experiments are based on a hypothesis: if A occurs, then we can conclude X, or if B happens, then we can conclude Y. Keep experimentation to a short duration so that you have enough time for analysis. A key point to remember is that it’s not about the quantity of people who behave or adopt your software in one desired way, it’s about how many customers acted in one way or another during a certain period of time that would be a projection of a real-life situation.
Experimentation is all about focusing in on very specific objectives and activities. If you define too many objectives and milestones your core team and the initiative will not obtain clear, specific data.
6. Conduct the most appropriate experiments
The essence of experimenting is to obtain an answer that will serve your initial purpose. To ensure the experience you are proposing is the right one and the best way to generate adoption, the experiment should take into consideration all variants and the intent of the hypothesis. There are several tests you could design, but not all of the experiment’s results will be useful or relevant. For instance, some experiments will just seek a clear response as to whether the customer will adopt a particular design, feature, functionality, or process.
When considering which experiments to run, start with the basic questions that will ultimately determine the success or failure of an experience:
- What are you trying to offer and how will it be perceived?
- Which target customers will be utilised in your experiment?
- To generate clear feedback, what kind of experience do you want to convey? Intuitive, clear messaging? Self-explanatory design?
- Which channels should you use?
- What are the several hypotheses or variants best suited for that target customer?
- What experience(s) or software design(s) will be part of the experiment?
Good experimentations help achieve success, when well-planned. Bear in mind: If your experiment will compare different versions of a solution, at several points along the customer journey, you should pre-define the data that will be worthwhile to collect, as well as the variants for comparing all of the results. This sounds like a lot of work, but this process performs well for technology in particular.
Finally, remember to define up front what your metrics are supposed to measure. At the end of the day, experiments are designed to deliver useful information to clarify for us which decision to make. Always focus on relevant information and targeted experiments. Use the data to further fine-tune your design, solution, product, or customer experience to get it right in the eyes of your customers.
7. Plan and prioritise: you can’t have everything all at once
Plan and prioritise should have been the first step. I added it last on purpose, because I want to make sure that if something stays with you, it would be planning and prioritising the various parts of your digital experiments. These will impact customer adoption, experience, and, ultimately, loyalty and renewals. Your first step is to keep your list of assumptions relatively small. It should include as potential benchmarks what you already know about the problem you are trying to solve and your customer insights.
Develop a plan to test your many hypothesis, and include on it every single move you are going to make. Define the timing, scope, dos and don’ts, and the expected results if this hypothesis is right or wrong. Create a testing calendar using software created for experimentation – or even a project management tool – to plan and coordinate all activities. Prioritise activities based on the more difficult parts of your assumptions – the potential show stoppers. If these don’t work, you will have to rethink your experiment all over again, so it’s better to get the most risky ones out of the way.
To start, gather your team and ask the question “Which experiment will we run to generate that behaviour?” Then, turn the answer into your hypotheses and determine what a confirmation of that behaviour would look like.
How to go about implementation after analysing the results of your initial experimentation is a subject for a future article. It’s a key topic for generating the adoption of any kind of technology. We often hear the saying, “You cannot manage what you cannot measure.” The same applies to the experimentation experience.
Now is the time to confirm your goal for the experiment and determine the metrics and KPIs you should measure. What is the desired result as far as customer behaviour and perception? Ultimately, experimentation rests on the shoulders of a company’s leaders. They must be honest about the process trajectory and about what the data says in order to generate a better digital experience for their customers. A company’s preferences don’t really matter; the data and results will speak for themselves.
This piece adapted from an article that originally appeared on the Eglobalis blog page.
I am a global executive and strategic consultant for medium and large global technology organizations, focused in Customer Experience, Professional Services, Customer Success and delivery.
I enable major global enterprises to generate new revenue and enhance market competitiveness by delivering global customer experiences, services,...