Ponderings on Private Beta
Ponderings on Private Beta
Hi all, I’ve been asked my by PO to put out a request about what a good private beta looks like in terms of size, ramp up etc? I would think it would vary from service to service. Any advice / links to examples would be most welcome. Thanks in advance. Jane DS
Reply
Really interesting question. I’ve not seen any formal guidance on it –
I guess my answer would be yes it varies between services. Big enough so you can get some useful insight and Small enough that it is low key and no one notices if you have to pull it?
The last thing you want for your ‘Private Beta’ service is to spend ages setting up a private beta panel/community only to not have enough users going through to gain any useful insight, this would be very expensive means of testing and validating a thing
For the new NHS Manage Your Referral Service (part of NHS e-Referral Service). We ran a very very small public beta. The old national service has around 1 million transactions per year, we used a tiny link on the homepage of the old service to redirect about 1% of users, we later made a more prominent link which redirected about 10%.
We had (for a few months) only a few dozen users per day — around 10% provided survey feedback (5–10 users per day) but the survey and analytics base sizes weren’t robust enough to say if it was statistically an improvement (e.g. Satisfaction/completion rate) over an old service it was due to replace — was this just by chance due to low sample and high margin of error.
What you can do though with only a few users is analyse individual users and see why people drop off and the errors and make some informed changes based on analytics / database queries, that needs to be part of the private or public beta plan. Until you scale up you won’t know for sure if the private beta data is robust and the service is cooking on gas as many weird errors/alternative use cases that you won’t find in private beta testing are much less likely to occur with only a handful of users!
I guess it comes back to your service, questions for the PO and UR to think about might be:
- what are the objectives are for private beta?
- what is the means for validating incremental improvements?
- is there is a drive/need to go for public beta?
- how many changes are you planning to test? as you will need much larger numbers e.g. if using AB or multi variate testing,
- how quickly can the team resolve issues/make changes?
- when should we scale up? how long will it take to gather * data to validate that it works?
Hope this helps!
Tom
Pre-Discovery is a thing.
I’ve been asked about: “What is a Pre-Discovery”?
Pre-Discovery is helpful for teams working on existing products and services where data can be gathered from internal resources. It should underpin the User Research Discovery themes and will make a big difference to the users approached and the questions asked during Discovery.
Teams with an existing service will have questions such as:
- How frequently is our service currently used?
- What is the demographic of the current user base?
- What have our users already told us about the current product?
- What are users pain points?
- What do users do before arriving in our service?
- What do users do after using our service?
- What are the main outcomes from using our service?
Teams should list out the many dozens of questions they may need to answer through Discovery and then assess whether any internal data sources will answer this before kicking off new research.
Why more Pre-Discovery is helpful
Discovery is a awesome, challenging and rewarding too. Meeting users, getting immersed in their environment, understanding nuances in their needs. But Discovery is expensive and time consuming so we need to do try and answer questions effectively, in a targeted way.
Here is my Pre-Discovery checklist:
- Review all previous research reports and surveys to identify relevant insight about a thing
- Speak to service desk and review a sample of relevant service tickets
- Review analytics and performance data for the thing
- Review business case and benefits case for the thing, to see what benefits we expect users will see
- Run a quick session with Product, Sales or Implementation team to understand if there is knowledge or previous evidence about the thing
- Highlight in a service map where the thing sits in relation to other things
- Evaluate the benefits of fixing the thing
Teams may say these activities should just happen in Discovery…
While possibly true, most teams need to raise funding to pursue Discovery research and delivery projects and Pre-Discovery may be helpful to flesh out the business case and specify some problems to solve.
Gathering user research evidence for benefits early sets in motion good practice that should be continued through Discovery and Alpha. This also can help a team quickly surface all the key issues whether they need further understanding.
Gathering this data during Pre-Discovery won’t help you to confirm user needs, you will still need depth Discovery research. It also won’t help you identify the best solution, you will also need Alpha research and iterative design .
More benefits of Pre-Discovery
Pre-Discovery gives researchers some time to start thinking about finding the right users and starting recruitment so there is a seamless transition into Discovery when an Agile team may scale out and exploratory research and technical work really kicks off in earnest.
Pre-Discovery usually involves a limited number of roles too. A User Researcher, UX Designer, Service Designer and Business Analyst. It can take a few days to a few weeks to gather relevant insights.
Getting buy in for your Pre-Discovery
Share your Pre-Discovery evidence with key stakeholders and sponsors, to answer the question ‘What do we know?’, ‘What else do we need to know?’ and why should we buy into continue into Discovery phasee? Give them a flavour of the future: how will their world be better when they know more about how to solve a thing?
Keeping momentum going
If more business cases were based on User Needs and Pre-Discovery evidence with further staged funding towards the end of Discovery/Alpha, there would surely be a lot less disappointment for users and stakeholders when product launches go awry. Less feature commitments will be promised that are later abandoned quietly during Discovery or Alpha, and importantly less irrelevant and implausible features will continue through to live service deployment.
Getting started with UR projects isn’t always easy — Chicken or egg first? How do you complete a Pre-Discovery on the thing without funding or resources?
Breaking the cycle is challenging especially when most teams might already at capacity with their existing commitments.
Innovative organisations embed Pre-Discovery into their standard process for Strategy, Policy and Product. This support alignment at all levels to ensure outcomes are based on both user needs, and business needs.
How did your Pre-Discovery go?
If you found this post interesting or useful please comment below :) it would be great to hear your stories of Pre-Discovery!!