1. Context: Revamp Flight Delay Guarantee to increase Revenue
Previous offer and “learn more” screens for Flight Delay Guarantee.
The current design of the Flight Delay Guarantee offer was a bottom sheet presented on top of the Confirm Itinerary screen, and there the users had the option to “Add to my trip” or decline it with
“No, thanks”.
Previous surveys had shown that customers had different needs for different types of travel, so we wanted to experiment with combining the ability to choose your coverage and the price for it.
The hypothesis was that we had the potential to reach even more users if they had the flexibility to choose their coverage.
3. Smoke Tests: Testing multiple coverage options
3.1 Round 1: 7 different customized coverages and pricing
To run rapid and trackable experiments without the need for engineers to build the interface, I built 7 different variants we wanted to experiment with using Webflow (a low-code tool), and this allowed us to track the success of each variant once we launched it.
These are the variants we took to the first round of the smoke test:
1. Upgrade concept
As mentioned in the research section, this concept offered a single coverage for a smaller price and an option to add delays + cancellations, the upgrade option, for just $5 more. We had two variants: one where the single coverage was Delays, and one where it was Cancellations.
💡
What if we broke the current coverage into 2 options so that users could choose what better fits their needs?
.png)
Delays with upgrade to Cancellations.
.png)
Cancellations with upgrade to Delays.
2. Single product
Only one of the two coverages was offered — Cancellations or Delays.
💡
Could the current coverages be more successful if offered alone?
.png)
Cancellation only.
.png)
Delays only.
3. Coverage extension
The current Flight Delay Guarantee policy covered cancellations that happen on the day of travel and 1-hour delays.
💡
If we offered better coverage for a slightly higher price, would users add it?
And if we offered a less comprehensive offer for a smaller price, would this make more users interested in adding it?
V1. Cancellation coverage extension
From time to time we read on surveys that users wanted to be covered against cancellations that happen anytime between their purchase and departure. This variant gave them the option to choose between cancellations that happen on the day of travel coverage or a more comprehensive one, where they would be covered against cancellations that happen anytime between booking and departure.
V2. Delay coverage extension
This variant had the goal to understand if users would be willing to pay less for a 2-hour coverage instead of the current 1-hour coverage.
.png)
V1. Cancellation extension coverage.
.png)
V2. Delay extension coverage.
4. Disruption for Any reason
What if users could be covered against ANY type of disruptions during their trip? If you miss it because you arrived late, or the security lines were too long, really anything?
This is the idea behind the most comprehensive variant we tested:
💡
Will users see more value in complete coverage instead of one that only covered delays and cancellations?
.png)
Disruption for any reason.
3.2 First-round results: Upgrade wins
By tracking the attach rate of each variant we found a winner: The upgrade concept.
1.
The “upgrade concept” was the most purchased option 👑
Upgrade was the clear winner with both variants having the two highest attach rates.Based on the research, we learned that the small difference in price between the single and the upgraded option was perceived as too small to not add — it made the combo option look better.
2.
Cancellation coverages had a good performance 👍
Even with users mentioning that they thought it was the airline's responsibility to protect them against flight cancellations, the Cancellation only and Cancellation coverage extension got 3rd and 4th place respectively in the smoke tests, losing to the Upgrade variants by a small percentage.
3.
The Delays offers did not perform well 🕒
This finding was a total surprise — the two options that explored offering only a Flight Delay coverage (Delays coverage extension, Delays only) had low attach rates, and this is unexpected since users mentioned in the interviews that coverage against delays looked like a good deal.
4.
Disruption for any reason had the worst performance 👎
Even more surprising was seeing that the most comprehensive coverage had the lowest attach rate.
There are a few hypotheses for why this has happened (looked too good to be true, not enough information was provided on the offer screen, the policy was too generic, …), but for the short-term, it meant that we wouldn’t explore this coverage option for now.
3.3 Rounds 2 & 3: New design and removing friction
3.3.1 Testing the upgrade concept with a new design
With the upgrade concept being the winner, I created a new design for the Flight Delay Guarantee screen, now incorporating the radio button options. For this experiment, we used the same two upgrade concept options and I built the new designs on Webflow, so we could track the success rapidly again.
Delays with Cancellations upgrade.
Cancellations with Delays upgrade.
The second-round winner was the Delays with Cancellations upgrade, which has become the new official offer for Flight Delay Guarantee.
3.3.2 Round 3: Removing friction
We believed there was still more potential to be explored on the offer screen, and previous experiments ran by other teams at Hopper showed that using buttons to quickly add/decline an offer on the booking funnel could have an impact on the attach rate and overall booking conversion. With that in mind, we ran an experiment with the new official offer design that had a radio button to decline the offer, and one where they could easily add or decline it using the primary and secondary buttons, reducing the friction to move forward on the flow.
Some hypotheses and risks
1.
Having one option pre-selected could mean users can easily decline the offer by tapping on the “No, thanks” button without even reading it, leading to a lower attach rate.
2.
But pre-selecting could also help to highlight the best option and make it easier to add it to the booking.
The radio button offer had a higher-priced option placed after the lower one, and with this experiment, we want to try changing the order because there’s a pricing psychology tactic that says:
1.
Customers are more likely to choose an expensive option when you sort prices from high to low. Initial prices establish a reference price. Customers use these initial prices to evaluate subsequent prices. While scanning products that get higher in price, customers lose the ability to pay a lower price. They feel pressured to pounce on a cheaper item before they get too expensive.
2.
Showing prices in a decreasing sequence can trigger the reverse effect. Each new product feels like a loss in quality. Customers feel pressured to pounce on an option before they lose too much quality.
Results
The preselect won against the radio button. We ran comprehension surveys to understand if there was any difference in understanding what they bought, but no problems came up.
.png)
