As mentioned during the JMP Kickoff Meetings, the Strategic Initiatives & Analytics team is running regular experiments on jmp.com to maximize desirable actions by our website visitors. During Q1 2021, the team ran three experiments. The results can be found below. Reminder: If you have future website experimentation ideas or requests, please submit them here.
Q1 experiments were designed to answer the following questions:
- Does changing the layout of the homepage increase the number of visitors who ultimately download the trial?
- Does personalizing the call-to-action text on the STIPS page increase click-thru rate?
- Does rounding button corners on the Buy JMP page increase click-thru rate?
Experiment #1 – Does changing the layout of the homepage increase the number of visitors who ultimately download the trial?
Led by Alexa Baltazar
Background: There’s a common belief among those familiar with our website behavioral data that most visitors to the JMP homepage want to download the JMP free trial. Operating with this thought, we created a website test that featured the JMP trial more prominently on the homepage with more clickable real estate leading visitors to the trial download form. Our hypothesis was that shifting the layout and positioning of the trial on the homepage would increase the number of people downloading the trial.
We followed landing page best practices when designing the homepage variant:
- Large hero image with human faces looking at your product.
- Quick, to-the-point headlines and sub headlines.
- Large, easy to click call-to-action button.
- Social proof – in the form of customer quotes.
- Benefit explanation.
The Experiment: Launched in early January, the experiment tested the current homepage (the control) against our new design (the variant). We utilized the website testing tool VWO to run the experiment, which allowed us to randomly serve up either the existing homepage or the variant to visitors. In addition to tracking which version of the homepage each user received; we tracked their subsequent actions – including if they downloaded the trial.
Since the en_us version of the homepage gets substantially more traffic than any other country site, we ran the experiment on this country site first.
Control
Variant
The Results: The experiment ran for six weeks. We then downloaded the user-level hit data and prepared the data for analysis in JMP. (We removed traffic from JMP employees.) Our data identified which version of the homepage each user received, along with if that user had visited the trial download page and/or downloaded the trial during their session. After cleaning and preparing the data, we were able to run two chi-square tests of independence to determine:
- Does the layout of the homepage affect the number of people who reach the trial download from?
No, there was no evidence to suggest that the layout of the homepage affected the number of people who reached the trail download form. We were initially surprised with these results, so we decided to also look at if the website visitor was already in Salesforce or not. This layering did not affect the initial results.
- Among those who made it to the trial download page, did the layout of the homepage affect the number of people who actually followed through and downloaded the trial?
Yes, there is evidence to suggest that the layout of the homepage affected the rate at which the trial was downloaded. When we looked at homepage visitors who reached the trial download page from the homepage, there was a 11% increase in trial form completions from visitors who were served the homepage variant.
Conclusion: This experiment led us to believe that there is a set number of homepage visitors who are interested in downloading the trial, regardless of the homepage layout. Of these visitors, however, the layout of the homepage affects the likelihood that visitors interested in the trial will actually complete the trial download form. For this reason, we recommend that the web team implements the new version of the homepage after JMP 16 promotions conclude. Our plan is to then continue to iterate with other experiments to see if there are other aspects of the homepage that we can test to optimize our desired results.
Experiment #2 - Does personalizing the call-to-action text on the STIPS page increase click-thru rate?
Completed by Liz Zarzar
Background: Three factors make call to action buttons highly effective: urgency, relevancy, and personal connection. This test attempted to assess the effectiveness of the urgency and personal connection factors on the call to action buttons found on the STIPS landing page. Urgency refers to phrases that suggest the action must take place now – words like “Urgent,” “Now,” and “Today.” Personal connection presents actions in terms of the user and is seen through first-person oriented call to action – using words like “I,” “Me,” or “My.”
The Experiment: In early January, we launched the STIPS call-to-action button experiment. This experiment tested new text “Access my free course now” (the variant) against the current text “Enroll now” (the control). Our testing tool, VWO, allowed us to randomly serve up either the variant or the control. In addition to tracking which version of the button each user received, we tracked our visitors’ subsequent actions – including if they clicked the button.
Control
Variant
The Results: In order to collect sufficient data, we ran this experiment for a few months. After opening the user-level hit data in JMP, we removed traffic from JMP employees and identified which version of the enrollment button each user was served, along with if that user had ended up clicking the button during their session. After cleaning and preparing the data, we were able to run a chi-square test of independence to determine:
- Does personalizing the call-to-action text on the STIPS page increase click-thru rate?
There is insufficient evidence to conclude that the change of the CTA text was the reason we saw a 1% uplift in click-thru rate on the variant page with the “Access my free course now” button text.
Conclusion: This experiment was inconclusive -- adding personalization and urgency to the call to action button text may cause an uplift in click-thru rate (1% click-thru rate uplift was observed), but we lack sufficient statistical evidence to distinguish the effect observed from chance variation. However, because this is a minor website change requiring no development, and there is evidence in the literature supporting this change, we do recommend changing the text to “Access my free course now.”
Experiment #3 - Does rounding button corners on the Buy JMP page increase click-thru rate?
Completed by Caroline Huang
Background: Companies such as Amazon and Target have conducted testing on the influence rounded button corners have on the likelihood of purchasing a product. The theory is that rounded corners convey a friendly approachable image that draws in buyers, while sharp corners make customers more cautious. This factor is already implemented on the JMP Trials Form page, but if we were to implement rounded corner buttons on the JMP Buy page, would the science still apply, increasing conversion rates?
The Experiment: In early January, we launched the rounded button corner experiment which tests rounded button corners versus square button corners on the Buy JMP page. We utilized the website testing tool VWO to run our experiment, which allowed us to randomly serve up either the existing web page or the variant. In addition to tracking which version of the button each user received; we tracked their subsequent actions – including if they clicked the button.
Control
Variant
The Results: To allow us to collect sufficient data, we ran this experiment for a few months. After opening the user-level hit data to JMP, we removed traffic from JMP employees and identified which version of the enrollment button each user was served, along with if that user had ended up clicking the button during their session. After cleaning and preparing the data, we were able to run a chi-square test of independence to determine:
Does rounding button corners on the Buy JMP page increase click-thru rate?
There is insufficient evidence to conclude that the rounding of button corners affected the click-thru rate of the buttons on the Buy JMP page.
Conclusion: This experiment was inconclusive — we were not able to detect an effect of rounding the button corners on the Buy JMP page on the click-thru rate of the buttons. While research findings suggested that rounded button corners offer a less intimidating appeal to buyers, this research was conducted on major online retailers – so the scenario differs from our exact experiment.
