Onboarding results: Text-based vs. designed emails

After running an A/B test for 5 months and collecting a sample size of over 1,000 per variant, we have developed interesting findings when comparing text-based emails to designed ones.

Callout: The two variants - text-based (A) is not apples to apples with the designed version (B). These differences may impact the results.

Let’s look at how the two emails compared:

Similarities

• Both had the same subject line
• Both opened the door to replies
• Both main CTA’s direct to exploring the Demo email all new accounts have in their workspace
• Both emails contain personalized copy pertaining to the goal of the individual (ie. code, testing, etc)

Differences

• Copy is different between the two
• Images are only included in the designed variant

Resulting Metrics

Click-through rate

Variant A: 6.55%

Variant B: 8.85%

Replies

Variant A, the text-based email, had a 1% reply rate, that eventually led to Zoom conversations, new feature requests, and a better understanding of the user backstory.

Variant B received no replies.

Conversions

Variant A [text-based] 3.24%

Variant B [designed] 2.89%

Product engagement

Outside of email engagement and conversion, after individuals received either variant, there were no significant differences between key product usage events.

Summary of findings

Although Variant B had a higher statistically significant click-through rate, we witnessed more relationships being created through 1:1 communication in response to Variant A. We were able to:

• answer questions about Parcel
• schedule calls to talk about plans and pricing
• understand the back-story of why the user came across Parcel and wants to use it for email creation

Despite more individuals clicking on Variant B, their engagement did not result in higher conversions to subscribing to a plan. Those that engaged with Variant A, showed higher intent to convert to a paid subscription, while we were able to build and nurture relationships with those that did reply directly.

💡 Variant A resulted in more benefit to Parcel than Variant B.

Taking insights into next steps

I plan to iterate off of the text-based emails, and take the learnings of the higher click-through rate from Variant B by working in resources such as guides, or our new Parcel demo video, to the basic email copy.

The next approach I plan to take is a prescriptive vs. enabling approach to answer the following question:

Does action and direction in email copy, convert higher than an email that offers the freedom for an individual to educate themselves?

Stay tuned to find out more!