I tested 3 different email subject line tools with 45 email subject lines, along with 120 cold emails.
Today I’m going to share what I learned with you!
It all started when I got an email with the following subject line in my inbox:
This 🤓 will help you get better subject lines.
That was the subject line of an email sent from “Nathan” (if that is your real name) at CoSchedule.
I opened that email and learned about subject line testers for the first time, starting with CoSchedule’s.
Turns out, there are a bunch of online tools that score your subject lines so that you know in advance whether it completely sucks, or if it’s the best thing since ice cream tacos.
Isn’t that convenient? In a time where your target audience gets over 100 emails a day, it’s helpful to know whether you’ll stand out or not before you send an email.
Of course, today there are plenty of simple tricks to writing subject lines that boost your open rates and drive engagement. But knowing in advance whether your subject lines are good to start with, would be a huge convenience. It’s also a key part to automating successful email funnels.
The only question is, can you live and die by these subject line testers? Are they reliable, somewhat insightful, or just gimmicks to draw you into an elaborate sales funnel/wormhole?
I had no idea, so I decided to test three specific tools, by comparing their rankings of subject lines and the open rates of emails using those subject lines.
Why did I do this? Because then I could write an article about it and save you the trouble of having to test it yourself!
My test subjects
CoSchedule, Subject Line and Net Atlantic each have an online tool that ranks the quality of your email subject line. I chose these tools because:
- They’re all pretty straightforward to use (no sign up or purchase required).
- They are the first few tools to show up in a Google search.
- They rank each subject line out of 100.
- And they provide suggestions on how to improve your email subject lines.
These are quite possibly the cool kids of email subject line testers, so I chose them. And CoSchedule told me about their tool, so I had to try it anyway!
What I tested
I tested the accuracy of these tools in scoring the quality of each email subject line.
I could easily have gone further and tested the tools’ suggestions, etc. However, I was less interested in what each tool considers to be a good email subject line, and more in its ability to evaluate an email subject line that it’s given.
I tested the correlation between how email subject lines were scored and the open rates of emails using those same email subject lines. I felt it was an indicator of how successful or unsuccessful email subject lines would be.
For example, if an email subject line ranks as a 90, I’m expecting a pretty high open rate, upwards of 80%. On the other hand, if an email subject line ranks at 30, I’m expecting open rates of about 10% – 30%.
The email subject line experiment design
I created three categories of email subject lines. Then, I ran each of the email subject lines through the tools and recorded the scores.
Once I had gotten my score, I chose email subject lines from each category to use for cold outreach emails.
I compared the open rates of the emails with their email subject line scores to see if there were any discernible correlations.
The email subject line categories
I created email subject line categories to test the tools’ own possible biases. I wanted to know, do these tools only understand salesy language as drivers of high open rates? Or can these tools evaluate any set of phrases and rank them accordingly.
Here’s the email subject line category breakdown:
- Real email subject lines
These are email subject lines that I’ve received in the past from actual businesses. I wanted to test these to see if typical email subject lines perform well or not.
- Gibberish
These are a random series of letters and numbers to test whether the rankings are just random themselves. This is the gimmick-detector.
- Pop culture quotes
I wondered if emotion might play a role, over things that sounded businessy or salesy. So I tested a few iconic movie quotes.
In total, I tested 45 email subject lines to see how the tools generally assess phrases, sentences and words. This gave me a sense of how complex the tools are.
Below are the rest of the email subject lines that I tested and their scores (according to each tool), sorted by category.
1. Typical email subject lines
2. Gibberish email subject lines
3. Pop culture email subject lines
Early observations
I noticed a few things about email marketing after running the various email subject lines through the tools.
What stood out to me almost immediately were the results from Net Atlantic. Consistently, for each category, one third of the email subject lines ranked at 100. I thought that was odd, especially because of the gibberish category.
With Subject Line, more than 50% of the email subject lines I tested received ratings of 80. However, of the actual email subject lines from real businesses, only one-third received a rating of 80.
CoSchedule consistently gave the lowest scores. The highest score was given to two email subject lines, one in the gibberish category and one in the movie quotes category. Just 63% for both.
So keeping these results in mind, I went ahead and scheduled five small batches of emails, with twenty recipients each.
The experiment
I put together a list of contacts for completely cold email outreach. Truthfully, I didn’t want to spend a small eternity on this experiment. So instead of testing 27 different email subject lines on over a thousand people, I chose five specific ones to test on about 120 people.
It was important that this test was done using contacts that don’t know me or have any existing relationship to Venngage.
If someone recognizes the sender (in this case, me), then their decision to open or ignore the email isn’t based solely on the email subject line. As a result, the impact of the email subject line on open rates is muddled.
So I compiled a list of contacts, retrieved email addresses, and then used Mailshake to create the campaigns.
Choosing my email subject lines
I chose email subject lines that were ranked very differently by each tool. This helped me determine if the tools were indeed accurate, understood what makes something click-worthy, or were all hopelessly random. I also wanted to see if numerous approaches to crafting an email subject line can have positive results.
I didn’t focus on the body of the email very much.
In fact, I wrote to my recipients about what I was doing, thinking it wouldn’t affect the open rate (spoiler alert!). Here’s my note:
Hey [first name],
I’m Jeilan, a copywriter at Venngage. And I’m going to be completely honest with you.
I’m running an experiment to see if email subject line tools EVEN work.
And you’re part of the experiment!
Well, sort of.
I want to see if these tools can actually tell me whether an email subject line is effective or not. Once I’ve gathered my data, I’ll share my results with you!
How’s that sound? Let me know if you’d like to see the final product. I’ll send a link to the post once it’s live!
Have a great week, [first name].
JD
My expectations
Of course, I anticipated a certain set of results.
CoSchedule was ranking each line consistently in the 50s. I thought to myself, does each email subject line equally suck? That’d be disappointing, I was hoping to be surprised.
Subject Line was pretty generous with its ranking for almost every line. Although it wasn’t a big fan of Casablanca. I thought, Okay, they can’t all do well. One of them is actual gibberish.
Net Atlantic was pretty unpredictable in how it ranked the email subject lines. I assumed its rankings wouldn’t be predictive of open rates at all.
The results
I ended up sending my email on a Tuesday, around 2PM. It wasn’t the best time to send an email, nor was it the worst. I just wanted to get the experiment underway sooner than later, and everything was ready to go that morning.
I waited an entire week before actually looking at the open rates. This way, I gave people the weekend to clear their inbox and either open or ignore the email then. I also ended up receiving three replies from people.
The results were in. I was a little surprised, but not really at the same time.
CoSchedule was fairly accurate with email subject lines 2, 3 and 4. The open rate percentage of each email very nearly corresponded with CoSchedule’s ranking of the email subject lines. However, it was pretty off with email subject lines 1 and 2.
Subject Line was not very helpful as a predictor of email subject line performance. Whether I got an open rate of 75% or 35%, its ranking only fluctuated by 9 points. Email subject lines 2 and 3 both got 50% open rates, yet their ranking in Subject Line varied quite a bit. All in all, Subject Line is a poor indicator of email subject line quality.
Net Atlantic gave the lowest ranking to the best performing email subject line in the experiment. It also gave the highest ranking to the worst performing email subject line. I think it’s fair to say it has no predictive value whatsoever.
Factors that impacted my open rates
There are several factors that affect your email open rates. Superoffice looked at a study by CMB Consumer Pulse, which identified five significant reasons why people open emails.
- 64% of people actually open their emails because they recognize the sender.
- 47% of people open emails due to the email subject line.
- 26% of folks are motivated by the offer of the email.
- 14% are swayed by the first few sentences of the email.
- 4% open emails based on their anticipated length.
I know of at least two instances in this experiment where my email subject line was not the reason for opening the email.
Email subject line 1
One recipient saw the “I’m a copywriter, yadda yadda” message and assumed it was a guest post pitch. She replied, realizing I was testing email subject lines, letting me know that she only opened it thinking it was a guest post pitch.
Email subject line 4
You’ll recall, I mentioned who I am and where I work in the body of the email. Well, one person recognized the company name. She knew Venngage because she attended Inbound17 and saw our marketing director, Nadya Khoja, speak. So, without meaning to, I skewed the results.
In addition, the time that an email gets sent affects open rates. I sent the emails between 2PM-3PM on a Tuesday. Our own study on the best time to send an email newsletter showed that sending an email between 2PM-3PM is ideal. However, Tuesdays is one of the worst days to send an email.
So the experiment itself was not perfect, but since the outreach was completely cold, I can confidently say the email subject lines influenced open rates more than anything else.
My thoughts
CoSchedule
CoSchedule, by virtue of its own consistency, managed to be relatively predictive of open rates. But even then, it was an inaccurate measure of email subject lines 1 and 5.
While open rates between the email subject lines varied by 40%, CoSchedule only ascribed a difference of 6 in rankings.
The tool clearly has a built-in formula for the ideal email subject line. It considers character and word range, word lengths, word uses, emoji uses and more, to determine how effective the email subject line is. That tells me it follows a specific formula. However, it just did not accurately reflect my best-performing email subject line with its score.
Subject Line
I would say the tool proved to be an unhelpful predictor of open rates. Almost every email subject line was treated as outstanding, with some room for improvement.
Yet, the experiment shows that even email subject lines that performed poorly rank high in Subject Line. It begs the question, if poorly-performing email subject lines rank high in Subject Line, then what’s the purpose of achieving a high rank?
Net Atlantic
I don’t know what the deal is with this tool. It clearly has no predictive value whatsoever, and relying on its ranking to shape your own subject line would honestly be a waste of time.
What does it all mean?
If you’re suffering from poor open rates, can’t figure out what to do about it, then CoSchedule may provide a fresh approach to engaging your audience. It’ll encourage you to craft subject lines that are concise, intriguing and fun.
However, my best performing subject line was ranked quite low by CoSchedule. That tells me if you’re subject lines already perform above average, then you’re doing something right and need to keep building on that.
If you understand your target audience, their interests and why they engage with your business in the first place, then you already have some of the insights needed to create compelling subject lines that resonate with them.
If you’re struggling with writing impactful, enticing subject lines, you can also start by looking at some best practices. This lets you set a foundation for yourself to improve upon in the future (using CoSchedule’s tool or another approach).
Writing better subject lines
It’s important to try new things in order to determine which email subject lines work best for you and your business. Some approaches to subject lines are universal, working across most industries. And other should be avoided like the plague.
Mailchimp, an email marketing tool with millions of users, released a study analyzing 24 billion email subject lines. I’ll summarize a few of the findings that will help improve your email open rates:
Personalization works!
Mailchimp’s study found that using people’s names in the subject line of an email results in significantly higher open rates. While it’s great to use a first name or a last name, using both yields the best results.
This particular finding is validated further in other studies, including a cold outreach email study by Klientboost. The simple email subject line, “Hey there [FIRST NAME]!” was tested and saw open rates of over 30%.
“Free” doesn’t always help
Turns out, while people like free stuff, the word “free” doesn’t always get you higher open rates. Depending on your industry, using the word “free” can either increase or decrease open rates.
“Thank you” goes a long way
People like being thanked, it makes them feel good! The words ‘thank you’ in a subject line improve open rates considerably.
Capitalization helps (a little bit)
Capitalizing your entire subject line gives your open rates a slight boost. It’s not major, but the study showed that the increase is consistent across industries.
Our study on the best number to use in a blog title found that the number 10 attracts the most attention. Numbers like 7, 5, 4 and 3 are also popular. Blog titles are certainly different from email subject lines, but this tells us that things like numbers alone can influence someone’s interest.
Conclusion
However you decide to improve your subject line, whether you rely solely on CoSchedule’s rankings, implement Mailchimp’s suggestions, a combination of the two or neither, test your changes.
Make the extra effort to run A/B tests that compare different sets of subject lines, to see what actually works. At the end of the day, those are the results that will help you decide what changes to build on and what changes to drop.
CoSchedule, Subject Line, and Net Atlantic are only three of the tools available but there are plenty of others, such as Omnisend’s Subject Line Tester. Shop around to see what tool best suits your needs.
As you dive deeper into making changes and measuring the results, let me know what you learn! Comment below and share your experiences.
Design a newsletter that your audience will love to receive using Venngage