A/B Testing in Email Marketing: Frequently Asked Questions (FAQ)

This FAQ addresses common questions and challenges digital marketers encounter when running A/B tests for email campaigns. For foundational concepts, practical steps, and examples, see our main article: A/B Testing in Email Marketing: Why and How to Use Statistics .

Frequently Asked Questions

1. What Confidence Level Should I Use for A/B Testing in Email Marketing?
   The most common confidence level used in email marketing is 95%. This means you want to be 95% certain your observed result is not due to random chance. Some marketers might use 90% for quick, less critical campaigns, but 95% is the industry standard for reliability. Very strict tests may use 99%, but this requires much larger sample sizes.
2. Why Don’t Email Marketing Tools (like Mailchimp) Require You to Set a Confidence Level?
   Most eDM tools (like Mailchimp or MailerLite) automatically use a default confidence level for A/B test calculations—typically 95%, but sometimes as low as 80% for faster results. They simplify the process by hiding statistical details and simply show you whether the difference between versions is “statistically significant” or not, based on their internal settings. Some tools may show you the actual confidence percentage in the test result summary.
3. What to Do If Your A/B Test Is Inconclusive?
   If your test result is “inconclusive” (meaning the confidence intervals of the two versions overlap), it means you cannot be sure one version is better. In this case, don’t make a big change based on the test. Try running a new A/B test with:
   • a larger sample size
   • a more dramatic difference between the versions
   • different variables (e.g., try testing subject line instead of image)
   • a longer testing period
   It’s normal for some tests to be inconclusive—don’t be discouraged. Each test is a learning opportunity!
4. Can You Lower the Confidence Level (e.g., from 95% to 90%) to Make a Result Significant?
   Technically, yes—you can lower the confidence level to make it “easier” for your test to be called significant. However, lowering the confidence level increases the risk of a false positive (thinking you found a winner when it was just luck). Use 90% only if you’re doing quick, low-risk experiments. For important business decisions, stick to 95%.
5. Will the Population and Sample Size Affect My Test Results?
   Yes! The larger your sample size, the more reliable your results and the smaller your margin of error. If your overall audience (“population”) is small, your sample size should be a large percentage of it. Always try to use at least 500 recipients per version for trustworthy results.
6. Does the Sample Size Chosen by eDM Tools Affect My A/B Test Reliability?
   Yes. Most email marketing tools automatically select a sample size (often 10% to 20% of your list, or a minimum of a few hundred per version). Too small a sample leads to a high margin of error, making your results less trustworthy. If possible, check your platform’s sample size recommendations and increase it if you need more accuracy.
7. What’s the Difference Between Statistical Significance and Practical Significance?
   Statistical significance means the result is unlikely to be due to chance, based on your confidence level. Practical significance means the result is large enough to matter in real life. Sometimes a tiny improvement is “statistically significant” but not big enough to affect your business goals. Always ask: Is the difference meaningful, or just a tiny statistical win?
8. How Can I Improve My Chances of a Statistically Significant Result?
   
   • Increase your sample size
   • Test bigger differences (e.g., two very different subject lines)
   • Make sure your audience is randomly selected
   • Extend the duration of your test if you can
   These steps help reduce random noise and make it easier to spot real differences.
9. What If My Results Are Different Next Time?
   This can happen! Audience behavior changes over time, and random chance can also play a role. That’s why it’s best to keep testing regularly and look for consistent patterns, not just single lucky wins. Always base big changes on repeated evidence, not one-off results.
10. Can I Test More Than One Thing at a Time?
    It’s best to test one variable at a time (e.g., only the subject line) in an A/B test. If you change more than one thing, you won’t know which change caused the result. For more complex tests (multivariate testing), you’ll need more advanced tools and bigger sample sizes.

---

常見問題（繁體中文翻譯）

1. 電郵行銷的A/B測試應該使用什麼信心水平？
   電郵行銷最常用的信心水平是95%。這代表你希望對於測試結果不是因隨機機會出現，有95%信心。某些營銷人員會在不太重要或需要快速結果時選用90%，但95%是可靠的業界標準。極嚴格的測試可能會用99%，但這需要更大的樣本數。
2. 為什麼像Mailchimp這類電郵工具不用你自己設定信心水平？
   大部分eDM工具（如Mailchimp或MailerLite）自動採用預設的信心水平計算A/B測試，通常是95%，有時為求快速會用低至80%。這些平台會隱藏統計細節，直接顯示結果是否「統計顯著」，用戶毋須自行設定。有些平台會在報告中顯示實際信心百分比。
3. 如果我的A/B測試結果不明確，應該怎麼辦？
   如果結果顯示「不明確」（即兩個版本的信心區間重疊），就無法確定哪一個更好。這時不應根據結果做重大更改。建議：
   • 增加樣本數
   • 讓A/B版本的差異更明顯
   • 換測試變數（例如改測主旨行而非圖片）
   • 拉長測試期
   測試出現不明確是很常見的，每次測試都是學習機會，毋須灰心！
4. 我能否將信心水平由95%降到90%，讓結果變得顯著嗎？
   技術上可以，這樣「達標」會較易。但降低信心水平會提高偽陽性（即以為找到贏家，其實只是運氣）。90%僅適合低風險和快測試，重要業務決定請用95%。
5. 母體與樣本數會影響測試結果嗎？
   會！樣本數愈大，結果愈可靠，誤差範圍愈小。如果總受眾（母體）較細，樣本應佔較高比例。每組最少500個受眾較為可信。
6. eDM工具自動選擇的樣本數會影響A/B測試可靠性嗎？
   會。多數電郵平台會自動選取一定比例（通常是10%至20%，或每組最少數百人）。樣本太少會導致誤差大，結果不可靠。若可自行調整，建議根據平台建議適度提高樣本數。
7. 「統計顯著」與「實用顯著」有何分別？
   統計顯著代表結果出現的機會很低，是根據信心水平計算。實用顯著則是指結果夠大，有實際影響。有時候很細微的提升雖然「統計顯著」，但對業績影響微乎其微。要常問自己：差異對業務是否有實際意義，還是僅僅是數字上的勝出？
8. 如何提升取得統計顯著結果的機會？
   
   • 增加樣本數
   • 測試差異較大的版本（例如兩個主旨行風格很不同）
   • 確保樣本隨機抽樣
   • 若可，拉長測試時間
   這些做法可以減少雜訊，更容易看到真正差異。
9. 如果下次測試結果又不同怎麼辦？
   這是正常的！受眾行為會變，隨機因素亦會出現。建議持續測試並留意有否重覆模式，不要只因單次結果就作重大決定。只有累積多次的證據，才適合作出關鍵改動。
10. 可以同時測試多於一項東西嗎？
    最好每次A/B測試只改一個變數（例如只改主旨行）。如果同時改多項，無法判斷哪一項導致改變。多變數（multivariate）測試需進階工具和更大樣本。

---

如有疑問，歡迎參考主文： A/B Testing in Email Marketing: Why and How to Use Statistics ， 或在下方留言提問！

— Dr. Ken FONG