Direct mail test group size?

Question

I've read tons of articles on the 'net about doing test vs control mailings. I've taken my list parameters and expanded them as wide as I'm interested in going and that gets me just a hair over 6000 addresses I can direct mail (currently I'm mailing to 2900 addresses).

For those of you that have done test mailings, what size list would you recommend for a test group?

Dev Horn · Answer

You can do tests with as few as 500, but the smaller the count, the lower the statistical relevance to the larger market. I like to do direct mail in minimum runs of 2,500, but for some that is cost prohibitive. One of the challenges of testing direct mail is that it works better when the target group is touched at least 3 to 5 times over the course of a campaign. So, you could send out a test mailing to a group one time, and the results would not be a true measure of effectiveness. You'd need to hit that group with a similar message at least three times. In your case, consider dividing the 6,000 into three groups and running three concurrent campaigns with three variations on test messaging (and elements like use of photos vs. just graphics, etc.). If you don't have the funds to hit all 6000, divide it down to three groups of 1000 or 500 - whatever fits the constraints of your budget. Just remember, the test groups need to be statistically similar, and you must consider the campaign aspect rather than trying to test with a single mailing.

Sending out two variations of a message to two groups just one time will tell you something. But it may not really mean anything. =)

Sending out two variations of a message to two groups just one time will tell you something.  But it may not really mean anything.  =)

Michael Quarles · Answer

My point to sending only 3 pieces is at that point those calls are going to tell you a story... And more then 1000 seems like a waste of money... Once you have tested and have your infrastructure in place mail away...

Depending on your prospect 3 hits may be all you would want.

Absentees and OOs should get hit until they sell  or die where Mortgage Lates and PreNODs 3 is your max which turns into 6 hits if the seller continues down their path.

I have mailed millions and millions of pieces of mail and I'm a frigging numbers nerd so I track...

Heck I am sending out 5000 pieces using 5 different front designs on Monday to determine which front works best and that is after a single blind test with 20 plus people asking them what they like.

testing is an art... And expensive...

Certainly there are ways to circumvent some testing and that is to hire a advisor who has been there done that kind of thing.

There are three or four posters here on BP who seem to have experience with direct mail. Not certain if they have mailed out the quantity I have at the variety I have however when they type responses they're typically right on.

Happy house hunting

Michael Quarles · Answer

IMHO

More then 1000 prospect hit 3 times is too much.  A lot of data will be learned.

However there a large amount of variable besides prospect demographics and geographical area.

1- days between hits
2-message
3- mail piece type
4 - color of envelope if letter
5 - ink
6 - use of pictorial messages 
6b- which picture works best
7 - order of messages

All important factors.

Terrence Smith · Answer

Hi @Al Bunch

how did this turn out?

of the 6k mail drop what did you break your test groups into? Any lessons learned between then and now on testing you can share?

I was searching for info on how to test direct mail campaigns when I came cross this thread. Comments from Dev and Michael suggest testing on small batch mailing of 1000 or so can't really be counted on.

Al Bunch · Answer

Michael Quarles  
The list I'm working on right now is only Absentee IS/OOS - I was able to glean some interesting (to me) statistics before I started mailing that helped me shape my list.  Now that I've kinda got the mailing process down, I'm ready to start doing some testing.  I'll be marketing to one list for 48 months and the other for 54 months - only because I believe that's where the sweet spot is...so there's plenty of time to try different things.

At some point the cost of my time will increase to where it makes more sense to farm out the direct mail and I'll seek out a shop like yours to handle it for me..but for now I'm enjoying it and interested to see how high I can drive the rate of quality leads being generated.  Increasing response rates is simple - increasing close rates is a whole other story - I want the people that are ready to be closed calling me - so that's my focus.

I was talking to someone about direct mail the other day and the conversation shifted to sales goals..eventually I gave my theory that if I mail the same exact piece out every 4 weeks to the same exact "window" of people (assuming that people move into and out of that window over time, but that the makeup stays the same) and I mail 5000 of those pieces and as a result I net $10,000 in profit every month, that all I need to do to net $20,000 is increase the size of that window to allow me to mail out 10,000 of those pieces...and to net $30k/mo it should be 15k pieces, etc.  I have a feeling those numbers wouldn't exactly bare out, but I imagine it's pretty close.

He didn't buy my argument..at all.  I'm hoping to prove him wrong (and learn something along the way).

Thanks for the responses!

Al Bunch · Answer

Michael Quarles I don't think I'm understanding your statement, "More then 1000 prospect hit 3 times is too much."  I'm betting though that you're not suggesting to only mail to a prospect 3 times...??

The only variables on your list that aren't (currently) on mine are 6, 6b and 7 - because I'm doing straight text campaigns with no graphics and I'm just going for repetition.  I have another group I will start work at the end of the year that will get an ordered series of messages..that's a ways off though.

I'm a big fan of turning one dial at a time, so I don't want to vary a bunch of things all at once and wonder which one caused the change.

Al Bunch · Answer

Three contacts before making another change dovetails nicely with how I'm setting my system up to work.  Right now I batch 3 months worth of direct mail at a time and just deliver to the post office when my pre-determined date hits.

The budget will handle all 6000, I'm just looking for the best way to carve the list up to get the most out of it.  At one point I had a pretty elaborate system devised before I realized I'd be spending more time working on the direct mail than I would working on the leads generated by the direct mail.  Eventually I scaled it back and thought I'd split the list down the middle and just run 3 month races with the winner becoming the control for the next quarter.

Direct mail test group size?

Most Popular Reply