Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
Even if you're not really a political nerd.
Speaker 2 (00:02):
When you get this close to a presidential election, everybody's
thinking about politics to some degree. We try not to
do too much politics here on the show to kind
of keep our brains fresh and all.
Speaker 1 (00:12):
But I mean, two weeks from tomorrow, it's the election. Now.
Speaker 2 (00:16):
As we're thinking about the election, one of the main
types of data point that we keep getting told about,
that we might read about voluntarily, that we might hear
about on the news is Poland you hear this poll,
that poll, and maybe you hear a few other little
terms likely voters, registered voters, and so on. And you
(00:39):
hear a poll Trump up zero point eight hair US
up two points in the national which isn't really a thing.
You hear all this stuff, but do you really know
what it means? Do you really know what it's measuring?
And most importantly, do you really know how to think
about polls and what they are actually measuring and what
(01:03):
they're not measuring. Really, but a lot of people talk
about him as if they are so I realize, on
the one hand, this is nerdy. On the other hand,
it's really fascinating to me. And I'm so pleased to
be joined by Carl Allen. Carl is a data analyst
and he recently wrote a book that I read all
of entitled.
Speaker 1 (01:24):
The Polls Weren't Wrong.
Speaker 2 (01:27):
And so with that already too long introduction, Carl, welcome
to KOA, good to see you, good to have you here.
Speaker 3 (01:35):
Hey, good morning, thanks for having me.
Speaker 2 (01:37):
So I'm just gonna jump in with some questions, and folks,
if you've got questions.
Speaker 1 (01:42):
About political polls and really.
Speaker 2 (01:45):
Like methodology and questions about how polling works. Not do
you believe Trump is really up or down? Not that
kind of question, but how this stuff works, text them
to me at five sixty six nine zero and I
will ask Carl the good question. So this is gonna
probably do some people, sound like a dumb question, But
what is a poll actually measuring?
Speaker 3 (02:04):
You know what?
Speaker 4 (02:05):
That extremely simple question baffles even the.
Speaker 3 (02:09):
Most brilliant of experts in the field.
Speaker 4 (02:11):
Right now, I would bet money, and I know we're
not I know it's kind of legal now, kind of not,
but I would bet money that if you ask one
hundred experts in the field what do polls measure?
Speaker 3 (02:22):
Really, you would.
Speaker 4 (02:23):
Probably get about seventy five different answers, and the answer
the scientifically valid answer is that polls are an estimate
of a base of support. Poles are an estimate or
a snapshot of some current characteristic of a population. When
we think about poles, obviously, political polls are the example
(02:43):
that come to mind. Oh, you can take polls on anything, race, gender,
do you have a dog, are you married, et cetera.
But polls as a tool, as a scientific pool, are
intended to give us an estimate of a current or
the snapshot of a current characteristic of a population.
Speaker 2 (03:05):
Okay, one thing that I think baffles a lot of people,
and this is a math question now, is how can
a pole with what seems like a small number of people,
let's say, one thousand, eleven hundred, be even close to
possibly being representative of a whole state or a whole
(03:30):
nation of three hundred million plus people.
Speaker 4 (03:33):
Beautiful question, And this gets into the nerdy mathematical part.
Speaker 3 (03:39):
My book is not too technical on this end.
Speaker 4 (03:42):
My book is not just intended to help experts fix
the way they think about and talk about poles, but
it's also intended for people who are not experts that
want to better understand poles to understand what they mean.
So the underpinnings of poles statistically rely on this concept
of a random sample, and it is unintuitive.
Speaker 3 (04:02):
But it to me, it's very interesting and it's brilliant.
But to the.
Speaker 4 (04:06):
Average person it's they're very skeptical of it. The fact is,
if you can take a random sample from a population,
say one thousand people.
Speaker 3 (04:17):
If you take a random sample.
Speaker 4 (04:18):
Of one thousand Americans of our what three hundred million population.
If you take a random sample of one thousand Americans,
you can get plus or minus three percent.
Speaker 3 (04:28):
Of just about anything. Of just about anything, whether they are.
Speaker 4 (04:33):
Married, whether they have a dog at home, how many
kids they have, et cetera.
Speaker 3 (04:36):
You can get very, very very close.
Speaker 4 (04:38):
And the idea that a sample of that size could
be representative of such an enormous population is not intuitive.
But in the book I point out, hey, guys, this
is why we do science in the first place. This
is the value of statistics. It helps us wrap our
minds around concepts that would otherwise be not intuitive.
Speaker 3 (04:57):
But the best part about math is that we can
prove it.
Speaker 4 (05:00):
We can prove it mathematically, and I encourage readers who
are interested.
Speaker 3 (05:04):
In the more technical aspects in the book to test it.
Speaker 4 (05:07):
Themselves, because it's a very easily testable concept.
Speaker 3 (05:11):
The challenge when we get into political.
Speaker 4 (05:13):
Polling is this idea of a random sample, and and
how random is that sample really?
Speaker 2 (05:20):
All Right, we'll get into that in a second. One
more question on the math. Is it accurate to say
that if I were to take a pole of a
medium sized state that has five million people, and I
wanted to get error bars around the pole of let's
say plus or minus about three percent, and then I
(05:42):
wanted to do the same pole of the entire country
of three hundred million people, I would.
Speaker 1 (05:49):
Need almost the exact.
Speaker 2 (05:52):
Same number of people in the two polls to have
the same error bars, right, And or another way to
put it, If I'm taking a pole of three hundred
million people and I need a thousand to be representative,
then if I I mean I'm not pulling three hundred
million people, I want to do a poll representing a
population of three hundred million people, and I need to
pull a thousand people to get X error bars. If
(06:12):
I then go to a country that has six hundred
million people, I do not need two x the number.
Speaker 1 (06:18):
Of people in the poll.
Speaker 2 (06:19):
It's still actually very very close to X, right, yes.
Speaker 4 (06:23):
Sir, that is exactly correct, and that is the unintuitive
nature of a random sample is that once the sample,
once the random sample becomes arbitrarily or sufficiently large, the
size of the population does not matter. Which is to say,
the margin of error for a one thousand person random
(06:44):
sample is approximately the same. I'm talking within hundredths of
a percent, within hundredths of a percent. Whether you're talking
to a population of fifty thousand or fifty million, it's
within hundreds plus or mind minus three point zero percent,
plus or minus three point five percent.
Speaker 3 (07:03):
It is remarkably, remarkably close.
Speaker 4 (07:06):
And the reason, the mathematical reason is the margin of
error is in verse. I'm probably gonna get this wrong
in virtually proportional to the sample size. The population only
matters when the population is small. The sample size is
the only thing that matters when you're dealing with a
sample size of anything larger than, say, twenty to fifty thousand.
Speaker 2 (07:25):
We're talking with Carl Allen. His new book is called
The Polls Weren't Wrong. One one last on this because
a listener question came in on it.
Speaker 1 (07:33):
How do polsters know the.
Speaker 2 (07:37):
At least theoretical error rate error bars on their polls,
and let me just see if I've been a good
student of yours. So, Sean, I think the answer is
it's straight up math, and as long as you believe
you have a random sample, and we will get to
that in a second, as long as you believe you
have a random sample, the answer is to the error
(08:00):
in bar in the polls just comes out of the math.
And it doesn't matter what question you're asking. It does
nothing like that. It's just math, and it's not even
very difficult math once you understand it.
Speaker 1 (08:11):
Go ahead, Carl, Yes, that's correct.
Speaker 4 (08:14):
So the margin of error that is reported in every poll,
to my knowledge, ever taken, is the margin of air
based on the sample size.
Speaker 3 (08:23):
But as many of your listeners may know.
Speaker 4 (08:25):
If you've read my book you'll know, and even people
who don't study politics may know, the margin of error
is not the only potential source of air in a poll,
and yet it's the only source of error reported. So
that creates this really interesting debate about how to capture
this this total error. And I talk about that some
(08:46):
in my book. But the bigger question, or the bigger answer,
I guess, is that, yes, the only number ever reported
is based on the sample size, even though even though
polsters know that in most cases not anymore, maybe fifty
years ago, but not anymore, Ulsters know that their samples
are not truly random. So we're getting really really deep
(09:08):
into this conflict of what is scientifically valid versus what
is how are things currently discussed right?
Speaker 2 (09:17):
So I will say to Sean, the error rate around
the around the pole is absolutely known, just based on
the math. But it assumes a random sample, and that's
not something we can assume anymore. And in particular, this
is a thing that a lot of people.
Speaker 1 (09:37):
Have one particular political persuasion.
Speaker 2 (09:40):
Conservative Republicans MAGA Republicans have a deep mistrust Carl in
the underlying samples, and.
Speaker 1 (09:49):
I get it.
Speaker 2 (09:50):
So, without getting into conspiracy theories of intentionally non random samples,
which I really don't believe in, talk about how you
could get a sample that really isn't random and therefore
the error bars really aren't right.
Speaker 3 (10:11):
That's great question.
Speaker 4 (10:12):
So to answer the first question, it is remarkably easy
to get a random sample.
Speaker 3 (10:18):
To obtain a random sample, all you.
Speaker 4 (10:21):
Need to do is have a list of registered voters
and you can take a random sample from it.
Speaker 3 (10:25):
It's not very hard. I've done it. Most of your
listeners could probably do it. This data is public.
Speaker 4 (10:32):
The challenge is contacting the random sample. Contacting the random
sample is where all of the statistical issues come into play,
because we know some people will never answer their phone,
some people will never respond to a text, and those
people might have something in common. So when posters are
(10:53):
contacting these people, they say, well, we need responses from
this many white men, this, many young black people people,
this many.
Speaker 3 (11:00):
Et cetera, et cetera.
Speaker 4 (11:02):
But the people that they're talking to might not be
representative of that demographic.
Speaker 3 (11:07):
So what polsters have.
Speaker 4 (11:08):
To do on the back end is they say, Okay,
we know our sample wasn't random, and to their credit
this is true.
Speaker 3 (11:13):
They say, we know our sample wasn't random.
Speaker 4 (11:15):
If our sample were random, what would it have shown?
And therein lies the conflict of how do we weight
this data? How do we weight this data one or
two points in any direction? I've done this not that
long and there are people out of it better, people
who are better at analyzing weighting methods than me. But
(11:36):
I can tell you that one or two points in
any direction for either candidate is not hard to do
with a poll. And this all comes back to if
our sample was random.
Speaker 3 (11:48):
What would it have shown.
Speaker 4 (11:49):
And it all depends on your perception of these demographics
and who's ultimately going to turn out to the.
Speaker 2 (11:56):
We're talking with Carl Allen. His new book is called
The Polls Weren't Wrong. And I'll just tell elaborate on
this once and then we'll get onto something else. And
then again, Carl, you can kind of tell me whether
I've whether I've said this wrong. But let's just say,
as an example, say to listeners, let's just say as
an example that a significant majority of work quote unquote
(12:17):
working class white men will vote for Trump. And also
a significant percentage of working class white men have no interest.
Speaker 1 (12:28):
In talking to pollsters.
Speaker 2 (12:29):
And so if their phone rings and they see it's
a poll call, or if they answer it and they
say we want to ask you some questions, they don't
answer or they hang up and they just they won't participate.
And so let's say then the pollster asks a thousand people,
and of the thousand people, maybe twenty three of them
are working class white men. Well, they have to guess then,
(12:54):
and let's say seventy five percent of them.
Speaker 1 (12:56):
Say they're voting for Trump. Well, they have to guess,
is it.
Speaker 2 (12:59):
Really two point three percent of the electorate that's gonna
be working class white men, or is it gonna be
double that? And if it's double that, then they got
to multiply by two. But if they get that wrong,
they'll get the result wrong. And then it gets even
harder than that, because what if the working class white
(13:20):
people who respond to the poll are actually not typical
working class white people and might lean slightly less towards
Trump than the ones who don't, who don't respond to
the pull. It gets so difficult trying to guess what
the actual electorate is gonna gonna look like. Okay, so
(13:42):
give me a really quick response to that, and then
I want to get onto something else for our last
few minutes.
Speaker 3 (13:47):
No, you're you're exactly right.
Speaker 4 (13:49):
Polsters having a remarkably hard job.
Speaker 3 (13:52):
And so, as someone who wrote a book entitled the
Polls Weren't Wrong.
Speaker 4 (13:55):
I have to say the fact that they are still
able to get somewhat accurate data out of this nearly
impossible task of contacting a fraction of a fraction of
a populace and only getting responses from an even smaller fraction.
The fact that they are still able to make what
I would consider very very good estimates really gets to
(14:16):
the point that these people, the good polsters, kind of
know what they're doing.
Speaker 3 (14:21):
Yes, they have to make some assumptions. Yes they have
to make you might call them guesses.
Speaker 4 (14:26):
They're informed guesses, but they are, at the end of
the day, guesses. The fact that they've been able to
do so well is pretty remarkable to me.
Speaker 2 (14:35):
Okay, so we only have a few minutes left, and
this may be the most important question, but I think
we needed a lot of foundation there. Way too many
people treat polls as being the same as forecasts. And
people will look at polls that show, you know, Trump
up one percent in such and such a state, or
(14:56):
even betting odds showing Trump sixty percent to win the
and betting on really is a.
Speaker 1 (15:03):
Kind of a forecast. But so let's stick with poles.
Speaker 2 (15:06):
But people will look at a poll as a as
a forecast. And I want you to just explain why
a poll is not the same as a forecast.
Speaker 1 (15:14):
And I realized you just wrote a whole book on it,
and I'm asking you to do this in two minutes.
Speaker 4 (15:19):
Yeah, So interpreting poll data as a prediction of election
outcomes has been around since the beginning of political polling
one hundred years ago Gallup literary digest. Interpreting poll data
as a prediction of the election outcome is not new.
The problem is modern research, which is much more analytical,
(15:41):
much more scientifically based, has maintained those unscientific standards of
if the poll says this candidate is up by let's say,
up by two in the poll, then therefore they must
win the election by two and or the poll was
not accurate and so in order, and it's this very
simple math. In order for that state to be maintained,
(16:04):
two things must be true. One, no one can change
their mind between the poll and the election. Two undecided
must split evenly between these major candidates.
Speaker 3 (16:14):
Well, we can test these things in science.
Speaker 4 (16:16):
When things are testable, we do not use assumptions in
their place. And yet that is no joke, no exaggeration,
no artistic liberties. That is the literal definition used by
the consensus of experts around the world about how to
analyze poll data. They analyze polls, a tool not intended
to be predictive, by how well they predict election result. Again,
(16:40):
polls are an estimate of a base of support. In
the book, I use the term of simultaneous census. If
you take a survey months before the election, maybe it's
an election that not many people are aware of, a Senate.
Speaker 3 (16:52):
A house race, et cetera.
Speaker 4 (16:54):
And it says one candidate is at forty five percent,
the other candidate is at forty percent, and there are
fifteen percent on this. What that means is, if you
had taken a simultaneous census of that population, approximately forty
five percent plus or minus the margin of error would
respond with that candidate. Forty percent plus or minus the
(17:14):
margin of error would respond to that candidate.
Speaker 3 (17:15):
And this is the part that is extremely.
Speaker 4 (17:18):
Important, because I hate to say experts do not understand this.
Fifteen percent wels Er minus the margin of error would
also say they're undecided, period. End of what the poll
tells us. How those undecided to allocate themselves how they decide,
can swing the election. In many swing states, in many
(17:40):
close elections where most polling is.
Speaker 3 (17:41):
Done, this does happen. The assumption that.
Speaker 4 (17:45):
Undecideds must sweit fifty to fifty or else the poll
was wrong is the single most unscientific belief that is
literally held by the consensus of experts in the United States,
and it is wrong.
Speaker 3 (17:59):
It's not true. We can test it, we know it's
not true.
Speaker 4 (18:02):
So my proposal is that instead of using these assumptions,
we must ask the question of how did they decide?
Because they can wing the outcome of the election to
a quote unexpected outcome.
Speaker 2 (18:16):
All right, I have literally one minute left. I want
to check your math on one thing. Let's say there's
a poll that's got it forty five to forty with
fifteen undecided.
Speaker 1 (18:26):
In order for it.
Speaker 2 (18:27):
To end up forty five forty as the actual result
at the end, would it the undecided had to have
broken forty five to the whatever That forty five to
forty ratio is like.
Speaker 1 (18:43):
Not nine out of seventeen to eight out of seventeen.
Speaker 2 (18:48):
Versus having broken fifty to fifty, which is what you
said before.
Speaker 4 (18:52):
They're two different standards for analysis. In the United States,
experts say fifty to fifty. Non US experts stay proportional,
and they're an unspoken truce between these unscientific analysts that
they don't challenge each.
Speaker 3 (19:03):
Other because they're both wrong.
Speaker 1 (19:04):
Uh huh.
Speaker 4 (19:06):
So what they make different, they make different unscientific assumptions. Yeah,
so election results will always add up to one hundred percent.
Speaker 2 (19:13):
We know that.
Speaker 3 (19:14):
I mean, that's that's just very simple, Matt. Uh huh. So,
an election result cannot be forty.
Speaker 4 (19:18):
Five, right, it must They must add up to one
hundred percent, and those that comes from how those undecided
ultimately decide.
Speaker 2 (19:24):
Right. And that's true because forty five forty really, as
I said, if you were to take that as the
whole electorate, then that ad then would be nine out
of seventeen and eight out of seventeen at the end,
which isn't the same as forty and forty out one
hundred and forty five out of one hundred. So all right,
we're just about just about out of out of time here.
(19:44):
I want to follow up on one quick thing you said.
Uh So, I read in your book that not only
should you not assume that undecided voters will break in
a particular way that matches the decided.
Speaker 1 (19:58):
Voters in in percent or anything, but.
Speaker 2 (20:02):
There seems to be a little bit at least of
bias among undecided voters in America that when they do decide,
they tend to vote for for change. I think I
read that like they they side against the incumbents with
some frequency.
Speaker 1 (20:15):
Is that what you wrote?
Speaker 4 (20:17):
That was analysis done by a man named Nick Panagakus
about twenty to thirty years ago, and his work should
have led directly into mine.
Speaker 3 (20:26):
He passed away a few years ago.
Speaker 4 (20:28):
Unfortunately I never got to meet him. His work in
a proper scientific field would have led directly into mind.
But what ended up happening is he passed away. His
work kind of was lost to journals in history, and
these unscientific analysts Silver and Morris and the other guys,
they've taken his place, or they've taken the place of
proper analysis and grown this spread mentality of undecided to
(20:54):
must split evenly. His the best quote, one of the
best quotes that that I read of his, was rules
of analysis are necessary, rules that are not as simple
as eight points is comfortable and two points is a
close race. He wrote that in nineteen eighty seven, this
is not new my work and the analysis, and you
can probably attest to this having read the book. The
(21:15):
graphics that I showed towards the end of the book
kind of back that up. And the fact that he
was able to infer that in the nineteen eighties, and
that work still hasn't been built upon. Speaks to the
problem of this contaminated traditional thinking. The analogy that I
use is baseball analysts who still try.
Speaker 3 (21:37):
To figure out who the best offensive player is by
batting average.
Speaker 4 (21:41):
Batting average isn't the worst metric in the world, but
it is not the best way to measure offensive output.
And that's the same exact way that spread analysis is
currently contaminated.
Speaker 2 (21:52):
Carl Allen's new book is called The Polls Weren't Wrong.
I know there were a lot more listener questions. If
I didn't get to your question, email it to me
at Ross at iHeartMedia and I'll get all the questions.
Speaker 1 (22:02):
Over to Carl and he'll answer them back.
Speaker 2 (22:04):
To me by email, and then I will send your
answer to you. So my email is Ross at iHeartMedia
dot com. Send me your questions. I'll get you answers
Carl's book, The Polls Weren't Wrong. Thanks for being here, Carl,
really fascinating and important conversation.
Speaker 1 (22:18):
Talk to you again soon.
Speaker 3 (22:20):
Thanks again, Ross
Even if you're not really a political nerd.
Speaker 2 (00:02):
When you get this close to a presidential election, everybody's
thinking about politics to some degree. We try not to
do too much politics here on the show to kind
of keep our brains fresh and all.
Speaker 1 (00:12):
But I mean, two weeks from tomorrow, it's the election. Now.
Speaker 2 (00:16):
As we're thinking about the election, one of the main
types of data point that we keep getting told about,
that we might read about voluntarily, that we might hear
about on the news is Poland you hear this poll,
that poll, and maybe you hear a few other little
terms likely voters, registered voters, and so on. And you
(00:39):
hear a poll Trump up zero point eight hair US
up two points in the national which isn't really a thing.
You hear all this stuff, but do you really know
what it means? Do you really know what it's measuring?
And most importantly, do you really know how to think
about polls and what they are actually measuring and what
(01:03):
they're not measuring. Really, but a lot of people talk
about him as if they are so I realize, on
the one hand, this is nerdy. On the other hand,
it's really fascinating to me. And I'm so pleased to
be joined by Carl Allen. Carl is a data analyst
and he recently wrote a book that I read all
of entitled.
Speaker 1 (01:24):
The Polls Weren't Wrong.
Speaker 2 (01:27):
And so with that already too long introduction, Carl, welcome
to KOA, good to see you, good to have you here.
Speaker 3 (01:35):
Hey, good morning, thanks for having me.
Speaker 2 (01:37):
So I'm just gonna jump in with some questions, and folks,
if you've got questions.
Speaker 1 (01:42):
About political polls and really.
Speaker 2 (01:45):
Like methodology and questions about how polling works. Not do
you believe Trump is really up or down? Not that
kind of question, but how this stuff works, text them
to me at five sixty six nine zero and I
will ask Carl the good question. So this is gonna
probably do some people, sound like a dumb question, But
what is a poll actually measuring?
Speaker 3 (02:04):
You know what?
Speaker 4 (02:05):
That extremely simple question baffles even the.
Speaker 3 (02:09):
Most brilliant of experts in the field.
Speaker 4 (02:11):
Right now, I would bet money, and I know we're
not I know it's kind of legal now, kind of not,
but I would bet money that if you ask one
hundred experts in the field what do polls measure?
Speaker 3 (02:22):
Really, you would.
Speaker 4 (02:23):
Probably get about seventy five different answers, and the answer
the scientifically valid answer is that polls are an estimate
of a base of support. Poles are an estimate or
a snapshot of some current characteristic of a population. When
we think about poles, obviously, political polls are the example
(02:43):
that come to mind. Oh, you can take polls on anything, race, gender,
do you have a dog, are you married, et cetera.
But polls as a tool, as a scientific pool, are
intended to give us an estimate of a current or
the snapshot of a current characteristic of a population.
Speaker 2 (03:05):
Okay, one thing that I think baffles a lot of people,
and this is a math question now, is how can
a pole with what seems like a small number of people,
let's say, one thousand, eleven hundred, be even close to
possibly being representative of a whole state or a whole
(03:30):
nation of three hundred million plus people.
Speaker 4 (03:33):
Beautiful question, And this gets into the nerdy mathematical part.
Speaker 3 (03:39):
My book is not too technical on this end.
Speaker 4 (03:42):
My book is not just intended to help experts fix
the way they think about and talk about poles, but
it's also intended for people who are not experts that
want to better understand poles to understand what they mean.
So the underpinnings of poles statistically rely on this concept
of a random sample, and it is unintuitive.
Speaker 3 (04:02):
But it to me, it's very interesting and it's brilliant.
But to the.
Speaker 4 (04:06):
Average person it's they're very skeptical of it. The fact is,
if you can take a random sample from a population,
say one thousand people.
Speaker 3 (04:17):
If you take a random sample.
Speaker 4 (04:18):
Of one thousand Americans of our what three hundred million population.
If you take a random sample of one thousand Americans,
you can get plus or minus three percent.
Speaker 3 (04:28):
Of just about anything. Of just about anything, whether they are.
Speaker 4 (04:33):
Married, whether they have a dog at home, how many
kids they have, et cetera.
Speaker 3 (04:36):
You can get very, very very close.
Speaker 4 (04:38):
And the idea that a sample of that size could
be representative of such an enormous population is not intuitive.
But in the book I point out, hey, guys, this
is why we do science in the first place. This
is the value of statistics. It helps us wrap our
minds around concepts that would otherwise be not intuitive.
Speaker 3 (04:57):
But the best part about math is that we can
prove it.
Speaker 4 (05:00):
We can prove it mathematically, and I encourage readers who
are interested.
Speaker 3 (05:04):
In the more technical aspects in the book to test it.
Speaker 4 (05:07):
Themselves, because it's a very easily testable concept.
Speaker 3 (05:11):
The challenge when we get into political.
Speaker 4 (05:13):
Polling is this idea of a random sample, and and
how random is that sample really?
Speaker 2 (05:20):
All Right, we'll get into that in a second. One
more question on the math. Is it accurate to say
that if I were to take a pole of a
medium sized state that has five million people, and I
wanted to get error bars around the pole of let's
say plus or minus about three percent, and then I
(05:42):
wanted to do the same pole of the entire country
of three hundred million people, I would.
Speaker 1 (05:49):
Need almost the exact.
Speaker 2 (05:52):
Same number of people in the two polls to have
the same error bars, right, And or another way to
put it, If I'm taking a pole of three hundred
million people and I need a thousand to be representative,
then if I I mean I'm not pulling three hundred
million people, I want to do a poll representing a
population of three hundred million people, and I need to
pull a thousand people to get X error bars. If
(06:12):
I then go to a country that has six hundred
million people, I do not need two x the number.
Speaker 1 (06:18):
Of people in the poll.
Speaker 2 (06:19):
It's still actually very very close to X, right, yes.
Speaker 4 (06:23):
Sir, that is exactly correct, and that is the unintuitive
nature of a random sample is that once the sample,
once the random sample becomes arbitrarily or sufficiently large, the
size of the population does not matter. Which is to say,
the margin of error for a one thousand person random
(06:44):
sample is approximately the same. I'm talking within hundredths of
a percent, within hundredths of a percent. Whether you're talking
to a population of fifty thousand or fifty million, it's
within hundreds plus or mind minus three point zero percent,
plus or minus three point five percent.
Speaker 3 (07:03):
It is remarkably, remarkably close.
Speaker 4 (07:06):
And the reason, the mathematical reason is the margin of
error is in verse. I'm probably gonna get this wrong
in virtually proportional to the sample size. The population only
matters when the population is small. The sample size is
the only thing that matters when you're dealing with a
sample size of anything larger than, say, twenty to fifty thousand.
Speaker 2 (07:25):
We're talking with Carl Allen. His new book is called
The Polls Weren't Wrong. One one last on this because
a listener question came in on it.
Speaker 1 (07:33):
How do polsters know the.
Speaker 2 (07:37):
At least theoretical error rate error bars on their polls,
and let me just see if I've been a good
student of yours. So, Sean, I think the answer is
it's straight up math, and as long as you believe
you have a random sample, and we will get to
that in a second, as long as you believe you
have a random sample, the answer is to the error
(08:00):
in bar in the polls just comes out of the math.
And it doesn't matter what question you're asking. It does
nothing like that. It's just math, and it's not even
very difficult math once you understand it.
Speaker 1 (08:11):
Go ahead, Carl, Yes, that's correct.
Speaker 4 (08:14):
So the margin of error that is reported in every poll,
to my knowledge, ever taken, is the margin of air
based on the sample size.
Speaker 3 (08:23):
But as many of your listeners may know.
Speaker 4 (08:25):
If you've read my book you'll know, and even people
who don't study politics may know, the margin of error
is not the only potential source of air in a poll,
and yet it's the only source of error reported. So
that creates this really interesting debate about how to capture
this this total error. And I talk about that some
(08:46):
in my book. But the bigger question, or the bigger answer,
I guess, is that, yes, the only number ever reported
is based on the sample size, even though even though
polsters know that in most cases not anymore, maybe fifty
years ago, but not anymore, Ulsters know that their samples
are not truly random. So we're getting really really deep
(09:08):
into this conflict of what is scientifically valid versus what
is how are things currently discussed right?
Speaker 2 (09:17):
So I will say to Sean, the error rate around
the around the pole is absolutely known, just based on
the math. But it assumes a random sample, and that's
not something we can assume anymore. And in particular, this
is a thing that a lot of people.
Speaker 1 (09:37):
Have one particular political persuasion.
Speaker 2 (09:40):
Conservative Republicans MAGA Republicans have a deep mistrust Carl in
the underlying samples, and.
Speaker 1 (09:49):
I get it.
Speaker 2 (09:50):
So, without getting into conspiracy theories of intentionally non random samples,
which I really don't believe in, talk about how you
could get a sample that really isn't random and therefore
the error bars really aren't right.
Speaker 3 (10:11):
That's great question.
Speaker 4 (10:12):
So to answer the first question, it is remarkably easy
to get a random sample.
Speaker 3 (10:18):
To obtain a random sample, all you.
Speaker 4 (10:21):
Need to do is have a list of registered voters
and you can take a random sample from it.
Speaker 3 (10:25):
It's not very hard. I've done it. Most of your
listeners could probably do it. This data is public.
Speaker 4 (10:32):
The challenge is contacting the random sample. Contacting the random
sample is where all of the statistical issues come into play,
because we know some people will never answer their phone,
some people will never respond to a text, and those
people might have something in common. So when posters are
(10:53):
contacting these people, they say, well, we need responses from
this many white men, this, many young black people people,
this many.
Speaker 3 (11:00):
Et cetera, et cetera.
Speaker 4 (11:02):
But the people that they're talking to might not be
representative of that demographic.
Speaker 3 (11:07):
So what polsters have.
Speaker 4 (11:08):
To do on the back end is they say, Okay,
we know our sample wasn't random, and to their credit
this is true.
Speaker 3 (11:13):
They say, we know our sample wasn't random.
Speaker 4 (11:15):
If our sample were random, what would it have shown?
And therein lies the conflict of how do we weight
this data? How do we weight this data one or
two points in any direction? I've done this not that
long and there are people out of it better, people
who are better at analyzing weighting methods than me. But
(11:36):
I can tell you that one or two points in
any direction for either candidate is not hard to do
with a poll. And this all comes back to if
our sample was random.
Speaker 3 (11:48):
What would it have shown.
Speaker 4 (11:49):
And it all depends on your perception of these demographics
and who's ultimately going to turn out to the.
Speaker 2 (11:56):
We're talking with Carl Allen. His new book is called
The Polls Weren't Wrong. And I'll just tell elaborate on
this once and then we'll get onto something else. And
then again, Carl, you can kind of tell me whether
I've whether I've said this wrong. But let's just say,
as an example, say to listeners, let's just say as
an example that a significant majority of work quote unquote
(12:17):
working class white men will vote for Trump. And also
a significant percentage of working class white men have no interest.
Speaker 1 (12:28):
In talking to pollsters.
Speaker 2 (12:29):
And so if their phone rings and they see it's
a poll call, or if they answer it and they
say we want to ask you some questions, they don't
answer or they hang up and they just they won't participate.
And so let's say then the pollster asks a thousand people,
and of the thousand people, maybe twenty three of them
are working class white men. Well, they have to guess then,
(12:54):
and let's say seventy five percent of them.
Speaker 1 (12:56):
Say they're voting for Trump. Well, they have to guess,
is it.
Speaker 2 (12:59):
Really two point three percent of the electorate that's gonna
be working class white men, or is it gonna be
double that? And if it's double that, then they got
to multiply by two. But if they get that wrong,
they'll get the result wrong. And then it gets even
harder than that, because what if the working class white
(13:20):
people who respond to the poll are actually not typical
working class white people and might lean slightly less towards
Trump than the ones who don't, who don't respond to
the pull. It gets so difficult trying to guess what
the actual electorate is gonna gonna look like. Okay, so
(13:42):
give me a really quick response to that, and then
I want to get onto something else for our last
few minutes.
Speaker 3 (13:47):
No, you're you're exactly right.
Speaker 4 (13:49):
Polsters having a remarkably hard job.
Speaker 3 (13:52):
And so, as someone who wrote a book entitled the
Polls Weren't Wrong.
Speaker 4 (13:55):
I have to say the fact that they are still
able to get somewhat accurate data out of this nearly
impossible task of contacting a fraction of a fraction of
a populace and only getting responses from an even smaller fraction.
The fact that they are still able to make what
I would consider very very good estimates really gets to
(14:16):
the point that these people, the good polsters, kind of
know what they're doing.
Speaker 3 (14:21):
Yes, they have to make some assumptions. Yes they have
to make you might call them guesses.
Speaker 4 (14:26):
They're informed guesses, but they are, at the end of
the day, guesses. The fact that they've been able to
do so well is pretty remarkable to me.
Speaker 2 (14:35):
Okay, so we only have a few minutes left, and
this may be the most important question, but I think
we needed a lot of foundation there. Way too many
people treat polls as being the same as forecasts. And
people will look at polls that show, you know, Trump
up one percent in such and such a state, or
(14:56):
even betting odds showing Trump sixty percent to win the
and betting on really is a.
Speaker 1 (15:03):
Kind of a forecast. But so let's stick with poles.
Speaker 2 (15:06):
But people will look at a poll as a as
a forecast. And I want you to just explain why
a poll is not the same as a forecast.
Speaker 1 (15:14):
And I realized you just wrote a whole book on it,
and I'm asking you to do this in two minutes.
Speaker 4 (15:19):
Yeah, So interpreting poll data as a prediction of election
outcomes has been around since the beginning of political polling
one hundred years ago Gallup literary digest. Interpreting poll data
as a prediction of the election outcome is not new.
The problem is modern research, which is much more analytical,
(15:41):
much more scientifically based, has maintained those unscientific standards of
if the poll says this candidate is up by let's say,
up by two in the poll, then therefore they must
win the election by two and or the poll was
not accurate and so in order, and it's this very
simple math. In order for that state to be maintained,
(16:04):
two things must be true. One, no one can change
their mind between the poll and the election. Two undecided
must split evenly between these major candidates.
Speaker 3 (16:14):
Well, we can test these things in science.
Speaker 4 (16:16):
When things are testable, we do not use assumptions in
their place. And yet that is no joke, no exaggeration,
no artistic liberties. That is the literal definition used by
the consensus of experts around the world about how to
analyze poll data. They analyze polls, a tool not intended
to be predictive, by how well they predict election result. Again,
(16:40):
polls are an estimate of a base of support. In
the book, I use the term of simultaneous census. If
you take a survey months before the election, maybe it's
an election that not many people are aware of, a Senate.
Speaker 3 (16:52):
A house race, et cetera.
Speaker 4 (16:54):
And it says one candidate is at forty five percent,
the other candidate is at forty percent, and there are
fifteen percent on this. What that means is, if you
had taken a simultaneous census of that population, approximately forty
five percent plus or minus the margin of error would
respond with that candidate. Forty percent plus or minus the
(17:14):
margin of error would respond to that candidate.
Speaker 3 (17:15):
And this is the part that is extremely.
Speaker 4 (17:18):
Important, because I hate to say experts do not understand this.
Fifteen percent wels Er minus the margin of error would
also say they're undecided, period. End of what the poll
tells us. How those undecided to allocate themselves how they decide,
can swing the election. In many swing states, in many
(17:40):
close elections where most polling is.
Speaker 3 (17:41):
Done, this does happen. The assumption that.
Speaker 4 (17:45):
Undecideds must sweit fifty to fifty or else the poll
was wrong is the single most unscientific belief that is
literally held by the consensus of experts in the United States,
and it is wrong.
Speaker 3 (17:59):
It's not true. We can test it, we know it's
not true.
Speaker 4 (18:02):
So my proposal is that instead of using these assumptions,
we must ask the question of how did they decide?
Because they can wing the outcome of the election to
a quote unexpected outcome.
Speaker 2 (18:16):
All right, I have literally one minute left. I want
to check your math on one thing. Let's say there's
a poll that's got it forty five to forty with
fifteen undecided.
Speaker 1 (18:26):
In order for it.
Speaker 2 (18:27):
To end up forty five forty as the actual result
at the end, would it the undecided had to have
broken forty five to the whatever That forty five to
forty ratio is like.
Speaker 1 (18:43):
Not nine out of seventeen to eight out of seventeen.
Speaker 2 (18:48):
Versus having broken fifty to fifty, which is what you
said before.
Speaker 4 (18:52):
They're two different standards for analysis. In the United States,
experts say fifty to fifty. Non US experts stay proportional,
and they're an unspoken truce between these unscientific analysts that
they don't challenge each.
Speaker 3 (19:03):
Other because they're both wrong.
Speaker 1 (19:04):
Uh huh.
Speaker 4 (19:06):
So what they make different, they make different unscientific assumptions. Yeah,
so election results will always add up to one hundred percent.
Speaker 2 (19:13):
We know that.
Speaker 3 (19:14):
I mean, that's that's just very simple, Matt. Uh huh. So,
an election result cannot be forty.
Speaker 4 (19:18):
Five, right, it must They must add up to one
hundred percent, and those that comes from how those undecided
ultimately decide.
Speaker 2 (19:24):
Right. And that's true because forty five forty really, as
I said, if you were to take that as the
whole electorate, then that ad then would be nine out
of seventeen and eight out of seventeen at the end,
which isn't the same as forty and forty out one
hundred and forty five out of one hundred. So all right,
we're just about just about out of out of time here.
(19:44):
I want to follow up on one quick thing you said.
Uh So, I read in your book that not only
should you not assume that undecided voters will break in
a particular way that matches the decided.
Speaker 1 (19:58):
Voters in in percent or anything, but.
Speaker 2 (20:02):
There seems to be a little bit at least of
bias among undecided voters in America that when they do decide,
they tend to vote for for change. I think I
read that like they they side against the incumbents with
some frequency.
Speaker 1 (20:15):
Is that what you wrote?
Speaker 4 (20:17):
That was analysis done by a man named Nick Panagakus
about twenty to thirty years ago, and his work should
have led directly into mine.
Speaker 3 (20:26):
He passed away a few years ago.
Speaker 4 (20:28):
Unfortunately I never got to meet him. His work in
a proper scientific field would have led directly into mind.
But what ended up happening is he passed away. His
work kind of was lost to journals in history, and
these unscientific analysts Silver and Morris and the other guys,
they've taken his place, or they've taken the place of
proper analysis and grown this spread mentality of undecided to
(20:54):
must split evenly. His the best quote, one of the
best quotes that that I read of his, was rules
of analysis are necessary, rules that are not as simple
as eight points is comfortable and two points is a
close race. He wrote that in nineteen eighty seven, this
is not new my work and the analysis, and you
can probably attest to this having read the book. The
(21:15):
graphics that I showed towards the end of the book
kind of back that up. And the fact that he
was able to infer that in the nineteen eighties, and
that work still hasn't been built upon. Speaks to the
problem of this contaminated traditional thinking. The analogy that I
use is baseball analysts who still try.
Speaker 3 (21:37):
To figure out who the best offensive player is by
batting average.
Speaker 4 (21:41):
Batting average isn't the worst metric in the world, but
it is not the best way to measure offensive output.
And that's the same exact way that spread analysis is
currently contaminated.
Speaker 2 (21:52):
Carl Allen's new book is called The Polls Weren't Wrong.
I know there were a lot more listener questions. If
I didn't get to your question, email it to me
at Ross at iHeartMedia and I'll get all the questions.
Speaker 1 (22:02):
Over to Carl and he'll answer them back.
Speaker 2 (22:04):
To me by email, and then I will send your
answer to you. So my email is Ross at iHeartMedia
dot com. Send me your questions. I'll get you answers
Carl's book, The Polls Weren't Wrong. Thanks for being here, Carl,
really fascinating and important conversation.
Speaker 1 (22:18):
Talk to you again soon.
Speaker 3 (22:20):
Thanks again, Ross
The Ross Kaminsky Show News
Popular Podcasts
40s and Free Agents: NFL Draft Season
Daniel Jeremiah of Move the Sticks and Gregg Rosenthal of NFL Daily join forces to break down every team's needs this offseason.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Las Culturistas with Matt Rogers and Bowen Yang
Ding dong! Join your culture consultants, Matt Rogers and Bowen Yang, on an unforgettable journey into the beating heart of CULTURE. Alongside sizzling special guests, they GET INTO the hottest pop-culture moments of the day and the formative cultural experiences that turned them into Culturistas. Produced by the Big Money Players Network and iHeartRadio.