Help me out for school!
Registered User
Joined: May 2006
Posts: 9,409
Likes: 0
From: Calgary
Originally Posted by wizard8100,Aug 14 2008, 05:58 AM
True 
But mostly I have a very low opinion of surveys. Seems that the people who write surveys tend to look for a specific answer, then some other yokel uses his 2nd grade knowledge of statistics to make it look like the survey shows something that it does not.
I once saw two separate surveys in a statistics class to show this principle. At the begining of the class and then two weeks later, the class was given surveys about WW2. After running both surveys through the statistical "grinder," it showed that 90% of the class hated the actions of Adolf Hitler, yet over 75% approved of his actions.
And the real kicker is that neither survey mentioned the name of Hitler or WW2. It was to show how you can misapply statistics and get an incorrect answer.
Don't get me wrong, I love surveys, but Politicians use the same priciple not to see what people want, but to find out how to word their agenda so that the average voter will swallow it.
If people were more aware of exactly what was being asked, and could tell the tone of the survey, we might get people to realise when they are being lied to, and then to vote for better candidates.
Then again, maybe not, but I can always hope.
But mostly I have a very low opinion of surveys. Seems that the people who write surveys tend to look for a specific answer, then some other yokel uses his 2nd grade knowledge of statistics to make it look like the survey shows something that it does not.
I once saw two separate surveys in a statistics class to show this principle. At the begining of the class and then two weeks later, the class was given surveys about WW2. After running both surveys through the statistical "grinder," it showed that 90% of the class hated the actions of Adolf Hitler, yet over 75% approved of his actions.
And the real kicker is that neither survey mentioned the name of Hitler or WW2. It was to show how you can misapply statistics and get an incorrect answer.
Don't get me wrong, I love surveys, but Politicians use the same priciple not to see what people want, but to find out how to word their agenda so that the average voter will swallow it.
If people were more aware of exactly what was being asked, and could tell the tone of the survey, we might get people to realise when they are being lied to, and then to vote for better candidates.
Then again, maybe not, but I can always hope.
I've worked on several anonymous surveys used by companies to determine how their employees feel about various aspects of the company. It took quite a bit of care and objective thinking to ensure we weren't phrasing questions in a leading manner, were covering all important issues, and that the person filling out the survey felt the information they gave was helpful and would be put to use.
People don't like surveys because it often feels like a waste of time. They need to know where the information will be used, and they need to feel confident that the answers they give will result in some sort of positive change with a benefit to them.
It's hard to summarize how to write a survey, but here goes:
1) Be very concise in each question so there is no extra thinking or consideration of "what do they mean by that."
2) Make each question very clear and unable to be interpreted in multiple ways - or suddenly even the dumbest of the respondents will become super intelligent and start thinking of 6 different meanings, sometimes just to be difficult
3) group questions of a similar nature together so the respondent's mind isn't jumping too far from topic to topic.
4) Use verification questions (appropriate distanced from each other) to confirm past answers. Do not repeat the question, phrase in such a way that it should yield the same result as last time, without having the respondent have immediate recollection of the previous.
Creation of a survey is only 1 part of the process. It's what you do with the results that is equally (if not more) important.
There are so many points to consider, I'm not sure why I turned this into an attempt and an education post
BTW I filled out this survey.
Joined: Oct 2006
Posts: 13,494
Likes: 71
From: Chiswick
Originally Posted by wizard8100,Aug 14 2008, 06:58 AM
True
But mostly I have a very low opinion of surveys. Seems that the people who write surveys tend to look for a specific answer, then some other yokel uses his 2nd grade knowledge of statistics to make it look like the survey shows something that it does not.
I once saw two separate surveys in a statistics class to show this principle. At the begining of the class and then two weeks later, the class was given surveys about WW2. After running both surveys through the statistical "grinder," it showed that 90% of the class hated the actions of Adolf Hitler, yet over 75% approved of his actions.
And the real kicker is that neither survey mentioned the name of Hitler or WW2. It was to show how you can misapply statistics and get an incorrect answer.
Don't get me wrong, I love surveys, but Politicians use the same priciple not to see what people want, but to find out how to word their agenda so that the average voter will swallow it.
If people were more aware of exactly what was being asked, and could tell the tone of the survey, we might get people to realise when they are being lied to, and then to vote for better candidates.
Then again, maybe not, but I can always hope.
But mostly I have a very low opinion of surveys. Seems that the people who write surveys tend to look for a specific answer, then some other yokel uses his 2nd grade knowledge of statistics to make it look like the survey shows something that it does not.
I once saw two separate surveys in a statistics class to show this principle. At the begining of the class and then two weeks later, the class was given surveys about WW2. After running both surveys through the statistical "grinder," it showed that 90% of the class hated the actions of Adolf Hitler, yet over 75% approved of his actions.
And the real kicker is that neither survey mentioned the name of Hitler or WW2. It was to show how you can misapply statistics and get an incorrect answer.
Don't get me wrong, I love surveys, but Politicians use the same priciple not to see what people want, but to find out how to word their agenda so that the average voter will swallow it.
If people were more aware of exactly what was being asked, and could tell the tone of the survey, we might get people to realise when they are being lied to, and then to vote for better candidates.
Then again, maybe not, but I can always hope.
Registered User
Joined: Jul 2001
Posts: 6,592
Likes: 0
From: Yorba Linda, CA
I recall an example of survey writing that went something like this:
Two (essentially identical) groups were given a situation where, say, 3,000 people are in peril and asked to make a choice. The first group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will save (with certainty) 1,000 of the people.
The second group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will lose (with certainty) 2,000 of the people.
Although the choices are identical in outcome, the overwhelming majority in group 1 choose Option B, while the overwhelming majority in group 2 chose Option A.
Sometimes, no matter how you phrase the question, you will influence the responses.
Two (essentially identical) groups were given a situation where, say, 3,000 people are in peril and asked to make a choice. The first group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will save (with certainty) 1,000 of the people.
The second group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will lose (with certainty) 2,000 of the people.
Although the choices are identical in outcome, the overwhelming majority in group 1 choose Option B, while the overwhelming majority in group 2 chose Option A.
Sometimes, no matter how you phrase the question, you will influence the responses.
Registered User
Joined: Mar 2006
Posts: 434
Likes: 0
From: Indianapolis, Indiana
Originally Posted by magician,Aug 14 2008, 01:38 PM
I recall an example of survey writing that went something like this:
Two (essentially identical) groups were given a situation where, say, 3,000 people are in peril and asked to make a choice. The first group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will save (with certainty) 1,000 of the people.
The second group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will lose (with certainty) 2,000 of the people.
Although the choices are identical in outcome, the overwhelming majority in group 1 choose Option B, while the overwhelming majority in group 2 chose Option A.
Sometimes, no matter how you phrase the question, you will influence the responses.
Two (essentially identical) groups were given a situation where, say, 3,000 people are in peril and asked to make a choice. The first group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will save (with certainty) 1,000 of the people.
The second group got these choices:
Option A: there's a 1/3 chance that everyone will survive, and a 2/3 chance that everyone will perish.
Option B: you will lose (with certainty) 2,000 of the people.
Although the choices are identical in outcome, the overwhelming majority in group 1 choose Option B, while the overwhelming majority in group 2 chose Option A.
Sometimes, no matter how you phrase the question, you will influence the responses.
"Option B: you will save (with certainty) 1,000 of the people." Does not say that you will save ONLY 1,000 people. The unspoken implication is that you will save AT LEAST 1,000 people. Given that choice and the odds, an intelligent person would choose option B, as the odds are better based on the unspoken assumption implied in the context.
"Option B: you will lose (with certainty) 2,000 of the people." Again, this, based on context, seems to say that you will lose AT LEAST 2,000 people. It does not say you will lose exactly 2,000. Therefore the possibility exists that you could lose more. On this basis, option A would be the logical choice.
This goes to show that the error was NOT on the part of the people who took the survey, but on the people who created the survey. They twisted the language too far, and applied absolute statistics to a question that was a <= and >= situation. In other words, they got their math wrong in trying to twist the semantics of the questions.
Registered User
Joined: Jul 2001
Posts: 6,592
Likes: 0
From: Yorba Linda, CA
Originally Posted by wizard8100,Aug 14 2008, 04:58 PM
Ah....but they are not the same . . . .
[QUOTE=wizard8100,Aug 14 2008, 04:58 PM]"Option B: you will save (with certainty) 1,000 of the people." Does not say that you will save ONLY 1,000 people.
Registered User
Joined: Mar 2006
Posts: 434
Likes: 0
From: Indianapolis, Indiana
Ah, but language is slippery. That is why attention to the survey questions is at least, if not more important that interpreting the answers. A simpler example of what you are trying to demonstrate is any survey about taxes.
You pose one set of questions with hard numbers.
You pose a second set with percentages.
In this case, you ALWAYS get bad results, which, IMHO, is due to the fact that politicians mix and match the two on a regular basis to show what they want to show, not really to show what people think is "fair."
Again, when people think they understand the question, when the question is not clear, or when the people who create the survey think that they have asked a clear question when they have not, the results are essentially meaningless.
That is why I love surveys, but have little respect for them. It is like a game. Figure out how to get my true opinion across without falling into the "trap" of answering the questions the way the underlying marketing firm is trying to lead you to.
You pose one set of questions with hard numbers.
You pose a second set with percentages.
In this case, you ALWAYS get bad results, which, IMHO, is due to the fact that politicians mix and match the two on a regular basis to show what they want to show, not really to show what people think is "fair."
Again, when people think they understand the question, when the question is not clear, or when the people who create the survey think that they have asked a clear question when they have not, the results are essentially meaningless.
That is why I love surveys, but have little respect for them. It is like a game. Figure out how to get my true opinion across without falling into the "trap" of answering the questions the way the underlying marketing firm is trying to lead you to.