1- Survey

Why Should I Care?

Surveys are very common. However, many are not done well. There are many traps most people don’t know about that reduce their scientific validity.

Definitions

Survey:
the act of measuring objects or phenomena.

Social survey:  
a research technique that observes a sample of individuals by asking questions and analyzing the responses.

Questionnaire:
an instrument made of a written set of questions, mostly made up of closed-ended questions, and organized in a sequence appropriate to the purpose of a survey, or psychological test.

Interview:
an instrument made of a loose set of questions, mostly designed to allow for an open-ended conversation.

Usefulness of Surveys

Surveys allow for a “real-time” expression of opinions and attitudes on a particular topic. It might only take less than 6 days to conduct a telephone survey. This is very useful for marketers and politicians who might want to change their strategy, or tactics, quickly.

For political scientists, sociologists, and economists, survey results allow for a snap-shot of public opinion at different points in time. This allows to see shifts in opinion, or the impact of specific events on the views. Surveys can also allow for cross-sectional analysis since they often measure socio-demographic variables, such as gender, or mother-tongue language, as well as political opinions, or market preferences.

“Surveys” are often used by corporations as a commercial ploy to engage a sales pitch, or to build an email mailing list for future advertising.

Video - Basic Steps to do Surveys

Survey methodology is controversial because it is an unregulated industry, and many surveys are not conducted following strict scientific methods. Survey firms are private for-profit companies, and provide their services to both private sector industry, and also to political parties. Many surveys are commissioned and paid-for by lobbies and interested parties. The results - if they were to contradict the interests of the client - are not usually published. In fact, most surveys are never published publically, since they are commissioned by private corporations for their internal needs in terms of strategy, marketing and advertising.

The following documentary was prepared by the CBC in 1997, at a time when the media were publishing too many poor quality surveys, as social facts.

Watch - CBC Documentary

Ask A Silly Question

CBC Documentary - 1997
https://curio.ca/en/catalog/a4b8ee23-1dbe-41e6-ad06-0ab78dc40812

Discussion Questions

1. Identify a surprising element of the documentary. What did you find surprising?
2. In the case of the negative advertising towards Jean Chrétien, what question did Marzollini ask in his survey, that Gregg had forgotten? How did this affect both parties strategy?
3. What can the survey industry do to improve the quality of political and public-interest surveys?

Objects of Measurement

Surveys are useful for measuring many variables, but they are not appropriate for the whole array of variables that social scientists may be interested in. Typically, surveys are excellent for measuring opinions, ideologies, and preferences. Surveys can also be useful for measuring personal and socio-demographic characteristics such as someone's age, religion, and gender.

Surveys are also take on various shapes and forms. They are used widely by governments to gather information, through the use of bureaucratic forms. For example, when taxpayers fill their annual income declaration, their earnings for the year are included in national economic accounts. It's thanks to these kinds of surveys that we can measure variables like Gross Domestic Product, and Employment Occupations. School tests can also be considered a type of survey, which is measuring the level of knowledge, or competencies, of a group of students.

This said, surveys are not adequate for measuring aptitudes, abilities, behaviors, actions, and performances. Respondents are prone to lie, to exaggerate, or simply to misinterpret questions. For example, if you ask someone if they can touch their toes, they might say yes. However, if you ask someone to demonstrate touching their toes, they may not be able to do it. In this case, you are no longer using the survey method, but rather using direct observation in what is called Field Work.

Variable Table for Surveys

Here is summary table of the variables you might want to research in social science, and how they would apply to a survey method.

Type of Object	Yes	No	Maybe	Example
Personal Characteristic	X			Eye colour, Height, Accent
Socio-Demographic Characteristic	X			Religion, Language, Income, Gender, Ethnicity, Marital Status
Opinion	X			I don’t like Rice Krispies Cereal
Motivations			X	Fear, Hope, Ambition, Greed
Ideology	X			Right-Wing, Left-Wing
Biases / Prejudice	X		X	I trust tall people more than short people
Preferences	X		X	Commercial example: pricing, marketing
Personal History	X		X	Immigration, Settler, Indigenous, Divorce
Family Dynamics	X		X	Number of siblings, Rank in Family, Class
Cultural History	X		X	Military History, Political History, Class
Perception / Self-Perception	X		X	Other people are smarter than me
Aptitude / Ability		X		Can hit 3-point shot. Can manipulate others
Behavior / Actions		X		Domestic Violence. Narcissist. Kind.
Personality	X			Agreeable, Irritable, Outgoing
Moods / Emotions	X			Sad, Happy
Performance			X	Productivity, Quality
Level of Knowledge	X			School test. Diagnostic quiz.

Types of Surveys

There are essentially two kinds of surveys, cross-sectional, which are a snap-shot of variables at a point in time, and longitudinal surveys, which are repeated over several periods to observe a trend.

Cross-sectional surveys allow the analyst to compare many independent variables to a dependent variable. This allows to establish association, between variables, and work towards the establishment of causation. For example, adding variables like gender, and age, to a survey on political preferences, allows one to produce a cross-section of the results for gender and opinion, and then another cross-section for age and opinion. Each pairing is called a cross-section of the results.

Longitudinal surveys are meant to study a phenomenon over time. The researcher will conduct many surveys at appropriate intervals, to be able to measure changes in attitudes or opinions. If each survey is using a different sample of the population, we call this a trend study. If each survey is using the same sample of the population, where the same people are being surveyed several times, we call this a panel study.

Cross-sectional:
compare many independent variables to a dependent variable

Longitudinal:
compare a few variables over time

• • Trend: similar samples taken at different time points
  • Panel: same sample followed through time

Instruments

There are two types of instruments used to produce social surveys. The first is called the questionnaire, which is a formal question list, with mostly closed-ended questions. It is meant to be used on larger samples, going up to 6,000 respondents. Questionnaires can be executed in person, over the phone, on the internet, or in a group meeting. Thanks to questionnaires, analysts can collate statistical frequencies of the variables measured in the survey.

The second is called the interview, which is an informal question list, with mostly open-ended questions. It is meant to be used on smaller samples, usually less than a dozen respondents. The format of the interview depends on the method and the discipline. In Field Work, interviews are done in the field and may take on many formats. However, in the case of social surveys, interviews are usually conducted as a sit-down face-to-face conversation and are usually recorded for accuracy. Many social surveys are done by political scientists, and sociologists, and serve to allow an expert to fully explain their views on a topic. Interviews allow for more detail. However, because the question list can vary, they are not well suited for statistical analysis of the results.

1. Questionnaire: set question list, closed-ended questions, larger sample
   • In-person
   • Telephone
   • Internet
   • Group
2. Interview: starting question list, open-ended questions, smaller sample
   • Field interview
   • Formal Face-to-face interview

Scientific Power

Social surveys are mostly descriptive studies, which focus on socio-demographic characteristics, and opinions. Surveys can be used for explanatory studies which validate hypotheses and their causal relationships (why), using cross-sectional tables. However, you have to keep in mind that you need to establish temporal order, and eliminate other factors, to argue causation.

Surveys are not suitable for exploratory work because they usually are designed once the topic is relatively known and understood.

Sampling

The sampling techniques used in surveys depend on the type of instruments chosen. Hopefully, your sample is large and random if you are using a questionnaire. This is possible using a list of the population such as a local phone book, which is difficult to achieve with email and internet surveys. In this case you can easily have a quota sample.

With interviews, samples are non-random and rather small. Sampling technique is usually purposive, convenient, or snowball.

References and Further Reading

Del Balso, M. & Lewis, A. D. (2012). First Steps, A Guide to Social Research, Fifth Edition. Nelson.

Neuman, Laurence W. & Robson, K. (2015). Basics of Social Research, Qualitative and Quantitative Approaches, Third Canadian Edition. Pearson.

Sheppard, V. (2020). Research Methods for the Social Sciences: An Introduction. BCcampus.