Posted by: James P Burke | September 6, 2009

That’s Not Research

I eagerly await part 2 of Derek’s rant on Social Media.

It probably bears mentioning that my opinion on social media is probably slightly more favorable than my colleagues, although not much more. I’d classify it thusly: I appreciate the uses I have personally had for it and I admire efforts of people I know to actually use social media to empower people.

However, I believe that people are jumping to some unsupported conclusions and they worsen the situation if they claim they have science to back up their assertions when they have nothing of the sort.

It is refreshing to see some criticism of social media. That’s one other thing that is lacking. Enthusiasm for the new, in and of itself, is not a virtue. That is especially true when a thing already has as many adopters as Twitter, Facebook or blogs have.

Around any new concept you’ll always get two camps: those that hate it as a reaction to possible change and those who love it because they love the new. Both camps grow as adoption grows, but neither have anything useful to say. A lot of social media boosterism is of the latter-camp type, and a lot of the criticism is of the type best described as “kids, get off my lawn!”

I want to see both more substantive support and criticism for the uses of social media. And right now, the hype for social media makes me wish that the criticisms were less bewildered and more substantive.

On a related note, there is also the notion of science and research at stake here. When you use the word science to describe any old question and answer, you dilute the notion to the point where it is meaningless and useless. It is up to scientists to tell you to stop.

I thought you might be interested in this passage from Wayne K. Hoy’s Quantitative Research in Education: A Primer (pp. 17-18) discussing a type of so-called research (Descriptive Research) that really isn’t. It seems applicable to this and future discussions.

Descriptive Research

Descriptive research is another phrase that is used to characterize inquiry in the social sciences. It is a term that I typically avoid because it has several connotations, one of which is not really research. Research always should involve examining relations between at least two variables. Yet many “researchers” simply use statistics to describe the characteristics of various groups of individuals. For example, they ask what the proportions are of males and females in a given group of teachers, the average age and experience of the group, the number who have BA, MA, and PhD degrees, the percentage of teachers who drop out of teaching after one year, the average level of experience before teachers become principals, and on and on. The answers to such queries may be quite useful, but such compilations are not examples of descriptive research because they do not relate variables–a more appropriate term for the process is social book keeping, not research.

Descriptive research is the process of simply describing relations without speculating about the cause. The two variables go together; they are correlated. For example, weight and height are correlated, but gaining weight does nor cause you to grow taller. Likewise, growing taller does not necessarily lead to growing heavier. Correlational studies give us interesting information about relations, but until we can begin to make causal inferences, their utility is limited. Careful description is important, but it is only the beginning of the study of causal relations.



  1. I love that quote, by the way, I’m going to print that and put it all over the place.

    Part 2 is centered more around what you’ve described with Descriptive Research, in Social Media. Now, I know I’m being pushy and whiny about Social Media and how much it sucks: “Both you kids and you jerks get off my lawn”, but I’ll bring it back around to something more like a critique than a criticism. In the meantime, there is more and more support at just how ridiculous and fad-like Twitter is, and I’ll integrate this in, too:

    NPR’s On the Media did a piece today with some folks who have done more real studies on Twitter, showing that Twitterers are basically Birthers – it’s really just a small group of people that yell real loud. And that kind of makes sense.

    What I found surprising, is what (I believe) Clive Thompson said: “…Twitter is used by an extraordinarily small sliver of society. 1 in 10 adults use it, and it’s mostly well educated men, by and large, when you look at the statistics. And almost no young people, at all… certainly not college age.”

    This blew my mind. Now, he could be doing descriptive research, or correlational or inferential stats – I don’t know yet. But that quote is intriguing, and quite literally supports my description of Twitter being a circle jerk, even more. I’ll get to that full analogy later…

  2. At this point there aren’t any scientific studies of Twitter (that I know about.) The social book keeping is all we have, and it doesn’t take us very far. And it’s not science.

    I enjoy reading your criticisms; this is a blog, not a research paper. A lot of what we post here is going to be blowing off steam. So, nobody should they need to hold back much.

    Release the hounds!

  3. love the quote. it points out our exact criticisms of our two dearest SM/PR friends’ methods:

    1) give a survey
    2) make some pretty graphs
    3) science just happened
    4) where’s my money?

  4. […] What kind of data is this, though? Marketing. Also known as “Descriptive Research”, which is nothing more than descriptive stats – which when used alone, are relatively meaningless. Perfectly, and eloquently supported by James’s recent post, “That’s Not Research“. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: