Who Can You Trust?: Source Evaluation

Thanks to some conversations recently, I’ve been thinking about evaluating sources. Obviously, this is something I’ve taught for years in the classroom setting, but I think it is even more important to consider in a wider context given the growing propensity for people to link articles in online discussions (arguments, rants, whatever) as evidence (or “evidence”).

So, why should we evaluate sources? Why not just go grab the first online articles that seem to agree with our positions?

Several reasons, really, but the most important is: just because a source agrees with your position, does not mean that it is a good source.

Again, so what?

To best convince an open minded audience, we need to present the best possible evidence. Part of doing so is presenting evidence from solid, reputable, credible sources.

So, how do we determine a source’s credibility?

Just because it’s written (or on the interwebs), this does not mean that it is credible or reputable. Honestly, anyone can write anything. And these days it’s easier than ever before for anyone to publish whatever the heck they want to, whether it’s true or a drug induced mental rambling. This is one reason that scholars and other professional researchers are often skittish about internet and self-published materials.

Here are some useful criteria, gleaned and boiled down from a variety of collections of criteria found in dozens of writing and research handbooks (this is by no means a complete discussion):

1) Authorial Reputation and Associations

As we know, most writers and speakers have reputations for various things and everyone is associated with something. For example, we know that Mike Huckabee is an evangelical conservative and is associated with Fox News while Al Gore is an outspoken environmentalist. These reputations create certain expectations. They can also help us determine whether the individual is typically trustworthy, approaches multiple perspectives fairly, or not. A recent example I saw was an article written about problems with the Canadian health care system, written by a person who works for a public policy group. One of that ppg’s primary goals is to dismantle the Canadian health care system, so the author’s reliability is suspect.

2) Publisher/Host Reputation

Just as authorial reputations influence whether we can call a source reliable, so does the reputation of the publisher (or web-host). And this varies from topic to topic. For instance, we can consider the Harlequin publishing group an excellent source for the romance genre of novels; however, they would certainly not be a reputable publisher for sources on nuclear physics. On the other hand, the Oxford University Press has spent over 600 years building a strong reputation for solid publications in a range of fields.

3) Obvious Bias

Does the source make clear its bias without attempting objectivity? Obviously, a purely objective position is impossible to achieve, however any good piece of journalism, scholarship, policy, etc. should at least attempt to be as objective in tone as possible. For example, Ludwig Von Mises’s Socialism: An Economic and Sociological Analysis is a fairly obvious propaganda piece that does nothing to appear objective—one of the first lines is even a logical fallacy (either/or fallacy).

4) Whom Do They Reference?

This is important with all sources, really, although it is more obvious with print. Most print non-fiction will include a list of source material consulted by the author. A read through that list should bring up some recognizable names within the field the source is in. For example, in a source of medieval magic, I would expect to see Carlo Ginzburg’s name come up; for medieval childhood, Barbara Hanawalt should be in there somewhere; etc. Likewise, virtually every article online references other articles, interviews, or other sources. These sources of information for the writer are important to look at, both for veracity of data and credibility of the original source.

5) Do They Look at Multiple Angles or Perspectives?

One way to present objectivity, and build the author’s credibility, is to look, honestly, at multiple perspectives on an issue or topic. By honestly, I mean by clearly and truthfully presenting the other positions without resorting to stereotypes, ad hominem, or misrepresentation of the arguments. This lends both credibility and objectivity in that it shows the author looked at the topic from a number of directions and chose one in particular as best, or synthesized one of out many, rather than simply starting with a single premise, putting on blinders, and ignoring any other ways of looking at the topic.

6) What’s the Context?

No material is created in a vacuum. Everything ever written is produced within a cultural-historical moment that influences it, or even acts as a catalyst for its creation. For instance, MLK’s historic “I Have a Dream” speech was written in 1963. If the year had been 1943, the speech would have been very different, likewise if it had been 1983. But, that doesn’t tell us about credibility. Because culture influences thought and source creation, cultural biases and issues come into play as well. For example, it is difficult to trust an anti-socialism source from the early-1950s in the U.S., because of the effect of McCarthyism (e.g. a pro-socialist book would never get published, the author would be blacklisted, and the author would potentially be deported or face imprisonment as a Soviet spy). So, even if the author thought socialism was the greatest thing since sliced bread, (s)he would write that it is evil as a means of self-preservation.

7) Can the Information Be Confirmed by Unrelated Sources?

One of the most important, and potentially difficult, measures of source reliability is confirmation. That is, can the claims be confirmed with reference to other sources. The difficulty is that the other sources must be unrelated to the first and must reference different primary sources (studies, interviews, etc). For example, recently someone in a discussion posted links to three articles—one from a conservative website, one from a left leaning site, and one from CBS—that all said the same thing. On the surface, this is good. However, all three cited the exact same study, from the exact same conservative public policy group (in fact, the second one, on the left leaning site, was written by a senior fellow of said group). So, there was no credibility in terms of data confirmation, since they all used the same primary data source.

8) Has the Work Been Peer Reviewed?

For print, and eprint, works, particularly those from scholarly writers, the question of peer review arises. Peer review (double blind) basically means that two people, experts in the same field, read the document and decide: print as is, print with modification, or do not print. The author gets feedback from the reviewers, modifies as needed, and the piece gets printed. This is essentially quality control for the written word. It keeps publishers and journals from printing materials written by individuals who have no knowledge or expertise in the appropriate field(s), usually. Versus non-reviewed sources that literally anyone can publish.

9) For Online Sources, Who’s the Sponsor?

With online sources, it can be difficult to determine who produced the material or what the goal of the organization hosting it is. So, we can also look for the sponsor of the website, e.g. who pays the bills. Then we apply the same criteria to the sponsor as to the publisher or author.

10) For Online Sources, What’s the URL Suffix?

I know it has been said before as well, but the URL suffixes can be helpful in determining web source reliability. As we know, the most common is .com (commercial site; e.g. they’re selling something), probably not the best option. After that, .org (non-profit organization), which will likely have obvious biases that need to be accounted for. Perhaps the most reliable are .gov (government), .edu (American academic), and .ac (academic network; non-U.S.), as they generally undergo some standards for posting and review of materials.