Subscribe to my Author Newsletter

Check out my Publisher

Hello everyone and welcome to Episode 12, Season 1 of the autodidactic podcast. If you’re a new listener welcome, and welcome back to returning listeners.

This is episode 12 and we are nearing the end of Season One. We’ll get a couple of more shows in before taking a break for the holiday period. The last show of the season I’ll try to cover off all the listener questions which I’ve had. If you have any questions you want to get on to the prompt sheet for the show make sure to send them in soon. The email address is rick@autodidactic.info.

Today we’re going to talk about evaluation of resources when you’re creating your study plan, or just generally when selecting resources. Not all information is created equal, and not all of it is accurate. First we’ll talk about information generally, then we’ll cover off how to evaluation textbooks and other traditional resources, and then finally we’ll look at Internet resources and how to evaluate them for study purposes.

I’ve drawn heavily from papers created for university students for resource evaluations for essays and other school assignments, but it is all relevant to the self-learner, perhaps even more so. These include:

• Meriam Library, California State University
• Flinders University Australia
• The American Library Association and the Reference and User Services Association
• International Association of University Libraries

What are the Characteristics of Information?

• Good information is that which is used and which creates value.
• Good information is relevant for its purpose, sufficiently accurate for its purpose, complete enough for the problem, reliable and targeted to the right person.
• It is also communicated in time for its purpose, contains the right level of detail and is communicated by an appropriate channel, i.e. one that is understandable to the user.
• Information should be easy to obtain or access.
• Information needs to be accurate enough for the use to which it is going to be put.
  • To obtain information that is 100% accurate is usually unrealistic as it is likely to be too expensive to produce on time.
  • The degree of accuracy depends upon the circumstances.
• Reliability deals with the truth of information or the objectivity with which it is presented.
  • You can only really use information confidently if you are sure of its reliability and objectivity.
  • When researching for an essay in any subject, we might make straight for the library to find a suitable book. We are reasonably confident that the information found in a book, especially one that the library has purchased, is reliable and (in the case of factual information) objective. The book has been written and the author’s name is usually printed for all to see. The publisher should have employed an editor and an expert in the field to edit the book and question any factual doubts they may have. In short, much time and energy goes into publishing a book and for that reason we can be reasonably confident that the information is reliable and objective.
• Information should be relevant to the purpose for which it is required.
• Information should contain all the details required by the user.
• Information should be in a form that is short enough to allow for its examination and use. There should be no extraneous information.
  • For example, it is very common practice to summarise financial data and present this information, both in the form of figures and by using a chart or graph. We would say that the graph is more concise than the tables of figures as there is little or no extraneous information in the graph or chart. Clearly there is a trade-off between level of detail and conciseness.
• The presentation of information is important to the user. Information can be more easily assimilated if it is aesthetically pleasing.
• Information must be on time for the purpose for which it is required. Information received too late or too old will be irrelevant.

If you’re evaluating traditional resources it can be simpler than evaluation of non-traditional resources such as websites. This is because traditional resources have associated workflows. So for example a textbook publisher employs fact-checkers, editors, etc.. They ensure that information published is as accurate as possible at the time of publication or broadcast.

When you’re selecting resources try to select sources use the CARS Checklist. CARS stands for Credibility, Accuracy, Reasonableness, Support. If you learn to use the criteria in this list, you will be much more likely to separate the high quality information from the poor quality information.

Credibility	Trustworthy source, author’s credentials, evidence of quality control, known or respected authority, organizational support. Goal: an authoritative source, a source that supplies some good evidence that allows you to trust it.
Accuracy	Up to date, factual, detailed, exact, comprehensive, audience and purpose reflect intentions of completeness and accuracy. Goal: a source that is correct today (not yesterday), a source that gives the whole truth.
Reasonableness	Fair, balanced, objective, reasoned, no conflict of interest, absence of fallacies or slanted tone. Goal: a source that engages the subject thoughtfully and reasonably, concerned with the truth.
Support	listed sources, contact information, available corroboration, claims supported, documentation supplied. Goal: a source that provides convincing evidence for the claims made, a source you can triangulate (find at least two other sources that support it).

You can also use some of the librarian guidelines for picking resources. Try search for Research Guides, also known as Library Guides, or LibGuides, which are a series of tools librarians create to help their community with searching and common questions. Google for libguides for more information. Scholarly information usually refers to information that you find from your Library’s resources. In general, scholarly works are written by experts in the field and are vetted for accuracy and scientific rigour via accepted scholarly publishing standards such as peer review or editorial processes in the case of books. A level of credibility is assumed when an item is found within the Library. However even if your evidence is sourced from the Library, the quality of the information itself should be assessed critically.

Authority

• Who is the author?
• What else has the author written?
• In which communities and contexts does the author have expertise?
  • Do they represent specific gender, sexual, racial, political, social and/or cultural orientations?
  • Do they privilege some sources of authority over others?
  • Do they have a formal role in a particular institution (e.g. a professor at Oxford)?

Purpose

• Why was this source created?
  • Does it have an economic value for the author or publisher?
  • Is it an educational resource? Persuasive?
    • What (research) questions does it attempt to answer?
    • Does it strive to be objective?
  • Does it fill any other personal, professional, or societal needs?
• Who is the intended audience?
  • Is it for scholars?
  • Is it for a general audience?

Publication & format

• Where was it published?
• Was it published in a scholarly publication, such as an academic journal?
  • Who was the publisher? Was it a university press?
  • Was it formally peer-reviewed?
• Does the publication have a particular editorial position?
  • Is it generally thought to be a conservative or progressive outlet?
  • Is the publication sponsored by any other companies or organizations? Do the sponsors have particular biases?
• Were there any apparent barriers to publication?
  • Was it self-published?
  • Were there outside editors or reviewers?
• Where, geographically, was it originally published, and in what language?

Relevance

• How is it relevant to your studys?
  • Does it analyse the primary sources that you’re researching?
  • Does it cover the authors or individuals that you’re researching, but different primary texts?
  • Can you apply the authors’ frameworks of analysis to your own research?
• What is the scope of coverage?
  • Is it a general overview or an in-depth analysis?
  • Does the scope match your own information needs?
  • Is the time period and geographic region relevant to your research?

Date of Publication

• When was the source first published?
• What version or edition of the source are you consulting?
  • Are there differences in editions, such as new introductions or footnotes?
  • If the publication is online, when was it last updated?
• What has changed in your field of study since the publication date?
• Are there any published reviews, responses or rebuttals?

Documentation

• Did they cite their sources?
  • If not, do you have any other means to verify the reliability of their claims?
• Who do they cite?
  • Is the author affiliated with any of the authors they’re citing?
  • Are the cited authors part of a particular academic movement or school of thought?
• Look closely at the quotations and paraphrases from other sources:
  • Did they appropriately represent the context of their cited sources?
  • Did they ignore any important elements from their cited sources?
  • Are they cherry-picking facts to support their own arguments?
  • Did they appropriately cite ideas that were not their own?

If you are thinking about using a printed book, Google the book title with the word “review” appended and read what others think of the book.

After you find some resources, enter them into Google and append one of the following words or phrases: controversy, dispute, disagreement, alternate views, debate, arguments for and against. This will help you broaden the scope of your resources and information.

The facts we learn today may be timely now, but tomorrow will not be. Especially in technology, science, medicine, business, try to get the latest information where possible.

Next we come to evaluation of resources from the Internet. For this I recommend using a system developed by the Meriam Library, California State University (2010). It is called CRAAP which stands for Currency, Relevance, Authority, Accuracy, and Purpose.

Evaluate	What to look for in Web sites
Currency	Does the paper/assignment require the most current information, historical information, or information over a period of time? When was the Web site published or created? (look for a copyright date on the homepage) When was the site last updated or revised? Are the links up to date?
Authority	Who is supplying the information? Is it an educational institution (.edu extension)? A government agency (.gov)? A commercial supplier (.com)? A non-profit organization (.org)? Is the supplier a reputable organization? (look for an “About Us” link on the homepage) Is there an author or contact person named? What are the author’s credentials (see “What to look for in books and periodicals”)? Has this site been reviewed by experts or professional organizations?
Validity/Accuracy	Are sources of information cited? Compared to other sources, is the information complete and accurate? Are the links also complete and accurate, or are there discrepancies? Is selection criteria provided for the links found in the Web site? Does the site appear to be carefully edited, or are there typographical errors?
Audience	Is the site appropriate for your needs, or is it too technical or too elementary, or too full of jargon? Who is the intended audience? Experts or the general public?
Point of view (bias)	Does the information appear to be filtered or is it free from bias? Could the organization sponsoring the site have a stake in how the information is presented? Is the site free of advertisements? Are various points of view, theories, techniques, or schools of thought offered?
Purpose/context	What is the purpose of the site or article? Is it to share new, scholarly research? is it to report developments in an evolving news story? Or is it to rant about a government conspiracy? How closely does the web site relate to the purpose for which you need that information?

Many sites on the Internet have legitimate useful information. But it also has a lot of information which is in accurate, but often repeated, sometimes virally.

Appearances can be deceiving. Don’t assume that a great-looking Web site is automatically credible. Very professional and sophisticated Web page templates are available for a few dollars, so that anyone can put up a site that looks expensive and authoritative.

Wikipedia can be a great place to start to get an understanding of a topic. It may also lead you to relevant, high quality resources. Try looking at the references of a Wikipedia page (at the bottom of each Wikipedia page) and assess the quality of the references you find. Many Wikipedia entries will cite scholarly resources (including books and journal articles) in their references.

We’ve covered a couple of acronyms CARS & CRAAP, but you might want to remember just AAOCC (Authority, Accuracy, Objectivity, Currency, and Coverage). The same basic questions should be asked of all information sources: books, journal articles, web pages, blogs, videos, sound recordings and e-books.

These questions will help you identify good resources. Bad resources can be identified relatively easily. If it makes a lot of value statements: He is the worlds best player vs He won 5 consecutive world championships. If it references a lot of vague or unnamed groups or people. It has few or no references to other publications.

However, remember that to locate fair, objective material, you must be fair and objective, too. A major error that too many researchers make is to look only for sources whose ideas, findings, or arguments they already agree with. Confirmation bias.

There is no single perfect indicator of reliability, truthfulness, or value. Instead, you must make an inference from a collection of clues or indicators, based on the use you plan to make of your resource.

Hopefully this has given you a good idea of how to evaluate resources for your study. I’d be interested in hearing your evaluation of this podcast as a resource. Thank you for listening.

Next week the topic will be getting certifications or qualifications in the topics which you’ve self-studied and how an autodidactic can gain qualifications.

See you next week.