The ability to critically evaluate information is an essential skill for postgraduate researchers. This skill is particularly pertinent to the production of literature reviews, where a critical appraisal or analysis of the literature is required.

In this section, we suggest using the ‘PROMPT’ system (Provenance, Relevance, Objectivity, Method, Presentation, Timeliness), which is a structured approach to the critical evaluation of information.

Provenance

The provenance of a piece of information (i.e. who produced it? Where did it come from?) may provide a useful clue to its reliability. It represents the ‘credentials’ of a piece of information that supports its status and perceived value. It is, therefore, very important to be able to identify the author, the sponsoring body, or the source of your information.

Factors to consider about authors:

• Are they acknowledged experts in the subject area?
• Are they writing for respected and reliable publications?
• Are their views controversial?
• Have other authors in the field frequently cited them? (Answering this requires either prior knowledge or a citation search)
• Are they known to have a particular perspective on the topic?

Factors to consider regarding sponsoring organisations:

• What type of organisation is it? Is it a commercial company, a voluntary organisation, a statutory body, or a research organisation?
• How well established is the organisation?
• Does the organisation have any vested interests in the subject area being researched?

Factors to consider about the method of publication:

• Any individual can publish anything on the World Wide Web or post to a discussion list. This has to be judged on merit and with reference to the author’s credentials.
• What do you know of the editor and/or the editorial board and how does their editorial policy influence what will be published?
• Is the journal well regarded? Does it have a high rating in the Journal Citation Reports? Does this matter?
• Is the information peer reviewed? Many electronic journals do not have a peer review process.

The provenance of a piece of information is not a direct clue to its quality. There is something called the ‘stable theory’, which suggests that academic work is often valued highly just because it emanates from a prestigious research group or is published in a prestigious journal. So we should judge information on its own merits. However, provenance can be an indirect clue to the reliability of information – a safety net that gives you the opportunity to check things out. Provenance can affect other people’s confidence in the sources you are citing.

Relevance

Relevance is an important aspect of information quality. It is not a property of the information itself, but rather of its relationship to the need you have identified. It may be a piece of high-quality information but not relevant to the question you are asking or the scope of your search. There are a number of ways in which the information may or may not be relevant to your needs.

Geographical (It may relate to countries or areas which you are not interested in).

Level (It may be too detailed/specialised or too general/simple for the level at which you are working).

Emphasis (It may not contain the kind of information you are seeking – this is often a question of emphasis, which may not be identifiable from the abstract).

A tip for determining relevance is to:

• Be clear about your requirements – this will help you to be ruthless in discarding things on the basis of relevance.
• Try to avoid having to read things in full – look at the title, abstract or summary, keywords and descriptors. If you are evaluating a large body of material, learn to skim read and/or scan information to get a quick indication of what it is about. For more details on reading techniques see the Effective reading (link) site from Deakin University.
• Consider research in context. Do the research results provide a unique insight into an aspect of your subject? Do they confirm or refute the findings of other researchers?

Objectivity

In an ideal world, ‘objective’ or ‘balanced’ information would present all the evidence and all the arguments, and leave you to weigh this up and draw conclusions. In the real world, however, we recognise that all information is presented from a position of interest, although this may not necessarily be intentional. Objectivity, therefore, may be an unachievable ideal.

This means that the onus is on you, the reader, to develop a critical awareness of the positions represented in what you read, and to take account of this when you interpret the information. It is also important to recognise that your own belief systems and opinions will influence your ability to be dispassionate and objectively evaluate information.

In some cases, authors may be explicitly expressing a particular viewpoint – this is perfectly valid as long as they are explicit about the perspective they represent. Hidden bias or errors of omission, whether or not it is deliberate, can be misleading. Consider the following:

• Perspectives: do the authors state clearly the viewpoint they are taking?
• Opinions: academic articles will often present unsubstantiated theories for debate. Look out for opinion presented as if it were fact.
• Language: can be a useful danger sign. Look out for language that is either emotionally charged or vague.
• Sponsorship: whether commercial, political or personal. For example, industries or governments may sponsor academic research. This does not necessarily make the research less objective but it may make its interpretation selective. Make sure that all potential vested interests are clearly identified and that the sponsors are happy to give access to the actual research data.

When producing a literature review there is a particular onus on you to recognise any selective interpretation of data. You will need to comment on any significant omissions or biases that you may encounter in other people’s findings.

Method

For this aspect of PROMPT, we do not refer to the evaluation of research methods per se, but to the information produced as a result of using particular methods. With your knowledge of the methods used in your subject area think about the following.

• Is it clear how the research was carried out?
• Were the methods appropriate?
• Ask some basic questions about sample size, the use of control groups, and questionnaire design.
• Are the results produced consistent with the methods stated?
• Are the methods suitable for your needs? Do you need the methods to be the same as yours or different to yours?

Do not assume that because a research report has been accepted for publication, it is error-free and meets a certain standard. There have been cases of fraudulent research that have successfully fooled the research establishment and which have been published in high profile journals.

Presentation

The way in which information is presented has a profound effect on the way we receive and perceive it. There are many aspects of presentation, any of which, if badly applied, can create a barrier between the message and the audience.

For example:

• Choice of font type and size
• Use of diagrams and images
• Lack of or illogical structure or layout
• Poor use of language and writing style

Be aware that poor presentation and inappropriate or confusing use of language will hinder your ability to critically evaluate the academic content. Try not to let poor presentation stop you from using what might otherwise be good quality, relevant information.

Timeliness

The date when information was produced or published can be an important aspect of quality. This is not quite as simple as saying that ‘good’ information has to be up to date; it depends on your information need.

Three factors to consider include:

1. Is it clear when the information was produced?
2. Does the date of the information meet my requirements?
3. Is it obsolete? (Has it been superseded?)