A better protocol, at last?

A better protocol, at last?

by Mark Tidwell

This is the second of two commentaries on papers published by T.J. Robertson (TJR) and Archie E. Roy (AER) in the Journal of the Society for Psychical Research:

A Double-Blind Procedure for Assessing the Relevance of a Medium’s Statements to a Recipient. Robertson and Roy, JSPR Vol 65.3 No 864 July 2001 pg 161 – 174

Introduction

In a previous paper, the authors tested the hypothesis that “Statements made by mediums to recipients are so general that they could as readily be accepted by non-recipients.” A statistical examination of statements made by mediums revealed a huge gap between the statements that applied to recipients over the non-recipients. The authors concluded that they had falsified the hypothesis. The authors did however acknowledge several “normal” factors (as opposed to paranormal factors) that they considered as possible contributors to this gap. These factors consist of:

  1. A “different attitude” between the recipient and the non-recipient.
  2. A different cultural background.
  3. Deduction of information from recipient’s appearance.
  4. Deduction of information from recipient’s body language initially and/or verbal responses during the proffering of statements by the medium.
  5. Deliberate cheating on the part of the medium.

The authors disregard the fifth factor based on their knowledge of the trustworthiness of the mediums, but do offer new controls to address the remaining four factors. (Note: Recipients are those to whom a medium has addressed a number of ostensibly relevant statements. Non-recipients are any of those to whom the statements have not been addressed.)

A new design, they suggest, could eliminate these factors by designing a double-blind experiment which included the following objectives:

  1. The medium and recipient are hidden from each other’s view.
  2. The recipient does not speak to the medium.
  3. The recipient does not know he is the recipient.
  4. The medium cannot identify the recipient in any normal way.
  5. Each of the other participants is unaware of whether he is the recipient or a non-recipient.
  6. Neither of the two investigators chooses the recipient.

A summary of the procedure

Labeled chairs are arranged in rows numbered **n. One of the seats numbers is chosen at random and placed into a sealed envelope by investigator A. No one else sees this number. The medium is escorted into the room by investigator B and is seated behind a screen with his back to the chairs. This prevents the medium from seeing the seats. A number of participants are brought into the room and take seats by a random process that does not allow seating assignments to be chosen by the sitters or the investigators. Investigator A observes which person takes the seat matching the number in the sealed envelope. That person is the recipient. Investigator B, who does not know the identity of the recipient, informs the medium that the recipient has been selected. The reading can now begin.

While the medium is speaking, Investigator B records the statements made, which are written as a list on a standardized form. When the reading is finished, Investigator B makes n number of copies and distributes them to the participants. The forms are labeled by the number of the chairs and by the names of the recipients, the date and the place. The participants read the statements carefully and tick off the statements that they feel are relevant to their life. When finished, the participants total the number of ticks at the bottom of the page. The forms are collected and the data undergoes standard reduction weighting as described previously (1). Once the analysis is completed, Investigator A reveals the number of the recipient for further processing.

Some of the advantages of using this procedure:

  • The medium cannot see or hear the participants, nor does he know the identity of the recipient.
  • None of the participants know if they are recipients or non-recipients.
  • All scoring is done at the same time under the same circumstances.
  • The determination of the recipient is completely random.
  • Only investigator A knows the identity of the recipient. This is not recorded on any of the forms.
  • Investigator A does not process the data.
  • Investigator B does not know the identity of the recipient until the analysis is complete.
  • Because of the knowledge restrictions, neither investigator can influence the process.

Initial comments

This is, in my opinion, an excellent experimental design. I believe the authors have introduced superior controls against cold-reading as well as against experimenter and rater-bias. They have also added stringent procedures to insure that the selection of the recipient is random. It is, overall, a measurable improvement over the previous study. I do have a few minor concerns however:

  1. Record-keeping
    As in the last paper, the authors here would record the mediums’ statements by hand. This is a dangerous precedent in my opinion, as it opens the opportunity for the suspicion of experimental error. By simply adding a tape-recorder to the process, the authors can assure fidelity of reporting and provide original records for future investigators to analyze. This may slow down the process as the recording will have to be transcribed, but surely accuracy is a more important consideration.

  2. The participants can hear the medium
    While the participants are waiting, the medium is giving his reading aloud to be recorded. As the participants are in the same room, they may be able to hear the reading in process. This may be undesirably influential for several reasons. If the medium is a famous one or if their voice is otherwise easily recognized, the mediums’ “celebrity status” may confer a degree of credibility that might otherwise not be an issue. Even the voice of an unknown medium may convey an emotional impact if it seems especially confident or sensitive.

  3. The participants can see each other
    During the course of the reading, each sitter may already be analyzing and comparing it to their own experiences. This in itself may lead to second-guessing. Furthermore, this allows the opportunity for the recipients to judge the reactions of others as well. For example, a sitter might notice that the man next to her is crying, and may conclude that if his reaction is so strong then the reading might be for him instead of her. This could influence her scoring later on.

  4. What applies?
    As before, the recipients and non-recipients both are asked to tick off the statements that they feel “apply” to them. I won’t repeat the whole of my previous objection, except to point out that a statement may “apply” to many people in many situations and still not be entirely accurate. This potentially subjective influence is the basis for all analysis that comes after, and may color the results.

Some proposed changes

A solution to the record-keeping issue has already been suggested. Concerns two and three can also be addressed quite easily by simply separating the medium from the participants. Instead of keeping them in the same room as the medium, they can be sequestered in a different one, either as a group or each in their own room. This would prevent the medium from having any type of normal sensory connection to the participants, (a preferred “situation” that was proposed in the previous paper). The recipient may still be chosen by the same random “numbered chair” system. Security can be maintained by leaving an experimenter with the group or by the use of video cameras. This has the added advantage of allowing several mediums to be tested simultaneously.

One possible way to improve the scoring procedure may have the added benefit of reducing the complexity of the statistical analysis. The mediums’ statements can be converted into simple questions that can only be answered by a yes or a no, or left in statement form to be answered True or False. Binary scoring has the benefit of removing the ambiguous nature of determining what “applies” as well as eliminating the need for a cumbersome weighting process. Some statements of course, may still not seem crystal-clear to the scorer. Should a statement be considered unanswerable, it would not be scored. The nature of binary scoring permits all statements to be weighed equally.

In a design such as this, the addition of a cold-reader as a control would have little benefit. Cold-readers depend primarily upon feedback. When the possibility of feedback is eliminated completely, as it is in this protocol, then there is no value in comparing a cold-readers score to that of a medium.

I believe the proposed design offered by Robertson and Roy is far and away superior to any that I have seen before. If the protocol is strictly adhered to, then there will be little room for dispute when the data is offered, in my opinion. It is my understanding that the team is currently undertaking a study based on this design. I look forward to seeing the results.

The commentary of the A Preliminary Study of the Acceptance by Non-Recipients of Mediums’ Statements to Recipients by Robertson and Roy can be found here.