Situations Vacant

Yellow Ribbon response to IPRC

REPORT ON DRAFT CASE STUDY CONDUCTED BY THE INJURY PREVENTION RESEARCH CENTRE AUCKLAND UNIVERSITY

Yellow Ribbon is an initiative of the Youth Suicide Awareness Trust. The Trust was established in 1998 with the objective of raising awareness of the shocking incidence of youth suicide in the 15 – 24 age group. In 1998, statistics showed three people in this age group were taking their lives each week. Of these, 80 percent were male. The statistics also showed more people died by suicide than were killed on New Zealand roads that year.

A pilot of the Yellow Ribbon peer support programme, “It’s OK to ask for Help”, was introduced to MacLean’s College in Howick, Auckland, in October 1999. Since then it has spread to cover much of the country. In 2003 Yellow Ribbon was working with 156 schools.
Earlier that year (2003), the Injury Prevention Unit of Auckland University was commissioned by the Ministries of Youth Affairs, Health and Education, to conduct research into the criteria for Suicide Programmes in Schools. Yellow Ribbon volunteered to become part of the Study.

The research comprised five case studies of five different organisations that work with schools comprising: Project Hope, Project K, Travellers and Kiwi RAP as well as Yellow Ribbon. Because Yellow Ribbon supported the concept of setting Criteria for Suicide programmes in schools, it was disappointed that this research was seriously under-funded. With a total budget of $5,000 any such project must necessarily be restricted in its parameters of research. Under thse constraints, the value of the study from a scientific viewpoint must therefore be limited. It also brought into question its validity as a nationwide guideline for schools, or as a yardstick upon which to base any conclusions about any peer support programme. Yellow Ribbon actively supported properly-funded, scientifically-based research and wished to work in partnership with Government, to prepare a thorough evaluation that is national in its influence and impact.

On 11 May 2003, the Sunday Star-Times published an article damning work undertaken by youth suicide programmes based upon the Draft Case Study, which formed part of the research. In view of constraints imposed by its funding and methodology, the Study contained inaccuracies that adversely affected its conclusions.

Yellow Ribbon wished to clearly disassociate itself from both the Study and the leaking of information from the draft to the media.

Yellow Ribbon had specific concerns about the content of the Draft Case Study, which through the Sunday Star-Times article, may have provoked concern and misconceptions. In April, Yellow Ribbon was provided with a copy of the Case Study by the Auckland Injury Prevention Unit and invited to comment only upon any factual inaccuracies, in an extremely limited timeframe. The draft was reviewed by Professor Ian Evans (i) and Narelle Dawson (ii) on behalf of Yellow Ribbon.

It is upon their review that the following commentary was made:

1.The study was conducted in only a limited number of schools in South Auckland and, therefore, cannot be considered representative of a national review.

2. There appeared little evidence as to whether or not the various Ministry informants, quoted in the draft report, supplied valid factual information, rather than their own, personal, subjective opinions.

3. The methodology described indicated that providing quotes from transcripts was a major source of information or data. Informants were described as coming from the following groups: Schools principals; school guidance counselors; Representatives from Ministry of Youth Affairs; Representatives from the Ministry of Health; Representatives from the Ministry of Education; Representatives from community agencies; (unspecified)

4. No information was provided as to who had selected the informants and, indeed why they had been chosen to take part in the study.

5. In the case study, a total of 60 quotes from these informants were provided in the following distributions: None from school principals, although three were from unspecified “school representatives” who might or might not have been principals; 10 were from school counselors; 35 quotes came from “Ministry representatives”. In no instance was it specified which Ministry had provided the quote. Five quotes came from community agencies; even further quotes came from unspecified “respondents” or “informants”; their connection with the designated groups is unknown. Thus essentially well over 50 percent of all comments quoted in the draft case study came from unspecified “Ministry representatives”.

6. In addition to what appears, to Yellow Ribbon’s expert advisors, to be obvious bias and selectivity of quotation, the case study uses the misleading device of using quotes such as “several”. Other pseudo-quantitative phrases throughout the case study are “general agreement”, “a number of concerns”, “high degree of concern”, “serious concerns”, “strongly questioned”. From their review, Yellow Ribbon’s experts concluded that the “Case Study” has no scientific value and is not a professionally responsible document. It uses no acceptable social science methodology, either qualitative or quantitative. It is a collection of totally unsubstantiated opinion, from an unknown and not appropriately selected group of individuals. In addition, to label any peer support programme as being “dangerous” without evaluation, is unethical. [My ital] (i)Professor Ian Evans, Professor and Head of the School of Psychology at Massey University, has extensive clinical and professional experience working primarily with children and adolescents as consultant psychologist in the United Kingdom and United States. He was particularly active in working with school-based programmes in New York State, designed to prevent social, emotional and educational problems in young people in schools and also in carrying out evaluation research.
(ii) Narelle Dawson is a registered Clinical Psychologist, a clinician and Australasian Youth Suicide Consultant. She has worked for 22 years as a Clinical Psychologist for Specialist Education Services (SES). During that time, she became a National Professional Consultant for SES, instigating several initiatives for at-risk youth that were later implemented at national level. Her PhD research involves investigation of outcomes and protective factors for suicidal youth placed on government benefits.

FACTUAL INACCURACIES AND SUPPORTIVE INFORMATION:

Page 1, 3rd paragraph:
Senior student volunteers are trained as Yellow Ribbon ambassadors. At all times a guidance counsellor, or supporting teacher, is present when training is undertaken.
Page 1, 4th paragraph:
“Discussions with representatives of Yellow Ribbon indicated ….”
Thelma French was the only representative of Yellow Ribbon spoken to and this consisted of one interview of approximately one hour on Monday 10 February 2003.
Page 1, 4th paragraph:
Our ultimate goal is to reduce youth suicide, but we don’t see it as a youth suicide programme as such.”
The comment should have continued, “It is a positive peer support programme”.

PROVIDERS Page 2, 4th paragraph:
Given their experience of working alongside New Zealand schools and communities, it is anticipated that Yellow Ribbon would be able to provide letters of support from school Principals. Apart from those on the website and programme documents, Yellow Ribbon is able to provide many testimonials of support from School Principals, Guidance Counsellors, Ambassadors and community networks throughout the country, if this is required.
Page 2, 5th paragraph:
Do they have sufficient educational and training qualifications to demonstrate an understanding of the New Zealand curriculum?
At no stage was Yellow Ribbon asked to provide information relating to educational and training qualifications of staff employed by Yellow Ribbon at the National Office or for the regional co-ordinators around the country.
Page 2, 6th paragraph:
We are very concerned as to paragraph 6 as it is based on supposition and uses emotive language.
However, comments from Yellow Ribbon representatives (only one representative) indicate that regional co-ordinators undergo “a whole induction and training process. Three to five days plus two conferences a year.”
his should read: However comments from the Yellow Ribbon representative indicate that regional co-ordinators undergo an induction and training process with the Programme Development Manager who holds a Master’s in Social Science. Each co-ordinator receives a Toolbox of administration documents and the Yellow Ribbon Facilitators Guide for the training. The Facilitators Guide was developed in consultation with eight staff members, two of whom hold a Diploma in Counselling and a Diploma in Social Work. Consultation with guidance counsellors, ambassadors and a clinical psychologist also informed this process. Following every training evaluation, forms are completed by the students and information is applied to further development of the Programme. On-going training and support for staff continues throughout the year.

PROGRAMME CONSIDERATIONS Page 3, Paragraph 4: Yellow Ribbon has developed guidelines for the guidance counsellor which assist in the selection process of students to train as ambassadors, which includes an application form and consent form to be signed by parents, prior to training commencing.
Page 3, Paragraph 5:
Is the programme congruent with the aims and broad principles of the New Zealand Youth Suicide Prevention Strategy?
Attached is the report prepared by Prof. Ian Evans and Narelle Dawson identifying the links between the Yellow Ribbon Programme and the Youth Suicide Prevention Strategy.
However in the media portrayal of our school-based, “It’s OK to ask for HELP” Programme, strict guidelines for reporting is discussed with the journalists involved. Written guidelines are currently being developed to inform this process.
Page 4, Paragraph 1:
Can the programme demonstrate that it has an appropriate an explicit theoretical or research base?
We have concerns with the wording of the sentence “A researcher has been employed to do the research for us”. We believe this implies criticism and is somewhat flippant and does not reflect the responsible and professional manner in which we contracted a reputable research firm as well as highly qualified academics to conduct research on Yellow Ribbon’s behalf.
Page 4, Paragraph 3:
Yellow Ribbon is very aware of the importance of identifying iatrogenic effects, which can be seen in the evaluation design prepared by Professor Evans and Narelle Dawson. This research has not been implemented because Yellow Ribbon was advised to wait until the Criteria established by this Review were available.
Page 4, Paragraph 4:
Does the programme assist with implementing a whole school approach to mental health promotion?
Yellow Ribbon believes that the programme assists with the implementation of a whole school approach to mental health promotion and does fill a gap within schools, especially where the guidance counsellors also hold a teaching role which may create a conflict of interest and lead to diminished levels of trust. Yellow Ribbon believes the promotion of help seeking behaviour is beneficial to students in providing protective factors, especially for when they leave the school environment.
Page 5, Paragraph 4:
Has the programme and programme outcomes been rigorously and independently evaluated? Is evaluation ongoing? How are the benefits of the programme substantiated?
The first pilot was commenced in October 1999 and the first evaluation was conducted in February 2000. Since mid 2001 all students who have attended ambassador training have completed ambassador training evaluation forms. In January 2003 a questionnaire was distributed to approximately 1500 ambassadors through a reputable research organisation. The majority stated the training increased their knowledge of how to seek help, where to seek help and how to communicate a lot more effectively.
Page 5, Paragraph 6:
Significant concern was expressed by Ministry representatives and community representatives about the lack of a robust evaluation framework for the Yellow Ribbon programme.
We are very concerned with this comment as Government is very aware of the evaluation design prepared by Prof Evans and Narelle Dawson in August 2002, the implementation of which we have been asked to delay, despite our seeking specific ring-fenced funding for evaluation studies. In order to ensure our evaluation plan would meet Ministry requirements, we initiated several meetings in which Yellow Ribbon requested from the Ministry representatives more detailed specification as to what in their view would be minimally required for a sound evaluation. To date they have been unable to provide any such guidelines. That safety issues have not been dismissed and are taken very seriously by Yellow Ribbon.
Page 6, Paragraph 2:
The Yellow Ribbon guidelines to school counsellors advocate a monthly supervision meeting be held with ambassadors. The ambassadors’ monthly Report Form is one way in which these meetings are reported back to the regional co-ordinator. We have established procedures for investigating adverse events. The parental consent form clearly indicates that students may withdraw from their role of ambassador at any time. This is also recorded on the student application form and is reiterated throughout the training.
Page 6, Paragraph 3:
Yellow Ribbon refutes the entire paragraph regarding lack of transparency. We have been totally transparent. Yellow Ribbon is committed to continuous improvement and believes this has been demonstrated from when the first pilot was conducted in October 1999 and when the first co-ordinators were employed in March 2001. As mentioned earlier, a questionnaire was distributed in January 2003to approximately 1500 ambassadors through a reputable organisation. The majority stated the training increased their knowledge of how to seek help, where to seek help, and how to communicate more effectively.
Page 6, Paragraph 4:
“However, the lack of evaluation evidence makes it extremely difficult to substantiate the impact of the programme, and the level to which programme aims have been achieved. Consequently, respondents strongly questioned the probable contribution of the Yellow Ribbon programme to young people’s help seeking behaviours and in particular to preventing suicidal behaviours among young people: “Yellow Ribbon has no right whatsoever to claim that they make any positive contribution to suicide prevention.”
This type of comment places Yellow Ribbon in a classic double-bind. Obviously a programme cannot produce outcome evidence until it has been implemented for a period of time. Clearly the general thrust of the Yellow Ribbon programme is based on reasonable principles, and as already explained, work is under way to evaluate both process and outcome. Some initial efforts at review of processes have been initiated, for example in the above-mentioned questionnaire to ambassadors in January 2003, the majority (45.5%) said the training increased their knowledge a lot; and 27.5% said the training increased their knowledge somewhat. The majority of ambassadors said training increased their knowledge of where to seek help a lot (40.9%) and 29.6% ambassadors said the training increased their knowledge somewhat.
In addition Youthline has recorded a 500% increase in calls and relationship services have also seen a marked increase.
Whenever asked if we believe we have contributed to the drop in youth suicide we state that our belief is that education and awareness is very important, but we always reiterate if there is a significant decrease, it is due to the efforts of many organisations and strategies.
Page 6, last paragraph:
No further information was provided around time and other opportunity costs associated with the programme.
No information was requested from Yellow Ribbon.

PARTICIPANT SAFETY
Page 7, Paragraph 2:
Currently there appears to be little to no inclusion or exclusion criteria provided by Yellow Ribbon programme representatives for participating in the Ambassador training component of the programme.
This is incorrect. Screening of intending ambassadors by the Guidance Counsellor is now compulsory and requires students to complete an application form and have a consent form completed by a parent or guardian. As mentioned above, we make it clear in these forms that ambassadors may withdraw from the programme at any time.
Other counselling representatives expressed concern about the lack of criteria for participating in the Yellow Ribbon programme: “It’s a worry, there are no criteria. I just use volunteers. My concern is that they are usually kids who have difficulties themselves and think they can help others.”
This is of concern that a Guidance Counsellor used volunteers who they were concerned about and did not take the responsibility upon themselves to exclude these students. This is one reason why we introduced the Guidelines for Guidance Counsellors and take the opportunity to discuss with them the screening of students who may be deemed to be “at risk”. From our experience, there may be involvement by students who have experienced difficulties in the past and now want to support others. This should be seen as a positive and not a negative.
Page 7, Paragraph 4:
For example, a representative commented that Yellow Ribbon”don’t seem to have modified their programme for Maori or Pacific students”.
It is recognised by Yellow Ribbon that each school has its own unique culture whether this be due to gender/socio-economic/private/ethnicity/boarding/rural/decile rating or other factors. The programme is implemented in consultation with each school to address its specific needs.
Page 7, Paragraph 5:
Does the programme directly or indirectly raise awareness around suicide?
Yellow Ribbon has indicated to the Ministries that they are prepared to revise current material linking the programme to youth suicide. The information to parents on the website is provided as a response to concerns, in accordance with Goal 1 of the NZ Suicide Prevention Strategy. Until the evaluation has been completed, reference to how the programme started has been deleted from the video and the written resources are currently being reviewed. However from the anecdotal evidence received by Yellow Ribbon, the previous video and materials were not harmful. People appreciate having some point of contact to find out further information. This should be seen as a positive because while community agencies know of SPINZ, the everyday person, from our experience, does not. They phone Yellow Ribbon and we refer them on.
Page 8, Paragraph 2:
Ministry representatives raised a number of concerns about explicit links between the Yellow Ribbon programme and suicide awareness. It was considered that the Yellow Ribbon programme “ends up by raising awareness of suicide”, particularly “amongst vulnerable students”. Explicit concerns were articulated that the Yellow Ribbon programme “normalises the concept of suicide”. Similarly, school counsellors also expressed the view that the Yellow Ribbon programme “highlights suicide”.
Since we have made changes to the video and the training programme, the links are no longer ‘explicit’ and we would like this to be emphasised. YR ‘normalises’ help seeking behaviour. At one stage a video was used in some South Auckland schools which due to the explicit nature in which suicide was discussed, YR has withdrawn the use of this. Apart from the YR brand, there is no mention to suicide in the “It’s OK to ask for HELP” Programme. Suicide is a reality for the students, along with bullying and other issues. They see YR as a positive feature in their school, offering the opportunity for them to make a difference.
Page 8, Paragraph 3:
“School representatives expressed concerns about a perceived overemphasis on the branding of the Yellow Ribbon programme: “all they want to do is advertise the programme, they don’t put any time into the kids.”
This comment is absurd – why would schools be reintroducing the programme for the second and third time if this were the case? We have letters of support and acknowledgement from around the country to refute this ridiculous comment and we believe that including comments such as these in the report detracts from the professional and academic purpose for preparing these guidelines.
Page 8, Paragraph 4:
Comments from community agency representatives also indicated a high degree of concern about the branding of the Yellow Ribbon programme. While overall the Yellow Ribbon programme was considered “benign”, the programme branding was considered “toxic”. Overall, the Yellow Ribbon brand was considered “synonymous” with youth suicide. It was considered that if the Yellow Ribbon programme changed its brand, then it would be “possibly a simple educational seminar where people will get some buzz out of it”.
We question this person’s knowledge and understanding of the Yellow Ribbon programme.
Page 8, Paragraph 5:
It was considered that representatives of the Yellow Ribbon programme “seem to actively seek media coverage and actively stir up media debate on the issue when the evidence says we should be very cautious about this.” Inappropriate media coverage instigated by the Yellow Ribbon programme was considered a significant concern as “inappropriate reporting can lead to youth committing suicide.”
Media guidelines primarily express caution about the reporting of actual suicides. Apart from local newspapers covering school launches, YR has not contacted the media. Our understanding it is the Ministries who have contacted the media and YR has been approached and asked for a response. In the ChCh Press on 13 Feb 2003 Annette Beautrais wrote an article with the heading “Grip on youth suicide” and the Dominion Post 18 February 2003 a heading “Government Investigates Youth Suicide Group’s work”. Yellow Ribbon did not initiate these reports in any way.
Page 9, Paragraph 1::
It was also considered that celebrity spokespeople on behalf of Yellow Ribbon were insufficiently informed about the broader issues of suicide prevention activities, including “little if no background information or guidance about what’s safe and what’s not safe.”
Ministry of Youth Affairs provided guidelines for the boxers last year and have been involved in discussions in which they have agreed to be represented on an advisory panel for this year’s FFL. Celebrity guidelines are currently being prepared.
Page 9, Paragraph 4:
Does the programme encourage young people to take a high degree of responsibility for the wellbeing of their peers?
The message on the cards say “stay, listen and get help”. Ambassadors are trained to support the student in seeking help from a supportive adult or outside agency. Ambassador resource cards identify the supportive staff members within the school and provide phone numbers and information for community agencies within the region. Yellow Ribbon promotes It’s OK to ask for HELP and where to get it. Changes were made to the programme with parents’ consent forms being made compulsory early 2002, not “recently”.
Page 10, Paragraph 1:
It was considered “this programme is likely to attract two groups of young people. One is kids who are already vulnerable so they shouldn’t be participating, the other group are those young people who want to do some good and want to make a positive change and my concern with them is they are least likely to hand over a difficult situation because they want to do it themselves.”
Guidance counsellors are made aware before the training of their role in providing ongoing ‘supervisory’ support for the group of ambassadors and the need for regular support meetings. We believe this is a dangerous assumption and does not reflect our experience. Indeed we question the purpose of including remarks like this that appear to be authoritative but which are not founded on any evidence. We are being asked for evidence, which we agree with, but critics are encouraged to make claims that have no empirical foundation.
Page 10, Paragraph 2: Training time
The training is designed to take place in two x three-hour sessions or one full day. An extra meeting is required for planning the school launch. Alternative training times are made to accommodate the school requests, and often this is because YR is incorporated as part of an existing peer support-training programme, which covers similar listening and communication skills material.
Some school counsellors considered a weakness of the programme to be “the facilitators of the programmes, and the delivery of it.”
While YR encourages guidance counsellors to bring concerns to our attention so they can be addressed, to date we have received only two complaints, which were immediately addressed to the satisfaction of both schools that continue to enthusiastically support the programme.
Page 10, Paragraph 3:
Criteria for guidance counsellors to meet with ambassadors.
As previously mentioned, the Guidance Counsellors guidelines indicate the need for monthly meetings and this requirement is discussed with Guidance Counsellors prior to acceptance of the programme.
Page 10, Paragraph 4:
Ministry representatives expressed concern about “the lack of ongoing support for schools and ambassadors that are participating in the programmes.”
Most of the regional co-ordinators do have informal ongoing contact with schools and a more structured format is being developed. Guidance Counsellors do have the option to contact the co-ordinator again and we have not received any complaints regarding this issue. Schools are encouraged to organise in conjunction with the co-ordinator two further initiatives during the year so the co-coordinator is in the school meeting with the guidance counsellor and ambassadors throughout the year. The guidance counsellor is aware of their role to provide ongoing support.
Page 10, Paragraph 5:
Ministry representatives commented, “It is becoming evident that those ambassadors don’t refer on to adults.”
How is it evident that ambassadors are not referring on to adults, and on what documented data is such a claim made? Guidance counsellors who have reintroduced the programme this year have done so because they can see the merits in it, and can testify to the fact that more young people are coming to see them. YR has now developed a policy to inform YR ambassadors of the process should a student make a disclosure of abuse or self-harm.
Page 11, Paragraph 5:
However, relationships between Youthline and YR have sometimes been problematic. E.g. Youthline did not provide permission for its crisis line numbers to be publicised during the Fight for Life promotional campaign because of concerns relating to the framing of the promotion.
The comments made have not been brought to YR’s attention CPH in ChCh and YR jointly funded a report to assist Youthline towards the implementation of a 24/7 Helpline. Further plans are underway to support Youthline more following FFL3. We thus can document effective collaboration between Yellow Ribbon and Youthline.
Page 12, Paragraph 1:
Overall, it was considered that, in spite of recent changes, the Yellow Ribbon programme was “not well linked into other community or mental health providers and services”.
In each school a community services card is prepared by the Co-ordinator with the support of the Guidance Counsellor and ambassadors that promotes awareness of community services in the local region. Co-ordinators actively work with community agencies to ensure appropriate services are being promoted to young people.

YELLOW RIBBON’S SUMMARY:
• We believe the co-ordinators are trained and bring a wealth of experience to their roles. Students identify readily with the fact that they are not teachers, and welcome the passion, enthusiasm and commitment they bring to their school environment. The Programme Development Mgr to whom the Co-ordinators report, has a MSocSci (Hons) and is currently completing a Postgraduate Diploma in Community Psychology.
• Links with the strategy have been identified, and we attach a review, which YR commissioned Professor Evans and Narelle Dawson to undertake.
• As noted in the final paragraph of the study, appropriate, rigorous and valid research or theoretical base to support the YR programme is being addressed. We reiterate that evaluations are undertaken after every training and final analysis of the questionnaire sent to 1500 ambassadors is expected to be completed in May 2003.
• As noted in the final paragraph of the study, a comprehensive and robust evaluation was designed in 2002 but our advice was to await the results of the Government review.
• We respect the professionalism of the Guidance Counsellors and work closely with them in establishing selection processes and criteria for inclusion/exclusion. Students complete application forms and consent forms are required from parents. Parents and students are also made aware of the opportunity to withdraw from the ambassador role at any time.
• It has not been YR’s experience that the programme encourages young people to take a high degree of responsibility for the wellbeing of their peers.
With regard to the methodology used in the case study, we would appreciate your comments on the following points that have been raised by Professor Ian Evans, after he had the opportunity of reading the case study.
(i) If it is considered that the “Ministry representative” is a source of valid factual information, rather than simply his or her own personal subjective opinion, then it would usually be necessary to obtain some sort of confirmation from another source.
(ii) Similarly we do not know how familiar the informants actually were with the YR operation; for example it might be difficult for a “community agency representative” to really have much in the way of first-hand knowledge.
(iii) Did the same 23 informants who were asked about the 5 programmes involved in the case study, all have equal access to first-hand knowledge about these 5 different programmes. Who selected the informants and on what grounds? These selection criteria should have been specified in the |Methodology section.
(iv) Generally the criteria on which the case study was based are good ones, but other criteria may have been more appropriate. For example, the criterion “Has (sic) the programme and programme outcomes been rigorously and independently evaluated?” is a very good criterion, but we already know the answer as there are no rigorous outcome studies published under peer review on any of the 5 programmes being considered. Such studies take time and considerable resources. Thus a more reasonable question might have been ‘Has the programme shown responsive changes in its procedures or approaches as a result of external feedback or internal review of its own methods?” Another reasonable question might have been “Has the programme begun to develop plans for a rigorous, independent evaluation?” By posing certain questions and not others the criteria used in the case study can distort the overall impression of the case study in subtle ways.
(v) Generally, however, the biggest problem is the total lack of any acceptable methodology in the conduct of the case study.
The methodology described indicated that providing quotes from transcripts was a major source of information or data. Informants were described as coming from the following groups: School Principals, School Guidance Counsellors, Representatives from Ministry of Youth Affairs, Representatives from the Ministry of Health, Representatives from the Ministry of Education, Representatives from community agencies (unspecified). In the case study, a total of 60 quotes from these informants are provided in the following distributions: 0 came from school School Principals, although 3 came from unspecified “school representatives” who might have been Principals. 10 came from school counsellors – 4 of these were negative, 4 were neutral (simply gave information about what YR had done in their school) and 2 were positive, but one of these were interpreted negatively in the case study. 35 quotes came from “Ministry representatives”, but in no instance was it specified which Ministry provided the quote. Of these 35 comments, 32 were negative, 1 was positive but given a negative slant in the case study (YR’s branding was “very successful”, which was judged to be a bad thing) and 2 were neutral. 5 quotes came from community agencies, 4 were negative and one was neutral. 7 quotes, all negative, came from unspecified “respondents” or “informants” and their connection with the designated groups is unknown. Thus essentially well over 50% of all comments quoted came from unspecified “Ministry representatives”.
In addition to this obvious bias and selectivity of quotation, the case study uses the misleading device of saying things like “several”. (Would it be possible obtain the original transcripts to enable a count of, for example, exactly how many “school personnel” used the words or phrase “(YR) do not have any safety nets”, in this way one could ascertain what “several” actually means in this case study, since “several” school personnel were said to have used these particular words).
Other pseudo-quantitative phrases used throughout the case study are “general agreement”, “a number of concerns”, “high degree of concern”, “serious concerns”, “strongly questioned”.
Finally there is a problem with using plurals, such as “Ministry representatives” as though there might be a large number, when in fact just one person from YR was described as “representatives”, and thus the plural form seems to be used rather loosely in the case study.
Overall, this “case study”, in Professor Evans’ opinion, has no scientific value and is not a professionally responsible document. It uses no acceptable social science methodology, either qualitative or quantitative, and is a collection of unsubstantiated opinion from an unknown and not appropriately selected group of individuals. Because more than half of the quoted comments come from “Ministry representatives” who have expressed concern about YR in the past, the case study is biased. And as 83% of all quotes are negative, the case study, Professor Evans believes, is also unbalanced.

Evaluation Principles Guiding this Proposal (from Evans and Dawson, 2002)
Evaluations can be either formative or summative, and in some circumstances can be both. Formative evaluations provide the project managers with information and data that can guide the activities of the program and future developments. This may also include management decisions surrounding related themes, such as publicity activities, or ways of introducing the Yellow Ribbon program in schools. Summative evaluations provide information that allows one to assess the outcomes of the program, essentially asking the question “does this program work?”
Evaluation often involves issues other than simple outcome assessment:
• There is the question of whether the approach developed is “evidence based”, that is to say whether the programme uses principles and procedures that have been empirically validated by other researchers.
• A somewhat more manageable version of the same question is whether a program is based on “best or most promising practices”, which can usually be judged by accepted experts in a given field.
• Another issue is that programmes often have unintended or hidden outcomes that might be judged as favourable or unfavourable. A concern that has been raised, for example, is that “awareness-raising initiatives may impact [negatively] on those young people who are already vulnerable.” A related concern is that there may be trickle-on effects in which promotional materials and publicity have unintended negative effects for parents, families, or whanau.
• In addition, programmes have costs (financial, as well as others) that need to be considered in relationship to their benefits.
• Also there are the perennial questions of whether, if a program can be considered successful, it is better than other alternatives; whether it can be sustained over time; and whether it remains true to its original objectives.
• Closely related to the previous issues is the degree to which the program can be integrated with other programmes or government initiatives that have the same general goals; is Yellow Ribbon a valued component of the national strategy?
• A further important principle is whether the program, if it is achieving some success, is doing so equally for all segments of the intended recipient social groups. Is the program acceptable to the different cultural and ethnic groups in New Zealand and is it reaching all segments of the vulnerable population? Are cultural needs being addressed? Do families value the program?

A number of leading commentators on suicide prevention note that there may be benefits in targeting only young people who are at particularly high risk. For example, suicide is thought to be related to significant levels of prior mental health needs. It would be potentially useful at some stage in the evaluation to introduce a more formal psychological measurement component that considers the nature of mental health symptoms in students. The Yellow Ribbon philosophy, however, would argue that while there may be a need for schools to be vigilant in recognising signs of significant emotional distress, there is an important principle in not stigmatising young people as having psychiatric syndromes before appropriate supports can be introduced. In other words, the message of it being “OK to ask for help” potentially applies to anyone at certain critical times, not just to those who might have a diagnosable mental disorder. This evaluation model therefore, presupposes that there could well be an outcome of increased acceptability of professional referral, while at the same time avoiding the presumption that one first has to experience a mental disorder before help-seeking behaviour is appropriate. At the same time it is important to ensure that suicide is not portrayed as an inevitable, likely, viable, and predictable response to stress. This is a delicate balance and data will need to be gathered as to how well this balance is being maintained in the “mythologies” around suicide that exist in targeted and non-targeted schools.

The potentially most difficult question to answer is the scientific validity of a program, which deals with the question of whether a program produces meaningful results and does so for the reasons stated—in other words the outcomes are not simply due to the injection of money, interest, energy, and so on, into a system. This in turn raises some interesting questions regarding what exactly might be the mechanism of change: if the Yellow Ribbon program is successful in promoting help seeking behaviour and peer support, what were the critical, necessary, effective ingredients and is this immediate goal causally related to the ultimate goal of reducing youth suicide? (Despite its tragic personal consequences and tremendous social cost, the base rate for youth suicide is quite low, making preventative efforts uncertain.) This level of scientific credibility is rarely reached in the evaluation of programmes that are already ongoing, because there are practical difficulties in introducing proper controls when programmes are already in operation. Nevertheless, if the resources are available, sound quasi-experimental evaluation is possible.

Finally, in a document prepared by Dawson and Evans (2002) entitled Yellow Ribbon’s Commitment to the Principles of the NZ Youth Suicide Prevention Strategy, it was pointed out that young people themselves could be actively involved in the process of evaluation and development of the program. Programmes that are youth-led and that encourage young people to take personal responsibility for the type of social conditions they would like to enjoy, are more likely to be shaped by young people’s own needs and interests. In this spirit of empowerment, we will endeavour to include high school students in the planning and gathering of information. This may be done formally in conjunction with social science classes, or informally in the spirit of “participatory research.” The same concept will apply to other influential figures in the educational system, such as educational psychologists, teachers interested in research (or completing higher degrees such as Masters and doctorates in education), and guidance counsellors.