Utility of evaluation and limits of the implementation in Europe, by Anja Meyer and Erich Marks, 2004.


(More) Quality in Crime Prevention
by Anja Meyer / Erich Marks
As in so many other fields nowadays, in crime prevention we are increasingly concerned with fulfilling quality demands and attaining an increase in quality in order to guarantee effectiveness and efficiency. If we aim to infuse national and European crime prevention methods with long-reaching and lasting impulses, we have to shift our emphases and define new quality goals. If  crime prevention is to be innovative and up-to-date, we need to employ management methodology. Amongst other things, a closer orientation towards project management, and especially the introduction of obligatory evaluation into preventive project work are called for.
More Quality instead of Quantity
In our times of empty registers, there is an increasing pressure to supply concrete evidence of the efficiency of preventive work. For a long time now, terms like evaluation and quality management have been around in the field of crime prevention. But what exactly do these often-used catch words mean?
If we speak of quality in every-day use, we mean precisely that a product has certain “qualities”, that it lives up to defined standards. The term quality stems from the Latin qualis, meaning ‘the nature or condition of a thing’. Evaluation (lat. valor), on the other hand, means ‘to judge the value of something’. According to König, it is “a special way of accompanying practice with scientific methodology.” [1] More to the point, it concerns the systematic use of social sciences methodology in order to evaluate the conception, the design, the putting into practice and the utility of social intervention programs. The goal is to evaluate certain aspects of our practical work, either in order to ascertain its quality standard, or else to find clues about what needs to be changed. The practical use therefore lies either in  a) stabilisation and continuance of the status quo or b) improvement by introducing well-founded changes (with the aim of eventual optimisation). Quality and evaluation are inseparable. Evaluation is a possible step towards the preservation of quality [2], while ascertaining quality is a step towards quality management [3].
In spite of all our previous successes in the prevention of crime, we have to ask ourselves how the quality of preventive work can be kept up and increased. The “spectrum” of prevention is manifold. The sheer number and diversity of prevention projects (at model stage) is astonishing. In some cases, reach and efficiency of certain programs are questionable, though. Many projects lack a theoretical foundation, others decline to take criminological theories into account. As Sherman puts it: “How do we know what we think we know?”[4]. Do we have a hypothesis in advance about why exactly this project will induce changes and bring about the desired effect? Does one or the other preventive measure obtain the expected effect at all? Which indicators prove the accomplishment of our aims? Working processes which are triggered by the respective measure must be determinable, i.e. it must be possible to examine them empirically. We do not need an increase in the quantity of prevention projects, but one in evaluation as well as in long-term efficiency research and longitudinal analyses. In order to achieve prevention projects of the highest quality, we have to ensure that evaluation becomes a natural component of the every-day working process.  This implies, of course, that any project must be conceived in such a way that it becomes possible to evaluate it. Only if its course and consequences are automatically measurable is it possible to judge its practical effects with precision. Inevitably, prevention requires evaluation (effort, effect, efficiency). An increased evaluation activity in crime prevention practice is called for if we aim to guarantee a protected quality standard for our practical projects. The principle of “everything works” cannot be brought to agreement with the demand for quality. The integration of evaluation strategies, on the other hand, leads to the conservation and increase of quality and, in the long run, to a more professional and effective prevention of crime. What we should ultimately be looking for in prevention is a shift of emphasis from quantity to quality.[5]
Evaluation in Germany‘s Crime Prevention “Landscape“
Although evaluation is no longer a foreign word in criminology and prevention, and Germany is not a complete evaluation “wasteland“, we nevertheless are still relatively at the beginning [6]. Evaluation associations, as for example the Deutsches Jugendinstitut (German Youth Institute) (DJI) [7], or comprehensive evaluations such as the one(s) instigated by the Bundesministerium für Familie, Senioren, Frauen und Jugend (German Ministry of Family, Senior Citizens, Women and Youth) (BMFSFJ) in the area of intervention programs against domestic violence [8], are still rather the exception than the rule.
Longitudinal analyses are still rare due to financial reasons and the time-consuming methodological effort they call for. One study that occupies an immensely important position in the field is one of several undertaken at the University of Erlangen-Nuremberg under the management of Prof. Lösel about “development and its support at nursery school age”. This project, sponsored by the BMFSFJ, is a composite of longitudinal analysis and intervention research. Its goal is the examination of the social (or anti-social) development of children aged three to seven via the analysis of complex bio-psycho-social development processes [9]. Also worth mentioning in this context is the longitudinal analysis “Berufsbildung, Arbeit und Delinquenz”(“Vocational Training, Work and Delinquency”) [10] as published by Schumann in 2003.
A first attempt at the efficiency analysis of crime prevention took place in 2001, when the City Council of Duesseldorf instigated a secondary analysis about national and international efficiency research. This so-called “Duesseldorf report” [11] aims to make clear recognisable factors taking effect in crime prevention for the benefit of preventive practice. The first part evaluates 61 crime prevention studies, remarkable for the empirical research that accompanies them. The second part contains excerpts from the Sherman report. Part three offers some specific results of effect research in the realm of “xenophobic and extreme right-wing violence”. Part four highlights possible political consequences for dealing with crime in the big cities along lines of reasoning which are primarily borrowed from the American debate about “broken windows” and “zero tolerance”. Due to the complexity and sheer bulk of the material, even the “Duesseldorf Report” cannot achieve an extensive analysis of the current state of affairs in crime prevention , or the projects evaluated, though.
In 2003, another call for the necessity of a professional and scientific method to examine the quality of prevention and intervention programs was put forward by the section “Political Psychology” of the Berufsverband Deutscher Psychologinnen (professional organisation of German psychologists). The organisation took the opportunity to offer assessments of project applications for customers – as well as the  evaluation of  projects for suppliers and program developers – as a service itself. [12]
Evaluation in International Crime Prevention
In 1996, the US Congress assigned a task to a research group under the direction of Laurence W. Sherman (of Maryland University) to write an evaluation report on the effectiveness of the local and federal crime prevention programs sponsored by the US Department (DOJ). In order to reach conclusions about the effectiveness of single programs, the researchers examined approximately 500 scientific evaluations of various preventive measures. [13] In 1998, they presented the US Congress with the so-called Sherman report “Preventing Crime”. The report differentiates the programs under examination into the following 3 categories:
1.      What works? Programs that, according to methodological criteria, can be said to be successful, being effective in preventing crime or reducing the risk factors leading to crime.
2.      What doesn’t work? Programs which can be regarded as ineffective since they do not prevent crime or do not reduce the risk factors.
3.      What’s promising? Programs which seem to give cause for optimism since preventive effects could be detected. Other programs, their effectiveness not being determined yet, have been labelled “unknown” for the time being.
The authors come to the conclusion that only very few projects have been scientifically examined so far. According to them, the effectiveness of most crime prevention strategies is still in the dark . An improvement to the situation can only be achieved by an increase of scientific evaluation of crime prevention programs. This requires a stronger input of financial resources. [14]
Just like Sherman et al., the “Office for Juvenile Justice and Delinquency Prevention” (OJJDP) places great importance on evaluation. The OJJDP is part of the “Office of Justice Programs” (OJP) of the U.S. Department of Justice. Among their responsibilities are the development of  effective strategies as well as the coordination, implementation and support of effective programs dealing with juvenile delinquency. Tools designed to offer help in the assessment and evaluation of programs and initiatives are available on their web site www.jrsa.org/jjec via the link “Juvenile Justice Evaluation Center Online”. They should be of considerable interest to practitioners, the police and federal agencies alike. “Juvenile Justice Program Evaluation” provides an overview of summarised descriptions of various evaluation areas concerning juvenile delinquency programs. For each topic, examples for process and effect evaluations are provided, as well as frequently used evaluation designs.  
Research and evaluation are among the key activities of the National Crime Prevention Center (NCPC) in Canada as well. The NCPC is responsible for the implementation of the “National Crime Prevention Strategy”. Its major aims are the distribution of information, the supply with tools as well as support for communities in dealing with the causes for crime. But the NCPC has also noticed a lack of evaluated projects which they put down to missing temporal as well as financial resources and the lack of the necessary expertise. People active in crime prevention can find an extensive presentation about the field of evaluation at their web site www.prevention.gc.ca/en/. The documents provided there do not only show “what works”, but also “how” interventions help to implement and keep up the standard of crime prevention.
Unlike in America and Canada, all across Europe one can still detect a far-reaching “abstinence” regarding evaluation, in spite of the general consensus that evaluation research brings about an improved level of quality and should be used as a permanent instrument to maintain it. In 2002, the members of the “Committee of Experts on Partnership in Crime Prevention (PC-PA)” [15] of the Council of Europe unanimously agreed that one can hardly speak of an actual evaluation research in the area of crime prevention yet. In this respect, the British “Home Office Crime Prevention Centre” [16] is in the vanguard. Since the instalment of Great Britain’s “Crime Reduction Program” in 1999, 10 per cent of the financial resources have to be spent on process and effect evaluation. We feel that a similar design, namely setting aside a certain share of a prevention project’s budget for the execution and financing of evaluation measures well in advance, could only be beneficial in Germany as well as in Europe.
The “European Crime Prevention Network” (EUCPN)
On May 28th, 2001, the “Council of the European Union” instigated the development of the “European Crime Prevention Network (EUCPN)” [17]. The aim of this network is the improved organisation of various initiatives devoted to the elevation and provision of approved practices in crime prevention. Apart from the maintaining of the web site http://www.crimprev.dk/eucpn, one of EUCPN’s essential tasks is the systematisation of the collection and recording of information regarding tried-and-tested methods of crime prevention. The necessity for evaluation, and for a mutual dedication to it, was one of the topics under discussion at the “European Crime Prevention Network” on October 7&8, 2002 in Aalborg. This “good-practice conference”, and another similar meeting on November 11&12, 2003 in Rome, mainly served the purpose of exchanging experiences with prevention projects from the 15 member states which had proven to be effective in practice and could therefore be considered to be exemplary. In theory, there is an obligation to put evaluation results into action in one’s own organisations, to inform others about them and to investigate together to what extent certain activities can be judged as a success or a failure. Nevertheless, at present there still are problems to put these results into practice, caused by the lack of longitudinal analyses, of the use of random samplings and of evaluations regarding the effectiveness of individual implementation practices.
Conclusion
The present state of affairs in Germany and Europe that we have outlined above makes clear that: 
1.)   Nationally and internationally, the systematic evaluation of crime prevention projects and measures is still a scarcity. However, if one aims to achieve progress based on a solid foundation, ultimately it will turn out to be essential.
2.)   In the long run, evaluation and a further development of European crime prevention call for a sound scientific training of prevention practitioners. Education and training in the field of crime prevention have not kept up with its increasing importance today. Crime prevention still has not found its way into the various vocational training courses of the professions it pertains to. The impression that everybody could master prevention, since there seems to be no special training or talent involved, is certainly deceiving. Like any other challenging activity, prevention presupposes professionalism. More often than not, what is lacking is theoretical, methodological know-how. As Ekblom puts it: “No other profession (public health, architecture, for example) would send out its practitioners into the field and expect them to deliver with such limited conceptual resources!” [18] Only by education and training, prevention can gain in quality. The “Australian Institute of Criminology” (AIC) [19] concerns itself with the aforementioned difficulties. For example, the AIC offers training programs (“Training Crime Prevention Specialists for the Modern World”), seminars and workshops on a number of topics, one of them being “Crime and Justice Monitoring and Evaluation.” [20] It would be desirable to have such programs in Germany and Europe as well.
3.)   Certain terms are not used according to a common standard, but are inflected with different meanings in different countries. If we are to exchange ideas about evaluation and training programs, and to compare our projects, we must come to an agreement about key concepts in crime prevention. As Werner [21] claims, what we need is a unified terminology to make distinction and description possible. Ekblom [22] also points out this necessity: ” CCO provides a language for partner professionals and practitioners in different countries to use for rigorous and explicit communication of crime prevention knowledge, for common understanding of diagnosis, and for collaboration on preventive action. In this, it connects with conventional knowledge management views that an enterprise-wide vocabulary is necessary to ensure that knowledge is correctly understood”. Today, we are still lacking a unified European “prevention language”, a common terminology. A collection of the most important technical terms in a glossary could be a step towards the overcoming of language barriers.
The Beccaria Project: Quality Management in Crime Prevention in the context of the AGIS program of the EU
The necessity for action stated above is taken up by the “Beccaria Project: Quality Management of Crime Prevention”. The EU-Rahmenprogramm für die polizeiliche und justizielle Zusammenarbeit in Strafsachen (AGIS) [23] made it possible for the Landespräventionsrat Niedersachsen (Prevention Council of Lower Saxony) to carry out this project.
AGIS promotes designs concerning the areas of judicial cooperation, organised crime, crime prevention and victim protection. Since 2003, the EU program AGIS (named after the King of Sparta, 338-331 BC) has replaced five previous EU programs (Hippocrates, Grotius, OISIN, Falcone, STOP). The aim of AGIS is to contribute to a better cooperation of the member states (and those applying for membership) in the areas of crime prevention and fighting as well as the protection of victims. AGIS supports projects devoted to the aforementioned aims, either dealing with the advancement of cooperation between different prosecution authorities, concerning the education and training of persons active in the fight against crime, or furthering scientific research and methodology. All public and private organisations or facilities may apply (no private individuals, however). Only projects not already in operation may receive financial aid. The maximum duration of support is two years. On the EU level, the financial volume of the whole program amounts to 65 million Euro for the years 2003 to 2007.  The financial aid applied for must not exceed 70 per cent of the total costs. Projects must include partners from at least three member states, or two member states and one country applying for membership.
In the Beccaria project, the Landespräventionsrat Niedersachsen cooperates with institutions from the following European partner countries: Belgium, France, Denmark, the Czech Republic and Estonia. The project was granted in December 2003 and will continue until November 2005. [24] The name of the project refers to Cesare Beccaria (1738-1794).
The further development of the project can be traced at www.lpr.niedersachsen.de  Bibliography
Bundesministerium für Familie, Senioren, Frauen und Jugend (2001): Perspektiven der Evaluation in der Kinder- und Jugendhilfe. QS Nr. 35. Materialien zur Qualitätssicherung in der Kinder- und Jugendhilfe
Bundesministerium für Familie, Senioren, Frauen und Jugend: Zielgeführte Evaluation von Programmen – ein Leitfaden -. QS Nr. 29. Materialien zur Qualitätssicherung in der Kinder- und Jugendhilfe (available free of charge). Berlin
Deutsche Gesellschaft für Evaluation (2002): Standards für Evaluation. Köln
Düsseldorfer Gutachten (2002): Empirisch gesicherte Erkenntnisse über kriminalpräventive Wirkungen. Eine Sekundäranalyse der kriminalpräventiven Wirkungsforschung. Gutachten für die Landeshauptstadt Düsseldorf vom Institut für Kriminalwissenschaften und Fachbereich Psychologie – Sozialpsychologie – der Philipps-Universität Marburg. (Ed.) Landeshauptstadt Düsseldorf Arbeitskreis Vorbeugung und Sicherheit
Düsseldorfer Gutachten (2002): Leitlinien wirkungsorientierter Kriminalprävention (Ed.): Landeshauptstadt Düsseldorf Arbeitskreis Vorbeugung und Sicherheit
Ekblom, P. (2001): Notes for consideration at the workshops on Crime Prevention Research. Paper for EUCPN Leuven. October 2001
Ekblom, P. (2002): European Crime Prevention Network – towards a logic model and mission statement. Vision of 1 July 2002, p. 11
König, J. (2000): Einführung in die Selbstevaluation. Ein Leitfaden zur Bewertung der Praxis Sozialer Arbeit. Freiburg
Polizeiliche Kriminalprävention der Länder und des Bundes (2003): Eine Arbeitshilfe für Evaluation. Stuttgart
Schumann, K.F. (2003): Berufsbildung, Arbeit und Delinquenz. Bremer Längsschnittstudie zum Übergang von der Schule in den Beruf bei ehemaligen Hauptschülern. Vol. 1
Schumann, K.F. (2003): Delinquenz im Lebensverlauf, Bremer Längsschnittstudie zum Übergang von der Schule in den Beruf bei ehemaligen Hauptschülern. Vol. 2
Sherman, L. (1997): Preventing Crime. What Works, What Doesn`t, What`s Promising. (Ed.:) U.S. Department of Justice/Office of Justice Programs
Sohn, W. (2003): Kriminalprävention in europäischer Perspektive. In: Kriminalistik. Ed. 1, p. 37
Werner, J. (2003): Aufruf zu einem Paradigmawechsel. Von der instruktiven zur konstruktiven Kriminalprävention zugleich Hinführung zu einer allgemeinen Theorie der Kriminalprävention, p. 7 (unpublished manuscript)
[1]      „eine besondere Art der wissenschaftlichen Begleitung von Praxis“, König 2000, p.34.
[2]      „the organisational measures by which we aim to ensure that products or services have certain qualities“ („die organisatorischen Maßnahmen, mit denen sichergestellt werden soll, dass die Produkte / Dienstleistungen bestimmte Eigenschaften aufweisen“) (online-Verwaltungslexikon at www.olev.de)
[3]      “a management system designed to guarantee a certain pre-defined level of quality, also usually contributing to the avoidance of extra costs arising from errors“ (“Managementsystem, das der Sicherstellung einer definierten Qualität der Produkte dienen und damit zumeist auch einen Beitrag zur Senkung von Fehlerkosten haben soll…“) , online-Verwaltungslexikon, www.olev.de.
[4]      “Wie wissen wir, was wir zu wissen glauben?“, Expert conference “Preservation of Quality in Crime Prevention“ on May 19, 2003 in Neuss.
[5]>      “Quality instead of Quantity: On the Quality of Communal Crime Prevention in Mecklenburg-Vorpommern“ (“Klasse statt Masse: Zur Qualität der kommunalen Kriminalprävention in Mecklenburg-Vorpommern“) was the title of Prof. Bornemann’s presentation at a prevention conference on June 11, 2003 in Greifswald.
[6]     In 2003, ProPK (Polizeiliche Kriminalprävention der Länder und des Bundes) has edited a brochure designed to facilitate the evaluation of prevention projects for the police.
[7]      www.dji.de
[8]      http://www.bmfsfj.de/Kategorien/Forschungsnetz/forschungvorhaben
[9]      “in a multi-method-multi-informant approach, the study examines appraisals of the behaviour of parents and tutors, cognitive development data, information about the previous development of the children, about their parents‘ educational behaviour, and about support and burdening by the family, as well as physiological and medical data.  A second aim of the study is the development and testing of preventive measures for the nursery school age. Its concept is to attain a combination of child- and parent-oriented prevention elements.” In einem multi-method-multi-informant Ansatz werden dazu u.a. Verhaltensbeurteilungen der Eltern und Erzieherinnen, kognitive Entwicklungsdaten, Angaben zum bisherigen Entwicklungsverlauf der Kinder, zum Erziehungsverhalten der Eltern und zur familiären Unterstützung und Belastung sowie physiologische und medizinische Daten erhoben. Ein zweites Ziel der Studie ist die Entwicklung und Erprobung präventiver Maßnahmen für das Kindergartenalter. Konzeptionell ist eine Kombination aus kind- und elternorientierten Präventionselementen vorgesehen“.) cf. www.phil.uni-erlangen.de
[10]      cf. Schumann, 2003.
[11]     The Düsseldorf Report was conceived by Rössner, Bannenberg et.al. It can be downloaded for free at www.duesseldorf.de/download/dg.pdf
[12]      Contact Prof. Preiser at the Institute of Pedagogical Psychology in Frankfurt/Main if interested. There will be a charge for their services.
[13]      The full report of 500 pages in English can be found at www.preventingcrime.org
[14]     Space does not allow an appraisal of approaches and results here, please refer to the extensive description of the Duesseldorf Report itself.
[15]      Cf. Sohn, 2003, p. 37
[16]      www.homeoffice.gov.uk/justice/index.html [17]      http://europa.eu.int/comm/justice_home/eucpn [18]      Ekblom 2002, p. 12
[19]      AIC is Australia’s national centre for the analysis and dissemination of criminological data and information.
[20]      www.aic.gov.au/training
[21]      cf. Werner 2003, p. 7
[22]      Ekblom 2002, p.11
[23]      http://europa.eu.int/comm/justice_home/funding/agis/funding_agis_en.htm
[24]      www.lpr.niedersachsen.de