Come Mentire e Indurre in Errore con le Statistiche sugli Stupri

L'articolo: https://www.datagoneodd.com/blog/2015/01/25/how-to-lie-and-mislead-with-rape-statistics-part-1/

How To Lie And Mislead With Rape Statistics: Part 1

January 25, 2015 by Francis Walker

Si scopre che solo il 7,8% delle denunce di stupro sono vere.

So che può sembrare difficile da credere, ma non me lo sono inventato. Tecnicamente, è completamente vero. È anche completamente una cavolata. È così fuorviante e costruito su così tante avvertenze non dichiarate, che la maggior parte delle persone lo considererebbe come buono come mentire se sapessero come è stato effettivamente derivato. La cosa interessante è quella statistica che potresti aver sentito, che "solo il 2-8% delle denunce di stupro si rivelano false"? Non solo è altrettanto fuorviante (se non di più), ma proviene dallo stesso identico insieme di dati.

Allora, da dove viene il numero del 2-8%? Se lo hai letto nell'articolo di Zerlina Maxwell sul Washington Post potresti essere ragionevolmente confuso che l'intervallo provenga dall'FBI.

Infatti (nonostante i vari miti popolari), l'FBI riporta che solo il 2-8 percento delle accuse di stupro si rivelano false, una percentuale che è inferiore a quella (10 percento) di persone che mentono sul furto di auto.

Maxwell ha fatto riferimento a un articolo del 2009 che ipotizzava la fascia del 2-8%, ma questo non ha nulla a che fare con le statistiche dell'FBI o i furti d'auto. Tuttavia, esiste una versione precedente dell'articolo del 2009 pubblicata nel 2007 con lo stesso nome che includeva questo dato:

Per esempio, il dipartimento di polizia di Portland, Oregon, ha esaminato 431 denunce di aggressioni sessuali completate o tentate nel 1990 e ha scoperto che l'1,6%1 di esse sono state determinate essere false. Questo era in confronto con una percentuale del 2,6% per le false denunce di furto di veicoli (Ufficio del Procuratore Generale dell'Oregon, 2006).

Questo proviene da un singolo dipartimento di polizia, in un singolo anno, e non è una statistica dell'FBI.  Come si scopre, c'è una statistica rilevante dell'FBI, ma invece dell'1,6%, è dell'8%, il che significherebbe che non si adatterebbe al suo punto riguardo ai furti di veicoli. D'altra parte, c'è anche una statistica separata2 che dice che circa il 10% delle denunce di furto di veicoli sono false.  OK, quindi ora abbiamo una fonte per la fascia del 2-8%, una stima dell'FBI che è in quella gamma, e un'altra statistica che mostra una percentuale più alta di furti d'auto.  Quindi, mentre lei confonde un po' le sue fonti e ciò rende quello che dice un po' fuorviante, nel complesso il suo punto è accurato, giusto?

È qui che ci imbattiamo in un piccolo problema3 – stiamo attingendo tutte queste statistiche da posti diversi, quindi come facciamo a sapere che sono abbastanza simili da essere confrontabili? Se Maxwell avesse controllato quella statistica dell'8% dell'FBI, scoprirebbe che effettivamente hanno qualcosa da dire su questo argomento – anche se non si inserisce così bene nel suo racconto. Il numero dell'8% proviene dalla sezione II dei rapporti dell'FBI sul crimine negli Stati Uniti dal 1995 al 1997 che dicono:

1995

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. In 1995, 8 percent of forcible rape complaints were “unfounded,” while the average for all Index crimes was 2 percent.

1996

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. Eight percent of forcible rape complaints in 1996 were “unfounded,” while the average for all Index crimes was 2 percent.

1997

A higher percentage of complaints of forcible rape are determined “unfounded,” or found by investigation to be false, than for any other Index crime. While the average of “unfounded” rates for all Crime Index offenses was 2 percent in 1997, 8 percent of forcible rape complaints were “unfounded” for the same timeframe.

So when you look at things on a comparable basis and from crime data across the country and across multiple years, instead of the percentage of unfounded reports being lower for rape than other types of crime, forcible rape actually has the highest unfounded rates and was consistently 4x the average.

Let’s go back to that 2-8% range from 2009.  Think for a minute about what would make for a good statistical source on false rape reports.  Given how easy it can be to manipulate statistical research, you’d probably want someone unbiased if the data is to be at all useful. Now click through to the source Maxwell listed.  It is an article that appears in The Voice, the newsletter for The National Center for the Prosecution of Violence Against Women.  The article is titled False Reports: Moving Beyond the Issue to Successfully Investigate and Prosecute Non-Stranger Sexual Assault. OK, so not starting out so hot on the unbiased front, but let’s give it the benefit of doubt and dig into the article.

The article starts out with a note that estimates for sexual assault false reports vary quite a bit (1.5% to 90%). They then go on to say that they don’t find many of the prior data points to be credible, and attempt to narrow that range. The first study4 they attempt to discredit was done in 1994 by Professor Eugene Kanin. In it, Kanin found that 41% of the 109 sexual assault reports made to a midwestern police department over a 9 year period were determined to be false. This would seem to stand in pretty stark contrast to their assertion that the real number is 2-8%. The authors present two main reasons why the Kanin study is invalid. This first is that the determinations that the reports were false were made only by the detectives.  They work in a quote from an article written by David Lisak, one of the authors of the paper (the 2009 iteration anyway):

Kanin describes no effort to systemize his own ‘evaluation’ of the police reports—for example, by listing details or facts that he used to evaluate the criteria used by the police to draw their conclusions. Nor does Kanin describe any effort to compare his evaluation of those reports to that of a second, independent research— providing a ‘reliability’ analysis. This violates a cardinal rule of science, a rule designed to ensure that observations are not simply the reflection of the bias of the observer (p. 2).

They then go on to sum up this first point as follows:

In other words, there is no way to explore whether the classification of these cases as false was simply made as a result of the detectives’ own perceptions and biases, without any real investigation being conducted.

I’ll take this opportunity for a brief detour into irony land. Remember how I said there was an earlier version of this paper from two years prior? Though David Lisak was not an author of that paper (Dr. Alan Berkowitz took his place), the other two authors – Dr. Kimberly A. Lonsway, and Sergeant Joanne Archambault (Ret.) – were.  In that version, the “credible” range was 2-4% instead of 2-8% and they offered three pieces of evidence to support it.  The first is the Oregon Police Department data (it would seem the police are perfectly capable of an unbiased determination – so long as you like the number they come up with).  The second was this5:

Similarly, Sgt. Joanne Archambault of the Sex Crimes Division of the San Diego Police Department routinely evaluated the rate of false reports over several years and found them to be around 4%

Observer bias can be a real issue, but it does work both ways. Later on we’ll get to the evidence that they are using for their new estimate (thankfully they scraped the first two sources from 2007 this time around). When we do, keep in mind what kind of views they had going in. Let’s return to the Kanin study. Their primary concern is essentially that the police are making the true/false determination on their own, potentially on a whim without much investigation, which on the surface is a pretty valid complaint.  Except, of course, for the fact that it completely misrepresents how the reports were determined to be false.  What they want you to think is that this is just a bunch of cops who don’t believe the women and so they classify it as false without digging into the details at all.  They can do that because they know that pretty much no one is actually going to go and read the underlying research.  If you did, you would find this right up front in the methods section:

This city was targeted for study because it offered an almost model laboratory for studying false rape allegations. First, its police agency is not inundated with serious felony cases and, therefore, has the freedom and the motivation to record and thoroughly pursue all rape complaints. In fact, agency policy forbids police officers to use their discretion in deciding whether to officially acknowledge a rape complaint, regardless how suspect that complaint may be. Second, the declaration of a false allegation follows a highly institutionalized procedure. The investigation of all rape complaints always involves a serious offer to polygraph the complainants and the suspects. Additionally, for a declaration of false charge to be made, the complainant must admit that no rape had occurred. She is the sole agent who can say that the rape charge is false. The police department will not declare a rape charge as false when the complainant, for whatever reason, fails to pursue the charge or cooperate on the case, regardless how much doubt the police may have regarding the validity of the charge. In short, these cases are declared false only because the complainant admitted they are false. Furthermore, only one person is then empowered to enter into the records a formal declaration that the charge is false, the officer in charge of records. Last, it should be noted that this department does not confuse reported rape attempts with completed rapes. Thus, the rape complainants referred to in this paper are for completed forcible rapes only. The foregoing leaves us with a certain confidence that cases declared false by this police agency are indeed a reasonable–if not a minimal reflection of false rape allegations made to this agency, especially when one considers that a finding of false allegation is totally dependent upon the recantation of the rape charge.

Since this is the internet and that was a long quote, allow me to sum it up

• Police officers are forbidden to use their discretion in deciding whether a complaint is false
• A complaint is determined to be false if, and only if, the complainant admits that no rape occurred

Credibility is important here.  In order for the 2-8% range to be believed, you have to accept the premise that the article’s authors reviewed the relevant research and fairly and accurately distilled the data down to that range. With what you now know about the methodology, ask yourself the following question – Does the view put forth by the authors fairly and accurately represent the facts of the study?

Let’s move on to their second problem with the Kanin study.

This concern is compounded by the fact that the practice of this particular police department was to make a “serious offer to polygraph” all rape complainants and suspects (Kanin, 1994, p. 82). In fact, this practice “has been rejected and, in many cases, outlawed because of its intimidating impact on victims” (Lisak, 2007, p. 6).

I have to take another detour here.  People use a lot of different tactics when they are trying to convince you of things and some are more effective than others.  I don’t know about you, but quoting yourself as authority sits pretty far down on the list of things that convince me. It kind of has that “My point is really great, but I seem to be the only one making it” ring to it. Sometimes it may be a good option, and I’m sure if I keep this blogging thing up long enough I’ll end up doing it a few times myself, but in disputing the Kanin study the authors use three different quotation – and they’re all the same Lisak article.  Oh and just for bonus points, that Lisak article they keep citing? In it Lisak cites Lonsway and Archambault’s 2007 version of False Reports. So I guess that makes it authority squared? Setting aside the fact their sources for critiques of the Kanin study only seem to come from one of the authors themselves, their complaint is not completely without merit. In the underlying article Lisak notes that when investigators express doubt when interviewing potential victims it can cause feelings of confusion, shame, and self-blame.  I’m not going to dispute that or say that I think the police using a polygraph makes for good policy.  However, I’m not really sure how it disproves the fact that, in this case, 41% of the women admitted they made the allegation up. In fact, Kanin himself deals with this very point in the study

A possible objection to these recantations concerns their validity. Rape recantations could be the result of the complainants’ desire to avoid a “second assault” at the hands of the police. Rather than proceed with the real charge of rape, the argument goes, these women withdrew their accusations to avoid the trauma of police investigation. Several responses are possible to this type of criticism. First, with very few exceptions, these complainants were suspect at the time of the complaint or within a day or two after charging. These recantations did not follow prolonged periods of investigation and interrogation that would constitute anything approximating a second assault. Second…in those cases where a suspect was identified and interrogated, the facts of the recantation dovetailed with the suspect’s own defense. Last, the policy of this police agency is to apply a statute regarding the false reporting of a felony. After the recant, the complainant is informed that she will be charged with filing a false complaint, punishable by a substantial fine and a jail sentence. In no case, has an effort been made on the part of the complainant to retract the recantation.

Don’t get me wrong, I’m not naive enough to say that there is no such thing as a false confession.  However false confessions don’t just happen – they are made for certain reasons6, and I’m not really sure those reasons have much to do with the situations being discussed here.  False confessions made in an attempt to end long long hours of coercive interrogations, or to avoid physical violence make sense.  What doesn’t make sense is the idea that these women admit to making up the the allegations and provide explanations that comport with other facts of the case instead of just getting up and leaving if they felt uncomfortable with the questioning.  It makes even less sense when they are then told they will be charged with false reporting and continue to stick with the the story that they made it all up.

You don’t have to just take my word for it though, we can use some simple math to help us evaluate these claims.  If the 2-8% range is correct, how many of the confessions in the Kanin study would be false? To avoid issues with rounding, let’s multiply all the reported numbers by 1,000 (so total reports goes from 109 to 109,000 and the number of women who confess to making false reports goes from 45 to 45,000).  If we assume that the true false reporting rate is 8%, there would be 8,720 reports that were actually false. That means that 80.6% of the time (36,280 cases), the women were admitting to making the report up and accepting punishment for doing so, despite there genuinely being a rape.  Actually, the number is probably even higher.  Remember that the Kanin study only counts a report as false if the woman admits that it was false.  The above numbers assume that 100% of the women who filed the false reports are caught and admit to doing so.  If only 75% of the those who file false reports confess, than it goes from an 80.6% false confession rate to 85.5%.  The following chart shows the false confession rate based on a variety of actual false reporting rates (down the left) and false reporter confession rates (across the top).

So the question you have to ask yourself is – Does this seem like a plausible range for the the percentage of false confessions in situations where the women are free to leave without confessing, and face fines and jail time if they do?

There is one final thing I find curious about their critique of the Kanin study. If they feel that the use of the polygraph is throwing off the results of the study so much, you would think they would at least mention the addenda to the study that describes a different data set that they later gained access to7.  In 1998 Kanin obtained access to the police records of two large Midwestern state universities.  The result?  Over a three year period, 50% of the forcible rape complaints were false.  Once again, only cases that included a recantation of the charge were classified as false, and there was no use of polygraph this time.  Here is what Kanin had to say about that data

In both police agencies, the taking of the complaint and the follow-up investigation was the exclusive responsibility of a ranking female officer. Neither agency employed the polygraph and neither declared the complaint false without a recantation of the charge. Most striking is the patterning of the reasons for the false allegations given by the complainants, a patterning similar to that found for the nonstudent city complainants. Approximately one half (53%) of the false charges were verbalized as serving an alibi function. In every case, consensual sexual involvement led to problems whose solution seemed to be found in the filing of a rape charge. The complaints motivated by revenge, about 44%, were of the same seemingly trivial and spiteful nature as those encountered by the city police agency.

Given the amount of time I have spent defending the Kanin study, you might think that I regard it as some sort of gold standard – I do not.  There are a number of valid complaints about the study.  First of all, the initial study is from a Midwestern city with a population of 70,000 and the universities were Midwestern as well, so you can’t reliably extrapolate these rates as being indicative of the entire country.  Second, the data is fairly dated at this point, with the study being published in 1994 and addenda from 1998, so it doesn’t necessarily give us a lot of information about how things are currently. However, the criticisms that Lonsway et al chose to bring up are not only weak, but significantly misleading. [Update: I have since written more about the flaws in the Kanin study here]

So if that is the data that they don’t like, what kind of data do they think we should be relying on?  Check back soon for How to Lie and Mislead With Rape Statistics: Part 2.

UPDATE: Part 2 is now available.

1. This stat was from version 2 of the Oregon SART handbook.  When they updated this stat 12 years later in 2002, as described in version 3, they found it had nearly doubled to 3% ↩
2. Mentioned at the bottom of page 5 here, though original source is unknown ↩
3. Actually there is more than one issue – people make false reports of stolen vehicles for insurance fraud, which has very little to do with why they make false rape reports, so I don’t really see much informational value in one being higher than another. Though if comparing unrelated numbers is your thing, I’m not sure you are going to like how this one turns out. ↩
4. I think archive.org needs to refine their keyword listing methodology ↩
5. Their third and final piece of evidence in the original paper was a study that took place outside the US ↩
6. The Innocence Project has a pretty good explanation here ↩
7. I suppose they did technically mention it. They buried in an end note like this one. ↩