Mike Godwin's Critique of the TIME Article and the Rimm Study

Time Waited For No One
(Or At Least Not For Me)

Why I Picked A Fight With The Newsmagazine That Fed
The Great Internet Sex Panic

By Mike Godwin
mnemonic@eff.org



Copyright Mike Godwin, 1995


I don't know when it was that I first heard the name "Martin Rimm." I
first *remember* hearing it last fall, when I got involved in the
censorship battle at Carnegie Mellon University in Pittsburgh. As
censorship battles at universities go, the CMU fight didn't seem terribly
different -- it followed the normal pattern: administrators discover that,
horrors, there is sexual content on the Internet, and, in a combination of
disapproval and fear of publicity, they leap into a crackdown, often
cloaking their censorship motives in terms of fear of legal liability.

But there were two aspects of the case that made it a bit different. The
first was CMU's prominence as a networked university -- in its ubiquity of
connections to the Internet and its plethora of computer resources freely
available to its students, CMU is second only to MIT (and many at Carnegie
Mellon would claim that it's MIT who's in second-place).

The second was that, in this case, the triggering event seemed to be an
undergraduate research project on, of all things, pornography on the Net.
Based on images he'd encountered on Usenet, and a superficial
understanding of the law of obscenity, Rimm was said by my sources at CMU
to have informed the administration that their systems were carrying
material likely to be found obscene. Furthermore, he was reported to have
told them, now that they knew about the material, they couldn't claim a
standard defense in obscenity cases: that administration lacked "scienter"
(a legal term meaning something like "guilty knowledge"). Since CMU had
been put on notice, I was told by more than one source, it had to act. And
so CMU's decided to announce it was cutting all alt.sex.* newsgroups and
most of the alt.binaries.* newsgroups as well.

The story of the CMU censorship debate has been told in many places, but
only Time magazine's report of the story focused on Rimm's alleged role in
triggering the abortive attempt at summary censorship. It was a role whose
details, at least, he continues to dispute: in email to me, he took pains
to tell me he opposed the action CMU had taken, and he urged that we meet
when I visited CMU to attend a freedom-of-speech rally. He told me we'd
corresponded in the past -- I didn't remember it, but, then, with the
volume of email I handle, it was certainly possible we'd had prior
contact.

------------------

Date: Tue,  8 Nov 1994 09:10:05 -0500 (EST) From: Martin Rimm
 To: mnemonic@eff.org Subject: CMU Porn Study Cc:
Martin Rimm  Status: RO

As you may recall, I corresponded with you a number of times regarding a
study entitled, "Marketing Pornography on the Information Superhighway,"
of which I am the principal investigator. The study, while not distributed
or read by anyone outside of the research team, has occasionally (and
incorrectly) been invoked as a reason for the ban of the .sex hierarchy.

Given that you will be on campus tomorrow, there are two things I would
like to discuss with you. First, I would appreciate an independent check
of our legal footnotes, which to some extent are based on your postings
and articles. Second, our preliminary data indicate that there are no
significant differences among individuals in communities across the
country in what kinds of erotic materials, including pornography (visual
and verbal) and obscenity, they find of interest. In our experiment we
began by assuming that there were indeed community standards which
differed across communities - that is, that some communities of
individuals had no tolerance for or interest in say, pictures of
heterosexual anal intercouse.  We then began collecting data to allow the
evaluation of the ~null hypothesis~ - that is, that there are no
differences.  Our conclusions are very clear: there are no differences
when communities are defined by telephone area codes. This there are no
~community standards~on which communities differ.

We would like to refine our findings by continued data collection and
analysis. We have examined only one kind of erotic material - pictures
about anal intercourse - and would like to look at other interests such as
pedophilia, more kinds of paraphilias, and so on. We have examined only
areas of the country (by telephone area codes) and want to consider other
ways of structuring the data to compare major sections of the country,
states, major metropolitan areas, etc.

Please let me know if and when we can meet.

Martin

-------------------

Although I posted email to Rimm telling him where I'd be during my visit
to CMU, he made no contact with me during my visit, which centered on my
meeting privately with CMU administrators, then my giving a speech at the
student rally. (An excerpted version of my speech was later published in
Wired.)

I can't say I thought much more about Rimm at the time. There was
something that smelled a bit goofy about his research project and the
weird seriousness with which he was pitching it to me. His faculty
adviser, Marvin Sirbu, actually wrote me independently to suggest that EFF
sponsor the Rimm project in some way. But EFF doesn't normally sponsor
this sort of project, and my instincts told me we should keep our
distance. That instinctive reaction was only bolstered when a contact at
CMU sent me an draft abstract of the Rimm study.

----------------

[This is a draft of the ABSTRACT of the Martin Rimm study.]

      As Americans become increasingly computer literate, they are
discovering  an unusual and exploding repertoire of sexually explicit
imagery on the  Usenet and on "adult" computer bulletin board services
(BBS). Every time  they log on, their transactions assist pornographers in
compiling  databases of information about their buying habits and sexual
tastes.  The more sophisticated computer pornographers are using these
databases  to develop mathematical models to determine which images they
should try  to market aggressively. They are paying close attention to all
forms of  paraphilia, including pedophilic, bestiality, and urophilic
images,  believing these markets to be among the most lucrative. They are
using  the Usenet to advertise their products, and maintaining detailed
records  of which images are downloaded most frequently. 
      Modem technology also enables researchers, for the first time, to
use  computers to acquire vast amounts of information about the
distribution  and consumption of pornography on a scale hundreds of times
larger than  previously established methods. Because BBS pornographers
rely primarily  upon verbal descriptions to market their images,
researchers can develop  computer programs that classify these
descriptions according to category  (e.g. oral, anal, vaginal,
sadomasochism, etc.). The descriptions may be  sorted by frequency of
downloads (consumer demand), size, and the date  on which each image was
first posted onto the bulletin boards. What is  even more useful, the data
can be easily reanalyzed under many different  sets of definitions and
assumptions. This multidimensional  characteristic of digital pornography
enables researchers to provide  unbiased information to those involved in
the heated public policy  debate over pornography. 
      The research team at Carnegie Mellon University has undertaken the
first systematic study of pornography on the Information Superhighway.
The study is also the first ever - whether  print media or electronic -
to track detailed purchasing habits of  consumers of sexually explicit
materials. All prior studies have assumed that those surveyed about  their
sexual tastes would offer honest replies, while this study focuses
entirely upon what people actually consume, not what they say they
consume. This proved particularly important when analyzing such taboo
imagery as incest, bestiality, coprophilia,  urophilia, and torture. 
      All available pornographic images from five popular Usenet boards
were  downloaded over a six month period. In addition, descriptive
listings  were obtained from 68 commercial "adult" BBS located in 32
states. These  lists described 450,620 pornographic images, animations,
and text files  which had been downloaded by consumers 6,432,297 times,
from 35 "adult"  BBS; (approximately) 75,000 for which only partial
download information  was available, from six "adult" BBS; and another
391,790 for which no  consumer download information was available, from 27
"adult" BBS.  Finally, approximately 10,000 actual images were randomly
downloaded or  obtained via the Usenet or CD-ROM. These were used to
verify the  accuracy of the written descriptions provided in the listings.

      This article analyzes only the 450,620 images and descriptions for
which complete download information was available. A survey of the
remaining images and descriptions suggests no substantive differences
between the two datasets. At least 36% of the images studied were
identified as having been distributed by two or more "adult" BBS. These
"duplicates" enable researchers to compare how identical imagery is
consumed on commercial BBS in different regions of the country. 
      Part II of the study outlines the methods used to obtain and analyze
the data gathered. Two important aspects of reliability and validity  were
carefully considered: 1) How well do the verbal descriptions  correspond
to the Carnegie Mellon study's categories? and 2) How well do  the verbal
descriptions marketed by pornographers correspond to the  actual images?
Part III.A addresses three issues concerning pornography  on the Usenet:
1) the origins of such imagery; 2) the percentage of all  images available
on the Usenet that are pornographic on any given day;  3) the popularity
of pornographic boards in comparison to  non-pornographic boards. Part
III.B comprises the major portion of this  study. It examines 1) the image
portfolio and marketing strategies of  the Amateur Action BBS; 2) the
concentration of market leaders among  "adult" BBS; 3) the availability
and demand for hard-core, soft-core,  paraphilic and pedophile imagery; 4)
market forces common to all "adult"  BBS. Part IV presents a more informal
discussion of the data, including  a) the appeal of digital pornography;
b) the relationship between images  and the words that describe them; c)
the wide circulation of paraphilic  imagery; d) the importance of
descriptive lists; e) the sophistication  of pornographers. Part V offers
a summary of the significant findings of  this study; Part VI offers
suggestions for further research. Appendix A  lists the categories of
imagery according to the Dietz-Sears and  Carnegie Mellon models. Appendix
B offers the reader an indication of  the power of the linguistic parsing
software developed for this study.  Appendix C presents the data in the
form of pie charts, bar graphs, and  scatterplots. 
      It is assumed that the reader has a basic understanding of the
Usenet  and BBS. Only the technical aspects of BBS which relate to
pornography  will be explained in detail. 


----------- 

It was the kind of scientific abstract that, for me, raised a lot of
questions. These questions troubled me not because I'm a lawyer concerned
with free speech on the Net so much as because--once upon a time--I had
planned to be a research psychologist and had devoted serious time to
studying research methodology and statistics. I'd even managed to win a
graduate fellowship to pursue a doctorate in the University of Texas at
Austin's experimental-psych program; it was then I discovered that,
although I liked *knowing* science, I didn't much like *doing* it. So I
altered my plans for graduate study -- first to English literature, and
then, after a few years' as a journalist, computer consultant, and slacker
in Austin, to law.

But even with all the changes in plans, I never lost my head for math or
for method -- which turned out to be useful when I was reporting science
stories. And it was my psych-research alarm bells, not my legal ones, that
Rimm's abstract first set off.

You see, even in his draft abstract Rimm was making statements that he
could not possibly support. "Every time [users] log on, their transactions
assist pornographers in compiling databases of information about their
buying habits and sexual tastes," he'd written. It was the kind of
absolute statement that no responsible researcher dealing with human
behavior would ever make -- given the range and unpredictability of human
behavior, credible researchers of psych and social-science phenomena will
qualify both their hypotheses and their conclusions. The abstract was
chock-full of categorical generalizations like the one I quote here --
generalizations that, given the limits on the types of data Rimm purported
to be studying, were wholly inappropriate.

And, as it happens, I knew that many of his statements were also flat
wrong. In the course of my work, I'm regularly in contact with operators
of adult BBSs (they often have questions about obscenity law, and they
hope to stay on the right side of legality). The claim that they're
refining their offerings of sexual material to focus on what Rimm asserts
to be a more "lucrative market" in what he charmingly calls "paraphilias"
flew in the face of what I'd been hearing from the BBS sysops who called
me for advice, or whom I met at conventions like One BBSCon. Those sysops
wanted to minimize the risk of angering their communities -- especially
their local law-enforcement agencies -- but the strategies Rimm was
categorically attributing to them would *increase* their legal risks.

There were other potential methodological problems: the reliance on verbal
*descriptions* of the images to characterize them, the apparent conflation
of Usenet and BBS data, the conflation of "download" and "consume." Sure,
it was possible that Rimm might advance, in his discussion of his
methodology, reasonable explanations for his peculiar approach, but even
the most rigorous theoretical framework he could advance would not leave
him in the position of generalizing with the certainty to which he was
prone in the abstract. And, given what I knew about Usenet and the
difficulty of measuring user behavior there (I've long followed the
pioneering research of Brian Reid at DEC), Rimm's implication that he
might be able to determine "the percentage of all  images available on the
Usenet that are pornographic on any given day" was sheerest fantasy.

Nor were these the only problems I had with the abstract. But the biggest
howler was this one: "The research team at Carnegie Mellon University has
undertaken the first systematic study of pornography on the Information
Superhighway. " Even from the abstract, it was apparent that the bulk of
Rimm's data came from 68 "adult" BBSs -- to generalize from commercial
porn BBSs to "the Information Superhighway" would be like generalizing
from Time Square adult bookstores to "the print medium."

There were other weirdnesses that, strictly speaking, were neither factual
nor methodological. Like Rimm's evident fascination with types of porn
that are, uh, not mainstream. (I was about to say "off the beaten track,"
but I just remembered Rimm's expressed interest in material featuring
sadomasochism.) It was hard to avoid the conclusion that Rimm was, at
best, an odd duck, and that he had some sort of agenda.

But it wasn't an agenda I was particularly worried about -- given the
amateurishness of his abstract, I was certain the Rimm paper would never
come to anything. I figured that, once the CMU censorship fracas died
down, the Rimm research would sink, like most undergraduate research
projects, into oblivion. (Look, no one can be right all the time!) So I
read the occasional note I received from Rimm over the next few months
with benign tolerance. In his subsequent email Rimm renewed his request
that I review his legal footnotes -- he even sent me the text of the
footnotes for my convenience. But even if I'd had the time to check on
someone else's legal research (doing the job right would require many
hours), I couldn't ethically approve of legal footnotes without seeing the
text of the article they were footnotes *to*. I pointed this out to Rimm
and suggested that, if he were to send me the full article, I might be
able to find the time to review the footnotes for any obvious mistakes.

Rimm told me he'd get back to me on that. But he never did. And the next
time I heard about the Rimm study was early in the week of when Philip
Elmer-DeWitt of Time called me early the week of the 19th for comment on
the Rimm study and the conclusions Rimm, who by now had received his
bachelor's degree, had reached. Among these conclusions, Philip told me,
were that tastes for online porn were becoming more "extreme," that adult
BBSs were using Usenet to market their wares, that sysops had discovered
that the more "violent" the language of a description the more popular an
image was, and that Amateur Action BBS, whose Milpitas, CA, sysops had
recently been successfully prosecuted in Memphis, Tennessee, was "the
market leader" of online porn.

It was clear from the questions Philip was asking that Time was going to
treat the Rimm study as a major story -- perhaps even a cover story. And
this insight was Part One of what I'd later think of as Philip's Triple
Whammy. Given what I already knew about Rimm's research, I was appalled
that Time would publicize it -- I immediately tried to warn Philip of the
methodological and other problems I saw with the study. He told me that
study was going to be published in an article in the Georgetown Law
Journal, that Time had an exclusive, and that he (that is, Philip
Elmer-DeWitt) found Rimm's methodology convincing. I couldn't believe we
were talking about the same study. Philip found it easy to dismiss my
caveats -- after all, I hadn't seen the study.

So I asked to see it -- I promised Philip that, if he showed it to me, I
wouldn't "leak" it, but instead would use it to frame more detailed and
substantive criticisms (or, perhaps, be forced to admit that the
methodology and conclusions were convincing after all). That was when
Philip hit me with the Second Whammy -- thanks to an arrangement with the
law journal and/or with Rimm (Philip was vague about this), *no one*
outside of the editors of Time and the law journal would get to see the
study before the Time story appeared and the law-journal issue was
published. I was stunned -- if there were questions about the study's
reliability (and I still had every reason to believe there were), the
arrangement Philip told me about practically *guaranteed* that those
questions wouldn't be fully considered by Time's editors. Especially since
Philip had already convinced himself that the doubts I tried to raise
weren't serious ones. I knew Philip to be by Time editorial management --
I'd even heard rumors of an upcoming promotion -- so I was certain that,
if Philip vouched for the reliability of the study, his superiors would
take his word for it.

So at this point I made two suggestions: First, I referred him to Donna
Hoffman, a Vanderbilt University professor I knew from the WELL. I knew
Hoffman and her husband Tom Novak to be among the most knowledgeable
people in the world when it came to questions of surveying Net usage or of
modeling marketing strategies in this new medium. I assumed that Hoffman
and Novak would raise the same methodological questions I had, plus some
I'd no doubt overlooked, and perhaps that would convince Philip to look
again at the reliability of the Rimm study.

My second suggestion was for Philip to contact whoever it was who was
insisting on nondisclosure of the article and ask them to grant me
permission to see it for comment, with the proviso that I'd agree not to
leak it in any way. This came to nothing -- when I reminded Philip about
it the following week, he professed not to remember that I'd ever proposed
this arrangement.

And although Philip did have one of Time's field reporters interview
Hoffman, he never spoke to her himself. He did read the "file" from the
reporter's interview, though. We know this because he later argued on the
WELL that the intensity of Hoffman's language in commenting on the Rimm
study methodology (she knew about it from the abstract and -- mirabile
dictu! -- from her own prior correspondence with Rimm, who'd solicited her
advice and support months before) made her an unreliable source. After
all, how could she be so critical *when she hadn't seen the study*? And,
of course, she was barred from seeing it by the arrangement among Time,
the law journal, and Rimm.

The more I thought about the study's imminent publication, the more
troubled I was by the secrecy and the lack of critical review. That's when
it occurred to me to consider how odd it was that an article by an EE
major, purporting to be a *marketing* study, was appearing in a *law
review*. Although Philip took this to be an index of the study's likely
reliability, I knew something that, at least at first, he did not --
namely, that law reviews are unlike most other scholarly journals in that
they're edited not by professors or professional editors but by
*third-year law students*. And while I have the highest regard for the
ability of student law-review editors at a school like the Georgetown
University Law Center, I knew it was highly unlikely that the editorial
staff had the expertise to question the claims and arguments that Rimm
would be making about his computer-mediated research into the "information
superhighway." Suddenly the legal footnotes took on a new significance --
they were the thin entering wedge that qualified Rimm's article as a fit
piece for a law review.

It all came together for me then. If Rimm had set out to publish an
article about online porn in a way that *legitimized* his article yet
*escaped* the kind of critical review the piece would have to undergo if
published in a scholarly journal of computer-science, engineering,
marketing, psychology, or communications, *what better venue than a law
journal?* And a law-journal article would have an added advantage -- it
would be read by law professors, lawyers, and legally trained policymakers
and taken seriously. It would automatically be catapulted into the center
of the policy debate surrounding online censorship and freedom of speech.

I tried to point this out to Philip when he called me back for a second
interview, but he clearly wasn't terribly interested in hearing it -- he
grunted obligingly, but moved to the questions he really wanted to ask me,
about the net.censorship legislation pending in Congress and about what I
thought the effect of the publication of the study and its appearance in
Time would be. "It will be a disaster," I told him. "It won't matter if
you try to balance your presentation of the study with the questions
people have about its methods and reliability. It'll be used to stoke the
fires of the Great Internet Sex Panic." He noted my comments, then ended
the conversation.

As the days counted down to publication of the next issue of Time, I
indulged in hopeful thoughts. Philip had a great track record as a
reporter on cyber-issues -- for all that even the most balanced story
would be, in my view, "a disaster," I could understand how Philip had
convinced himself of the importance of the story, and, as a once and
future journalist myself, I respected his commitment to tell a story even
if the facts might generate the wrong kind of reaction among policymakers
or the public. Not once in all my discussions with him had I ever
suggested that he not do the story. And when it came to the critical issue
of balance, I fancied that I could trust in his professionalism. Indeed,
when rumors of the upcoming Time story had surfaced, and some WELL users
were ready to castigate Philip for writing it, I posted the following
one-line message on Sunday morning, June 25: "Let's hold off criticizing
Time until we see what the story looks like."

But all this hope left me wide open for what would turn out to be Part III
of the Triple Whammy.

Here's what I posted on Monday, when I had had a chance to read the piece
as it appeared in Time:


--------------------

media.1029.86: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 14:39


Philip's story is an utter disaster, and it will damage the debate about
this issue because we will have to spend lots of time correcting
misunderstandings that are directly attributable to the story.

For example, when Philip tells us what the Carnegie Mellon researchers
discovered, he begins his list with this:

'THERE'S AN AWFUL LOT OF PORN ONLINE. In an 18-month study, the team
surveyed 917,410 sexually explicit pictures, descriptions, short stories
and film clips. On those Usenet newsgroups where digitized images are
stored, 83.5 percent of the pictures were pornographic.'"

Who but the most informed among us will not come away with the impression
that the CMU study involved a survey of 917,410 items *on Usenet*? (Guess
what -- it didn't.)

And he concludes the list with this;

"IT IS NOT JUST NAKED WOMEN. Perhaps because hard-core sex pictures are so
widely available elsewhere, the adult BBS market seems to be driven largely
by a demand for images that can't be found in the average magazine rack:
pedophilia (nude photos of children), hebephilia (youths) and what the
researchers call paraphilia--a grab bag of "deviant" material that includes
images of bondage, sadomasochism, urination, defecation, and sex acts with
a barnyard full of animals.'

Problem is, this isn't the typical range of content you find in Usenet
newsgroups, or on commercial services, or even on most BBSs. Instead, this
is the range of content you find on the specialized subclass of commercial
BBSs that focus on pornography.

Just to make things worse, Philip refers to the Internet in the next two
grafs (and not at all to commercial porn BBSs).

This is an incredibly muddled abortion of a story, despite Philip's
attempts to introduce balance. The *packaging* of the story -- a cover
with an innocent child at a keyboard, the paintings of  men fucking a
computer or being pulled into one -- is deeply  sensationalistic.

And the profound problems with the study's
methodology go undiscussed.  Sure, we have a guy pointing the possibility
of a "gaper" phenomenon, which tells us something about how to interpret
the results of a correctly conducted survey. But not a hint of how
methodologically flawed the study is, or about how the people doing the
study were rank amateurs, or about how the legal footnotes were spiced with
citations from anti-porn zealots like Catharine MacKinnon and Bruce Taylor.

The Time story aims at legitimizing the study as raising important issues.
What it does instead is raise serious questions about whether the lure of
an exclusive eclipsed Time's professional judgment.


----------------------------


And in the course of the next few days, I questioned Philip pointedly
about the writing and editorial decisions he'd made in the Cyberporn cover
story -- decisions that both maximized the extent which the story
exacerbated the Great Internet Sex Panic and actually *obscured* critical
facts about the study. Philip occasionally responded with glib,
superficial answers, which enraged me. It was as if he were deliberately
ignoring the magnitude of what he'd done.

Now it wasn't my researcher buttons that were being pushed -- it was my
journalism buttons. Philip had written the story in such a way that, in
effect, he would be deceiving great numbers of his readers. With a copy of
the study in hand (finally!) I began to savage Philip in the media
conference on the WELL:

----------------------


 media.1029.102: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 20:27


Philip writes:

"Well, it *was* a graph about adult BBSs, wasn't it?"

Philip, this is the most infuriatingly disingenuous answer I can imagine
your making. *Did you not read my criticisms above?* You conflate Usenet,
Internet, and BBSs so readily in your listing of the study's conclusions,
that the *nuance* that a *particular* graph is about particular subniche
of commercial BBSs--and not the Internet--*is certain to be lost to any
reader who is not knowledgeable about the medium, and to many that are.*

I pointed out *already* that the next two graphs *following* your
hebephilia-bestiality graph mention the Internet, not BBSs. Are you simply
*oblivious* to the meaning communicated by that juxtaposition? Perhaps you
are, since *you get confused yourself*. In one of those next two graphs in
which you mention the Internet, you say: "The Internet, of course, is more
than a place to find pictures of people having sex with dogs."

Just one problem -- you haven't said even once, prior to that, that the
study shows that bestiality images are common on the Internet.

I'm going to put the lie to your disingenuous claim. Let's just look at
how you juxtapose these three paragraphs:

'IT IS NOT JUST NAKED WOMEN. Perhaps because hard-core sex pictures are so
widely available elsewhere, the adult BBS market seems to be driven
largely by a demand for images that can't be found in the average magazine
rack: pedophilia (nude photos of children), hebephilia (youths) and what
the researchers call paraphilia--a grab bag of "deviant" material that
includes images of bondage, sadomasochism, urination, defecation, and sex
acts with a barnyard full of animals.

'The appearance of material like this on a public network accessible to
men, women and children around the world raises issues too important to
ignore--or to oversimplify. Parents have legitimate concerns about what
their kids are being exposed to and, conversely, what those children might
miss if their access to the Internet were cut off. Lawmakers must balance
public safety with their obligation to preserve essential civil liberties.
Men and women have to come to terms with what draws them to such images.
And computer programmers have to come up with more enlightened ways to
give users control over a network that is, by design, largely out of
control.

'The Internet, of course, is more than a place to find pictures of people
having sex with dogs....'


Now, Philip, please tell me, with a straight
face, that you think a Time reader who is not knowledgeable about this
medium will *not* draw the conclusion that the Rimm study shows that the
*Internet* is the place where all those images can be found.

Explain the goddamned "sex with dogs" link, Philip.

I have no patience with this dishonest bullshit.


media.1029.103: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 20:44


Philip also writes, incredibly:

"Also, you seem to have bought into the as yet unproven assertions about
"profound problems with the study's methodology." So far, the chief
criticism that's been leveled against it here is that it was headed by an
undergraduate. I'm waiting for something more specific. And meaningful."

What makes this an incredibly dishonest statement is that, when Philip
called me for commentary about the study, *I begged for a copy of it, so I
could review the methodology.* Not only that, but I gave you contact
information for Donna Hoffman, prof@well.com, who is indisputably
competent to critique the methodology, and you *didn't give her a copy
either*.

Still, since Rimm discusses the methodology in the abstract (which I've
posted in this topic), both Donna and I were able to make several
methodological critiques of what the authors *said* they were doing.



You've totally lost it, Philip. The statement that you haven't heard
anything except a complaint about Rimm's age, undergraduate status, and
inexperience is flatly a lie.

But what's worse is the lie that you tell by implication. Please quote the
passage in your story where you *mention* that Rimm, "the study's
principal investigator," is an undergraduate EE major with no former
experience in studying or applying the statistical methodology used in
conducting surveys.

What? You omitted to mention it? Now, why did you do that, Philip? Could
it be because you wanted to give the impression that Rimm is far more
authoritative than, in fact, he is? Because that would improve the cachet
of your exclusive?

Don't even bother talking to me any more. After the immense dishonesty of
this piece, and your subsequent dishonesty in defending it, I don't even
want to know you. That I have defended you in the past, and that I
defended this piece before I saw it ("Philip will try to balance the
story, I'm sure," I told people) is now an embarrassment to me that I'm
going to spend a long time living down.


media.1029.125: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:00


ped writes:

"The story was careful to keep the usenet stats separate from the adult
BBS stats."

The TIME story states:

"THERE'S AN AWFUL LOT OF PORN ONLINE. In an
18-month study, the team surveyed 917,410 sexually explicit pictures,
descriptions, short stories and film clips. On those Usenet newsgroups
where digitized images are stored, 83.5 percent of the pictures were
pornographic."

The Rimm study states:

"In addition, the team obtained descriptive listings from sixty-eight
commercial 'adult' BBS containing 450,620 pornographic images, animations,
and text files that had been downloaded by consumers 6.4 million times;
six 'adult' BBS with approximately 75,000 files for which only partial
download information was available; and another twenty-seven 'adult' BBS
containing 391,790 files for which no consumer download information was
available. Thus, a total of 917,410 descriptive listings were analyzed by
the  research team.'

Hmmm. Looks like the number 917,410 (450,620+75,000+391,790) has to do
with '"adult" BBS' -- and yet that number appears in the very  same graf
as some statistics on Usenet newsgroups.


Yet I could have sworn Philip just said he kept the stats separate!





media.1029.136: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:21


Philip writes:

"I was satisfied that your cricicism was based on the wording of the
executive summary you had read, and was not a flaw in the study."

An abstract is not an "executive summary." Abstracts discuss methodology
in detail.

"The fact is, that if you look at .binaries on Usenet, most of them come
from adult BBSs."

That you think this somehow supports applying your characterization of the
contents of Usenet from the contents of commercial porn BBSs shows that a
good remedial methods course is in order. Hint: you *do not know* whether
the sample of porn-BBS content that appears on Usenet has the same
distribution of content as the porn-BBS content surveyed in the Rimm
study. Rimm doesn't know either. There's no part of the study that
supports this leap.

Want another blunder?

You never mention in your story that the CMU researchers didn't look at
the images themselves -- they based their study on the *description* of
the images that appeared on the porn BBSs.

You never mention that the way they justified this was by checking 10,000
*Usenet* images against *their* descriptions -- that was taken to be the
measure of the accuracy of the *porn-BBS descriptions*.

You never mention that the CMU researchers probably didn't download the
images they purported to be surveying (that costs money, and you have to
do it over phone lines, not the Internet, and that takes a lot of time),
so *you don't even know*  if there actually *are* 917,000+ separate and
independent images, etc. -- many of the items that appear with different
descriptions may in fact be the same item.

"[If we are to beliee the study, and I still have no reason not to.]"

And you have *every* reason *to* believe the study, don't you, Philip?
Because if you *disbelieved*, it would undermine the value of your
exclusive.

"And yelling at me is not going to make that go away."

Why don't you buy a clue and respond to the real reasons I'm yelling at
you, Philip? If you think I'm upset because Philip the upright  journalist
has reported some unpleasant facts, you're even more self-delusional than
I thought you were.

Hint: "sex with dogs"/Internet. How to explain that little comment, when
the only conclusions about the availability of bestiality images concerned
porn BBSs, not the Internet?

Note: I'm not making any statement at all about whether bestiality images
are available on the Internet -- but the Rimm study can't be read as
saying there are, *since the Rimm study is not about images on the
Internet at all*. In fact, it's *not about images* -- it's only about
*descriptions of images*.

Of course, we needn't bore Time's readers with these little nuances.

To read the Rimm study about porn BBSs as saying something about Usenet is
like surveying Times Square adult bookstores and using the results to
characterize *all* bookstores in San Francisco.



media.1029.144: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:39


ped writes:

"Mike, thse are two separate sentences and two separate statements."

Well, now, Philip, let's look at how those two separate sentences and two
separate statements are presented:

"THERE'S AN AWFUL LOT OF PORN ONLINE. In an 18-month study, the team
surveyed 917,410 sexually explicit pictures, descriptions, short stories
and film clips. On those Usenet newsgroups where digitized images are
stored, 83.5 percent of the pictures were pornographic."

Why, those statements don't look separate to me. In fact, they appear to
be right next to each other. And they are adduced to support the same
conclusion: THERE'S AN AWFUL LOT OF PORN ONLINE.

Guess what, Philip -- you're still screwed on this one. Because *either*
one must read the sentences together (in which case you do mix BBS and
Usenet stats) *or* one must read the sentences separately, *in which case
the second sentence does not in itself support the conclusion that THERE'S
AN AWFUL LOT OF PORN ONLINE.* (Basic stat hint: percentages by themselves
tell us nothing about quantity. Most people learn this basic fact about
percentages in high school.)

But if you are seriously arguing that those two sentences, right next to
each other and used to support the same point, won't be read as  being
about the same thing, you are seriously cracked.



 media.1029.170: Avant Garde A Clue (mnemonic)  Tue 27 Jun 95 06:13


ped writes:

"When I posted above that I hadn't heard any criticism of the study's
methodology beyond the fact that it was conducted by an undergraduate, I
meant I had hadn't heard any *here." I had, of course, heard Mike and prof
criticize it."

Even if we grant that you were trying to say, in response to my comment
about Rimm's status as an undergraduate, that you had not heard other
criticisms of the study from me *here*, you *had* heard other criticisms
of the study from me elsewhere. So the implication that I and others are
simply taking cheap shots and not making substantive criticisms is just
wrong.

But I think it's important to acknowledge the extent to which you were
unwilling to accept those criticisms of the study. Yes, I believe you
reviewed the study with my criticisms in mind -- you needed to find a way
to explain them away or dismiss them, or else they might have undermined
the whole raison d'etre of the piece.

So you reminded yourself that we hadn't seen the study (I offered to
review it for you with the promise not to leak it -- Donna would surely
have done the same). You relied on Sirbu's endorsement of Rimm (although
Sirbu seems to have had no prior experience in conducting this sort of
study either). You told yourself that, regardless of what (implicitly
minor) methodological flaws there might be in the study, or whether the
conclusions were supportable by the data they did gather, *this is an
important study*.

In this, you remind me of the infamous last line of Newsweek's cover story
on the Hitler diaries [I paraphrase Newsweek here]: "In the end, this has
been such a big story that it doesn't really matter whether the diaries
are genuine or not."


----------------


I was venting at this point. But it took a single epiphanic moment to
convince me that the thing to do was not merely criticize Philip, but
instead to do the kind of reporting he and Time had failed to do. What was
that moment? I recounted it in a WELL posting reproduced below:


-----------------


media.1029.197: Avant Garde A Clue (mnemonic)  Tue 27 Jun 95 15:33


Well, I'm going to be on Nightline tonight, debating Ralph Reed of the
Christian Coalition. It has already been taped -- we mostly debated
whether the Exon legislation was a better fix for protecting your children
than the software tools and filters that I advocated.

But the taped lead-in focuses on the Rimm study, and stresses how the Rimm
study shows how easily pornography is available to children on the Net.
And not just any old pornography, but the hebephilia, urination, etc.,
that the Rimm study shows there is so much demand for.

Before we taped, I mentioned to one of ABC's reporters, Richard Harris,
that there were a number of methodological criticisms of the Rimm study.

So, afterwards, Harris and his researcher arranged to have me in a
conference call with Philip, Marty, and apparently one or more other
people who were involved in the study. I was given a chance to raise my
concerns about the study's methodology, with mixed results, so  that the
ABC people could hear, at least, some of the reasons for  believing that
the study focused not on cyberspace as a whole, but on a nonrandomly
selected subset of commercial BBSs that focus on selling porn, that the
study was based on descriptions, not images,  and that the conclusory
links between Rimm's sample and the "information highway" as a whole were
not supported methodologically.

I doubt it made much difference -- ABC guys aren't terribly interested in
hearing nerds talk about statistical inferences. But they were kind enough
to give me a hearing on the methodological and ethical problems I have
with how this done.

During the call, Philip noted that I'm an advocate, so it follows that I
feel compelled to argue against a study that reports inconvenient  facts.
(I later pointed out to the ABC guys that the Levy piece has inconvenient
facts, but I'm not outraged about that one, and that before I was a lawyer
I was a researcher.) The implicit dishonesty of his casting doubt on my
motives, of course, lies in the suggestion that Philip and Martin Rimm
don't have far stronger motives of their own to pitch the study as
something groundbreaking and compelling and reliable.

Rimm answered a lot of the question I raised, some adeptly and others with
dodges. He claimed to have no agenda.

We wound down, although a few voices (mostly mine) were raised. But before
we lost the connection, I heard this:

Philip: "Marty, you there?"

Rimm: "Yes, I'm here."

Philip:  "Good job!"


--------------------------


To a journalist, Philip's "good job" was a revelation. At that point, I
turned to Harris and mouthed (with regard to Philip): "He's on the team!"
It was stunningly clear that Philip had so identified himself with the
story that he believed his and Rimm's interests were essentially the same.


There was only one fix, I thought -- do everything I could to make sure
that the truth about the Rimm study, and about Time's collusive arrangment
that prevented it from being properly criticized, be made as public as
possible, as widely as possible. I had already fedexed a copy of the study
to Donna Hoffman -- working with EFF's legal interns, I made several
copies of the article and fedexed them to people who knew enough to
criticize the piece the way it should have been criticized at the
beginning.

I later learned that Philip characterized my labors as an "orchestrated
campaign" to discredit him, conducted by a "professional lobbyist." Since
I've never been a lobbyist in my life, that comment did sting, but of
course in a sense he was right that I was conducting an "orchestrated
campaign." It was orchestrated largely from my PowerBook, and the campaign
consisted of putting copies of the study in front of independent reporters
and other commentators who were capable of reading it and seeing the
obvious. That's what led to Elizabeth Corcoran's insightful piece in the
Washington Post, and to Peter Lewis's thorough reporting in the Times.
It's what led to the critiques of the study that you see on Donna
Hoffman's Web page and here at Hotwired. It's what has led to the
revelation that Rimm's own faculty adviser, named in the study's
biographical footnote, doesn't think Rimm's data support his conclusions
about Usenet.

And all the time I was getting other people to read the study, I was doing
my own reading. Perhaps the single most damning discovery I made appears
in this posting:


---------------------

media.1029.511: Avant Garde A Clue (mnemonic)  Sat 1 Jul 95 18:34


Let's come back to the Footnote Quiz, which Philip declined to answer. I
had written this:

*********************

"9. As a result of federal legal action against a few well known 'adult'
BBS operators, including Robert and Carleen Thomas (Amateur Action) and
Robert Copella (Pequena Panacha), some systems have removed their
paraphilic, pedophilic, and hebephilic imagery from public display. This
has created a thriving underground market for 'private collections' and
anonymous ftp sites on the Internent, which cannot be studied
systematically. Thus, it may be difficult for researchers to repeat this
study, as much valuable data is no longer publicly available. See infra
notes 89-95 and accompanying text."

Now, Philip, try answering this quiz:

Of the many unsupported assertions in this single footnote, which one
would raise the *biggest* red flag for a reader/editor working for a
peer-reviewed journal?

*********************


Now, this footnote is rife with candidates for "red flag status." The
"some systems have removed" claim is undocumented and unsupported, as is
the "thriving underground market" and the "anonymous ftp sites [market
created for such sites because of porn crackdown]" comment. So's the claim
that anonymous ftp sites can't be studied "systematically." (Note: it may
well be true, but *it is not supported by the study*.)

But the single biggest red flag is the penultimate sentence--"it may be
difficult for researchers to repeat this study...."

It is, in my opinion, designed to make the study unkillable, so that
anti-porn activists will be able to use it forever, *regardless of
subsequent studies that seem to disprove it*.

Think about what would happen if subsequent studies seemed to  support
Rimm's conclusions: 

(Rimm: "See? I was right!")

Now think about what would happen if subsequent studies seemed to
*disprove* Rimm's conclusions.

(Rimm: "See? I was right!")

You begin to see why the author might have felt compelled to  sidestep
peer review if at all possible.

------------------------


The process of exposing the Rimm study, which I increasingly believe may
have been a deliberately political ploy in the guise of "research," is
ongoing. You can see the results here and elsewhere on the Net. And the
same is true of the process of epxosing the extent to which Time and
Philip Elmer-DeWitt traded their responsibility to the American public in
return for questionable exclusive, written and packaged to maximize panic
about the Net -- it's ongoing. 

But there's one thing that's not ongoing, and that's any "orchestrated
campaign" to discredit Philip Elmer-DeWitt. My response to that particular
charge is best expressed in one last posting that I'll share with you
here:


-----------------

media.1029.423: Avant Garde A Clue (mnemonic)  Fri 30 Jun 95 13:19


Philip writes:

"I can understand why you feel obliged to discredit the Rimm study."

I don't think you do. Based on what you have said up to now, you think
it's because it comes to some conclusions that are inconvenient for my
work. You think I'm just playing out some role as an advocate for
net.freedom, and therefore feel compelled to challenge the study out of a
sense of loyalty to my cause. I'm sure that's what you told yourself when
you decided to dismiss my comments out of hand.

So long as you labor under this self-delusion as to my motives, you won't
have a clue about why I'm doing what I'm doing now.

"But I'm having a hard time understanding why you felt obliged to
discredit me at the same time."

Philip, you should be clear on this:

I have never, ever had it in my power to discredit you, nor have I ever
thought I did.

You discredited yourself.



            -30 -