Rimm's "A Detailed Critique of A Detailed Critique of the TIME Article "On a Screen Near You: Cyberporn"

A Detailed Critique of

A Detailed Critique of

 Donna L. Hoffman &
Thomas P. Novak Associate Professors of Management Co-Directors,
Project 2000
	Owen Graduate School of Management
Vanderbilt University

by
Marty Rimm
College of Engineering
Carnegie Mellon University
B.S., Electrical and Computer Engineering, 
Carnegie Mellon University (College and University Honors)
	Graduate Studies in Broadband Communications, Carnegie Mellon University
	Principal Investigator, Marketing Pornography on the Information Superhighway
(Georgetown Law Journal, Vol. 83, Issue 5, 1995)
	Researcher, Department of Electrical and Computer Engineering 
Carnegie Mellon University

[Image deleted]
Time magazine published a cover story exclusive on Marty Rimm's published, yet not peer-reviewed,

The Carnegie Mellon study indeed was reviewed and commented upon extensively by three outstanding scholars, each from differing perspectives. These scholars were in a position to raise serious questions about the study and were free to refuse to respond to, or to severely criticize, the Carnegie Mellon study. In fact, numerous scholars read the first draft of the Carnegie Mellon study in December, 1994, and offered detailed comments and requests for further analysis, which were incorporated into the final manuscript. Many scholars at Carnegie Mellon University and elsewhere (see p. 1849 of the study for a partial listing) who participated in the study itself offered extensive comments during the process.

It is hard to believe that the three scholars responding to the Carnegie Mellon study in the Georgetown Law Journal would have attached their considerable reputations to responding to a study at length if they believed it was seriously flawed. It is also difficult to believe the Georgetown Law Journal, given its excellent reputation, would have published a seriously flawed study. All three scholars have praised the study strongly in their published responses: one has a Ph.D. in political science from Yale, another is a highly respected scholar of information and communications technology, and the third has published before in the area of pornography and women's legal rights. Two of these scholars have tenure, and the third is a visiting scholar at the Annenberg School of Communications, University of Pennsylvania.

Sadly, it appears that the critics don't want anyone to know this. Having now had the time to at least carefully review the biographical footnotes of the Carnegie Mellon study and the three responses published, it would demonstrate a minimal level of intellectual honesty for Ms. Hoffman and other critics to acknowledge this. It is not surprising that they have not done so, since Ms. Hoffman, Mike Godwin, and a few others were unexplainably attacking the study before they had ever read it.

undergraduate research project

The Carnegie Mellon study drew on the expertise of more than two dozen researchers, many of whom hold Ph.D.s in statistics, engineering, economics, management, fine arts, psychology, history and public policy from universties such as MIT, Berkeley, Stanford and Carnegie Mellon. The formal credentials of those who assisted with, or directly participated in, this study are substantial and reflect expertise in a wide variety of disciplines.

Genuine critiques, which will come from those interested in advancing constructive research, are welcome. Any researcher's work is refined by the comments, criticisms, and suggestions of his colleagues.

concerning descriptions of images on adult BBSs in the United States.

The Carnegie Mellon study also reports worldwide findings from the Usenet. These findings were collected by Brian Reid, a highly respected network engineer at DEC Labs in Palo Alto, California. The Carnegie Mellon study also reports Usenet university findings which were collected by network engineers at a University. The Carnegie Mellon study also briefly discusses anonymous ftp sites on the Internet in a few footnotes. The Carnegie Mellon study also includes, as an appendix, a brief survey of pornography on the Web. To define the Carnegie Mellon study solely as "concerning descriptions of images on "adult" BBSs" is, to put it kindly, a misstatement.

Given the vast array of conceptual, logical, and methodological flaws in the Rimm study, (documented in Hoffman & Novak's Detailed Analysis... 7/1/95),

The claim of a "vast array" of "flaws" is not substantiated here or elsewhere in this "critique.".

at least some of which Time magazine was aware of prior to publication, Time magazine behaved irresponsibly in accepting statements made by Rimm in his manuscript at face value.

Comments about TIME are left to TIME, with a few exceptions: Philip Elmer-DeWitt is a superb technology writer. His reporting of the results of the Carnegie Mellon study has been precise, accurate, and insightful. Frankly, what Ms. Hoffman, Mike Godwin, and a few others are apparently upset over is the refusal of TIME Magazine to assent to their private, repeated requests to censor the Carnegie Mellon research. Further, Mike Godwin was offered the opportunity to participate in the Carnegie Mellon research effort last year. He refused to be involved, or offer any constructive insights, unless the research time could guarantee him that the results of the study would buttress the policy positions he has taken in public on behalf of EFF. For obvious reasons, no such guarantee could be made while preserving academic integrity.

At the least, Time magazine should have sought the detailed opinions of objective experts as to the validity of the study.

Ms. Hoffman and Mr. Godwin certainly would not qualify as "objective experts". Peter Lewis of the New York Times (July 3, 1995, Business Section) labelled them and all other critics of the study as "advocates."

Time further compounded this error by making other erroneous statements about the nature of pornography in "cyberspace," and in some cases, even misinterpreted Rimm's results. Below we detail the numerous errors in the Time magazine article.

p. 38, 3rd graf
The Rimm study is not "an exhaustive study of on-line porn - what's available, who is downloading it, what turns them on..."

Ms. Hoffman and Mr. Novak are certainly entitled to their opinion in this regard. Tenured professors who have published in this area have concluded otherwise. Legal scholars have concluded otherwise.

The term "unsophisticated" is peculiar and inappropriate. A study does not have to be "sophisticated" to be important or interesting.

The Carnegie Mellon study is exhaustive of "adult" BBS according to the carefully defined selection criterion outlined on pages 1876-1877. Again, it seems almost beyond comprehension that any serious reader of the study could fail to note these logical criteria.

The study findings cannot be generalized beyond this narrow domain.

The findings can be generalized about the "adult" BBS industry.

It would be a red herring to suggest that the Carnegie Mellon study purports to examine BBSs generally. It makes no such claim anywhere. Again, those who have read the study with care or objectivity cannot fail to notice this. Ms. Hoffman simply errs in this regard.

p. 38, 4th graf
TIME says the study "tells us about what's happening on the computer networks, [and] also what it tells us about ourselves."

This statement is misleading, because the study tells us only what happens on selected private adult BBSs in the United States and can only generalize to those networks and those individuals using those networks.

Quoting from a brochure which leading "adult" BBS around the country have distributed:

Log on through BBS Direct. Pay just $30 per month for unlimited telecommunications from any of the top 100 metro markets in the U.S. Stay on-line forever. Only $30 per month. No more long distance bills to fret about. Access BBS Direct via the Concentric network, a high-speed, easy to use data communication network. This state of the art fiber optic network allows error free connections at baud rates up to 14.4 Kbps.... In addition to BBS Direct, your subscription includes free unlimited access to the Internet.

p. 38, 4th graf
TIME quotes Rimm as saying, "We now know what the consumers of computer pornography really look at in the privacy of their own homes," ... "And we're finding a fundamental shift in the kinds of images they demand." However, the study does not reveal what consumers look at in their own homes (or anywhere else). The study

Ladies and Gentlemen:

A consumer turns his computer on, logs onto an "adult" BBS, types in his name and password, goes to the file section of the BBS, reads the descriptions of images, downloads a few images, pays money for this through a credit card or other means, and then chooses, having gone through all of the above repeatedly, according to Ms. Hoffman, not to actually view any of the images that were downloaded!

As much as this assertion misses the mark, it is still worth noting footnote 14, where the Carnegie Mellon study is careful to define "consumption" very precisely.

did not examine consumer behavior, but aggregate download counts of descriptive listings of images available on adult BBSs. Although

The research team has more than aggregate information, as is made abundantly clear on page 1895, "obtained either directly from the logfiles of "adult" BBS..." The logfiles upon which some of research is based are similar to the following:

 00:04 XXX [XXX/XX-XX] - Str Male-34 at 14400 bps on Sun, 01/01/95
      * 8 Databits:YES, Detected:ANSI, MNP:YES, CONF:8, Time Limit:90
      * Voice #: XXX-XXX-XXXX  Data #: One of Our .GIF
      * Verified User's Birthdate
      * File [XXXX.GIF] downloaded from (Area 18) via Zmodem (CPS = 1670)
      * File [XXXX.GIF] downloaded from (Area 10) via Zmodem (CPS = 1663)
      * File [XXXX.GIF] downloaded from (Area 10) via Zmodem (CPS = 1646)
00:24 Signed off NORMALLY.  Time Logged: 20 with 70 minutes remaining.

The "X" is placed to protect the identity of the user and BBS, a privacy commitment from which we have not wavered. This sample log tells us when the consumer logged on, which images were downloaded from which menu, and when the consumer logged off. These logs illustrated here are normally from a Wildcat! platform.

download patterns would be expected to correlate with viewing, we do not know the extent to which individuals actually *looked* at the images (or, indeed, whether they looked *at all*).

Additionally, the study provides absolutely no evidence for the statement that there is a "fundamental shift" in demand for certain types of images.

Several members of the Carnegie Mellon research team, including a psychologist with twenty years experience teaching human sexuality, believe that there is a fundamental shift because they read with care and cited the Dietz-Sears study noted in footnote 15. Please read that study of pornography in "adult" bookstores and consider whether you agree that pornography available on computers differs from pornography available in a typical "adult" bookstore.

p. 38, 5th graf
TIME says, "There's an awful lot of porn on-line." But in fact, Rimm's own figures suggest that the amount of pornography on Usenet and the World Wide Web represents an extremely small percentage of the total information available on the Internet. TIME further neglects to clarify this by noting that the vast bulk of Rimm's study concerns files that reside exclusively on "adult" BBSs, which is a very minor portion of "on-line," and which does not include the Internet.

These BBSs at times do include the Internet, as cited from the brochure above in footnote 27. These files definitely do not "reside exclusively on "adult" BBSs," because we found thousands of pornographic images on the Usenet which originate from these "adult" BBSs, some of which are now (a year after the data collection) connected to the Internet in any case. We knew that they originated from "adult" BBS because we clearly identified their logos and telephone numbers on the actual images. See p. 1867 and 1875.

Again, apart from a concerted effort to simply misstate the facts, it seems almost incomprehensible that a researcher with any expertise in this area could actually have read the study or examined the Usenet and drawn the conclusions Ms. Hoffman and Mr. Novak do here.

TIME then supports this quote by saying that "917,410 sexually explicit pictures, descriptions, short stories and film clips" were "surveyed." However, the 917,410 files do not represent porn on-line, as all of these 917,410 images came from adult BBSs.

Ms. Hoffman and Mr. Novak seem to be suggesting that a unique definition of "on-line" as limited exclusively to the Internet. Users of America-on-Line, Compuserve, Prodigy, as well as thousands of other BBS throughout the country, consider themselves "on-line" totally apart from any access they may have to the Internet.

None of these 917,410 files came from Usenet or the Internet.

It is true that all of the images originated from "adult" BBS, but at least a few thousand of these images over a four month period made their way to the Usenet and other networks. The Carnegie Mellon study never purports to conclude that any of these files "originate" from the Usenet. What the Carnegie Mellon study makes clear is that it does examine the availability of pornographic imagery on the Usenet.

Rimm states that of the 917,410 "descriptive listings," 450,620 with complete download information came from 68 different adult BBSs, 75,000 with partial download information came from 6 different adult BBSs, and 391,790 with no download information came from 27 different adult BBSs.

Further, of the 917,410 files, all text and audio files were deleted

They were not "deleted" whatsoever. They were not considered or reported because the focus of this article was to report the results which pertained to images. In the spirit of genuine academic scholarship and debate, Ms. Hoffman should have first inquired further about our methodological procedures before making such baseless assertions.

from analysis, only a very small number of images were actually examined,

A good statistician knows that in order to analyze a vast quantity of data, one develops a methodology which takes a sufficiently large sample in a random way. The Ph.D. statistician who advised us on this project indicated confidence that our sample size of 5,000 images was sufficiently large and in the case of "adult" BBSs, taken randomly. We report this quite clearly in the study's methodology section. There is no reason to believe that the actual images examined in the sample set differ in any respect from those that were not. Finally, to suggest that categorizing 5000 images as a "very small number" is a highly subjective assertion.

and the actual number of descriptions of images retained for the content analysis on which the study's conclusions are based was 292,114.

In comparison with the 917,410 pornographic files located on the "adult" BBSs, how many pornographic images did Rimm locate on the Usenet? Rimm states: "Between April and July of 1994, the research team downloaded all available images (3254)...the team encountered technical difficulties with 13% of these images, which were incorrectly encoded or incorrectly uploaded by the poster. This left a total of 2830 images for analysis." Thus, while 917,410 pornographic files were found on adult BBSs, only 2830 pornographic images were found on the Usenet!

According to the statistics compiled by Brian Reid (see table 2, #10), 260,000 people read alt.binaries.pictures.erotica each month. Thus, the number of images found on the Usenet on the five newsgroups considered is a different issue from the number of people who may access these images. The Carnegie Mellon study also takes great care to point out that only 3% by message count of the Usenet is associated with newsgroups which contain pornography. See p. 1868. Of those images found on the Usenet, 71% originate from "adult" BBS. See p. 1874. It is also worth noting that while one BBS may regularly serve a clientele of only a few hundred or thousand, the Usenet is available to, and is consulted by, far more people.

In addition, out of 11,576 World Wide Web sites in December 1994, Rimm found only nine Web sites, which is only eight one-hundredths of one percent, contained R or X-rated Adult Visual Material. Time's statement that "there is an awful lot of porn on-line" is thus blatantly misleading and irresponsible.

It is not. At a minimum, it is worth noting that many Web browsers enable consumers to access Usenet newsgroups. To many, especially non-technically oriented consumers or researchers, the difference between these two protocols may not be entirely clear. See footnote 39, p. 1869, which discusses this difference and provide some clarity.

p. 38, 5th graf
TIME says that 83.5% of images in Usenet binaries groups are pornographic; however, this number is simply incorrect. What Rimm actually wrote (p 1867) was "Among the pornographic newsgroups, 4206 image posts were counted, or 83.5% of the total posts."

The statement in the Carnegie Mellon study is correct. The total number of posts, pornographic and non-pornographic, was 5033. Of these, 4206 images were found on newsgroups associated with pornography, 827 were not associated with these newsgroups.

This is based upon 17 alt.binaries groups that Rimm considered "pornographic" and 15 alt.binaries groups that Rimm considered "non-pornographic." However, Rimm does not provide a listing of the names of these groups, so there is no objective evidence of whether these groups are, in fact, "pornographic."

Also, no information is provided on the degree to which these 32 groups comprise the complete universe of Usenet imagery.

Some background information resolves this "issue." Under the guidance of faculty advisors, several Carnegie Mellon research assistants spent a great deal of time last year navigating the Usenet newsgroups for imagery, pornographic or otherwise. They were not able to find many newsgroups outside of the alt.binaries hierarchy which contained images. Any time they did, they included such a newsgroup in our sample. It was quite clear that, at least at the time the data was collected, the vast majority of the images on the Usenet were posted to the alt.binaries hierarchy. A significant factor is part of established "netiquette"--one should not post images to a discussion group. A careful researcher would have to provide evidence, based on an extensive search, that a statistically significant number of images were to be found outside of the alt.binaries hierarchy in order to draw the conclusion that the areas of image posting has now changed.

Further, as the methodology for counting the number of images is not specified,

We were unaware of any fancy methodology necessary for counting images. We used a calculator.

it is likely that even given Rimm's definitions and selection of 32 groups, the percentage is inflated due to the inclusion of non-pornographic text comments and multi-part images in the counts.

"Multi-part" image counts are a two way street: one would have to cede that just as many images on the non-pornographic newsgroups are comprised of "multi-parts;" one would also have to cede that just as many text comments are found on non-pornographic alt.binaries newsgroups as pornographic ones. We, in fact, checked, and found this to be the case.

To make matters worse, Rimm overgeneralizes his results in his summary (p 1914): "83.5% of all images posted on the Usenet are pornographic." This is a particularly misleading misinterpretation.

It is not. Ms. Hoffman and Mr. Novak would have to show that the seven day period during which the posts were counted was atypical of other weeks. There is no evidence which would suggest this. They also would have to argue that a seven day sample period is too small, and that a sample period of, for example, 30 days would yield significantly different results. They have not done so in their "critique." Or again, another possibility would be to show, at the time the data was collected, a significant number of images were available on Usenet newsgroups outside of the alt.binaries hierarchy.

p. 38,40, 6th graf
TIME says that '[t]rading in sexually explicit imagery, according to the report, is now 'one of the largest (if not the largest) recreational applications of users of computer networks.'" But there is no evidence for this statement as Rimm's study does not examine "trading behavior" on Usenet news groups, only aggregate *postings*.

It is not clear what Ms. Hoffman means by "trading behavior." One notes from her direct quote, however, that the Carnegie Mellon study was careful to prefix the term "applications" with "recreational." One would need to identify recreational genres on the Usenet which are notably larger than pornography. To dispel this idea, one merely needs to examine Brian Reid's statistics.

One suspects that Ms. Hoffman and Mr. Novak may be reading something into the word "trading" other than mere posting. When we used "trading", we meant someone posting a porn image for others to read on Usenet, and one or more of these others posting additional images in return. We used "trading" to describe that phenomenon. Ms. Hoffman and Mr. Novak probably believe that the word "trading" implies an exchange relationship between the parties which cannot be inferred from the mere fact of posting. If you post images (e.g. as advertising) with no expectation of having someone post others in return, it is probably inappropriate to describe what you have done with the word "trading", since there was no exchange relationship intended. What Ms. Hoffman and Mr. Novak may be saying is that one has no way of inferring whether the posts were part of an exchange relationship and therefore whether the observed phenomenon, posting of images, constitutes "trading in sexually explicit imagery."

'Trading behavior' is indeed a complex social phenomenon. When we used the word trading in the article, we were using it in an informal way to describe the posting of sexual imagery on Usenet. Many of the posters preface their contributions with comments like, "Thanks for the neat GIF from xxxx. Here's another one." from which we took the liberty of using the word "trading" to refer to the overall phenomenon of posting of pornographic images.

p. 40,first full graf
TIME says that the "great majority (71%) of the sexual newsgroups surveyed originate from adult" BBSs, "whose operators are trying to lure customers" to those boards. This percentage is unsubstantiated as Rimm provides *absolutely no support* for it.

The Carnegie Mellon study makes it quite clear exactly how we determined if an image posted to a Usenet newsgroup originated from an "adult" BBS. See page 1867 for a detailed explanation.

Further, no evidence is presented that operators are engaged in luring customers to the adult BBSs via Usenet newsgroups.

Again, we made it quite clear in the Carnegie Mellon study how we knew this. See page 1910, "A number of "adult" BBS have reported that their subscriptions "skyrocketed" after posting a few images on the Usenet." See also generally pages 1875-1876. We spent 18 months discussing these issues in detail with dozens of operators. We were (and remain) happy to explain this further to either Ms. Hoffman or Mr. Novak.

p. 40, third full graf
TIME says that "there is some evidence that ... the 1.1% ... women [on BBSs] are paid to hang out on the 'chat' rooms and bulletin boards to make the patrons feel more comfortable." But in fact, Rimm provides *no* evidence for this supposition (nor any credible evidence that there are 1.1% women and 98.9% men).>

Many of the "adult" BBS operators with whom we have spoken make it clear that at least some of the women are paid to be there. This is what we accurately reported in the Carnegie Mellon study. What Ms. Hoffman and Mr. Novak apparently mean to say is that for privacy and ethical reasons we did not elaborate on how we managed to obtain the demographic data from a particular "adult" BBS. See page 1895, "various methodologies developed by research team programmers." See also footnote 88. This is the case because we are committed to protecting the privacy rights of these consumers and we consulted well respected privacy experts at Carnegie Mellon on this issue. Without doing so, future research in controversial areas such as this may be impossible. This would be a serious loss to the broader research community as a whole.

p. 40, fourth full graf
TIME says that demand in the adult BBS market is driven by images that "can't be found in the average magazine rack." Yet, Rimm did not study the existence, availability or extent of "analog" pornography, so no such conclusion is warranted, nor possible. Further, Rimm's study, due to methodological flaws,

Ms. Hoffman and Mr. Novak have yet to identify a single methodological flaw. Also, for those concerned about analog pornography issues, see comments above about our comparison with Dietz-Sears study data.

does not demonstrate the demand for such images (over and above other types of images) on adult BBSs.

This is plainly, patently false. One of many examples, Figure 7 of the study shows the availability-demand characteristics for soft-core, hard-core, pedophilic and paraphilic pornography.

p. 40, first column, last graf
TIME says that this material appears on a "public network accessible to men, women and children" globally, yet as stated above, there is no evidence that material from private, restricted-access adult BBSs ever makes its way to public networks like the Internet.

It is puzzling how Ms. Hoffman or Mr. Novak could have actually read the Carnegie Mellon study and walked away with this completely false impression. The Carnegie Mellon study made it quite clear that 71% of the images identified on the Usenet originate from "adult" BBS. A skeptic can go to Usenet newsgroups which contains pornography, download a few dozen images, and confirm for themselves that some of them originate from "adult" BBS. It is not difficult nor, until this attack, did it even seem remotely controversial to report.

p. 40, second column, first full graf
TIME reports that "only about 3% of all the messages on the Usenet newsgroups [represent pornographic images], while the Usenet itself represents 11.5% of the traffic on the Internet." But TIME neglects to take the interpretation to its logical conclusion, which is that less than 1/2 of 1% (3% of 11%) of the messages on the Internet are associated with newsgroups that contain pornographic imagery. Further, of this half percent, an unknown but even smaller percentage of messages in newsgroups that are "associated with pornographic imagery" actually contain pornographic material. Much of the material that is in these newsgroups is simply text files containing comments by Usenet readers.

The calculation Ms. Hoffman and Mr. Novak propose would mix apples and oranges. The figure 11.5% was the percent of NSFnet backbone traffic in bytes. Not all traffic on the Internet is in the form of "messages" and thus multiplying a percentage of messages times a percentage of bytes would be logically incoherent. The Carnegie Mellon study in fact points out that the set described as 3% of Usenet messages is 22% of Usenet bytes. The Carnegie Mellon study then multiplies these two numbers, both percentages based on byte count, and reports that only 2.5% of bytes sent on the Internet backbone were sent as part of newsgroups which carry pornography. One reason that these image carrying groups represent only 3% of messages but 22% of bytes is because images tend to be much larger than text messages. Thus, the bulk of the 2.5% of bytes are image bytes, not the additional text messages also carried in these newsgroups. Understanding the difference between a message and a byte is essential to understanding this distinction.

p. 40, second column, 3rd full graf
TIME speculates that pornography is "different" on computer networks, and although the Rimm study suggests this, as well, absolutely no evidence is presented to support this hypothesis.

p. 42, third column, second full graf
TIME wonders "[h]ow the Carnegie Mellon report will affect...the cyberporn debate" and notes that "[c]onservatives...will find plenty" of "ammunition." Yet TIME fails to note that the "Carnegie Mellon report" is in fact a sole-authored study by an undergraduate student in Electrical Engineering that was not subjected to the usual rigors of peer-review and revision that are common for this type of research.

It is peculiar that two business faculty from a well-known university would invest considerable effort in a non-reviewed, rushed manner attack a study of computer pornography written by a person with a degree and an advanced background in electrical and computer engineering, published by a well respected law journal and discussed in the popular press. Ms. Hoffman attacked the Carnegie Mellon study in a Washington Post article without ever having read it. The study "is very misleading," and "doesn't tell us anything about consumer behavior," she is quoted as saying. The "portions" of the study that Ms. Hoffman apparently saw referred to a six month old, two page abstract. It is also apparent that certain advocacy groups believe that having solid, objective data in the public arena is contrary to their agendas. This is to be expected in any heated public policy issue. No matter what the findings, no matter how refined and exemplary the methodology, at least some advocacy group would have found fault with the Carnegie Mellon study. What was breathtaking was the willingness of an academic to dismiss a study she had never read. Again, this is certainly her right, but it is not scholarship.

Further, Ms. Hoffman's policy comments, which she is certainly entitled to, have nothing to do with the rigorous methodology we pursued. Her "critique" also seems to be an attempt to argue that while Marty Rimm authored the Carnegie Mellon study, no one else participated in the collection of data, careful review of methodology, and careful presentation of analysis. As the study's first footnote makes clear, this was an interdisciplinary effort of many people with widely diverse positions. Many of the professors who participated in the Carnegie Mellon study hold advanced degrees in their specific areas of expertise from MIT, Stanford, Berkeley and Carnegie Mellon. It is perplexing that those attacking the methodology of our research efforts at Carnegie Mellon have their primary expertise in either the law, market research, or other unrelated disciplines.

Seemingly ad hominem attacks on an author are also no substitute for examining of the merits of a work. These "critics" had less than a week to read and prepare an attack on a study whose data was gathered and analyzed over an 18 month period. It is not surprising, as is plainly evident, that the attacks miss the point simply because they do not understand the methodology or have not read the Carnegie Mellon study with the care necessary at this point.

p. 42, third column, fourth full graf
TIME notes that "1 million or 2 million people who download pictures from the Internet represent a self-selected group with an interest in erotica." Yet, this 1 to 2 million number is completely fictitious and unsubstantiated because it is not known *and it is not possible to know* how many people download pictures from the Internet. Time provides no reference for this figure, and the figure itself is not mentioned in the Rimm report.

p. 42, third column, last graf
TIME suggests that Rimm's study will be a "gold mine for psychologists, social scientists, computer marketers and anybody with an interest in human sexual behavior." Yet TIME fails to note that it is highly unlikely (at least without a cover story by Time) that an unsophisticated, poorly executed, weakly documented study

These are rather harsh terms which have not at all been substantiated by anything stated in this "critique."

conducted by an undergraduate in electrical engineering that was

This is the third time Ms. Hoffman invokes the term "undergraduate." Frankly, many people here are impressed that an "undergraduate" managed a team of more than two dozen researchers at Carnegie Mellon University, was published in one of the more prestigious law journals in the United States, and had the study he authored featured on the cover of TIME. Unfortunately, because pornography evokes such strong, often irrational, passions in people, this accomplishment has been overlooked.

not published in a rigorously peer-reviewed scholarly behavioral science journal would be ever be perceived as a "gold mine" by experts in these areas.

Ms. Hoffman and Mr. Novak are overstepping their bounds. First, their training is in psychology and their claimed expertise is as market researchers, not as sociologists, historians, or experts in human sexuality or pornography. In fact, a well known historian and an English professor, both of whom have published widely cited works on pornography, were impressed by the Carnegie Mellon study and may choose to comment on it in the next year. Second, Ms. Hoffman and Mr. Novak do not know how scholars in other disciplines will receive the Carnegie Mellon study or utilize its methodology. No one does. It is worth noting that three well respected law professors, each analyzing from a different perspective, praised the Carnegie Mellon study. Professor MacKinnon called the Carnegie Mellon study a "landmark" (page 1959). Professor Meyer said the Carnegie Mellon study "makes an important and original contribution for all those interested in examining sexuality in American and Western culture" (page 1969, footnote 3). Both professors MacKinnon and Meyer have published before in scholarly journals on pornography; Ms. Hoffman and Mr. Novak, to the best of our knowledge, have not. Finally, Professor Branscomb, a well known information and technology scholar (see Who Owns Information?, Basic Books, 1994), commended the Carnegie Mellon study for "offering a methodology for the academically rigorous tracking of pornographic images" (page 1957).

Curiously, Rimm has been surprisingly uninterested in making the study available to such experts.

We have solicited the advice of numerous experts. See first footnote, second paragraph. We did not solicit Ms. Hoffman's or Mr. Novak's advice because we did not consider them experts in either computer pornography or the law. In our hundreds of discussions with other researchers, neither was mentioned as worth consulting as an expert in this area.

The study was embargoed for at least six months prior to publication in the Georgetown Law Journal. Scholarly researchers who requested a copy of the manuscript from Rimm were refused access to the manuscript prior to publication.

p. 43, top graf
TIME says that the "more sophisticated operators were able to adjust their inventory and their descriptions to match consumer demand," yet the Rimm study provides very little evidence that this is actually occurring except in isolated incidents.

Ms. Hoffman and Mr. Novak do not explain what makes an incident "isolated." We identified several "adult" BBS operators with technical backgrounds who adjusted their inventories to more accurately reflect consumer demand. See footnote 109 for citation.

More than two dozen researchers with a technical background at Carnegie Mellon offered their expertise and full support to this project. Moreover, of the thousands of manuscripts submitted each year to the Georgetown Law Journal, only a few dozen are accepted for publication, and the Carnegie Mellon study was one of them. Independent scholars reviewed the manuscript and praised it strongly. TIME Magazine read the manuscript and thought its merits so solid that it warranted a cover story.

No where in their "critique" do Ms. Hoffman and Mr. Novak have a single positive thing to say about the Carnegie Mellon study. They write with anger, and often without thinking things through. Do you really believe that it is even possible, given the number of respected scholars which enthusiastically supported the study, that there is absolutely no merit to it? One might reasonably ask what Ms. Hoffman's and Mr. Novak's real agenda is in rushing into publication--without peer review--such an overwrought "critique"?

Last modified: Sun Jul 16 22:15:57 EDT 1995