Philip Pichotta
September 1, 1998

Comments on the ASA Ethics Guidelines - III (27 August 1998 draft)

{Dr. Gardenier's Note: this is a thoughtful and extensive commentary. The only way to deal with it is to interrupt Dr. (?) Pichotta's remarks to respond to each item in its place.}

I think people need to take a couple steps back and look at the guideline and identify items that actually refer to ethics. There is little substance about ethics in the document. Much of it deals with professional conduct and other areas that are already covered by laws and regulations. Response: While understanding that some readers will agree with you, that has not been the majority response. We feel the material in the guidelines is almost all ethics. Where we address good professional citizenship, it is labeled as such. We do not address aspects of professional conduct or law and regulations which we do not find to have compelling ethical rationale.

I think something less than 10% of our members live outside the U. S. Do we need them to follow our laws and regulations so that they are ethical? I question the need of much of the current guideline. Response: They need to follow ethical guidelines whether or not those are implemented in laws and regulations that apply to them. Most commenters have asked us to expand the guidelines in successive drafts, not to narrow them.

Can you start with examples of unethical conduct and then address how we should be ethical? I can think of some examples that I have seen. Response: No, we start from what is required to perform statistics ethically and then apply that to case studies. You are cordially invited to post case studies of your choosing on our web site.

On of the problems with the guidelines is so what if someone is unethical. Will we have a tribunal and drum them out of the ASA? I don't think so because the resulting litigation would be too costly to the ASA. Response: We hope that the guidelines will be widely accepted such that individuals who perform statistical studies unethically will be recognized as doing that. The controlling factor should be their professional reputations and their concern for their reputations. What we do not want to happen is to have the guidelines misused to address honest differences of opinion among competent, ethical practitioners.

Why would I want an ethics guideline? I want clear language that says it is wrong to fabricate data, falsify results, or bias the results to achieve the desired results. Having some clear statements might help me better deal with a client or employer. Response: We can consider adding words like "fabrication", "falsification", "plagiarism" and so forth in the guidelines. In practice, however, these have sometimes been far from "clear." One can avoid all of those and still be professionally unethical in many other ways.

I.C.1 I am not sure the intent of the statement. Does it mean it is okay to do a poor analysis if you believe that the results of the analysis has little social value. Shouldn't we always try to perform an appropriate/valid analysis? Response: Yes, but most of us cannot devote equal time and attention to every task we approach.

I.C.2 I think the language is very obtuse and could be much clearer. I believe we should be searching for the truth. We may believe that the we know the answer but need to be open to the assess the results fairly. Response: True, but that is not what we are getting at here. People use statistics unethically when they manipulate it to prove a predetermined point rather than to seek the truth. This is not uncommon in some areas of science, including biomedical research. If the goal is to determine whether a new drug is effective, fine; if the goal is to "prove" that it is effective by tweaking the analysis until the p-value is driven below .05, that is not fine.

I.C.3. I don't know what Statistics as a Science means. Are you trying to say there may be valid differences of opinions in the selection of appropriate statistical analyses? An analysis considered as appropriate 5 years ago may no longer be considered as appropriate today. Is this a question of ethics or incompetence? Response: All three - incompetence, ethics, and honest differences of opinion. We have at least three schools of thought in statistics today - frequentists, Bayesians, and some add "robustniks" - plus those of us who feel each approach is valuable in its place. We have many open issues about the logic and validity of hypothesis testing as well as about the "best" distribution to use in specific applications. One very experienced statistician/ethicist urges us to take out the word "best" in II.B.1. because statisticians often disagree what the best approach is in any given situation. (It is easier to agree about what approaches are unacceptable.)

I.C.5 and .6. Is it necessary to have these statements in an ethics statement? While these generally are politically correct, I think we should keep our guidelines focused on professional ethics. Response: They are in there on the basis of "Above all, do no harm." The argument has been made that the public and the rest of science is properly contemptuous of statistics because we (or too many of us) are aloof, non-communicative with the public, incomprehensible to nonstatistical colleagues, and far more interested in arcane technicalities than in coping with real world problems and messy data. We chuckle to ourselves about gross misuses of statistical arguments in the press or in Congress, but usually we do nothing about it. The document separates "good citizenship" from "ethics", but we really advocate both.

I.C.7- 10. If these are not ethical obligations, then I would suggest that they be deleted. How important is it for us to expose dishonest or incompetent uses of statistics? Should this be one of our ethical obligations? Response: No, it is clearly labeled as a "good professional citizenship" issue.

II.A.1. I understand the first sentence but not the rest. Is the second part a sentence or a bunch of phrases? If protocols are to be clearly defined, this is a different item from practical and statistical significance. Make it clear. Response: The second sentence is composed of the verb "combine" followed by the items to be combined. If you can suggest clearer wording, we would be interested in considering it.

II.A.2. I think this statement is clear as mud. It seems like the statement is trying to say to avoid data selection processes that will provide results biased to the conclusion desired. Response: Well, you seem to understand the "mud" adequately. The wording would be simpler if we only had to confront deliberate misuse. We also need to address the fact that various people involved in the research can harbor unconscious biases. A number of scientists have pointed out that such bias is not rare; it is the rule rather than the exception. Therefore, we need to take explicit steps to defeat our biases, and we need to explain those steps in our publications or testimony.

II.A.3. I don't know what ‘to valid results' means. Suggest a rewording. Response: It says, "suitable . . . to valid results." What rewording would you prefer?

II.A.4 Those of us in industry do not often have the luxury to chose which projects to work on. Most often we are assigned projects and don't have a choice unless we change employers. What are my options in industry? Either delete the point or make it applicable to industry to which a large number of statisticians belong. The use of the term ‘valid results' reminds me of the quote that all statistical models are wrong, but some are useful (or something similar). A paraphrase of that is that all results are invalid but some results are useful (less invalid than others). I don't think there is a dichotomy between valid and invalid. If you only join projects which expect valid results, then you don't have to worry about your name being used. Seems like something is missing from the last part of that point. Why should I ‘feel assured' rather than ‘be assured' that my name won't be used. Why not make a statement in the ethics that the statistician's name not be used in any report or publication without his/her expressed consent and the statistician has the right to withhold his/her consent without reprisals. Response: A.4. uses the criterion of "explicit consent." H.6. implies without reprisals. As to your larger point, A.1. emphasizes the criterion of practical significance, which is usually what industry demands. If a boss told a statistician to perform a market research study of the potential demand for stagecoach wheels, what should she do? Unless she understood how to give him information of value to the business, why bother? If the boss is contemplating meeting a niche demand for stagecoach wheel replicas to be made into coffee tables or card tables, then perhaps she can provide valid and useful results. You should "feel assured" rather than "be assured" because you can trust some people to follow this rule rather than having to address it with them in any specific case. If you do not "feel assured," then you may need to take further steps in order to "be assured."

II.A.5. At the end, delete ‘in each statistical study' as it appears to be unnecessary. I am not clear how this guideline applies relatively common situations where people who may perform an automated analysis of data but who don't understand the theory of the method they are using. The automated analysis may have been designed by statisticians but sometimes is not. Is this considered ethical? Response: Not unless the user understands how the software works. Just because you understand, say, stepwise regression, that does not mean that you are correct in using given stepwise regression software in your current study. All we are saying here is that statistical practitioners have an obligation to know what they are doing in each analysis they perform. Is that an ethical issue? You bet your sweet donkey it is!

II.A.8. I don't understand what being proud of our expert testimony means. It doesn't seem like being proud of something equates with being ethical. Response: This is clearer to those who have performed as expert witnesses. There are a lot of pressures in such situations to tell half-truths, to omit key facts, and to misrepresent the "state of the art." In fact, one of the best uses of the old guidelines document - and this new one - is to show it to lawyers to explain why you will not do some of the things they may ask you to. Good lawyers will love it; they can use it against the other side!

II.B.1. While it is good in certain situations to provide alternative statistical approaches, I may not offer these alternatives in every situation. I don't think that I should be considered unethical for not providing alternative analyses. Response: Ultimately, of course, the application of any ethical guideline involves subjective judgment. For example, I could understand that one might have a standard protocol in a factory for performing routine quality control sampling and analysis to be performed weekly or monthly. It would be foolish to analyze alternatives for each specific application. The committee is wrestling with item B.1. now - and your comments will factor into the mix. Basically, however, we have no members willing to back off from the general principle. You may know of ways to get approximately the same results at a significantly cheaper cost, or to get much more useful data by spending only a few percent more on the study. If you do not analyze alternatives and give your boss or client a choice, you may be performing a serious disservice. You should guard against that possibility.

II.B.6. I am not sure why it necessary for statistical methods to remain in the public domain as an ethical statement. For practical reasons, I think that it may be necessary for the methodology to be published for it to be accepted. Response: Other commenters have stated that they should be able to keep a method they discover as a trade secret and use it to competitive advantage. If that were the general practice, statistical science would be unduly inhibited in its advancement, as would the sciences that depend on it. The result is that obtainable social benefits from the application of the new statistical methods would be delayed or lost due to a preference for personal greed. That is an ethical issue.

II.C.1. This statement states we have personal responsibility for only those publications which bear our name. What about analyses we have performed but are not listed as an author (or even acknowledged). Do we have an ethical responsibility to ensure that the results of these analyses are reported accurately and appropriate conclusions made?
Response: C.1. does not contain the word "only." You should always do your best. Your name is associated with the work you did by the people who know you did it, whether you get credit or not. As to the ethical obligation, see H.4.

II.C.3. What does this mean? What is the intent? Response: It applies mostly to research teams composed of people from different disciplines, not all of whom have statistical training. Its intent is to let reviewers and readers know something about the competence of the people who performed the statistical analysis.

II.C.4. I think this should be a policy for the journals and doesn't belong here. Response: In an ideal world, you might be right (although these guidelines are addressed to journal editors as well as statisticians.) In the real world, it is all too common for senior researchers to hog the credit for their subordinates' work. The subordinates suffer later when they are found to lack the publication base required for promotion or tenure. It is more common in academia than industry, but the ethical disease can be quite pervasive.

II.C.5. Delete the word ‘considered'. Response: Your full communication goes to the Committee. Anything which finds substantial support will be used. A problem with this suggestion is its failure to address the deliberate decision not to look at certain data because you believe it is likely to indicate some information contrary to the position you are trying to prove. The argument about the 2000 census is a case in point. Supplemental sampling is likely to result in finding more poor people in the U. S. than would enumeration. If you do not want to know that, you oppose sampling. You may oppose it for other reasons, too, but I hope you see my point.

II.C.6. Revise to: Report the source and adequacy of the data. Response: Most unlikely; it is stronger the way it is.

II.C.9. I am not sure that this is needed. If I make it clear that the results of a bioavailability study was performed in volunteers, there is no need for a disclaimer. If I analyze a study performed in a VA hospital and have most men, am I unethical to not put a disclaimer that the study has a higher proportion of men. A VA population may be different from the overall population in other ways as well. It puts us in a difficult situation with our clients to present results of a study and then add a bunch of disclaimers saying that the results may not be real or applicable to the over population. Some of it is needed but I don't think this item is needed. Response: You may misunderstand; the Veterans Administration (VA) hospital population IS a "defined population" and your results may well be representative of that population. I suspect that you identify a random sample in advance, approach them all, and actually use only those who agree to participate - volunteer. This guideline discourages use of volunteer data (say, writers of letters to the Dear Abby newspaper column) as being representative, say, of the U.S. population. It also discourages any claims that the low rate of hysterectomies in VA hospitals proves that the procedure is overly performed nationwide. Obviously, you would not do that. Good.

II.D. I think there are problems with this section. In the pharmaceutical industry, we design many clinical study protocols but the studies are actually conducted by investigators who are contracted by the company. It is the investigators' responsibility to protect research subjects (by US law or Regulation). It is not even the pharmaceutical companies responsibility. I think many statisticians know some of the regulations about informed consent, I don't think that we should be required to know and understand all the nuances of these regulations. This is clearly not our job and detracts from our job about designing and analyzing the best trials we can. If we have any ethical concerns, we raise them with our clinicians who assess our concerns and make decisions about them. Other than protecting privacy and confidentiality of research subjects, I recommend that the entire section be deleted. Response: You are entitled to your opinion. I can only say that the majority of comments we have received on this section have led to making it more extensive and tougher than originally drafted. Michael O'Fallon, of the Mayo Clinic, to be President of ASA in 2000, also made your point about statisticians not being the appropriate people to "obtain" informed consent. We changed that. Still, the Nuremberg principles apply to everyone involved in research on human subjects. Say that you had performed a safety and efficacy analysis of clinical trial data involving a medication which had produced disabling side effects or death in a small percentage of the laboratory animals on which it was previously tested. You publish results finding the medication is much too dangerous for general use. Would you be comfortable if you later learned that the human subjects in the trial had not been informed of the risks? Hopefully, you would be outraged to be associated with that project. To avoid that possibility, you should assure yourself about the human subject protection (hsp) measures used before you examine the data. Does that mean you have to obtain and read a copy of each study's documentation? Not necessarily. If the study was performed by a group known to have rigorous hsp protocols, training, and supervision, you may make an informed judgment that those protocols were also used in that specific set of trials.

II.D.3 Seems to conflict with II.C.11 (sharing data). Is this statement needed because of Federal laws protecting privacy and confidentiality of research subjects? Response: Good point. Yes, there is a tension between those items. There are various federal, state, and local laws governing privacy, but there is also an ethical issue. If Mary confides to Susie that she had an affair, is Susie ethically entitled to announce to the world that Mary had an affair? Many of us think not. Does it make a difference if Susie is a professional survey interviewer conducting statistical research? Yes, because betrayal of the confidence does not further the research; there is no reason for Mary to anticipate such a thing. Apart from law, this is a professional ethical imperative.

II.D.5. I presume that the use of a double-blind technique is not considered deception. If it is, then I would delete this statement. At times we have deliberately deceived investigators. In one case, patients were being randomized to 2 treatments. What we didn't tell the investigator was that the randomization was very much imbalanced. We were attempting to obtain a clean estimate of adverse event rates. Another study used deception in selecting when a washout period ended and active treatment began. In both these instances, the investigator's and patient's actions affected the response because of their expectations. Response: Generally, no, double-blinding is not deceptive if it is announced. Those experienced with clinical trials on the Committee are not necessarily comfortable with the examples you provide. We feel it is ethically necessary to be truthful with both patients and investigators. That extends to not misleading people as well as not lying to them. The concept of "informed consent" applies here - although you may have less need of acknowledged consent from investigators than from patients. If patients and investigators know in advance that randomization may be imbalanced; if patients and investigators know in advance that they will not be informed as to when washout periods end and active treatment begins, fine. If either group has reason to believe otherwise (or would normally be expected to assume otherwise) and you fail to inform them, we would consider that unethical. We consider both validity and ethics to be imperative and not to be necessarily in conflict. We will try to clarify this in the guidelines. Thank you for bringing it up.

II.E. I view these items as being a professional rather than having any ethical content. Response: The alternative is to condone unethical or incompetent research as long as you can blame it on somebody else in your research team.

II.F. Again, I view these items as definition of being a professional and are don't consider them to have any ethical content. Response: Some of these are also in the 1989 Ethical Guidelines document. I guess the Committee on Professional Ethics does not regard professionalism as being cleanly separable from ethics - never has, hopefully never will.

II.F.1. In many cases, the data would be considered as proprietary and therefore would not be possible to share data. Response: True, and we cover that elsewhere.

II.H. Other than Item 6, I don't know if I believe the others are necessary. The employer should have a mechanism to investigate cases of unethical conduct/misconduct so that the statistician can have his observations investigated. Response: The principal here is that ethics arises from a moral agent operating in a moral environment.

I guess after reading this guideline, I am not sure this will help provide guidance to a professional statistician in accessing his/her ethical responsibilities. I think the code implies that incompetence is also unethical. In many people's eyes that may be true. But the question is the person's intent. Did the person intend to deceive his client or audience by performing a specific analysis when the person knew the analysis provided improper results. Response: It will help some, but not others. It only addresses competence to the pont of discouraging wilful incompetence and lying about one's competence. We have tried hard to avoid the "intent" criterion because it is horribly misused. "Yes, I stole my student's dissertation protocols, got a federal grant using her ideas without crediting her and faked some data, but I didn't intend to do any of those bad things." "Oh, that's ok then." This approximates a recent case at a university.

In no place did I find a clear definition of unethical conduct. I don't find anything indicating it is wrong to fabricate data or falsify results. I don't find much in it to indicate that it is wrong to select data or analyses in an attempt to deliberately deceive a client or audience. Response: A.1. through A.3 cover this, but we will consider your words.

I find much of the guideline describing professional conduct rather than ethical concerns.
Response: Professionalism and ethics put demands on each other.

No where is there any guidance about disciplining statisticians who perform unethical conduct or misconduct. Response: This is a policy document of the American Statistical Association; we have no authority or capability to do that.

The problem faced with writing an ethical guideline is that statisticians are a very heterogeneous group of people. This problem is similar to trying to certify statisticians that was discussed a few years ago. Response: We have revised wording in many cases to accommodate heterogeneity of statisticians; we will continue to do so as needs are brought to our attention.

The guideline seems to be written to imply that the statistician is not being unethical but needed to make sure that everyone else is being ethical. You need to define how a statistician can be unethical and then formulate your statements around them to define an ethical statement. Response: A statistician can be unethical by ignoring these guidelines.

Philip J. Pichotta
Senior Statistician
D436 AP9A/2
Abbott Laboratories
100 Abbott Park Rd
Abbott Park, IL 60064-3500

Replies by: John S. Gardenier, D.B.A., Chair, Committee on Professional Ethics, ASA.
Also Statistician, Centers for Disease Control and Prevention, National Center for Health Statistics, Room 1120, 6525 Belcrest Road, Hyattsville, MD 20782. Please address comments to: drgarden@erols.com

• Re: Comments on ASA Ethics Guidelines Phil Pichotta 18:06:07 10/02/98 (1)
  • Re: Comments on ASA Ethics Guidelines John Gardenier 16:36:51 10/06/98 (0)