30-Aug-86 12:32:50-PDT,13001;000000000000 Mail-From: NEUMANN created at 30-Aug-86 12:30:42 Date: Sat 30 Aug 86 12:30:42-PDT From: RISKS FORUM (Peter G. Neumann -- Coordinator) Subject: RISKS-3.46 Sender: NEUMANN@CSL.SRI.COM To: RISKS-LIST@CSL.SRI.COM RISKS-LIST: RISKS-FORUM Digest, Saturday, 30 August 1986 Volume 3 : Issue 46 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Human error (Nancy Leveson, Lindsay F. Marshall) Re: F-16 Tales (Earl Boebert, Phil Ngai) Correction to note about flight simulators (Martin Minow) Supermarket grinds to a halt (David Sherman) Video processing (Guy Schafer) ATMs (Jacob Palme) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM) (Back issues Vol i Issue j available in CSL.SRI.COM:RISKS-i.j. Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.) ---------------------------------------------------------------------- Date: Fri, 29 Aug 1986 18:48 EDT From: LIN@XX.LCS.MIT.EDU To: Nancy Leveson Cc: risks@CSL.SRI.COM, lin@XX.LCS.MIT.EDU Subject: Human error From: Nancy Leveson ... My real quibble is with the term "computer errors". Since I do not believe that computers can perform acts of volition (they tend to slavishly and often frustratingly follow directions to my frequent chagrin),.. While I agree with the above sentiment for the most part, it strikes me that there is a sense in which it is not true. Consider an situation in which a very large computer program is used to control a very complex real-time process. Imagine that the designers are generally competent, but do not understand very well certain aspects of the process or the environment in which the process operates. Something that they did not anticipate happens, and the software performs an action that results in a disaster. How are we to describe this occurrence? We can describe this as a design (and thus human) error. But given the circumstances, it is not implausible to argue that an "Act of God" occurred. It was a rare happening; we didn't understand it very well; we could not predict that it would happen. Is there a better definition for "Act of God"? While computers are in principle deterministic, a sufficiently complex program operating in a highly varied environment is for all practical purposes non-deterministic, in that its behavior is unpredictable under some circumstances. The problem with calling such an event an "Act of God" is that it leaves open the door to using that kind of defense when it is entirely unwarranted; I personally do not want to get away from the idea that human beings are (or should be) in control of their computers. I'm only pointing out that sometimes they may not be, through no "fault" of their own. Maybe computers should be used only in situations that we understand very well, and that are predictable. My safety instincts say that this should be true, but it's not clear that it is practical. ... the term "computer error" is also misleading since to me ... it seems to imply some sort of volition on the part of the computer as if it were acting on its own, without any human influence, to do these terrible things. While I agree it is misleading for the reasons I gave above, from the standpoint of being a user trying to behave defensively (as a driver drives defensively) it could be useful -- no user should approach a computer system as though its behavior is predictable and/or sensible under all circumstances, and in the absence of the system designers and programmers, that is probably an adaptive strategy from an evolutionary perspective. ------------------------------ From: "Lindsay F. Marshall" Date: Fri, 29 Aug 86 12:08:04 bst To: risks@csl.sri.com Subject: Re: Human Error Someone who has looked at the changing attitudes to "human error" as against "mechanical failure" is Michael Lesk. He has been studying reports of railway accidents in the UK to extract from them information about the attitudes of the reporters and investigators towards the causes of the accidents. I don't know if he has written this up anywhere or not, nor do I know if he reads RISKS. He is well worth talking to about the subject however and has uncovered some exceedingly interesting points. Lindsay F. Marshall [Will someone at Bell Labs who reads this please give Mike a nudge? PGN] ------------------------------ Date: Fri, 29 Aug 86 10:51 CDT From: Boebert@HI-MULTICS.ARPA Subject: Re: F-16 Tales To: Neumann@CSL.SRI.COM ReSent-To: RISKS@CSL.SRI.COM Weight on wheels is a basic sensor input that tells the flight program whether or not the aircraft is airborne. In advanced systems like the F-16 its is probably confirmed by air data computer and inertial platform inputs; in older systems, where the computer does just nav and weapons delivery, it is the prime indicator. It is therefore unlikely in the extreme that this would be overlooked in a design or an ordnance safety analysis ((weight_on_wheels = TRUE) & (master_arm = TRUE) & (weapon_release = TRUE) is clearly an undesired state). I am also skeptical that the gear would be controlled by the flight computer, but I am not familiar with the F-16 so cannot comment further. ------------------------------ Date: Fri, 29 Aug 86 19:57:30 pdt From: amdcad!phil@decwrl.DEC.COM (Phil Ngai) To: risks@csl.sri.com Subject: F-16 software It sounds very funny that the software would let you drop a bomb on the wing while in inverted flight but is it really important to prevent this? Is it worth the chance of introducing a new bug to fix this very minor problem? Is it worth the chance of making the code too big to fit in memory? What is the chance that a pilot would really make this mistake? [The probability is clearly NONZERO. It is very dangerous to start making assumptions in programming about being able to leave out an exception condition simply because you think it cannot arise. Such assumptions have a nasty habit of interacting with other assumptions or propagating. PGN] ------------------------------ Date: 29-Aug-1986 1406 From: minow%regent.DEC@decwrl.DEC.COM (Martin Minow, DECtalk Engineering ML3-1/U47 223-9922) To: risks@csl.sri.com Subject: Correction to note about flight simulators In a private mail exchange, Danny Cohen ("COHEN@B.ISI.EDU") was kind enough to point out that I had mis-remembered my article from Smithsonian where I claimed the article stated that a flight instructor flew as a flight engineer on a commercial flight. > The plane encountered a wind-shear situation on take off. The > instructor, from his flight engineer's position, reminded the pilot > that the correct recovery for wind-shear is opposite to the correct > recovery for a stall (which has a similar appearance to the pilot)." According to Danny (I can't find my copy of this issue), the article does not talk about anything being "opposite to the correct recovery for the stall." I'm sorry for the confusion this might have caused anyone. At least, I did learn a lot about flying and recovery from dangerous conditions. Danny did ask me to clarify my purpose in submitting the article to RISKS -- whether it was to show that computer-based simulators contribute to airline safety, or to "highlight the risks in using computers for whatever purposes." To set the matter straight, it was to show that computer-based simulation is a factor in increased airline safety, as it lets pilots learn about situations that are either dangerous or unusual (or both) in real life. Danny is still looking for pointers to accidents caused by computer-based simulators. Martin [I don't mean to take a potshot at Martin, who has been a delightful contributor. But PLEASE, all of you, if you see something that you think is appropriate for RISKS, make a note of it at the time rather than subsequently half-remember it. I keep a huge stack of old items next to my terminal just in case I have to dig back... PGN] ------------------------------ From: mnetor!lsuc!dave@seismo.CSS.GOV Date: Fri, 29 Aug 86 17:04:53 edt To: mnetor!seismo!CSL.SRI.COM!RISKS@seismo.CSS.GOV Subject: Supermarket grinds to a halt Last week I went to our local Miracle Food Mart supermarket (in northern Toronto) at 9 a.m. on a Sunday, when they were just opening. They discovered that they couldn't get any of the cash registers to work; something was down in the central system. So they had the cashiers writing each number down on a pad of paper and totalling them up by hand, which slowed checkout down to a crawl. After a while, someone found a desk calculator with a paper tape, which made things a bit faster. When I left they had someone at the door warning customers not to bother coming in because the terminals weren't working. Obviously, this kind of thing can happen only where cash registers are no longer cash registers but terminals connected to a central system, which is becoming more and more the case. I can't believe MFM doesn't have some type of backup system, since they're a large chain. My speculation is that someone wasn't prepared for the system to be running on Sunday morning; supermarkets must be closed in Ontario on Sundays, and the ones near us started opening only about a month ago... David Sherman, The Law Society of Upper Canada, Toronto { ihnp4!utzoo seismo!mnetor utzoo hcr decvax!utcsri } !lsuc!dave ------------------------------ Date: Thu, 28 Aug 86 15:38:31 edt From: decwrl!amdcad!amdimage!prls!philabs!linus!axiom!gts@ucbvax.Berkeley.EDU (Guy Schafer) To: CSL.SRI.COM!RISKS Subject: Video processing Now that sophisticated hardware for capturing and altering video images exists for even the modest IBM-PC (AT&T's Truevision products), several concerns arise: Because images can be captured in real time (for less than $5000), and it has been proven that at least one method exists for over-powering ('hi-jacking') a cable video broadcast, some program can be altered and re-broadcast (with a delay equal to the video processing time). This could be especially dangerous if it is done to, say, 2 minutes of a news broadcast or televised political proceedings. Video post-processing can also be an effective means to control the behavior of an individual by tapping directly into her cable coming into her house. An appropriate stock tip given by a seemingly authentic Ruekeiser (sp?) might cause a major stockholder to get on her phone to a broker with predictable (and thus profitable) results. We always knew a hacker with a PC had quite a bit of power; if this hacker can alter someone's main source of information (television broadcasts) he suddenly has quite a bit more. Also, video tapes which are used as evidence in court can be changed without simple means of detection. Actors can be cheated out of royalties--especially in commercials where post-processing 30 seconds of video could cost less than the royalties of an often-repeated performance. The features of the face can be "airbrushed" or distinguishing marks can be added or removed (by software--e.g. TIPS by AT&T) and the actor told that someone else got the part. Any comments? >< ...{ decvax!linus | seismo!harvard }!axiom!gts ------------------------------ Date: 30 Aug 86 16:57 +0200 From: Jacob_Palme_QZ%QZCOM.MAILNET@MIT-MULTICS.ARPA To: "RISKS FORUM" Subject: ATMs (RISKS-3.45) >>..Their dispensing machines cannot be cheated in this way, because they have >>a steel door in front of the machine which does not open until you insert a >>valid plastic card. > >People who swindle ATM's don't have cash cards????? If you have a legally obtained cash card, and insert it into the machine, this act is immediately recorded, so that if the swindle is detected, they can find out who did it. If you have an illegally obtained cash card, you probably do not know the password you have to input on the keyboard. When you input the wrong password (or do not input any password at all), the machine swallows the card, and you never get it back again. At least that is the way the Swedish ATM's work. ------------------------------ End of RISKS-FORUM Digest ************************ -------