11-Mar-87 20:55:23-PST,22023;000000000000 Mail-From: NEUMANN created at 11-Mar-87 20:54:03 Date: Wed 11 Mar 87 20:54:03-PST From: RISKS FORUM (Peter G. Neumann -- Coordinator) Subject: RISKS DIGEST 4.62 Sender: NEUMANN@CSL.SRI.COM To: RISKS-LIST@CSL.SRI.COM RISKS-LIST: RISKS-FORUM Digest Wednesday, 11 March 1987 Volume 4 : Issue 62 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: "Software Safety: What, Why, and How" (Minireview by Jim Horning) Beef with Restaurant's Hi-Tech Computer (Yigal Arens) Electronic Steering (Mike Brown) Enhanced 911 risks (Mike Brown) Computers in the arts (Don Craig, Glenn Trewitt) Mode C (Ken Calvert) Re: Plane Crashes (Ronald J Wanttaja) Re: Results of a recent security review (Arnold D. Robbins) Risks of Maintaining RISKS -- and a reminder for BITNET readers (PGN) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM) (Back issues Vol i Issue j available in CSL.SRI.COM:RISKS-i.j. MAXj: Summary Contents Vol 1: RISKS-1.46; Vol 2: RISKS-2.57; Vol 3: RISKS-3.92.) ---------------------------------------------------------------------- Date: Wed, 11 Mar 87 19:41:04 PST From: horning@src.DEC.COM (Jim Horning) To: RISKS@CSL.SRI.COM Subject: "Software Safety: What, Why, and How" Capsule Review of "Software Safety: What, Why, and How", Nancy G. Leveson, ACM COMPUTING SURVEYS, Vol. 18, No. 2 "This survey attempts to explain why there is a problem, what the problem is, and what is known about how to solve it. Since this is a relatively new software research area, emphasis is placed on delineating the outstanding issues and research topics." [From the Abstract] I've often wished that there was a good survey of the RISKS field that I could recommend to people who weren't as well informed on the subject as I thought they ought to be. I'm pleased to report that such a survey has now been published in a widely accessible journal by a frequent RISKS contributor. Software safety is the central theme, but many other aspects of risks to the public in computer systems are mentioned. ("Mishaps are almost always caused by multiple factors, and the relative contribution of each factor is usually not clear.") Regular readers of RISKS will find much of the material familiar. But they should appreciate the careful documentation of numerous software-related mishaps and the perspective obtained by pulling back a little from our daily batch of anecdotes and looking for general issues and principles. The paper is well organized, and lucidly written. More than 100 references point interested readers to more detailed and/or authoritative information. There is also a general bibliography with 28 entries. Despite the title, this paper doesn't tell how to produce absolutely safe software. Instead, it tells how hard it is, and why it is hard. Activist readers of RISKS will probably feel that it stops just short of some obvious conclusions and calls for needed action. (However, I'm sure that it would not be possible to get a consensus of this group about what the "obvious" conclusions and actions are.) (This issue is dated June 1986, but my copy arrived last week. That this is a Publishing Risk, rather than a Postal Risk, is indicated by an apology printed at the front of the issue.) Jim H. [Nancy herself proposed a solution to the last-mentioned problem -- simply declare the June 1986 issue to be a special combined issue dated June/ September/ December 1986/ March 1987. However, subscribers might feel cheated. PGN] ------------------------------ Date: Wed, 11 Mar 87 12:58:17 pst From: To: risks%csl.sri.com@usc-cse.usc.edu Subject: Beef with Restaurant's Hi-Tech Computer This happened to me several months ago, but I only just realized that it might be of interest to this group. My wife and daughter and I went to a rather fancy restaurant here in LA. We all ordered steak-type dishes and specified various degrees of doneness, all on the rare side. As we were eating our first courses there was a short blackout, lasting approximately two minutes. The restaurant remained illuminated by the candles on the tables. When the main courses came they were very overdone. Our waiter took a peek at the food and then called the manager. The manager apologized profusely and blamed the computer that controlled the kitchen(!). As far as I could figure out, he was claiming that the blackout wiped out the memory of a computer which (among other things?) controls cooking times. When the power returned, the chefs had to try and recall how long things had been cooking, and some mistakes occurred. I have no idea if this was the truth or whether the manager simply thought that a high-tech excuse would be most effective. Since we were too hungry to wait for new dishes, we ate what we had received anyway. We were not charged for the meal. Yigal Arens arens@usc-cse.usc.edu [This does indeed suggest a completely new line of high-tech excuses. PGN] ------------------------------ Date: Wed, 11 Mar 87 09:27:29 est From: mlbrown@nswc-wo.ARPA To: risks@csl.sri.com Subject: Electronic Steering I haven't had the opportunity to follow all of the discussion on the electronic steering issues that have been in the Risks Forum. I notice that there seem to be two sides: those who are scared and those who believe that the manufacturers will develop a safe product. Look back at several of those things that manufacturers should have caught: Pinto gas tanks, Audi 5000's sudden acceleration, Ford E-350 ambulances and their propensity to catch on fire. There are thousands of examples of such potentially catastrophic hazards that have made it through the design and development into manufacturing. Every car manufacturer has had problems of this nature. Yet, the tools and techniques for safety analysis of hardware have been around for many years. They can be very effective if properly applied. Yet we still have problems cropping up. Now we are proposing to allow steering to be controlled by software, control systems for which we do not have the tools and techniques that exist for hardware systems. Every software engineer will tell you that it is impossible to eliminate all of the defects in software: therefore, we have to ensure that the defects that remain do not cause a safety problem. Not a simple task. The process has to start at the concept stage. The software requirements must take into consideration the failure modes that can occur and develop traps to ensure that the system fails safe. The implementation of the safety requirements in the software requirements must be thoroughly analyzed and tested. Even then, it is difficult to develop a "warm, fuzzy feeling" about this system. Years of development can be destroyed by a simple failure that results in a fatal accident. Mike Brown, Chairman, Triservice Software Systems, Safety Working Group ------------------------------ Date: Wed, 11 Mar 87 09:13:27 est From: mlbrown@nswc-wo.ARPA To: risks@csl.sri.com Subject: Enhanced 911 risks Several people have commented recently about errors in enhanced 911 systems that resulted in misdirecting police, fire or rescue personnel. In these instances, a big safety issue is present. In my capacity as Emergency Services Coordinator for my county, I have been involved in investigating an enhanced 911 system for us. The system that has been proposed offers the dispatchers the capability of entering information in addition to the physical location of a caller into a master database. This database is located quite some distance from our locale and services a number of jurisdictions. Suggestions that have been made by our fire and rescue personnel include inserting information such as handicaps or special medical problems of residents, special problems that may be encountered in gaining access to people (e.g., having to ford a stream), etc. As the "good ideas" expanded, it was suggested that people who may have toxic materials (farmers with farm chemicals) or other hazardous materials (we have a number of gun clubs that store black powder or reload their own ammunition), gun collectors or others who may have valuables in their home, etc. be included in this database. Immediately I had visions of someone misusing the database to commit crimes, etc. How do we ensure the security of a database of this nature when the people who are required to have access to it cannot be trusted? Recently, two local jurisdictions have had sheriff's deputies arrested for participating in a burglary ring that has been functioning in the area for 15 years. Scary, isn't it? And then there's Big Brother.... Mike Brown [Maybe that is the same gang that was rampant in New Jersey in the 1960s, giving free estimates on police-linked burglar alarm systems, after detailed on-premise inspections , with the more profitable houses being burgled if they did NOT subscribe. PGN] ------------------------------ Date: Wed, 11 Mar 87 02:07:37 PST To: RISKS@CSL.SRI.COM From: dmc%videovax.tek.com@RELAY.CS.NET Subject: Computers in the arts [Manual vs computerized lighting systems] In my youth I worked as a stage manager and lighting designer/operator for a number of summer stock (and winter broth) companies. The most complex show we ever pulled off had about 300 cues over a two hour period. (I do think the art suffered as a result.) This was on pre-computer but modern (1966) equipment, with 72 x 6 KiloWatt electronic dimmer channels, and 4 presets (four slider pots per channel). The control room contained a desk with master controls, and a side-wing with 288 slider pots on it. On our 300 cue show, we had 3 people operating... When I later worked for the Canadian Broadcasting Corporation in Montreal (1972), the lighting system for Studio 42, a 600 seat auditorium, had 600 12 KiloWatt channels (one per seat :-). The channels were connected via a 48 volt patch panel to 80 control levels. The patch panel was a 600 by 80 matrix wherein the operator inserted a pin to make a connection. (Multiple assignments picked the lowest numbered control level). The 80 control levels had a primitive computer system for storage/recall, but the one slider pot on the control desk could be driven to the level of a recalled channel by a motor, and would detect the operator's touch (capacitively) to permit the setting of levels. Neither of these systems ever had all channels working at the same time. The electronic dimmers were packaged in racks and racks of drawers, and it was a simple matter to repair a vital channel by moving a drawer. The 1966 vintage Strand Electric console had sufficient internal parallelism that we lost channels and pots, but never the whole thing. (I vaguely recall a dual power supply). The CBC system was another story, and a fully qualified Group 8 (the top pay scale) NABET technician was always in the building when Studio 42 was in use. The lighting system failed occasionally, but never took more than an hour or so to restore to operation, since we stocked a full board replacement inventory. In my five years there it never failed on air. I see current lighting control systems at the television trade shows. They use multiplexed twisted pair to connect a small control desk to `intelligent' dimmer boards. Smaller systems build the dimmers right in. The control desk contains a microprocessor that operates each channel, and reads levels from a floppy disc. The key to making these systems redundant is buying two control desks. Individual dimmer channels can fail, but that won't shut down a show. The amount of rehearsal needed to choreograph the operation of a manual lighting console is significant. A failure of the modern control desks means the system is down, since there aren't manual controls any more. (It's usually possible to wire a channel on.) The technical solution is simple (buy or rent another control desk for performances) but the people making these decisions are often not technical (trust me), and view such backup as a luxury. Don Craig, Tektronix Television Systems ------------------------------ Date: 11 Mar 1987 1113-PST (Wednesday) From: Glenn Trewitt To: risks@csl.sri.com Subject: Computers in the arts -- The Show Must Go On. One of the most painful memories that I have: Last fall, I attended a Pilobolus modern dance performance at Berlekey. Their last segment was a performance of "Carmina Burana", accompanied by a compact disk. This is a long piece, perhaps 20 minutes. About two-thirds of the way through, when the performers were dancing "blind" (they had various things on their heads), the disk skipped to a different section. Many people didn't notice this, but I had seen the performance before and listened to the music at home. The performers were REAL good -- they recovered perfectly and the show went on. [What an ORFFul experience! PGN] For about 2 minutes more, that is. At that point, the disk went nuts and started playing random 10-second bits of music. Generally, just enough for the performers to start to recover. This went on for about a minute before they gave up and turned off the "music". But it seemed like an eternity, with the poor dancers up there on stage, just thrashing. I see the risks here as different from other risks associated with any other pre-recorded music, because almost all other failures are not so catastrophic. With a record, for example, you can just nudge the needle and continue. On the flip side, it occurred to me that it was quite possible that someone in the audience had, with them, the technology to fix the problem. Namely, a portable CD player, perhaps with the same CD. An amusing thought. - Glenn Trewitt [Although off the subject of RISKS, this is of course a common failure mode of CDs -- not just a skip and a jump, but wildly erratic behavior. Similar things happen in digital computer control systems (as opposed to analog) -- slight errors may translate into wildly erratic behavior, e.g., a wild control transfer... PGN] ------------------------------ Date: Wed, 11 Mar 87 09:41:53 CST From: calvert@sally.utexas.edu (Ken Calvert) To: neumann@csl.sri.com ReSent-To: RISKS@CSL.SRI.COM Subject: Mode C [Me on Karn on mode C for all:] (RISKS 4.61) > ... I expect that most such planes virtually never enter the busy airspaces > (i.e., Terminal Control Areas) where midairs tend to occur. One reason > is that regulations ALREADY require radios and transponders for aircraft > operating in TCAs, as well as permission from the controlling authority. > > [These last two sentences reach an apparently false conclusion. > (For example, Los Angeles and Chicago routinely report many such > incursions each day.) I don't see how my conclusion is false. My conclusion was NOT that incursions do not occur. The point is that an Airplane Without A Transponder is not a greater threat to other aircraft IF it never goes in airspace where a transponder does any good (busy terminal airspace or airways). Moreover, I have not seen anything to indicate that all or even most incursions into TCAs are made by Airplanes Without Transponders. Have you? [Absolutely. There seems to be lots of evidence that the incursions are by dingbat pilots, generally without appropriate avionics (adequate, nondefective, ...). In the Aeromexico case, the controller was totally distracted by dealing with one dingbat, and ignored another -- BOTH of whom were transgressing. PGN] In my understanding (as a temporarily inactive Private Pilot) the only thing that requiring Mode C on all aircraft does that the current regulations don't is require Mode C on aircraft NOT operating in busy airspace. If the current regulations don't work, this won't either. As you noted, there's a difference between regulation and reality. Transponders have to be turned on to work. Clearly some TCA incursions are made by planes with Mode C transponders - irresponsible/incompetent pilots may fly all kinds of airplanes. On the other hand, if the proposal is to have all aircraft equipped with Mode C that operates AT ALL TIMES, then the proposal must also require ATC to monitor all aircraft. As others have noted, I think it will be some time before the system can handle that, although that may be a worthy goal. Even then, incursions will occur. And your comment applies here: >...rely too much (or blindly) on the technology, then the existence of the > technology may be debilitating. PGN] As a side note, my brother has been training to become an Air Traffic Controller for about nine months. He won't even sit down at a radar screen for a long time yet. New controllers must be completely familiar with the "old" manual system, which is of course used when things break down (actually it is always used; radar and computers are simply an aid). My impression from speaking with him is that the ATC system has a healthy distrust of (at least some kinds of) technology. Ken Calvert, Univ. of Texas Computer Sciences ------------------------------ Date: Wed, 11 Mar 87 08:49:52 pst From: ucbcad!ames!ll-xn!ames!uw-beaver!ssc-vax!wanttaja@ucbvax.Berkeley.EDU (Ronald J Wanttaja) To: RISKS@csl.sri.com Subject: Re: Plane Crashes (RISKS 4.56) > In Europe there was a spate of (F-111?) crashes. The apparent cause of > these crashes was pilots (1) believing they could fly the plane on their > own without the help of any dumb computer, (2) turning the computer off, > and (3) promptly flying into a mountain. > Any Hints? DavidP Don't know much about this particular case, but there is a famous story about the early days of the F-111 in Southeast Asia... The F-111 was the first fighter-bomber with Terrain Following Radar. The radar controlled the plane through the autopilot to "hug" the earth; flying about 200 feet above ground level. It would look far enough ahead, and if an obstruction was sighted, the plane would pull up at the appropriate moment at a pre-programed G level (for those interested in further details of this type of flying, see the archives for rec.aviation). The first combat crews were trained in the US, then send to Thailand to fly missions to North Vietnam. They had a high loss rate for night TFR missions. Then they found out why: The TFR was set to fly the plane at 200 feet. The TFR couldn't see trees, and some trees in SE Asia grow over 250 feet high... Ron Wanttaja (ssc-vax!wanttaja) ------------------------------ Date: Wed, 11 Mar 87 12:59:30 EST From: "Arnold D. Robbins {EUCC}" Subject: Re: Results of a recent security review (RISKS 4.52) In article Risks 4.52 Andre Klossner writes on the licensing of OWNDIR. > [... and will someone sue AT&T if, after a license is duly obtained, a > devastating Trojan horse is perpetrated using this flaw/feature ? PGN] There has been a bunch of discussion about this in mod.os.minix; basically within a year of the patent, it was released for Public Use, i.e. anyone who wants to can use the setuid concept (which is why minix does). The article there cited real U.S. Patent Office publications giving the details. (I probably should have saved the article but didn't.) Anyway, I'm writing to try and cut off the spread of misinformation as early as possible. I find the moderator's point more interesting; the people to sue would be the manufacturer who incorporated the feature, not AT&T who invented it... Arnold Robbins CSNET: arnold@emory BITNET: arnold@emoryu1 ARPA: arnold%emory.csnet@csnet-relay.arpa UUCP: { akgua, decvax, gatech, sb1, sb6, sunatl }!emory!arnold ------------------------------ Date: Wed 11 Mar 87 11:56:14-PST From: Peter G. Neumann Subject: Risks of Maintaining RISKS To: RISKS@CSL.SRI.COM I received a complaint from Dave Parnas that he was suddenly receiving mail intended for the automatic BITNET mail list maintainer. It turns out that two readers forgot, or did not read, the old instructions. Sorry, there is nothing I can do on BITNET to prevent it, although I make a big effort (but still not foolproof) on CSL.SRI.COM. (At least it hasn't happened here yet, although I have received numerous retries to RISKS following rejected mail inappropriately sent to the LIST.) PLEASE READ THE MASTHEAD. Reminder for BITNETters, once again: For WISCVM, send mail to LISTSERV@CMUCCVMA, with a single line request: SUBSCRIBE MD4H your name or UNSUBSCRIBE MD4H your name For FINHUTC, send mail to LISTSERV@FINHUTC, with a single line request: SUBSCRIBE RISKS your name or UNSUBSCRIBE RISKS your name For UGA, send mail to LISTSERV@UGA, with a single line request: SUBSCRIBE RISKS your name or UNSUBSCRIBE RISKS your name (All three may be work interchangeably -- I'm not sure.) ------------------------------ End of RISKS-FORUM Digest ************************ -------