sss ssss rrrrrrrrrr
ssss ss rrrr rrrr
sssss s rrrr rrrr
ssssss rrrr rrrr
ssssssss rrrr rrrr
ssssss rrrrrrrr
s ssssss rrrr rrrr
ss sssss rrrr rrrr
sss sssss rrrr rrrr
s sssssss rrrrr rrrrr
+===================================================+
+======= Testing Techniques Newsletter (TTN) =======+
+======= ON-LINE EDITION =======+
+======= September 1995 =======+
+===================================================+
TESTING TECHNIQUES NEWSLETTER (TTN), On-Line Edition, is E-Mailed
monthly to support the Software Research, Inc. (SR) user community and
provide information of general use to the world software testing commun-
ity.
(c) Copyright 1995 by Software Research, Inc. Permission to copy and/or
re-distribute is granted to recipients of the TTN On-Line Edition pro-
vided that the entire document/file is kept intact and this copyright
notice appears with it.
TRADEMARKS: STW, Software TestWorks, CAPBAK/X, SMARTS, EXDIFF,
CAPBAK/UNIX, Xdemo, Xvirtual, Xflight, STW/Regression, STW/Coverage,
STW/Advisor and the SR logo are trademarks or registered trademarks of
Software Research, Inc. All other systems are either trademarks or
registered trademarks of their respective companies.
========================================================================
INSIDE THIS ISSUE:
o THE PENTIUM BUG_AN INDUSTRY WATERSHED (Part 1 of 4)
Dr. Boris Beizer
o SR RELEASES NEW COVERAGE TOOLS FOR ALL MICROSOFT PLATFORMS
o SOFTWARE RESEARCH, INC.
OFFERING Xdemo
o KRAZY KONTEST SERIAL NUMBER NO. 3
o CALENDAR OF EVENTS
o TTN SUBMITTAL POLICY
o TTN SUBSCRIPTION INFORMATION
========================================================================
THE PENTIUM BUG-AN INDUSTRY WATERSHED (Part 1 of 4)
Dr. Boris Beizer
Editor's note: This article has been divided into four parts, which
will be published over the next four consecutive editions of the TTN.
------------------------------------------------------------------------
Copyright, Boris Beizer, 1995. Permission is granted to print this docu-
ment for personal use and to redistribute it for internal corporate use
subject to the provision that this entire copyright notice must appear.
Use of this document or parts thereof in any commercial publication or
document without the written permission of the author is strictly prohi-
bited.
------------------------------------------------------------------------
1. General
Two guys in the Australian Outback, a dozen polar Eskimos and a tribe in
the Amazon jungle have not heard about the Pentium bug. People oblivi-
ous to floating point arithmetic before INTEL's customer relations
fiasco now speak angrily about this bug and how it will affect their
lives. There's a half-dozen class action suits afoot claiming warrantee
breaches and false advertising as well as few stockholders' suits
[INTE95]. This bug has probably been blamed for every human and techni-
cal misery from anorexia to zygopteran infestations.
Was this bug serious? Yes and no. Technically, it wasn't the first, it
wasn't the worst and it won't be the last of its kind. But from the
point of view of its impact on the computer industry, it is probably the
most significant bug ever. It has forever transformed the industry. It
is a watershed for all of us.
Some of you who are in software might mistakenly believe that this was a
hardware bug and that it therefore doesn't affect you: if so, you're
wrong on both counts. It was a software bug, but one that happened to
be compiled into silicon rather than RAM. But it doesn't matter if it
was hardware or software because the users, as we have just learned, do
not make that distinction.
2. The Bug
2.1. The Billion Dollar Bug?
This could (but won't) be the most expensive bug in the history of the
computer industry. INTEL announced a $475 million dollar charge against
earnings (fn#1) to cover replacement of faulty chips. Estimates running
around the industry are that 4-5 million users have the faulty chip:
that's consistent with the $475 million dollar write-off. If five mil-
lion users elected to accept INTEL's free replacement offer, assuming an
average cost of $75/hour and allocating four hours to call INTEL, open
the computer, remove the chip, replace it, etc. there's another $1.5
billion wasted, putting this bug at the two-billion dollar mark. This
doesn't take into account the millions of hours wasted by industry and
the press talking about the bug, the countless hours spent on the INTER-
NET flaming INTEL, and by people like you and me discussing it. That
could easily be another billion dollars, but that's hard to verify.
Then there's the possibility of consequential damage - how many
businesses and operations went bankrupt because of this bug? Here I'm
inclined to believe that the real damage, if it were possible to measure
at all, would be in the hundreds of dollars, rather than the millions
or billions. INTEL states [INTE95] that of the thousands of calls from
users regarding this bug, only two actual users have been confirmed as
having been affected. Talk about tempests (no, hurricanes) in a teapot!
------------------------------------------------------------------------
Footnote #1:
1. Let's not be naive about this write-off. Announcing the writeoff
makes good public relations sense, of course, but it could also have
substantial financial benefit. The write-off allows INTEL to reduce
earnings in a year when profits are high due to the initial high prices
of the chip. Assuming that most users won't bother to get the replace-
ment chip, the write-off provides a cushion for lower future earnings.
There are other possible financial advantages that would have made the
write-off attractive to INTEL financiers even if there had been no bug.
------------------------------------------------------------------------
2.2. The Technical Facts
The floating point processor in the Pentium chip uses a division speed-
up algorithm whose earliest version goes back to 1956 [NADL56]. The
INTEL algorithm is based on a paper published in 1968 [ATKI68] so the
algorithm itself has been as thoroughly verified as an algorithm can be.
The algorithm depends on values set in a table of about 2,000 entries.
The table is indexed by a combination of a portion of the divisor and
the remainder at each step of the algorithm. Five locations in this
table were not loaded - they were left blank. This resulted in the
values at those locations set to zero rather than to +2. The table is
implemented as a programmed logic array (PLA); transistors arranged in
an AND/OR cascade so that when the combined bits of the divisor and
remainder are fed into the top of the array, the table bits are produced
at the output. Because of these errors in the table and the way the
algorithm works [SHAR94], if some very specific combinations of divisor
and dividend are sent to the unit, output errors could occur in the 4'th
to 12'th significant digit.
2.3. A Classical Unit Bug
Based on the information given to me by INTEL [SHAR94, INTL95], I see
this is a classical software unit bug. By "classical" I mean that had
proper unit testing been done, this bug would have been found long
before it was cast into silicon. The culprit is a routine, called a
"script" by INTEL, used to download the table values to a silicon com-
piler that creates the PLA. The script is a program, albeit in a pro-
gramming language used by logic designers. The script failed to down-
load five values, leaving them blank. The systematic pattern of the
missing entries makes it obvious to this one-time logic designer that a
predicate in the logic of this routine was incorrect. I don't have the
facts from INTEL, but the script probably did not merely download stored
values from a table of data, but rather generated the values it needed.
Alternatively, the routine did use a set of stored constants and
translated those bit values into specific data needed by the silicon
compiler to generate the PLA. In either case, it looks like there was
a faulty predicate someplace. Further conjecturing on how this might be
done, it is likely that the script contained compound predicates, which
would be typical for such a routine. The script could have (should
have) been tested by itself, in isolation, as a unit. At least state-
ment coverage should have been used; but considering the nature of this
software, the correct test criterion would probably be basic predicate
condition coverage; that is, each term of every compound predicate
tested for at least one TRUE and one FALSE value and such coverage moni-
tored by an appropriate coverage tool. If lazy evaluation were used by
the script's compiler, then more tests to satisfy prime-implicant cover-
age would have been needed in addition to predicate condition coverage
[BEIZ90].
The importance of the bug from a technical point of view is that it
appears to be a unit bug that would have been caught by good unit test-
ing in a few dozen tests rather than in the billions of tests run by
INTEL on the entire system (i.e., the Pentium chip). If ever there was
supporting evidence for the importance of proper unit testing, this is
it.
2.4. Are You Affected?
If you have a Pentium you want to know if this bug affects you. It's
unlikely that you are affected, but here's how to find out. Fool the
system into believing that it is running an ST chip that doesn't have a
floating point coprocessor. The Lotus fix is to add a -s after
the last character in the command line path. Doing the floating point
calculations in software rather than in hardware may slow down recalcu-
lations a bit, but Lotus estimates [LOTU95] that the net impact is less
than 10%. Here's how to find out if you're affected in Lotus 1-2-3:
similar procedures are available from most software vendors whose pro-
ducts do extensive floating point calculations.
1. Install the recalculation icon on your tool bar.
2. Bring in the spreadsheet about which you're worried. Add a cell that
contains a check sum over the entire spreadsheet. Calculate that to
as many significant digits as you can. If there are only a few cells
with critical outputs check sum only those or just print them out for
later comparison. Force a recalculation by using the recalculation
icon and use a stopwatch to time the recalculation. In my case, on a
486DX2, it's an unmeasurable less than 50 milliseconds for my biggest
spreadsheet (8,350 cells). A Pentium should take even less time.
Print out the check sum to the maximum number of significant digits.
3. Invoke 1-2-3 with the -s option. Bring in the same spreadsheet.
Force a recalculation. You must force a recalculation to get the
"fix" done because most spreadsheets automatically recalculate only
changed cells and cells derived from them unless you force recalcula-
tions. Do several recalculations if you have loops and other complex
cell references. Time the last one. If there's no perceptible exe-
cution time difference, you might as well use the software floating
point algorithm and never worry about it again.
4. Print out the check sum (or the critical cells) for the spreadsheet
running with the -s option and note any differences from the value
calculated with hardware floating point. If there's no difference,
don't worry. If there is a significant difference and the recalcula-
tion time is acceptable, keep your current chip and software floating
point calculations. If there is a significant value difference and
if the recalculation time increase means something to you, get the
new chip and install it.
There are rumors floating around the industry that the new Pentium chip
does not actually fix the bug. INTEL categorically denied this rumor
and reiterated its offer to replace faulty chips at no charge to users
(but hardly at no cost - you pay for installing the replacement, of
course). INTEL and others have released and distributed software fixed
to be installed in compilers that rearranges the order of arithmetic
operations so as to make this bug's symptoms very unlikely. There are
as of today, dozens of small programs available free as downloads and
published in many trade journals that can be used to determine if you
are or are not affected. Most software vendors with products that make
even marginal use of floating point arithmetic have published and offer
simple software fixes that eliminate the problem with only a marginal
performance impact (e.g., SASS95).
<>
5. References
ATKI68 Atkins, Daniel E. Higher Radix Division Using Estimates
of the Divisor and Partial Remainder. IEEE Transactions
on Computers, Vol. C-17, #10, Oct. 1968, pps. 925-934.
BEIZ90 Boris Beizer, Software Testing Techniques, 2nd Edition,
Van Nostrand Reinhold, 1990.
BEIZ95 Samuel Beizer. Private communication.
IBMS95 Pentium Study, IBM Research White Paper, December 12,
1994.
INTE95 Interview of Howard High, INTEL, February 2, 1995.
LOTU95 Interview of Peter Cohen, Lotus Development, February 2,
1995.
NADL56 Nadler, Morton. A High Speed Electronic Arithmetic Unit
for Automatic Computing Machines, Acta Tech (Prague), #6,
1956, pp. 464-478.
SASS95 McGrath, Sue. Private Correspondence, SAS Institute,
February 9, 1995.
SHAR94 Sharangpani, H.P., and Barton, M.L., Statistical Analysis
of Floating Point Flaw in the Pentium Processor. INTEL
Corporation white paper, November 30, 1994.
------------------------------------------------------------------------
Boris Beizer, PhD
ANALYSIS
1232 Glenbrook Road
Huntingdon Valley, PA 19006
PHONE: 215-572-5580
FAX : 215-886-0144
Email: BBEIZER@MCIMAIL.COM
========================================================================
SR RELEASES NEW COVERAGE TOOLS FOR ALL MICROSOFT PLATFORMS
Software Research, Inc.
901 Minnesota Street
San Francisco, CA 94107 USA USA
Tel: [1] (415) 550-3020
FAX: [1] (415) 550-3030
Email: info@soft.com
Editorial Contact:
Ms. Rita Bral
Software Research, Inc.
Tel: [1] (415) 550-3020
bral@soft.com
FOR IMMEDIATE RELEASE
SR Releases New Coverage Tools for All Microsoft Platforms.
New Tool Integrates State-of-the-Art Compiler Technology.
San Francisco, CA, August 22, 1995 - Software Research, Inc. (SR), the leading
supplier of integrated automated software testing tools (under the product
line TestWorks (tm) [STW]) announces STW/Coverage for Windows Ver.
1.1. This is a next-generation product intended for developers and testers
working on host-based, client-server, and embedded-product environments.
STW/Coverage for Windows Ver. 1.1 helps determine how well products have been
tested through a variety of easily-produced textual, graphic and tabular
reports. This new product embodies all of SR's experience and insight
acquired in over 8 years of shipping test coverage tools.
STW/Coverage for Windows Ver. 1.1 is a major leap forward with its new
compiler-based technology. The new implementation permits the coverage
analyzer to support all standard versions of the C and C++ programming
languages for Microsoft Windows 3.1, Windows95, and Windows NT. It makes it
easy to extend the product language dialects and supported features, and to
completely support programmers' existing environments. For example, it
includes full support of templates within C++ and complete support of
Microsoft C++ Ver. 6 and later, plus Microsoft Visual C Ver. 1.5 and Ver. 2.0.
STW/Coverage for Windows Ver. 1.1 has extensive error checking to identify
syntax errors and allow the developer to correct them before continuing. It
is no longer necessary to have a 100% compilable program to run through the
coverage analyzer. With STW/Coverage for Windows Ver. 1.1 the process of
going from source code to object code that is ready to produce coverage
reports is extremely fast, streamlining the compile-edit-debug
test/development cycle.
"STW/Coverage for Windows Ver. 1.1 outpaces the few competitive products which
work in the Windows environment with several key features," says Rita Bral,
SR's Director of Promotion. "STW/Coverage has full support for both C and C++,
including all of the latest C++ language enhancements in Microsoft C++ and
other similar compilers. It also provides users with multiple ways to look at
coverage results."
"Users can use simple textual reports which are easy to include in internal
documents, and get graphical views of the code, which are annotated
numerically and by color to show coverage level. And if the developer needs
quick analysis of results, there is also a quick-look tabular report. This
new reports allow users to look at coverage results at high level, and then
interactively dive into whatever details are needed. Finally, by using
compiler technology it is easy to integrate coverage analysis into the user's
standard build-test-edit process with a simple one-line change," Bral
concluded.
STW/Coverage for Windows provides branch and call-pair coverage for both C and
C++. Branch coverage is used for unit and system testing, tracking the number
of times each segment in a function has been exercised. Call-pair coverage is
used primarily for integration testing, determining if each function call pair
has been exercised. Coverage values tell testers how good a test suite is,
focusing effort on building tests for untested code, rather than creating new
tests for sections of the code that have already been exercised.
Organizations know when they have done enough testing and can set a coverage
level which must be attained before a product ships. The dollar benefits are
obvious.
STW/Coverage for Windows is part of TestWorks, the most complete suite of
integrated testing tools on the Windows platforms. The other component is
STW/Regression(tm) for Windows, a fully-automated test execution suite for
host-based and client-server GUI testing. STW/Regression for Windows contains
a sophisticated capture/playback engine, an ASCII and bitmap differencing
engine, and a test manager for controlling the testing process. TestWorks is
available for MS-DOS, MS-Windows, Windows95, Windows NT and most major UNIX
platforms as well.
STW/Coverage for Windows Ver. 1.1 supports MS-Windows version 3.0 or later,
Windows95, and Windows NT. Price for single licenses of STW/Coverage for
Windows on these platforms starts at $2,500.
Software Research, founded in 1977 as a consulting and service organization,
is the leader in software testing and quality assurance. SR provides the
broadest suite of integrated testing tools in the market, TestWorks (STW),
with over 4500 licenses sold and over 12,500 users. TestWorks (STW), includes
three families of tools that operate independently or as an integrated test
tool suite: STW/Regression for test execution automation and test management;
STW/Coverage for coverage analysis in a variety of languages for branch,
call-pair and path-level coverage; and STW/Advisor for metric and static
analysis to aid in test resource allocation. Software Research, Inc. is
privately owned and internally funded.
Product details and training information are available directly from SR by E-
mail: info@soft.com; by Phone: (415) 550-3020; or by FAX: (415) 550-3030.
# # #
Trademarks: STW, Software TestWorks, CAPBAK/X, SMARTS, EXDIFF, CAPBAK/UNIX,
Xdemo, Xvirtual, Xflight, STW/Regression, STW/Coverage, and STW/Advisor are
trademarks or registered trademarks of Software Research, Inc. Other
trademarks are owned by their respective companies.
========================================================================
SOFTWARE RESEARCH, INC.
OFFERING Xdemo
In the sale of any software product, a demonstration is very often far
more effective than a write-up or lengthy presentation. As they say, a
picture is worth a thousand words. With the technologies that have been
available until today you have two choices for an on-line demonstration:
either have an automated slide presentation (some with voice over), or
capture key screens in your product and have them flipped through with
annotations and voice over. Now Software Research has given you a third
option when you want the person seeing your demonstration to get a real
feel for your product. Xdemo is based on SR's industry-leading
capture/playback product CAPBAK/X. It allows you to "record" a demons-
tration of your product, encrypt the resulting script, and have it
played back live into your application, with voice over. Now you can
have prospects, customers, and even your own sales force demonstrate
your product, live, without having to know how to run it. This is a
great way to show off new features to your existing customers, set up
demonstrations at tradeshows, or have a new prospect walked through an
example before starting to evaluate your application.
For your protection Xdemo will only play back demos and scripts that are
produced with your copy of CAPBAK/X for which the Xdemo option is
licensed. Xdemo is available on an OEM basis only, and a separate
license agreement authorizing distribution of the product feature is
required. Please contact SR Corporate Sales for complete details.
Software Research, Inc. 901 Minnesota Street San Francisco, CA 94107 USA
USA Tel: [+1] (415) 550-3020 US Only: [+1} (800) 942-SOFT Fax: [+1]
(415) 550-3030 Email: sales@soft.com
========================================================================
KRAZY KONTEST SERIAL NUMBER NO. 3
Krazy Kontest is a technical challenge for testers! Each Krazy Kontest
focuses on a specific technical question or situation, and invites
responses. While serious answers are expected they are not required.
What's Krazy is that we don't necessarily know if there *IS* a correct
answer to each Krazy Kontest situation.
We promise in each Krazy Kontest, scheduled to run a couple of months,
to listen and analyze everyone's response, to summarize all responses,
and to include the summary (including the best, most quotable quotes!)
in a future issue.
It is 1999, and December 31 is rolling around.
You're part of the Year 2000 team and your task is to test the new
date function, the one that returns a 4-digit year rather than the
right-most two digits of the (integer) year.
HOW DO YOU TEST THE DATE FUNCTION?
Be sure to identify your answer with the correct Krazy Kontest Serial
Number. E-mail your responses to: ``ttn@soft.com'' making sure that the
Subject: line says ``Krazy Kontest''.
========================================================================
------------>>>CALENDAR OF EVENTS<<<---------------
========================================================================
Here is a list of upcoming events of interest.
"o" indicates that Software Research, Inc. will lead or participate in
these events.
- 25-28 Sep: Fifth European Software Engineering Conference (ESEC'95)
Barcelona, Spain, Contact: ESEC Secretariat, Tel: +34 -3
-415.41.41, Fax: +34 -3 -415.55.56, E-mail:
difinsa@ibm.net
o 28-29 Sep: Pacific Northwest Software Quality Conference (PNSQC),
Oregon Convention Center, Contact: - Exhibits, Mr.
James Mater, Revision Labs, Inc., Tel: 503-690-2341, -
Conference Management, Ms. Terri Moore, Pacific Agenda,
PO Box 10142, Portland, OR 97210, Tel: 503-223-8633
o 23-24 Aug: XVT Software Partners, Village At Breckenridge, CO,
Contact: John. L. Donnelly, Exhibit Manager, Tel: 303-
440-9813, Fax: 303-440-9836, E-mail: johnd@xor.com
o 19-21 Sep: UNIX Expo 1995, Jacob K. Javits Convention Center, New
York, NY, Contact: Don Berey, Tel: 800-829-3976, ext.
169, or Laura McQuaid, show manager, Tel: 800-829-3976
ext. 9100, Fax: 201-346-1602
- 3-5 Oct: Software Development '95 East, Washington, D.C.
Convention Ctr., Washington, D.C.
- 5-6 Oct: Third Annual ISO 9000 Conference, Washington Hilton
and Towers, Washington, DC, Contact: ASQC Conferences
and Exhibits Department, P.O. Box 3005, Milwaukee, WI
53201-3005, Tel: 800-248-1946, 414-272-8575
- 10-12 Oct: IBM TechCon-The Client/Server Solutions Conference,
San Jose, CA, Contact: IBM TechCon Express Registration,
P.O. Box 2002, Andover, MA 01810, Tel: 800-767-2336,
Fax: 508-470-0526, E-Mail: 76225.3340@compuserve.com
- 10-12 Oct: Software World & Client/Server Developer's Conference &
Expo, San Jose Convention Center, Contact: Loretta
Taylor or Dan Horgan, Tel: 508-470-3870, Fax:
508-470-1992
- 16 Oct: Testing Smalltalk Applications; OOPSLA '95 Workshop,
Contact: Barbara Yates, Tel: 503-657-7703, E-mail:
barbara.bytesmiths@acm.org - 16-20
Oct: International Conference on Software Maintenance '95,
Opio, France, Contact: Mari Georges, Tel:
+ 33 1 49 10 5398, Fax: + 33 1 49 10 0615, E-mail:
mari@capsogeti.fr
- 23-26 Oct: 5th International Conference on Software Quality '95,
Stouffer Renaissance Hotel, Austin, TX, Contact: Red
Sturdevant, Tel: 512-328-4264, 512-794-0033,
512-282-8362 Fax: 512-328-3260, 512-794-0001 E-mail:
wstur@aol.com
- Oct 24-26: Client/Server Application Packages, Hyatt Regency,
Orlando International Airport, Orlando, Florida,
Contact: Diane Munini or Jean Desmarais, Tel:
508-470-3870, Fax: 508-470-1992
- 30 Oct-1 Nov: ABUI Fall '95 Conference, San Jose Convention Center,
San Jose, CA, Contact: Tricia Laursen, Tel:
508-779-5520/2724
========================================================================
------------>>> TTN SUBMITTAL POLICY <<<------------
========================================================================
The TTN On-Line Edition is forwarded on the 15th of each month to sub-
scribers via InterNet. To have your event listed in an upcoming issue,
please e-mail a description of your event or Call for Papers or Partici-
pation to "ttn@soft.com". The TTN On-Line submittal policy is as fol-
lows:
o Submission deadlines indicated in "Calls for Papers" should provide
at least a 1-month lead time from the TTN On-Line issue date. For
example, submission deadlines for "Calls for Papers" in the January
issue of TTN On-Line would be for February and beyond.
o Length of submitted items should not exceed 68 lines (one page).
o Publication of submitted items is determined by Software Research,
Inc., and may be edited as necessary.
========================================================================
----------------->>> TTN SUBSCRIPTION INFORMATION <<<-----------------
------------------->>>NEW INSTRUCTIONS!!<<<-------------------
========================================================================
To request a FREE subscription or submit articles, please send E-mail to
"ttn@soft.com".
TO SUBSCRIBE: please use the keywords "Request-TTN" or "subscribe" **AND
INCLUDE YOUR EMAIL ADDRESS** in the Subject line of your E-mail header.
To have your name added to the subscription list for the biannual hard-
copy version of the TTN -- which contains additional information beyond
the monthly electronic version -- include your name, company, and postal
address in the body of the mail message.
TO CANCEL: include the phrase "unsubscribe" or "UNrequest-TTN" **AND
YOUR EMAIL ADDRESS** in the Subject line.
Note: To order back copies of the TTN On-Line (August 1993 onward),
please use the keywords "Back issue request" in the Subject line, and
please specify the month(s) and year(s) in the body of your message when
E-mailing requests to "ttn@soft.com".
TESTING TECHNIQUES NEWSLETTER
Software Research, Inc.
901 Minnesota Street
San Francisco, CA 94107 USA
Phone: (415) 550-3020
Toll Free: (800) 942-SOFT
FAX: (415) 550-3030
E-mail: ttn@soft.com