sss ssss rrrrrrrrrrr
ssss ss rrrr rrrr
sssss s rrrr rrrr
ssssss rrrr rrrr
ssssssss rrrr rrrr
ssssss rrrrrrrrr
s ssssss rrrr rrrr
ss sssss rrrr rrrr
sss sssss rrrr rrrr
s sssssss rrrrr rrrrr
+===================================================+
+======= Quality Techniques Newsletter =======+
+======= February 2005 =======+
+===================================================+
QUALITY TECHNIQUES NEWSLETTER (QTN) is E-mailed monthly to
subscribers worldwide to support the Software Research, Inc. (SR),
eValid, and TestWorks user communities and to other interested
parties to provide information of general use to the worldwide
internet and software quality and testing community.
Permission to copy and/or re-distribute is granted, and secondary
circulation is encouraged, provided that the entire QTN
document/file is kept intact and this complete copyright notice
appears in all copies. Information on how to subscribe or
unsubscribe is at the end of this issue. (c) Copyright 2004 by
Software Research, Inc.
========================================================================
Contents of This Issue
o Real vs. Synthetic Web Performance Measurements: A Comparative
Study, by John Bartlett & Peter Sevcik
o SQRL Reports Available
o International Journal of Information Technology and Web
Engineering
o DETester Portal Website
o eValid: Latest News and Updates
o ROI Dashboard Offered
o Mini-Survey on Test Training, by Franco Martinig
o QTN Article Submittal, Subscription Information
========================================================================
Real vs. Synthetic Web Performance Measurements, A Comparative Study
by
John Bartlett & Peter Sevcik
Summary: This new study finds that synthetic Web
performance tests give organizations a false sense of
user experience. Specifically, synthetic measurements
frequently understate actual response time seen by real
users by as much as 50% and overstate the response time
of other applications by as much as 350%.
Executive Summary
With respect to accuracy, our analysis shows that synthetic
measurements randomly and frequently understate the actual response
time seen by real users by as much as 50% and overstate the response
time of other applications by as much as 350%. These error margins
are much too broad to serve as the basis for infrastructure
management or SLA enforcement. When comparing the results of real
versus synthetic measurements over time, we also found poor
correlation of response time trends. Establishing accurate
baselines is an essential first step in proactive detection of
service level degradation or traffic anomalies.
With respect to coverage, our analysis shows that synthetic agents
usually interact with less than 10% of total website URLs and less
than 15% of the ISPs connecting real users to the site. It is
dangerous to extrapolate the performance characteristics of an
entire application environment and its user communities from this
small sample. In one site analyzed, we found that 190 synthetic
agents were used to represent 1.7 million total website visitors.
The principle risk of using synthetic measurement for service level
management is that service problems can impact an unknown quantity
of users in unknown locations with an immediate and negative impact
on the business. This can leave the IT organization without the
data they need to detect and analyze service problems so they can
take the necessary corrective actions to restore service.
Permission Statement
You may excerpt small parts of this report as long as you add the
following citation: Source: NetForecast Report 5077:
http://www.netforecast.com
========================================================================
SQRL Report No. 26
Inspection of Software with Incomplete Description
By: Michael Sharygin
http://www.cas.mcmaster.ca/sqrl/sqrl_reports.html
Abstract: Software quality problem has become increasingly visible,
as software has emerged as the dominant factor in determining system
costs. Software inspection, or static analysis of software
artifacts, is commonly recognized as an efficient way of finding
defects. Unfortunately, in software inspection we often face the
problem of non-existent, unsatisfactory or inconsistent software
documentation.
This thesis investigates existing inspection methods and derives a
generic inspection process for assuring quality, evaluating and
specifying post-release software product with incomplete undergoing
description. The inspection strategy refers to a set of scenarios
and to the associated comparison scheme to be used for diagnosing
software defects.
To be rigorous and systematic, the process of inspecting software
with incomplete description needs an evolvable base of software
artifacts analysis in order to avoid desperate understanding of the
software system under study. To realize this, a generic process
within the framework suggested employs a lifecycle-centric approach
to software inspection. To facilitate the process, we propose a
stepwise abstraction system recovery while engineering the system in
reverse.
We will also introduce an experiment investigating the development
of a rigorous approach for effective software inspection.
SQRL Report No. 25
Use of Tabular Expressions in the Inspection of Concurrent Programs
By: Xiao-Hui Jin
http://www.cas.mcmaster.ca/sqrl/sqrl_reports.html
Abstract: This thesis presents a systematic, rigorous inspection
approach for concurrent programs. The approach has been successfully
applied to a classic concurrent program of the Readers/Writers
problem.
In the inspection process, we rewrite the concurrent program by
assigning each primitive statement a label; the transfer of control
from statement to statement is made explicit. Auxiliary variables
are used to record extra information for inspection without
affecting the original intent of the program. The resulting program
is a non-deterministic sequential program with the same behavioral
effect as the original concurrent program. The rewritten program is
then examined through checking the truth-value of the system
invariant that fully captures program structure. A decreasing
quantity of the program states is also used to show the clean
completion of the program.
We use tabular expressions, program-function tables, to describe the
function of the program. Each column in the table is inspected
individually; the program is divided into small components to be
conquered with ease. The correctness of the whole program is implied
(evaluated) by the correctness of the columns examined through the
inspection.
========================================================================
International Journal of
Information Technology and Web Engineering
(http://www.idea-group.com/ijitwe)
Inaugural Issue: January 2006
Submission Deadline April 10, 2005
Editor-in-Chief: David C. Rine
Professor of Computer Science
George Mason University
DavidCRine@aol.com
Co-Editor-in-Chief: Ghazi I. Alkhatib
Senior Lecturer of MIS
Qatar College of Technology
alkhatib.JITWENG@qu.edu.qa
An official publication of the Information Resources Management
Association Published: Quarterly (Print ISSN: 1554-1045 and
Electronic E-ISSN: 1554-1053)
Organizations are continuously overwhelmed by a variety of new
information technologies, many are Web based. These new
technologies are capitalizing on the widespread use of network and
communication technologies for seamless integration of various
issues in information and knowledge sharing within and among
organizations. This emphasis on integrated approaches is unique to
this journal and dictates cross platform and multidisciplinary
strategy to research and practice.
The Mission of the Journal The main objective of the journal is to
publish refereed papers in the area covering Information Technology
(IT) concepts, tools, methodologies, and ethnography, in the
contexts of global communication systems and Web engineered
applications. In accordance with this emphasis on the Web and
communication systems, the journal publishes papers on IT research
and practice that support seamless end-to-end information and
knowledge flow among individuals, teams, and organizations.
This end-to-end strategy for research and practice requires emphasis
on integrated research among the various steps involved in
data/knowledge (structured and unstructured) capture (manual or
automated), classification and clustering, storage, analysis,
synthesis, dissemination, display, consumption, and feedback. The
secondary objective is to assist in the evolving and maturing of
IT-dependent organizations, as well as individuals, in information
and knowledge based culture and commerce, including e-commerce.
Coverage Topics to be included are the following. Please refer to
the web site for more detailed list of topics.
* Web systems architectures, including distributed, grid computer,
and communication systems processing
* Web systems design and performance engineering studies
* Web user interfaces design, development, and usability
engineering studies
* RFID research and applications in web engineered systems
* Mobile, location-aware, and ubiquitous computing
* Ontology and semantic Web studies
* Software agent-based for web-engineered applications
* Integrated user profile, provisioning, and context-based
processing
* Security, integrity, privacy and policy issues
* Case studies validating Web-based IT solutions
* Data and knowledge capture, quality, validation, and
verification issues
* IT readiness and technology transfer studies
* IT Education and Training
* Virtual teams and virtual enterprises: communication, policies,
operation, creativity, and innovation
In addition to complete research articles, the journal will publish
book reviews and research notes on current innovative, substantial
and novel concepts and research areas to foster collaboration
research and encourage further exploration and exploitation by
academicians and practitioners. Both research and instructional
case studies of success stories of IT-augmented systems development
will be solicited form practitioner of business and government
organizations.
Publisher The International Journal of Information Technology and
Web Engineering will be published by Idea Group Inc. , publisher of "Idea Group Publishing", "Information
Science Publishing", "IRM Press", "Cybertech Publishing", and "Idea
Group Reference" imprints. The inaugural issue of this journal is
due for publication in January 2006.
========================================================================
DCTester Portal Website
by
Jeff Rashka
http://www.effectivesoftwaretesting.com/
The DCTester Portal website has been updated to reflect events in
the area for 2005. Check out the seminars, tutorials and
conferences. Also note that new jobs are posted on the website each
week.
Of special note, visit the site to learn about the free seminar on
23 February that addresses practices for testing across the entire
development life cycle for effectively validating application
functionality and performance. The approach applies a business
context to the test management effort, where critical success and
high-risk functions are prioritized.
Also, obtain a free white paper, associated with the approach
addressed in the seminar, to learn how you can implement a risk-
based test approach that will enable you to build higher quality
applications while meeting deadlines. Visit the site below today to
read "Implementing a Risk-based Approach to Ensuring Quality".
Discover the benefits of risk-based testing and what you can do to
accurately identify and prioritize testing, while controlling costs
and resources.
Free white paper: http://www.compuware.com/media.asp?cid
========================================================================
eValid: Latest News & Updates
eValid is the premier WebSite Quality Testing & Analysis Suite.
eValid solutions help organizations maintain e-Business presence,
improve WebSite quality and performance, reduce down time, prevent
customer loss, and control costs.
eValid's Web Analysis and Testing Suite is comprehensive, yet
scalable and easy to use, and applies to a wide range of web
applications. Built entirely inside an IE-compatible full-featured
browser, 100% realistic user experience results are guaranteed.
New Variable/Value Manipulation Commands
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The latest revisions of eValid V5 include new features for:
* A new capability to provide a context to a script at playback
time, via a new "EnvironmentVariableFile" command. For details on
how this command operates, see:
http://www.soft.com/eValid/Products/Documentation.5/Technical/environment.variables.html
* A new selectable option for site analysis, including a Browse...
file selection option, to permit you to choose where to store you
site analysis filter reports.
Assuming you have enough machine capacity you now can run launch
multiple simultaneous eValid site analysis scans.
* Expansion of the currently available set of Reserved Variables
that allow you to instantiate important fixed local values into an
eValid test script at playback time. See this description for
details of how this works:
http://www.soft.com/eValid/Products/Documentation.5/Technical/reserved.variables.html
CeBit Show Details
^^^^^^^^^^^^^^^^^^
eValid will be represented at the upcoming CeBit show in Hannover,
Germany, 10-16 March 2005. If you are planning on being at CeBit
please visit:
Business Consulting Saxony (BCS)
http://www.sw-products.de/evalid_eng.htm
You'll find BCS people at CeBit Hall 4, Booth F-064.
German-speaking readers are invited to review the German Language
Brochure (PDF):
http://www.soft.com/eValid/Alliances/Germany/eValidFlyer.pdf
And, don't forget to check the German language White Paper,
"Qualitat von WebPrasenzen -- Eine Herausforderung."
You can read this entire paper in PDF format at:
http://www.soft.com/eValid/Alliances/Germany/WP.WebSiteQualityChallenge.pdf
New Log Filtering Option Support Monitoring
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^i^^^^
The latest builds of the eValid website test engine include two
capabilities that will be of interest to users who are using eValid
in monitoring mode. There are two changes now available:
* Selectable options via a new Message Log Filter that will let you
select which messages are put in the Message Log. For details
see:
http://www.soft.com/eValid/Products/Documentation.5/Settings/project.log.filters.html
* A capability to create a Custom Log as a subset of the Event Log,
based on a set of string matches that are user specified. For
details see:
http://www.soft.com/eValid/Products/Documentation.5/Settings/project.log.filters.html
HTTP Error Reporting
^^^^^^^^^^^^^^^^^^^^
A new capability for monitoring HTTP errors has been added to the
eValid playback engine. Users can select to have HTTP errors
reported as WARNINGs or ERRORs. In addition, detailed timing logs
generated by eValid now include the specific byte size and download
time of each page component separately. For complete details see:
http://www.soft.com/eValid/Products/Documentation.5/Settings/project.log.filters.html
New Data Import Capability
^^^^^^^^^^^^^^^^^^^^^^^^^^
A new eValid capability now simplifies importing data from
CallScript, GoScript, and _eValid commands, and with the -DATA
command line switch.
The new Value Extraction< feature gives eValid users a simple way to
import complex settings for use in processing complex parametric
scripts.
For complete usage information please see:
http://www.soft.com/eValid/Products/Documentation.5/Generate/value.extract.html
New 3D-SiteMap Capabilities, Options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The 3D-SiteMap portrays page-to-page dependency information that the
eValid site analysis process obtains by systematically scanning a
WebSite and then analyzing the dependencies between all of the pages
it viewed.
A new release of the eValid 3D-SiteMap display engine is now
available. The new additions to the 3D-SiteMap engine include:
o A capability to limit the displayed dependencies to an adjustable
depth for children and/or parents of a chosen root node.
o An option to show only the "immediate family" of a chosen root
(base) URL. This lets you see only the key parts of any 3D-
SiteMap.
You can see a complete working example along with the updated 3D-
SiteMap Summary Documentation at this page:
http://www.soft.com/eValid/Products/Documentation.5/Mapping/3Dsitemap.html
Enhanced Adaptive Playback for ButtonClick Command
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^i^^^^^^^^^^^^^^^^^^^^
Changes in the adaptive playback support have been made for eValid's
ButtonClick command. The new implementation provides for increased
flexibility in how eValid adapts to a changed button name or changed
button location. With this change fewer tests will fail for
inconsequential changes in the underlying website. The improved
method is available to regular eValid users as part of their regular
maintenance subscription.
Command Line Switches and Error Codes Expanded
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To support expanded use of eValid in monitoring applications and
other unattended operational roles, we have added a number of new
exit codes. The complete Playback Error Codes documentation gives
all of the details. See this page for details:
http://www.soft.com/eValid/Products/Documentation.5/Technical/error.codes.html
Of special note is the addition of error/exit codes for the new
eV.Manager batch-mode to automatically repeat application of an test
suite a fixed number of times in immediate succession.
See the eV.Manager Command Line Switches description at:
http://www.soft.com/eValid/Products/Documentation.5/Technical/interface.html#eV.Manager
Product Download Location, Details
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here is the URL for downloading eValid if you want to start [or re-
start] your evaluation:
http://www.soft.com/eValid/Products/Download.5/down.evalid.5.phtml?status=FORM
Contact Us With Your Questions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We welcome your questions about eValid and its applications. We
promise a response to every question in ONE BUSINESS DAY if you use
the WebSite request form:
http://www.soft.com/eValid/Information/info.request.html
========================================================================
ROI Dashboard Offered
Finding accurate and reliable benefit data from software technical
and management improvements can be a hit or miss proposition.
The Data & Analysis Center for Software (DACS) and its newly
launched web-based ROI Dashboard at:
http://r.vresp.com/?TheDataandAnalysisCe/5d905f23d1/283909/254d27cbd2/79227cd
DACS is offering a new solution where the software community can
find benefit data from software technical and management
improvements.
The ROI Dashboard augments and updates the DACS Report "A Business
Case for Software Process Improvement"
http://r.vresp.com/?TheDataandAnalysisCe/70891d5a0c/283909/254d27cbd2/79227cd
with the latest published data on benefits. The ROI Dashboard
graphically displays open and publicly available data and provides
standard statistical analysis of the data.
The data used in the ROI Dashboard was collected from open
literature:
* Articles in Communications of the ACM, CrossTalk, the IBM
Systems Journal, IEEE Transactions on Software Engineering,
Computer, Software, the Journal of Systems and Software,
Software Practice and Experience, and Software Process
Improvement and Practice.
* Papers presented at conferences, such as conferences sponsored
by the Software Engineering Institute and Software Engineering
Process Group (SEPG) conferences.
* Data taken from technical reports.
* Data collected by the DACS through surveys
The DACS ROI Dashboard) provides the user with capabilities to
examine the empirical distribution of the impact of software
technology improvements on the following variables:
-Quality
-ROI
-Productivity
The processes/methodologies covered include:
-SEI CMM/CMMI
-SEI Team Software Process (TSP)
-SEI Personal Software Process (PSP)
-Inspections
-Reuse
-Cleanroom
========================================================================
Mini-Survey on Test Training
by
Franco Martinig
Martinig & Associates
Editor of the Methods and Tools Newsletter
www.methodsandtools.com
Our last poll question was: How many weeks of training on software
testing have you completed in your professional life?
--------------------------- -----
None 43%
Less than one week 19%
One week (5 days) 7%
One to two weeks 7%
Two weeks to one month 6%
More than one month 18%
--------------------------- -----
Number of participants: 240
--------------------------- -----
As you can see, a large majority of the participants received none
or few testing related training from their employers. I think that
this situation is typical of the importance given to the testing
phase in many software development projects. The time is often
limited and there are few processes, tools and infrastructure
available to optimise the efforts of the developers. The lack of
training is just another factor that limits the efficiency of
testing efforts.
At the other end of the spectrum, you see an important percentage
with more than one month of training. This result should be
connected with the activity of M&T readers. Around 20% of them are
working in the software quality area and people working in this area
should have more testing related training than the average
developer.
========================================================================
========================================================================
------------>>> QTN ARTICLE SUBMITTAL POLICY <<<------------
========================================================================
QTN is E-mailed around the middle of each month to over 10,000
subscribers worldwide. To have your event listed in an upcoming
issue E-mail a complete description and full details of your Call
for Papers or Call for Participation at
<http://www.soft.com/News/QTN-Online/subscribe.html>
QTN's submittal policy is:
o Submission deadlines indicated in "Calls for Papers" should
provide at least a 1-month lead time from the QTN issue date. For
example, submission deadlines for "Calls for Papers" in the March
issue of QTN On-Line should be for April and beyond.
o Length of submitted non-calendar items should not exceed 350 lines
(about four pages). Longer articles are OK but may be serialized.
o Length of submitted calendar items should not exceed 60 lines.
o Publication of submitted items is determined by Software Research,
Inc., and may be edited for style and content as necessary.
DISCLAIMER: Articles and items appearing in QTN represent the
opinions of their authors or submitters; QTN disclaims any
responsibility for their content.
TRADEMARKS: eValid, HealthCheck, eValidation, InBrowser TestWorks,
STW, STW/Regression, STW/Coverage, STW/Advisor, TCAT, and the SR,
eValid, and TestWorks logo are trademarks or registered trademarks
of Software Research, Inc. All other systems are either trademarks
or registered trademarks of their respective companies.
========================================================================
-------->>> QTN SUBSCRIPTION INFORMATION <<<--------
========================================================================
To SUBSCRIBE to QTN, to UNSUBSCRIBE a current subscription, to
CHANGE an address (an UNSUBSCRIBE and a SUBSCRIBE combined) please
use the convenient Subscribe/Unsubscribe facility at:
<http://www.soft.com/News/QTN-Online/subscribe.html>.
QUALITY TECHNIQUES NEWSLETTER
Software Research, Inc.
1663 Mission Street, Suite 400
San Francisco, CA 94103 USA
Phone: +1 (415) 861-2800
Toll Free: +1 (800) 942-SOFT (USA Only)
FAX: +1 (415) 861-9801
Web: <http://www.soft.com/News/QTN-Online>