Goal-Question-Metric = (GQM) Approach

Analyze

(The = object under=20 measure)

Effectiveness of = structured=20 reviews

For=20 the purpose of

(Understanding,=20 controlling, or improving the object)

Understanding

With=20 respect to

(The = quality focus=20 of the object that the measurement focuses=20 on)

- The=20 detection of faults

- The=20 learning ability of the technique

From=20 the viewpoint of

(The = people that=20 measure the object)

The=20 project team

In the=20 context of

(The = environment in=20 which the measurement takes place)

Project=20 = B

SOFTWARE=20 ACQUISITION GOLD=20 PRACTICE^TM

GOAL-QUESTION-METRIC (GQM) = APPROACH

FOCUS=20 AREA:=20 process - Measurement

Definition and Summary: A = goal-driven=20 method for developing and maintaining a meaningful = metrics=20 program that is based on three levels, Goals, = Questions and=20 Metrics. The approach uses metrics to improve the = software=20 development process (and its resulting software = products)=20 while maintaining alignment with organization business = and=20 technical goals.

The consensus of a recent GAO = Report on=20 Defense Acquisitions (GAO-04-393), published in 2004, = was that=20 stronger management practices are needed to improve = DOD=92s=20 software-intensive weapons programs. The report = goes on=20 to state that the success (quality software product, = on time=20 and within budget) of leading software companies is=20 attributable to creating a manageable evolutionary = development=20 environment, using disciplined processes, and = collecting=20 and analyzing meaningful metrics to measure = progress. This=20 GQM Gold Practice^TM provides = practical=20 guidance for implementing the GQM approach to = measurement in=20 order to establish and use meaningful metrics, (which = are well=20 aligned with the goals of the organization), to help = mitigate=20 risk, track progress and support improvement = initiatives.

GQM is a top-down approach to = establish a=20 goal-driven measurement system for software = development, in=20 that the team starts with organizational goals, = defines=20 measurement goals, poses questions to address the = goals, and=20 identifies metrics that provide answers to the=20 questions. GQM defines a measurement model on = three=20 levels as illustrated in the figure below:

GQM can be applied to all = life-cycle=20 products, processes, and resources, and is well = aligned with=20 the organizational environment. The approach was = developed by Dr. Victor Basili and colleagues during = the 1980s=20 in conjunction with their work at the NASA Software=20 Engineering Laboratory (SEL), was refined during the = 1990s,=20 and now, serves as the foundation framework for many=20 measurement initiatives. It is an appropriate means = to achieve=20 reliable empirical data and knowledge about the = organization=92s=20 software practices to drive systematic process=20 improvement. In that context, it is particularly = useful=20 for the following purposes:

=A7 =20 Understanding and=20 baselining an organization=92s software = practices

=A7 =20 Guiding = and=20 monitoring software processes

=A7 =20 Assessing new=20 software engineering technologies

=A7 =20 Evaluating and=20 certifying improvement activities

Although the primary focus of = the practice=20 is goal-driven metrics definition, the approach = also =20 addresses data collection, analysis and interpretation = and=20 packaging experiences for use in future initiatives. = These=20 activities are equally as important as defining the = metrics=20 because they guide how the data is actually used.

Some effective practices for = implementing=20 GQM, discussed in detail in the document, are:

=A7 =20 Get the = right=20 people(all levels of developers) involved in the GQM = process=20

=A7 =20 Set = explicit=20 measurement goals and state them explicitly

=A7 =20 Thoroughly plan=20 the measurement program and document it (explicit and=20 operational definitions)

=A7 =20 Don=92t = create false=20 measurement goals

=A7 =20 Acquire = implicit=20 quality models from the team

=A7 =20 Consider = context

=A7 =20 Derive = appropriate=20 metrics

=A7 =20 Stay = focused on=20 goals when analyzing data

=A7 =20 Let the = data be=20 interpreted by the people involved

=A7 =20 Integrate the=20 measurement activities with regular project activities =

=A7 =20 Do not = use=20 measurements for other purposes

=A7 =20 Secure = management=20 commitment to support measurement results

=A7 =20 Establish = an=20 infrastructure to support the measurement=20 program =

=A7 =20 Ensure that=20 measurement is viewed as a tool, not the end = goal=20

=A7 =20 Get = training in=20 GQM before going forward

ANTICIPATED BENEFITS OF=20 IMPLEMENTATION:

Some = benefits of=20 this practice that were identified in the PERFECT = project=20 [PERFECT,1997]=20 include:

=A7 =20 Achievement of=20 improvement goals

=A7 =20 Financial=20 gains

=A7 =20 Increased=20 capability to perform improvement = initiatives

=A7 =20 Improved = group=20 synergy

=A7 =20 Increased quality=20 awareness and Quality Assurance (QA) = involvement

Description of = the=20 practice:

Summary DESCRIPTION

The Goal-Question-Metric (GQM) = Approach is=20 a paradigm for developing and maintaining a meaningful = metrics=20 program that supports:

=B7 =20 Metric alignment with organization = business and=20 technical goals

=B7 =20 Software process improvement

=B7 =20 Risk management

=B7 =20 Product quality improvement

This Gold Practice document = describes the=20 basic concepts of the GQM paradigm and addresses key = aspects=20 of its implementation. The practice focuses on=20 measurement of goals and supports interpretation of=20 measurement results relative to those goals. = Although=20 GQM originated as a measurement methodology for = software=20 development, the basic concepts of GQM can be used = anywhere=20 that effective metrics are needed to assess = satisfaction of=20 goals.

Benefits of = GQM

Although the direct benefit of = GQM is=20 establishing meaningful metrics, when GQM is applied = within the=20 context of systematic process improvement, it is = particularly=20 useful for the following purposes:

=A7 =20 Understanding and=20 baselining an organization=92s software = practices

=A7 =20 Guiding = and=20 monitoring software processes

=A7 =20 Assessing new=20 software engineering technologies

=A7 =20 Evaluating and=20 certifying improvement activities

In Europe, between 1993 and = 1996, several=20 industries formed a consortium with the goal of = initiating=20 measurement-based software process improvement. = That=20 initiative, called project =93PERFECT=94 (Measurement = Based=20 Improvement of Software Processes), noted the = following=20 benefits from using the GQM approach [PERF= ECT=20 1997]:

=A7 =20 Achievement of=20 improvement goals

=A7 =20 Increased quality=20 awareness and Quality Assurance (QA) = involvement

=A7 =20 Increased=20 capability to perform improvement = initiatives

=A7 =20 Improved = group=20 synergy

=A7 =20 Financial=20 gains

GQM Basics

The open literature typically = describes=20 GQM in terms of a six-step process where the first = three steps=20 are about using business goals to drive the = identification of=20 the right metrics and the last three steps are about = gathering=20 the measurement data and making effective use of the=20 measurement results to drive decision making and=20 improvements. In a recent seminar, [Ba= sili=20 2005] Basili described his six-step GQM process as = follows:

1. =20 Develop a set of corporate, division and = project=20 business goals and associated measurement goals = for=20 productivity and quality

2. =20 Generate questions (based on models) = that define=20 those goals as completely as possible in a = quantifiable=20 way

3. =20 Specify the measures needed to be = collected to answer those questions and track process = and=20 product conformance to the goals

4. =20 Develop mechanisms for data = collection

5. =20 Collect, validate and analyze the = data in real time to provide feedback to = projects for corrective action

6. =20 Analyze the data in a = postmortem=20 fashion to assess conformance to the goals and to make = recommendations for future improvements

As illustrated in the figure = below, GQM=20 begins by identifying measurement goals (conceptual = level)=20 that support (are aligned with) business goals. = The team=20 (project managers, development team, customers, or = other=20 stakeholders) then poses questions (operational level) = to=20 further clarify and refine the goals as well as = capture the=20 variation of understanding of the goals that exists = among the=20 stakeholders with respect to their notions of quality = and the=20 environment that will impact goal attainment. = The team=20 then identifies metrics that will provide answers to = the=20 questions (Quantitative level). What = distinguishes GQM=20 from other measurement paradigms is the hierarchical = tree=20 structure used to maintain the relationships among = goals,=20 questions and metrics.

Once=20 appropriate metrics are identified, the last three = steps of=20 the GQM process address how to implement the metrics = program=20 in a way that ensures the focus will remain on goal=20 attainment. Basili and other GQM experts stress = the=20 importance of planning data collection mechanisms and = planning=20 how the resulting measurement data should to be = organized and=20 presented in order to maximize its value to the = stakeholders=20 who will interpret the results in relation to the = goals. =20 The literature notes that when measurement programs = fail, the=20 primary cause of failure is often a lack of attention = to how=20 the measurement results will be used.

Implementing GQM

GQM can be=20 applied at the strategic level of an organization, or = the=20 project level, or at both levels concurrently. = When=20 applied at the strategic level, the measurement data = consists=20 of results from targeted pilot projects providing the = feedback=20 to strategic level planners for decision making = relative to=20 product and process strategies.

Organizations=20 often use a phased approach for implementing GQM that=20 integrates GQM-related activities (derived from = Basili=92s GQM=20 process) with project planning and management=20 activities. The phases are GQM Planning, = Definition,=20 Data Collection and Interpretation. A more = extensive=20 explanation of the phased approach, and how it relates = to=20 Basili=92s six step process, is provided in the body = of this=20 document.

Some effective practices for = implementing=20 GQM, described in detail in the body of this document, = are:

=A7 =20 Get the = right=20 people(at all levels of developers) involved in the = GQM=20 process

=A7 =20 Set and = state=20 explicit measurement goals and state them=20 explicitly

=A7 =20 Thoroughly plan=20 the measurement program and document it (explicit and=20 operational definitions)

=A7 =20 Don=92t = create false=20 measurement goals

=A7 =20 Acquire = implicit=20 quality models from the team

=A7 =20 Consider = context

=A7 =20 Derive = appropriate=20 metrics

=A7 =20 Stay = focused on=20 goals when analyzing data

=A7 =20 Let the = data be=20 interpreted by the people involved

=A7 =20 Integrate the=20 measurement activities with regular project=20 activities

=A7 =20 Do not = use=20 measurement for other purposes

=A7 =20 Secure = management=20 commitment to support measurement results

=A7 =20 Establish = an=20 infrastructure to support the measurement=20 program

=A7 =20 Ensure that=20 measurement is viewed as a tool, not the end=20 goal

=A7 =20 Get = training in=20 GQM before going forward

DetailED description

The Goal-Question-Metric (GQM) = practice=20 focuses on following the GQM paradigm for establishing = a=20 metrics program to support software development and=20 maintenance. Organizations typically implement = GQM as=20 part of an overall software process improvement = initiative,=20 but it is not limited to that role. The basic = concepts=20 of GQM can be used anywhere that effective metrics are = needed=20 to assess satisfaction of goals. It can even be = used by=20 individual members of a project team to focus their = work and=20 assess their progress toward achieving their specific=20 goals.

GQM Basics

In = one of his=20 most recent professional development seminars [Ba= sili=20 2005], Basili defines the GQM process as = follows:

1. =20 Develop a set of corporate, division and = project=20 business goals and associated measurement goals = for=20 productivity and quality

2. =20 Generate questions (based on models) = that define=20 those goals as completely as possible in a = quantifiable=20 way

3. =20 Specify the measures needed to be = collected to answer those questions and track process = and=20 product conformance to the goals

4. =20 Develop mechanisms for data = collection

5. =20 Collect, validate and analyze the = data in real time to provide feedback to = projects for corrective action

6. =20 Analyze the data in a = postmortem=20 fashion to assess conformance to the goals and to make = recommendations for future improvements

The first three steps are about=20 establishing a goal-driven measurement program where = the=20 identification of goals triggers the identification of = appropriate metrics. The remaining steps are = about=20 collecting and using the measurement results for = better=20 decision making. Later in this document, we will = discuss=20 how the process is implemented as a phased approach = that is=20 integrated with project planning and project = management, and=20 show how the process can be implemented at various = levels=20 within the organization.

Figure 1 presents=20 a high-level visual synopsis of the definition phase = of the=20 GQM paradigm, illustrating the outputs of first three = steps of=20 Basili=92s GQM process, the hierarchy of goals, = questions, and,=20 ultimately, meaningful metrics.

Figure 1: Basic GQM = Hierarchy=20

This part (the first three = steps) of=20 Basili=92s GQM process, often called the definition = phase of=20 GQM, provides the process structure for migrating from = concepts to meaningful metrics that, when implemented, = quantify the goals and provide meaningful data for=20 decision-making. Goals identify = what we=20 want to accomplish; questions, when = answered,=20 tell us whether we are meeting the goals or help us = understand=20 how to interpret them; and the metrics = identify=20 the measurements that are needed to answer the = questions and=20 quantify the goal.

As illustrated in Figure 1, the = mapping=20 among goals, questions and metrics is not = one-to-one. A=20 single measurement goal may apply to multiple business = goals=20 and vice versa; for each goal, there can be several = questions=20 and the same question can be linked to multiple goals = as=20 appropriate. For each question, there can be = multiple=20 metrics, and some metrics may be applicable to more = than one=20 question. Adherence to, and preservation of, = this=20 hierarchical structure helps ensure that the = measurement=20 program focuses on the right metrics and that we avoid = extra=20 work associated with collecting metrics that are not = really=20 needed.

The remaining steps of = Basili=92s GQM=20 process (relating to data collection and analysis for=20 decision-making and for future recommendations) are = actionable=20 only when appropriate definition of metrics has=20 occurred. Goal-driven metric definition, using = the=20 goal-question-metric process, is what separates GQM = from other=20 measurement methodologies. Thus, this document=20 emphasizes the initial steps of Basili=92s GQM process = because=20 they provide the foundation for the remaining steps of = the=20 process.

A primary tenet of GQM, not = usually=20 evident in illustrations of the GQM paradigm, is that=20 stakeholders need to be involved throughout the = process in=20 order for it to be successful. Basili, and = others [Bas= ili=20 2005, PERF= ECT=20 1997], advocate for planning the = implementation of=20 GQM to ensure that those with a stake in any part of = the=20 process actually participate to ensure their knowledge = is=20 considered, that they understand their contribution = (role) in=20 the process, and to promote their buy in and = acceptance of the=20 measurement program. Those who implement GQM may = use a=20 variety of approaches to ensure the appropriate level = of=20 participation. The key message is that the = measurement=20 program should be planned and implemented from within = the=20 organization or project, rather than be = outsourced. =20 However, the experts agree that it is helpful to have = a=20 consultant (GQM expert) work with the team or = organization in=20 the initial stages to ensure that the principles of = GQM are=20 implemented, and to transition those principles to key = people=20 within the organization.

GQM Process Details

This section provides details = about each=20 of the six steps of Basili=92s GQM process.

Step 1 - Establishing=20 Goals

In his recent DACS course [Ba= sili=20 2005] Basili lumps all goal setting into = the first=20 step of his GQM process description because he is = teaching=20 organizational leaders what the organization must do = to build=20 core competencies in software development (applying = GQM at a=20 strategic level). Others [PER= FECT=20 1997] suggest that the GQM process begins = with=20 establishing measurement goals using previously = defined=20 business goals as a driver. The essence of the = step=20 is:

=B7 =20 There are two types of goals --- = business goals=20 and measurement goals

=B7 =20 Business goals drive the identification = of=20 measurement goals

Sometimes, it=20 is difficult to distinguish between a business goal = and a=20 measurement goal; they may not always be mutually=20 exclusive. What is important is that the driving = goals=20 originate from the group or organization which is = responsible=20 for the broader scope of a software initiative, the = business=20 environment in which the initiative occurs, rather = than from=20 within a particular project. It is not important = whether=20 the business goals are developed under the umbrella of = GQM, or=20 as a function of strategic planning. Business = goals must=20 exist; they must be identified and be the focus for=20 establishing the measurement goals. Without = them, the=20 measurement program has no focus. Without this=20 alignment, it is unlikely that implementing the rest = of GQM=20 will have a significant impact. When business = goals=20 exist, then multiple projects or sub-groups within an=20 organization have a basis for identifying the = measurement=20 goals relating to their role or scope of influence = within the=20 organization.

The goals at=20 the top of the GQM tree (see Figure 1) are the = measurement=20 goals that are the outcome of step 1 of the GQM = process. =20 They are conceptual, not quantitative. They are=20 quantified by their linkage to questions and metrics = as noted=20 in the mapping. Some examples are provided later = in this=20 document to illustrate this point (see Figure 3).

Basili and his followers express = GQM goals=20 (measurement goals) using five facets of information = to define=20 what the measurement should accomplish in precise = terms. =20 Each GQM goal statement explicitly contains these = facets:

=B7 =20 Object: The = product or=20 process under study; e.g., testing phase or a = subsystem of the=20 end product

=B7 =20 Purpose: = Motivation=20 behind the goal (why); e.g., better understanding, = better=20 guidance, control, prediction, improvement

=B7 =20 Focus: The = quality=20 attribute of the object under study (what); e.g., = reliability,=20 effort, error slippage

=B7 =20 Viewpoint: = Perspective=20 of the goal (who=92s viewpoint); e.g., project = manager,=20 developer, customer, project team

=B7 =20 Environment: = Context=20 or scope of the measurement program; e.g., project X = or=20 division B

Figure 2=20 illustrates the refinement of a measurement concept = into a GQM=20 goal statement containing these facets of = information. =20 Similar examples can be found in = [van=20 Latum 1998], [PER= FECT=20 1997], [van=20 Solingen 1999b] and other referenced = articles.

Figure 2: The five Facets of a = GQM Goal=20 Statement

Some=20 implementations of GQM use a table-formatted template = for goal=20 definition (see Table 1). The left column = identifies the=20 five facets of information and remains constant on the = form;=20 the right column changes for each goal defined = [van=20 Solingen 1999b].

Table 1: GQM Goal = Definition=20 Template and Example

Step 2 =96 Generating = Questions=20

The purpose of Basili=92s Step 2 = is to=20 clarify and refine the measurement goals, moving from = a=20 conceptual level to an operational level by posing=20 questions. By answering the questions, one = should be=20 able to conclude whether a goal is reached. = Questions=20 help identify interpretations of the goal that may = exist among=20 the stakeholders as well as constraints imposed by the = environment. Typically, at the project level (or = perhaps=20 for a group of related projects), conceptual = measurement goals=20 are identified relating to product quality, process,=20 resources, or the environment. The project team = then=20 identifies questions that the team (individually or=20 collectively) feels should be asked to capture various = perspectives of the goal and address whether the goal = is being=20 met. These questions would typically get at all = of the=20 nuances and perceptions relating to the goal, = addressing both=20 perceptions of quality and the context or environment = in which=20 the object will evolve. This is essentially a = process of=20 stakeholders converging on a common understanding and=20 interpretation of the goal at the appropriate level of = abstraction. In other words, the individual = project=20 managers and software engineers provide their = perspective of=20 what the goal means in the given environment. = They do=20 this by posing questions and responding to them with = their=20 metrics. Figure 3 provides a simplified example=20 illustrating some of the questions that might emerge = for the=20 specified goals.

On the surface this step may = appear to be=20 trivial, and for some goals that may be the case, but = GQM=20 experts [Bas= ili=20 2005, van=20 Solingen 1999b, PERF= ECT,=20 1997] and implementers have found that = getting the=20 right level of abstraction for GQM questions can be=20 difficult. If questions are too abstract, the=20 relationship between the metrics and the question may = be=20 muddied. If they are too detailed, it becomes = more=20 difficult to get a clear interpretation of the = goal. In=20 many instances, particularly where the purpose of the = goal is=20 to understand or characterize the process or product,=20 questions may need to be broken out into many = sub-questions to=20 drive appropriate identification of metrics. The = implementing organization may tailor this step of the = process=20 as needed to ensure that the level of questioning is=20 sufficient to drive the identification of the right=20 metrics. Some excellent examples of ensuring the = appropriate level of questioning are provided in the = Case=20 Studies section of van Solingen=92s book [van=20 Solingen 1999b].

In some=20 implementations [van=20 Solingen 1999b], a=20 GQM team interviews the stakeholders (members of the = project=20 team) individually to capture their perspectives of = the goal=20 (their questions) and to have them formulate expected = answers=20 as hypotheses. These hypotheses make explicit = the=20 current knowledge that is in the minds of the team = members=20 thus, forming a baseline for later analysis of = metrics. =20 Comparing actual results with these hypotheses during = the=20 interpretation phase of GQM increases the learning = effect from=20 measurement.

Step 3 =96 Specifying the=20 Measures

Step 3 is about examining how = the=20 questions could be answered, moving from the = qualitative (or=20 operational level) to a quantitative level. Once = goals=20 are refined into a list of questions (GQM process step = 2),=20 metrics need to be defined that provide all the = quantitative=20 information to answer the questions in a satisfactory=20 way. Stakeholders, those directly involved with = the=20 object of the goal, must be directly involved in the = metric=20 identification step as well as the Question = step. Direct=20 involvement of these stakeholders minimizes = ambiguities and=20 false assumptions and contributes to the overall = consistency=20 and completeness of the metrics identification.

In this context, the term metric = is=20 loosely defined; it can mean a base measure, a derived = measure, a composite or aggregate of measures, or, = what some=20 would call, an indicator. The level of = definition=20 depends on the scope of the goal and the environment = of the=20 GQM implementation. Further details are provided = later=20 in the section describing GQM implementation. = Figure 3=20 provides a simple example of how GQM would derive = metrics for=20 two goals. It shows some of the essential = questions that=20 would emerge, and some of the metrics that might be=20 identified.

The diagram is intentionally=20 oversimplified in order to convey the notion of how = one gets=20 from a conceptual level goal to the right quantitative = data=20 that renders the goal measurable. It also conveys the = multiple=20 mapping of metrics to questions and questions to = goals. =20

3D""=20
Figure 3: = Sample of GQM=20 Definition Phase

A refined GQM goal statement for = Goal 1 in=20 Figure 3 might read as follows:

Analyze: = ; =20 Change request = processing

For the = purpose=20 of: =20 Improvement

With = respect=20 = to: &nbs= p; =20 CR=20 processing = cycle time

From the = viewpoint=20 of: The=20 project team

In the = context=20 = of: &nbs= p;=20 The current = project=20 timeframe

Note a distinction between the = metrics=20 that are defined and the data elements that support=20 them. The metric is at a more abstract level = than the=20 actual data items and measurements that need to be = collected=20 to provide the correct data for preparing the metric. =

Step 4 =96 Preparing for = Data=20 Collection

Once the metrics are identified, = one can=20 determine what data items are needed to support those = metrics,=20 and how those items will be collected. The = metric=20 provides insight regarding how the data needs to be = organized=20 in order to be meaningful to the viewer/recipient of = the=20 information. A significant amount of planning is = necessary to provide the detailed procedures for data=20 collection that support the identified metrics. = Most=20 projects accomplish this detailed planning by = preparing a=20 Measurement Plan that includes at least the = following=20 [van=20 Solingen 1999b]:

=B7 =20 Formal definitions of direct = measurements

=B7 =20 Textual descriptions of direct = measurements

=B7 =20 All possible outcomes (values) of the = direct=20 measurements

=B7 =20 The person (role) that collects each = direct=20 measurement

=B7 =20 The moment in time (or frequency) when = the=20 direct measurement should be collected

=B7 =20 The medium (tool or form) that should be = used=20 for collecting the measurement

The plan also defines and = describes all=20 types of data collection forms and automated data = collection=20 tools that should be used. It addresses the = question of=20 how the data can be collected most efficiently and = effectively=20 and to whom it should be delivered.

Referring to the example in = Figure 3 of=20 Change Request processing time, one cannot assume that = all=20 stakeholders have the same understanding of what = constitutes=20 CR processing time. Does it begin when the CR is = first=20 documented, or after it is diagnosed and categorized = for=20 action? When is processing considered done? = Which CRs=20 are included for averaging --- only closed out CRs, or = also=20 those on hold? The Measurement Plan = anticipates=20 and addresses such questions.

Once a plan is developed, the = measurement=20 procedures need to be tested and validated before = implementing=20 the program. Exercising the forms and procedures = during=20 a trial period will reveal flaws that can subsequently = be=20 corrected before full-scale implementation of the = measurement=20 program begins, or before adding the new procedures = and data=20 to an existing program [van=20 Solingen 1999b].

In addition, it is important to = train=20 individuals involved in data collection to ensure that = they=20 understand why the data is needed, how it is going to = be used=20 and how their action contributes to the overall = validity of=20 the data collection process.

Step 5 =96 Collecting, = Validating and=20 Analyzing the Data for Decision Making

This step presumes that data = collection=20 follows the procedures pre-defined in the = Measurement=20 Plan. It is often a continuous (or periodic) = process=20 rather than a one-time activity. However, it is = only a=20 means to an end. Data collection is a worthless = process=20 if one does not do anything with the data. The = focus=20 needs to be on preparing the data for optimal = usage. =20 Regardless of the collection medium, data needs to be=20 validated before it is used for analysis.

Recently a Project Manager told = the DACS a=20 story about a group of developers responsible for = maintaining=20 an application that was particularly volatile. = Each time=20 they fixed a problem they entered a =931=94 in the = Problem Report=20 (PR) form field that asked for actual time spent on = the=20 fix. From their perspective, this was an = irrelevant=20 field because nothing was ever done with the = data. They=20 were very busy with a backlog of PRs and it took time = to=20 accurately estimate the time to fix each = problem. Their=20 Manager, who knew that many PRs required extensive = time to=20 fix, showed them a request for additional staff that = had been=20 denied based on a report of staff utilization that = showed a=20 graph of average time per PR, clearly demonstrating = that the=20 current staff was underutilized. Once the staff=20 understood the significance of that data item, what = decisions=20 it affected, and that it was not measuring their = personal=20 performance, they started entering their estimates of = time=20 appropriately and the manager began to get valid data = for his=20 management purposes.

Automation can assist, but it = cannot=20 replace all forms of data collection and = validation. The=20 key is to minimize the overhead imposed on the people = who are=20 required to provide data while ensuring that they = understand=20 the significance of their data collection = effort. =20 Validation consists of checking the data collected for = correctness, completeness and consistency. = Completeness=20 is the most significant data collection problem. = What=20 does one do with a form that is only half = completed? How=20 does one ensure that all instances of data are = actually=20 captured? These, and similar questions, need to = be=20 addressed in planning the measurement program so that = there is=20 a proper course of action available to those people = tasked=20 with validating.

As the story about the = maintenance staff=20 showed, leaders need to reinforce the purpose and = value of=20 data collection to promote better data collection=20 quality. A significant part of the validation = process is=20 checking validity as close to the data source as = possible and=20 taking immediate corrective action for invalid = data.

Once validated, it is important = to store=20 the measurement data in such a way that it can be = accessed for=20 varying analysis and reporting purposes. Because = of the=20 sheer volume of measurement data needed for all but = the=20 smallest of projects, it is useful to develop and use = a=20 measurement support system that contains a database = for the=20 storage of the metric data and tools for analysis=20 (spreadsheets) and presentation. Flexibility and = accessibility are the most important features of such = a=20 system.

Analysis is about organizing the = data and=20 preparing the metrics for presentation to the = stakeholders to=20 address the questions pertaining to the measurement=20 goal. Typically, a GQM team, together with the = project=20 team, develops an analysis plan as soon as they know = what=20 metrics are needed. The plan details how the = data should=20 be organized, what presentation formats are needed, = who will=20 review the data and when. Developing the = analysis plan=20 often helps with decisions about data = collection. Basili=20 [Ba= sili=20 2005] uses the term analysis to mean both = analysis=20 and interpretation, but some implementers of GQM make = a=20 distinction between the two terms, primarily, to = assert that=20 the analysis can be done by a GQM team (measurement = expert),=20 but the interpretation must be done by the project = team who=20 are the owners of the measurement goals.

Some form of feedback is = required to=20 communicate measurement results to the appropriate=20 stakeholders. These sessions are focused on the=20 measurement goal and reviewing the measurement results = to=20 answer the questions posed in step 2 of the GQM = process. =20 The project team can then decide on corrective action = when=20 progress toward goals is not deemed adequate.

Analysis and interpretation is = an=20 iterative step typically integrated with the progress=20 reporting cycle of a project.

Step 6 =96 Analyzing the = Data for Goal=20 Attainment and Learning

The last step in Basili=92s GQM = process is=20 about looking at the measurement results in a = post-mortem=20 fashion to assess goal attainment and also to = determine=20 lessons learned and what might be valuable to pass on = to=20 future projects.

When GQM is implemented to = support an=20 organization-wide improvement process, the experiences = and=20 lessons learned from each implementation are packaged = in the=20 form of policies, procedures and best practices, to = support=20 future projects and improvement initiatives and to = help the=20 organization achieve greater leverage from its = measurement=20 program.

Implementing GQM

Describing GQM in terms of a = six-step=20 process (see previous section titled GQM Basics) tends = to=20 convey it as a strictly sequential process, but that = is not=20 the case when GQM is actually implemented. Van = Solingen=20 [van=20 Solingen 1999b] and others assert that = implementation of=20 GQM should be viewed in terms of phases of activities = that are=20 integrated with project planning and management and = have=20 dependency relationships with each other. = Although they=20 embody Basili=92s GQM process, the focus of the phases = is on the=20 planning and implementation details needed to make GQM = a=20 reality within an organization. Solingen=92s = phases are=20 named as follows and their relationship to each other = is=20 illustrated in Figure 5:

=B7 =20 GQM Planning: deals = with the=20 logistics of implementing GQM and key plans that need = to be=20 documented. This phase, therefore, touches steps = 1 -5 of=20 Basili=92s process.

=B7 =20 Definition: focuses = on using=20 the GQM method to derive meaningful metrics. It=20 encompasses the first three steps of Basili=92s = six-step=20 process.

=B7 =20 Data Collection: addresses = planning for and executing data collection activities = to=20 obtain the data needed for the defined metrics. = This=20 phase addresses steps 4 and 5 of Basili=92s = process.

=B7 =20 Interpretation: = Addresses=20 preparing the measurement data in ways that facilitate = analysis and interpretation of results relative to the = pre-defined goals and actually doing the analysis and=20 interpretation. This phase implements steps 5 = and 6 of=20 Basili=92s process.

The following paragraphs provide = further=20 details about van Solingen=92s GQM implementation = phases.

GQM=20 Planning and Definition Phases

It = takes a=20 significant amount of planning to ensure that data = collection=20 and analysis are in fact coordinated with and actually = address=20 the defined measurement goals. Therefore, GQM needs to be integrated = with,=20 rather than separated from, project planning. = The=20 overall Project Plan specifies, that the GQM method = will be=20 utilized, and that GQM planning activities will = occur. =20 The GQM Planning phase oversees the = implementation of=20 GQM within the context of the project. It does = not=20 precede all other phases, but rather, is intermingled = with=20 them. Initially, the GQM planning phase = establishes how=20 the Definition phase will be implemented and who will = be=20 involved; then Definition occurs. Then GQM = planning=20 uses the output of the definition phase as a basis for = the=20 planning of data collection mechanisms and for = analysis and=20 interpretation. GQM planning thus provides = documentation=20 that serves as a guide for other phases of GQM, and = provides=20 the necessary integration with project planning. = That is=20 why the Planning phase block in Figure 5 stretches = across the=20 remaining phases.

Artifacts of=20 the GQM Planning phase include, but are not limited = to, the=20 following documents:

(1) =20 GQM=20 Plan: As illustrated in Figure 5, = the=20 Definition phase consists of identifying = measurement=20 goals, posing questions, and identifying appropriate=20 metrics. The GQM plan=20 is a document that contains each = measurement goal=20 and its corresponding breakdown into questions and = metrics,=20 thus preserving the relationships of goals to = questions to=20 metrics. This document provides the foundation = for=20 progressing through the other phases of GQM, just as a = requirements document provides the foundation for the=20 activities of a development effort. The GQM plan = contains the output from the first three steps of = Basili=92s GQM=20 process [Ba= sili=20 2005] described earlier.

(2) =20 Measurement = Plan: The GQM planning = phase,=20 additionally, includes developing a specific Measurement Plan = that defines=20 the actual base measures needed to generate the = metrics=20 defined in the GQM plan and establishes the = detailed=20 procedures for collecting the measurement data and = generating=20 the identified metrics. It should address what = data is=20 collected, how it is to be collected, by whom, and = when. =20 The Measurement Plan includes a description of = the=20 media needed to report, collect, and validate = data. It=20 serves to guide the activity of the Data Collection = phase. Developing this plan is part of step 3 of = Basili=92s GQM process.

(3) =20 Analysis=20 Plan: The GQM Planning = phase also=20 encompasses developing an Analysis Plan that = focuses on=20 how to analyze, aggregate, and present the collected=20 measurement data in ways that are meaningful to the=20 stakeholders. What the project manager needs to = see is=20 different from what the Vice President needs to see = and from=20 what the Quality Assurance manager needs to see. = The=20 plan sets the stage for the Interpretation = phase of GQM=20 by providing guidance on how the information needs to = be=20 organized to facilitate its use and ensure that the = focus=20 remains on the goals. Basili=92s process = description does=20 not specifically mention preparing this plan, but he = implies=20 (or perhaps assumes) that one makes a plan before one = does=20 analysis. It is likely to be considered part of = Basili=92s=20 steps 4 or 5 that address developing data collection=20 mechanisms and analyzing the data. The important = point=20 is that the planning for analysis and interpretation = should be=20 done prior to data collection so that it is clearly = tied to=20 the goals and does not fluctuate depending on the = actual data=20 values collected. The Analysis plan = defines the=20 appropriate level of abstraction for the presentation = of the=20 data.

Data=20 Collection Phase

The = Data Collection = phase=20 includes collecting the measurement data according to = the=20 Measurement Plan and preparing it for analysis = in=20 accordance with the Analysis Plan. This = phase is=20 covered by Basili=92s GQM process step 5. During = collection, data needs to be validated by checking for = such=20 things as data completeness, timeliness, and = accuracy. =20 Once validated, the data can be presented to the team = and=20 other appropriate recipients for interpretation. = A key=20 activity of this phase is to store the data in such a = way that=20 it facilitates access by the project team and others = both=20 during and after a project. Most GQM = implementations=20 develop a metric database for this purpose.

Interpretation = Phase

After the data=20 is collected, validated and prepared for review, in = accordance=20 with the Analysis Plan, the Interpretation = phase=20 involves review of the measurement results by the = stakeholders=20 (typically the project team). The team = interprets the=20 data in light of the questions asked and the ultimate = goals=20 defined. Do the measurement results provide = answers to=20 the questions addressed? To what extent has the = goal=20 been attained? Typically, during the = Interpretation=20 phase the team determines not only the extent to = which the=20 actual measurements address their corresponding = questions and=20 the extent to which the measurement goals were = attained, but=20 also what improvements can be made to the measurement = program=20 itself. Success in this phase is premised on the = principle that the measurement program must address = the=20 interests of those providing the data and must be = based on the=20 project team=92s knowledge. These feedback = sessions=20 typically lead to updates to the Measurement = and=20 Analysis Plans, and sometimes, to the = alteration or=20 addition of measurement goals. Keep in mind that = measurement goals may be related to the product as = well as the=20 process or resources. The visibility of = information=20 provided by the presentation format of the measurement = results=20 makes it easier for the project team to take = corrective action=20 during the project, since they are monitoring progress = on a=20 periodic basis. They use the quantitative = information=20 for fact-based decision-making. Basili includes = this=20 analysis and interpretation activity as part of his = step 5 of=20 the GQM process.

This phase also=20 addresses making decisions about the preservation of=20 measurement results for use in the future and lessons=20 learned. The GQM team (or persons with overall=20 responsibility for the measurement program) determines = how=20 such data should be packaged to provide optimum = accessibility=20 and use for future efforts. This activity is = consistent=20 with the last step of Basili=92s GQM process.

Applying GQM at Varying Levels = Within an=20 Organization

The = GQM=20 methodology is quite generic and thus its scope of=20 implementation can range from somewhat narrow (an = individual=20 using GQM to achieve desired goals) to very broad, = such as=20 when it is used in strategic planning. The open=20 literature delineates the GQM approach in terms of a = six-step=20 process, but the descriptions and boundaries (entry = and exit=20 points) of each step vary depending on the context of=20 implementation, and the publication period within the = body of=20 literature. Basili himself now defines the steps = somewhat differently than he did in his original = seminal=20 article in 1984 [Ba= sili=20 1984]. A review of several variations = of the=20 six-step process reveals that, irrespective of the = step number=20 assigned, or the different wording, each of the = descriptions,=20 collectively, contain the same essential concepts and = assign=20 the same relationships to activities of the overall=20 process.

In = the context=20 of organizational process improvement, GQM has been = applied=20 concurrently both at a project level and the strategic = level=20 [PER= FECT=20 1997]. This interplay of strategic versus = project=20 level GQM is illustrated in Figure 6. Although = the steps=20 are not described in exactly the same manner, these = steps are=20 consistent with the steps outlined by Basili as noted = earlier=20 in this document. The major distinction between=20 strategic level GQM and project level GQM relates to = the data=20 collection phase (step 4 of both levels). The = strategic=20 level implementation identifies pilots and projects as = the=20 source of data for strategic level metrics. = Therefore,=20 strategic level data collection is dependent on the = designated=20 projects actually implementing GQM at their project = level and=20 packaging results to feed back to the strategic level=20 implementation.

Figure 6: = GQM Process -=20 Strategic vs. Project Level

When applied at=20 a strategic level, GQM is still a six-step process, = with the=20 first three steps focusing on goal and metric = identification=20 and planning the measurement implementation, step 4 = addressing=20 data collection, and steps 5 and 6 addressing analysis = and=20 interpretation. The objects of focus differ = slightly at=20 the strategic level from those at the project = level. =20 Data collection, (strategic level, step 4) is about = gathering=20 the information, experiences and lessons learned from = the=20 multiple targeted projects, as the basis for the = strategic=20 level analysis phase.

For = example, a=20 software development organization may desire to = implement=20 formal inspections as a practice in all of its = software=20 development projects. Before making it a policy, = they=20 would identify some target projects in which to test = the=20 formal inspection process. The strategic level=20 measurement goal may be characterizing the ROI from=20 implementing formal inspections, and setting a = specific=20 threshold that would indicate that it would be = beneficial to=20 include the process as a fundamental practice for all=20 projects. They might identify a variety of = projects for=20 pilots to include both large and small projects and = projects=20 of short and long duration. Each pilot project = would=20 then apply GQM to measure the impact of formal = inspections in=20 their particular project environment. The = organization=20 would follow its plan for collecting and analyzing the = measurement results from the projects and use those = results to=20 plan the roll out of formal inspections across all = projects=20 within the organization.

This is easier to see when we = look at how=20 these steps are aligned with the GQM phases defined = earlier.=20 At both levels (strategic and project), the = first three=20 steps of the process entail goal alignment, goal=20 identification, metric identification and development = of the=20 measurement plan. These activities are = consistent with=20 what occurs in the Planning and Definition = phases of a=20 generic GQM implementation. Step 4, at both = levels, is=20 implementing the data collection phase of GQM; = however,=20 the type of data collected and reported differs = depending on=20 the level. Steps 5 and 6 at both levels address = the=20 Analysis and Interpretation phase of GQM. = At each=20 level, the analysis is tailored to address the needs = of the=20 stakeholders at that level. At both levels, = steps 5 and=20 6 are addressing actual use of the data to assess the = degree=20 to which goals are being met, and determining how best = to=20 package and/or preserve the results for future or = broader=20 use.

The = notion of=20 packaging experiences is essentially documenting and = sharing=20 lessons learned, making the information available to = future=20 projects or for the next refinement of the improvement = cycle. This is analogous to Basili=92s, process = step 6 ---=20 doing post-mortem data analysis.

Strategic level GQM analyzes the = experiences from the individual projects in order to = identify=20 what improvements yield the biggest gain and under = what=20 circumstances. This information is then packaged = for use=20 in future projects in the form of guidelines, policies = and=20 procedures and, perhaps, support tools provided to the = projects. This is difficult because future needs = for=20 measurement information are generally unknown. = The more=20 precisely you can describe your actual experiences and = their=20 specific context, the greater your chances for = adaptive reuse.=20

The notion of packaging = experiences for=20 adaptive reuse is captured in what Basili has coined = an=20 =93Experience Factory=94 (see Figure 7). It is = an=20 organizational concept established for = institutionalizing the=20 collective learning of the organization, the basis for = their=20 continual improvement.

Figure 7: Concept of an = Experience=20 Factory

The concept was developed by = Victor Basili=20 and first implemented at the NASA Software Engineering = Laboratory (SEL) in the late 1970s. A separate = group is=20 formed (which might start out as the GQM team) that = supports=20 reuse of experience by developing, updating, and = delivering=20 experience packages to the project groups responsible = for=20 developing and maintaining software. The = Experience=20 Factory team focuses on organizing and preserving the=20 information and data to optimize its reuse, and on = presenting=20 solid information back to the project groups to guide = them in=20 future projects.

GQM and the Quality Improvement=20 Paradigm

GQM is effective when = implemented as part=20 of a broader quality improvement initiative since one = of the=20 main purposes of measurement is improvement.

Figure 8 illustrates Basili=92s = six step GQM=20 process overlaid on the six step process (1 - = Characterize, 2=20 - Set Goals, 3 - Choose Process, 4 - Execute, 5 - = Analyze, 6 -=20 Package) of the Quality Improvement Paradigm (QIP), as = described in the Perfect Handbook [PER= FECT,=20 1997] and other sources = [van=20 Latum, 1998]. What this means is that, = although the=20 terminology may be slightly different depending on = which=20 perspective or focus one has, what is happening is = essentially=20 the same cycle of activities. This is because it = does=20 not make sense to talk about improvement without = metrics, and=20 it does not make sense to implement a metrics program = without=20 a greater purpose, be it process improvement, risk = mitigation,=20 or product quality, all of which can be viewed as = forms of=20 improvement. If one implements all phases of the = GQM=20 paradigm, one is seeking improvement. Note, = however,=20 that this does not preclude implementing the = definition phase=20 of GQM outside of a formal improvement framework = simply to=20 assist with the identification of meaningful = metrics.

In Figure 8, the boxes = containing the GQM=20 steps have intentionally been positioned across the = QIP=20 boundary lines to show an overlapping of Basili=92s = GQM steps=20 with adjacent steps in the QIP process.

GQM plays an important role in = software=20 process improvement in that it helps in achieving = reliable=20 data about software projects and software development=20 practices of the organization and provides the basic=20 information necessary for organizational learning and = guiding=20 of software projects.

Figure 8: GQM Integrated = with the=20 Quality Improvement Paradigm

Key Practices of the GQM = Approach:

Based on the work of Basili and = others [Bas= ili=20 1984,199= 4,=20 199= 9,=20 200= 2,=20 200= 5;=20 = van=20 Latum 1998; van=20 Solingen 1999; Lava= zza=20 2000], the practices listed herein are a = combination of=20 the main tenets of GQM, success factors identified by=20 developers in GQM feedback sessions and generic = practices that=20 are applicable to the implementation of any = measurement=20 methodology.

1=20 - Get the right people involved in the GQM process:=20 Although GQM is often described = as a top=20 down approach, it does not mean that upper management = defines=20 the metrics and hands their plan over to the project = team for=20 execution. Upper management provides guidance = and=20 direction by making available clearly defined project = and=20 organizational goals. The developers (usually a = GQM team=20 working with the project team) define the measurement = goals=20 and ultimately, the metrics. A GQM team may need = to=20 coordinate this task among several projects to ensure=20 consistency of some metrics across projects. = This=20 activity would be driven by a higher organizational = need for=20 consistent metrics to improve activities such as cost=20 estimating or benchmarking. The right people = must be=20 involved in every phase of GQM measurement.

The key roles involved in GQM = are:

=B7 =20 GQM Goal Owner =96 the person, or = persons, whose=20 viewpoint is stated in the measurement goal and = who is=20 responsible for the whole analysis task; a project = manager,=20 test manager, quality assurance (could also be a = customer)

=B7 =20 Measurement Manager =96 the person = responsible for=20 running the measurement program (operational = perspective) and=20 executing the measurement plan. This role, which = could=20 be within or outside of the project team, is = explicitly=20 identified in both the project plan and the = measurement=20 plan.

=B7 =20 Data Provider =96 any person who = provides data=20 during data collection. They need to be involved = in=20 order to understand exactly how the data they collect = will be=20 used and that it is being used as planned.

=B7 =20 GQM Expert - person who performs the = technical=20 tasks of the measurement program, such as = interviewing, and=20 deriving the measurement plan. Initially this = might be a=20 consultant but, after a training period, this role = typically=20 transitions to key project staff.

=B7 =20 GQM Team =96 depending on the size of = your=20 organization and the scope of the proposed GQM = implementation,=20 it may be valuable to create a GQM team that = specializes in=20 the GQM process. Such a team is often integrated = with=20 the group that provides Quality Assurance to the = projects, but=20 its focus is on implementing GQM. The team might = initially include an external GQM consultant. = The team=20 does a lot of coordination with project managers as = well as=20 much of the initial planning and analysis. = However, the=20 GQM team does not interpret the data. It merely=20 facilitates data interpretation by focusing on various = ways to=20 present measurement results to key people.

2=20 =96 Set explicit measurement goals and state them = explicitly:=20 Measurement goals are not = organizational=20 goals or project goals. They are goals that = describe how=20 to assess progress toward project goals and = organizational=20 goals. There are subtle differences. It is = important for all members of the project team to = understand=20 and distinguish among these three types of = goals. =20 Measurement goals direct the alignment of measurement = activity=20 with the business goals (project level and = organization level)=20 of the organization and, as such, guide all subsequent = activity of the GQM process, so it is important to = focus on=20 them.

Are they the right goals? = Have we=20 identified all of the key goals? Is their = meaning=20 crystal clear? Developing explicit goal = statements,=20 using the five facets of information identified in = Table 1,=20 helps provide that clarity and remove the ambiguity of = implied=20 assumptions among the team.

3=20 =96 Don=92t create false measurement goals:=20 Do not create goals simply = to match=20 the metrics you may already have. = You may=20 be collecting the metrics that are easy to collect, = without=20 proper analysis to determine if or how they fit into = your=20 measurement program. It is easy to get over = zealous=20 about measurement and identify metrics that are easy = to=20 collect and look at. The perspective of =93We = can get this=20 data, so what can we do with it?=94 or =93 Let=92s = look at the data=20 that we already have (so as not to create extra work) = in=20 different ways and see what we can discover=94 is a = form of=20 creating false goals. This perspective is = sometimes=20 evident in projects that do not follow a disciplined=20 development process. It is comfortable; it is = sometimes=20 a good way to get people who are =93anti-metrics=94 to = adjust=20 their attitude, but this mindset is, in fact, the = reverse of=20 GQM. It bypasses goal definition and refinement=20 (questioning) and goes directly to identifying metrics = first. Since the focus is on the metrics, this = practice=20 results in implicit assumptions about the goals or = purpose of=20 the metrics. This practice may be better than = not=20 measuring at all, but it may be difficult to migrate = from this=20 to a GQM mindset. The key point is, if you are=20 implementing GQM you must let goals drive the = selection of=20 metrics. You must focus on goals first, independent of = current=20 metrics or data you might have. The goals should = drive=20 the identification of the right metrics, or you are = not doing=20 GQM.

4=20 =96 Acquire implicit quality models from the people = involved:=20 Note that, while it is important = to=20 define measurement goals explicitly, GQM tries to = acquire the=20 implicit quality models, i.e., the = notions of=20 quality that the development team members or the = customers=20 have in their heads. Developers, at all levels, = need to=20 be involved in identifying their notions of quality, = project=20 constraints and specific elements of the environment = that can=20 affect quality. Their perception of quality is = important=20 because they are closest to the development = effort. =20 Their input, in the form of questions that address = their=20 perception of quality, is essential if meaningful = metrics are=20 to be identified. Without their understanding = and=20 acceptance of both the goals and the metrics that = quantify=20 them, GQM will probably not be an effective course of=20 action. Here are some questions that tend to = reveal the=20 variation of perceptions that might exist within the=20 development team:

=B7 =20 What is product quality? Process = quality?

=B7 =20 When is it ok to release product?

=B7 =20 Is our current product quality good or = bad? How=20 do we know?

=B7 =20 Are certain quality attributes more = important=20 than others to us? To our customers?

=B7 =20 Does our notion of good quality match = that of=20 the customer?

=B7 =20 Does the project manager have a = different notion=20 of quality than the designer, or the test = engineer?

=B7 =20 Does everyone assume that team members = all have=20 the same notion of quality, so, it has never actually = been=20 discussed?

Somewhere within the project, or = the=20 organization, there may be some definitions of various = quality=20 attributes, but that does not mean that a common = awareness of=20 those concepts exists among the team members. = Through=20 questions, the team members put their implicit notions = (their=20 quality models) of the object of measure on the table = for=20 discussion, thus providing a more detailed definition = of the=20 goal=92s quality focus = [van=20 Latum, 1998].

Although a GQM team (or = specialist)=20 typically leads the goal definition initiative, all = project=20 team members should be consulted for two reasons, (1) = to=20 solidify their understanding of the goals and (2) to = optimize=20 their motivation to reach the goals.

The = success of=20 this practice depends on many people issues, and = therefore,=20 techniques, such as facilitated discussion and = personal=20 interviews, should be applied as = necessary. Some = generic=20 techniques that may be useful for getting through the = GQM=20 definition phase are:

=B7 =20 Brainstorming - a = workshop=20 technique used to generate ideas that could be useful = in=20 solving some problem. It enables a group of = people to=20 come up with various suggestions for a possible = solution to a=20 specific issue. It is run by a facilitator who = should=20 merely record suggestions without comment. It is = important that no suggestions be in any way belittled = as=20 sometimes an off-the-wall suggestion leads to an = innovative=20 solution. Once the brainstorming session is = completed,=20 the ideas are reviewed rationally to see if any of = them=20 contribute to the solution of the problem in=20 question.

=B7 =20 Facilitation - = facilitation is=20 a technique of encouraging discussion among a group of = individuals who are engaged in some problem solving=20 activity. Many groups contain dominant = personalities=20 whose influence can overpower the discussion. = The=20 facilitator should be capable of encouraging the = widest=20 possible input from everybody taking part. The=20 facilitator is also required to suggest lines of = debate, if=20 the discussion dries up or comes to a natural halt. = They=20 should ensure that all aspects of the subject under = debate are=20 discussed.

=B7 =20 Mentoring -a = mentor is an=20 advisor to a less experienced person in some field of=20 endeavor. The mentor is (or should be) seen as a = trusted=20 advisor who can provide support and guidance which is=20 encouraging and non-judgmental. In all too many = cases,=20 mentors are not properly trained and are often chosen = because=20 of their technical skills rather than their personal=20 ones. Effective mentoring is very reliant on = personal=20 skills; a badly chosen mentor may do irreparable harm = to a=20 less experienced person. Conversely, well = conducted=20 mentoring can be a very powerful way of transferring=20 knowledge, skills and responsibilities. The very = best=20 mentors often claim that they have learned more from = the=20 experience than they taught.

=B7 =20 Using a=20 collaborative workflow tool =96 = web-based=20 software that allows for idea sharing, planning and = decision=20 making in a controlled environment, usually involving = a=20 moderator or facilitator. It is deemed useful = for=20 requirements negotiation, and GQM is about = establishing and=20 implementing requirements for measurement.

Team members need to feel = comfortable and=20 confident to express themselves. The process can = easily=20 break down into a situation where a single individual=20 dominates and only their thoughts surface. = Conflicting=20 notions may emerge and need to be resolved among the = team=20 members before proceeding to the metrics = identification=20 stage. Fifty questions might surface, but the = team may=20 end up with five or six that are deemed sufficient to = fully=20 address the goal. Facilitation is advisable for = this=20 process. Techniques such as affinity grouping = may also=20 be valuable. Some of this can be done via = interviews=20 with results captured in abstraction sheets. = Figure 9=20 shows the structure of a typical abstraction sheet. = The=20 content represents an interview with the project = manager of=20 the specified project. In large projects, where = having=20 face-to-face meetings is difficult due to the size of = the=20 staff, personal interviews, conducted by the GQM team, = take=20 the place of open discussion. Abstraction sheets = are=20 used to capture the perspective of individuals, and = then=20 analyzed by the GQM team. Differences in the = quality=20 focus or the variation factors emerge among the=20 interviewees. Such differences need to be = addressed=20 through further iterations of GQM in order to ensure = that the=20 process will result in convergence of the team = regarding the=20 quality focus of each of the goals.

Figure 9: Sample of a = Completed=20 Abstraction Sheet

5=20 - Consider context: What = are the=20 variation factors of the quality focus due to the = context of=20 our project? What constraints or limitations do = we=20 have? What factors do we expect will most = influence our=20 perceived quality model? Team members pose = questions (Q=20 stage) to address how the particular context of the = project=20 environment could impact the quality focus. One = example=20 might be related to the level of code reviews done on = the=20 project. Q =3D How will code reviews affect the=20 quality? If more code reviews are done, we are = likely to=20 find fewer faults during testing. For example, = if the=20 quality focus refers to achieving a 10% reduction in = defects=20 for the next release, but we have not adequately = tracked=20 defects in the past, then how will we know whether the = goal is=20 attained? This would probably trigger a revision = of the=20 goal, and perhaps some new goals relating to = establishing a=20 baseline for defect tracking. In essence, in = this part=20 of the Q stage, we are customizing the quality focus = to our=20 particular environment to make it realistic and, = therefore,=20 attainable.

6=20 - Derive appropriate metrics: = For a=20 given goal and question, there are many relevant=20 metrics. The key is to identify those metrics = that=20 clearly satisfy the question. More metrics are = not=20 necessarily better. Metric definition, data = collection,=20 and analysis and interpretation are extra work, and = costly to=20 produce. Do not create more metrics than are = really=20 needed. In some cases, the same metric may = address more=20 than one question. The mapping of questions to = metrics=20 need not be one-to-one.

Recognize that it is possible to = select=20 the wrong metrics.

=A7 =20 Metrics chosen should be specific to = what you=20 are trying to measure, not generic metrics that you = collect=20 because they can be collected

=A7 =20 Choosing the wrong metrics causes extra = effort=20 collecting and reporting on the wrong data and often = results=20 in having to collect more relevant metrics = after-the-fact

The task is complicated by the = fact that=20 the software community has not converged on a standard = definition of =93metric=94, often using it to = describe many=20 different measurement concepts, or interchanging the = terms=20 =93metric=94, =93measure (or measurement)=94, and = =93indicator=94. =20 Gopal [Gop= al,=20 2002] defines a metric as:

=93A method of=20 quantitatively determining the extent to which = software=20 process, product, or project possesses a certain = attribute=94=20

McGarry [M= cGarry,=20 2002] defines a measure as:

=93A variable to=20 which a value is assigned to represent one or more = attributes=94=20

He defines a base measure = as:

=93A measure of a=20 single attribute defined by a specified measurement = method,=20 functionally independent of all other measures and = capturing=20 information about a single attribute=94

Within the context of this = document, the=20 terms should be viewed as a hierarchy with = =93indicator=94 at the=20 top and =93measure=94 at the bottom. A metric is = an=20 expression composed of one or more measures (base or = derived)=20 and/or other metrics that can include conditional = selection=20 statements. For example, one metric might = be the=20 number of defects found during testing. = The=20 measures used, as components for the metric, = are things=20 like defect count, and phase defect = detected,=20 and so on. A reduction in the number of = defects found=20 in testing might be considered an indicator = of an=20 improved development process. In planning, developers = might=20 hypothesize that a reduction in the defects found in = testing=20 would mean, perhaps, that their implementation of = formal=20 inspections was working. Indicators convey the=20 significance of the metrics in the specified=20 environment. The example in Figure 10 = illustrates the=20 hierarchical nature of these terms.

Figure 10: = Hierarchy of=20 Measurement Terms

7=20 =96 Stay focused on goals when analyzing data:=20 GQM is about seeing if the = measurement=20 results answer the questions posed and address the = goal. =20 It is not about mining the data to see what else can = be=20 discovered. In the analysis plan, the analysis = process=20 should be clearly specified, including how the data = needs to=20 be organized and presented for interpretation. = The GQM=20 team (or project counterpart) has the responsibility = for=20 determining how to aggregate and present data so that = it=20 addresses the goal, but they do it with input and = review from=20 the project team. Their initial plan may not be = perfect,=20 but the project team will be able to identify issues, = as they=20 are the ones involved in interpreting the data. = This=20 feedback is then used to improve the analysis = support.

8=20 =96 Let the data be interpreted by the people = involved:=20 Just as those involved in = development=20 need to be part of the definition team, they also need = to be=20 involved in interpreting the measurement = results. They=20 are the people who can say,

=93Yes, but=20 --- we didn=92t consider this when we were defining = the metrics=20 =85=94, or,

=93This is=20 only meaningful in the following context =85 =93, = or,

=93Although this is = technically=20 correct, if it was organized this way it would be more = useful.=94

All too often measurement is = seen as=20 extraneous to real project work and is assigned to = someone=20 outside of the project who performs various analyses = and=20 interpretations that may or may not be communicated to = the=20 project team. Often, measurement-related reports = are=20 sent up the line =96 beyond the development = team. =20 Interpretation should be done by the people who = represent the=20 viewpoint of the goal, i.e., the goal owners. If = the=20 project manager (PM) owns the goal, then the PM should = interpret the results. Note that when goals are=20 explicitly stated and the owner identified up front, = then this=20 task is easy. This does not mean that others are = excluded from the interpretation process; the goal = owner,=20 however, leads the interpretation process. = Without such=20 direct involvement, the measurement process tends to = break=20 down, as people tend to view it as extraneous to their = real=20 work.

9=20 =96 Integrate the measurement activities with regular = project=20 activities: = Recognize=20 that a metrics program is, in fact, extra work. =20 Implementing the metrics program is itself a project = that=20 should be interwoven within software project or = process=20 activities. The PM should incorporate the = planning=20 aspects of GQM into the project plan, budget for = measurement,=20 track it, and be able to demonstrate the cost = effectiveness=20 (or lack of) of the implementation. = Responsibilities=20 regarding measurement need to be clearly delineated = and=20 assigned within the project team. If it is = feasible to=20 use a GQM team across several projects, then each = project=20 needs to understand how the GQM team and the project = team will=20 interact. Both the GQM and project teams review = the GQM=20 plan (which contains the Gs,Qs, and Ms) thoroughly to = arrive=20 at a set of standard terms for the process and product = models,=20 and to eliminate obscurities and ambiguities. = The=20 measurement plan describes the metrics, procedures, = and media=20 needed to collect, report and validate data for each = goal,=20 embedding data collection into the development = process.

The analysis plan describes how = to analyze=20 and aggregate measurement data into presentation = formats,=20 which others can easily interpret and use to answer = the=20 questions in the GQM plan. It is important to = describe=20 how to compare actual data with the hypotheses = established=20 during the interviews used to formulate the GQM plan.=20 Together, the GQM and analysis plans help during the=20 interpretation stage to focus on what the project team = deems=20 essential.

If this is the first = implementation of a=20 measurement program, then it is important to establish = project=20 goals that establish a baseline for measurement=20 activity. Measurement goals in that scenario = would focus=20 on understanding the current state of the development = process=20 in order to establish a baseline for quantifying=20 improvement.

Another important integration = element is=20 participation of the project team in feedback = sessions,=20 interpreting the data that is prepared and presented = by the=20 GQM team in accordance with the plan. Such = sessions=20 should occur on a regular basis (frequency to be = determined by=20 the project team), allowing sufficient time for new = data=20 collection, yet short enough to hold the interest of = the=20 team. The sessions lead to significant updates = in the=20 analysis, measurement and GQM plans and set the stage = for=20 evolving improvement.

10=20 =96 Do not use measurements for other purposes:=20 The fastest way to derail = the=20 measurement program is to use the measurement results = for=20 other purposes not related to the goal. This is=20 especially true when such measures are misused to = assess the=20 productivity of individual team members or as a basis = for=20 individual awards. This can occur when someone = other=20 than the goal owner interprets the measurements, = perhaps with=20 their own hidden agenda. It causes a breakdown = in the=20 trust relationship among team members and between the = project=20 team and management, often resulting in actual data = collection=20 being skewed to influence the outcome and degraded = data=20 integrity.

11=20 =96 Secure management commitment to support = measurement results:=20 This is, perhaps, the most = critical=20 of all GQM success factors, but it is not unique to = GQM; it=20 applies for any measurement implementation. If=20 management ignores recommendations supported by sound=20 measurement results, then measurement itself is = perceived as=20 an exercise in futility, and in that scenario, most=20 measurement programs do not survive. One way to = gain=20 support from upper management in the early stages of = the=20 initiative is to communicate to them how the = measurement goals=20 will support their business activity. This = prepares=20 management for anticipated results, removing the = element of=20 surprise. Since business or organizational goals = indirectly drive the identification of measurement = goals, this=20 is not difficult to accomplish. However, it is a = =93people=94 issue in that there must be the = appropriate level of=20 communication between the technical staff and the = business=20 managers, and any political issues need to be = addressed early,=20 and perhaps identified as risks. Without real=20 commitment, it does not make sense to proceed with any = formal=20 measurement program.

Take the time to develop a = measurement=20 project plan that identifies the resources, the = period, and=20 the coordination needed to implement = measurement. This=20 plan will help to align management=92s expectations. = [D= ekkers,=20 2003]

12 =96 = Establish an=20 infrastructure to support the measurement program:=20 Since=20 measurement is extra work, it is important to prevent = that=20 work from infringing on the product development = effort.=20 Lavazza [= Lavazza,=20 2000] notes that GQM plans can easily grow into = quite=20 complex and large graphs. A single goal can = generate=20 more than 100 metrics. Without specific = automated=20 support, keeping this amount of metadata under control = is a=20 challenging task. Wherever = possible,=20 automate data collection and provide tools for storage = and=20 analysis of metric data. A common database = should be=20 established that can be used by all projects for = storage and=20 analysis, and also serves as an organizational = resource for=20 estimating future efforts. In addition to the = actual=20 measurement data, the following metric information = should be=20 preserved:

=B7 =20 Name and definition of each unique = metric

=B7 =20 Classification for each metric

=B7 =20 Association point in product development = that=20 identifies when and how the data is to be = collected

=B7 =20 Definitions of the data collection forms = or=20 other mechanisms

=B7 =20 Procedures for data reporting, = collection, and=20 validation

=B7 =20 References to the metrics in the GQM = plan (link=20 to goals)

There are many useful tools for = data=20 collection and analysis. The key is the degree = to which=20 the tools interoperate or the resulting data can be=20 integrated. Making good tool choices is = necessary in=20 order to automate as much of the data collection as=20 possible. In many cases, such tools may already = exist=20 within the organization but have not been addressed = from the=20 perspective of interoperability or integration. = Here are=20 some types of tools that are important components of = the=20 measurement infrastructure:

=B7 =20 Defect=20 tracking/event tracking

=B7 =20 Software = configuration management

=B7 =20 Applications/packages=20 that generate web statistics

=B7 =20 Relational=20 database management systems

=B7 =20 Spread=20 sheets

=B7 =20 Drawing/charting=20 tools

=B7 =20 Requirements=20 management software

=B7 =20 Collaboration=20 tools

13=20 =96 Ensure that measurement is viewed as a tool, not = the end=20 goal: Measurement should aid, = rather than=20 hinder, the development process. Do not get so = focused=20 on metrics and measurement that you lose site of the = project=20 itself. Leaders must work diligently to keep the = focus=20 on the project, product, or process-focused = improvement goal=20 rather than the stockpile of measures collected.

14=20 =96 Get training in GQM before going = forward: GQM = sounds simple,=20 and there may be the temptation to do it yourself, but = beneath=20 the surface is a sophisticated process that is likely = to be=20 different from the current thinking and modeling of = most=20 practitioners. Some initial training is = important to=20 understand and master the salient points of = transforming the=20 team thoughts and concerns into the GQM = hierarchy. Team=20 coaching through the definition and interpretation = phases by a=20 skilled trainer or someone who has had a successful = experience=20 with GQM, can be very important for the first=20 implementation. The role of a GQM expert can be=20 transitioned to internal personnel after initial pilot = training has been successfully implemented.

Characteristics of=20 implementation

Summary = characteristics

NOT = AVAILABLE=20

ANTICIPATED = BENEFITS OF=20 IMPLEMENTATION:

The most direct benefit of = implementing=20 GQM is the identification of meaningful metrics. = However, some additional benefits of this practice = that were=20 identified in the PERFECT project [PER= FECT=20 1997] include:

=A7 =20 Achievement of=20 improvement goals

=A7 =20 Financial=20 gains

=A7 =20 Increased=20 capability to perform improvement = initiatives

=A7 =20 Improved = group=20 synergy

=A7 =20 Increased quality=20 awareness and Quality Assurance (QA) = involvement

Harris=20 Corporation [N= atwick=20 2003] used GQM to develop an integrated metrics = set for=20 quantitative management of performance, progress, = cost,=20 schedule, and resources across systems, software and = hardware=20 engineering disciplines. They assert that=20 institutionalizing this approach resulted in their = achieving=20 CMM for Software Level 4.

Detailed=20 characteristics

Key Characteristics of the=20 =93Goal-Question-Metric Approach=94 Gold Practice

Characteristic

Comments

Goal-driven = measurement

=A7 =20 Top-down=20 measurement program based on solid measurement=20 goals

=A7 =20 Measurement=20 goals are aligned with business and = organizational=20 goals

=A7 =20 Questions=20 serve to refine the goals

=A7 =20 Metrics (M)=20 are identified in response to = questions

Integrated with project=20 activities

=A7 =20 Planning for=20 GQM is integrated within project = planning

=A7 =20 Both a=20 measurement plan and an analysis plan provide = details=20 for measurement data collection and analysis of=20 results

=A7 =20 Interpretation = integrated within=20 normal project meetings

People oriented

=A7 =20 Project team=20 is involved in all aspects of GQM, including = defining=20 metrics, analyzing and interpreting the = measurement=20 results

=A7 =20 Goals have=20 owners who interpret the measurement relative to = their=20 goal

=A7 =20 Project team=20 provides feedback about the value of the = measurements=20 and makes suggestions for improvement =

Supported by a GQM = team

=A7 =20 A = team of GQM=20 experts (often part of Quality Assurance) = assists with=20 definition and analysis planning

=A7 =20 GQM = team=20 provides training in GQM and related analysis to = the=20 project team

Iterative/scalable =

=A7 =20 Can = start=20 small, if necessary, and enhance measurement = program=20 through improvement cycles

=A7 =20 Add = additional goals, questions and/or metrics over=20 time

Improvement = focused

=A7 =20 There must be=20 a greater purpose for measurement than = measurement=20 itself

=A7 =20 Can = be=20 focused on process, product, or project = improvement, or=20 any combination of the three = components

Promotes reuse

=A7 =20 The = artifacts=20 (plans, procedures, and metrics) of GQM = can often=20 be reused with minor adjustments, making future = GQM=20 implementations less costly

=A7 =20 GQM = artifacts, as well as measurement data, are = preserved in=20 a database so as to be accessible to future=20 projects

Relationships to other = practices:

The figure below (and the table of = inputs and=20 outputs that follows) represents the high-level = process=20 architecture for the subject practice, depicting the = nature of=20 the influences on the practice (describing how other = practices=20 might relate to this practice). These statements = are=20 based on definitions of specific =93best practices=94 = found in the=20 literature and the notion that the successful = implementation=20 of practices may =93influence=94 (or be influenced by) = the ability=20 to successfully implement other practices. A = brief=20 description of these influences is included in the = table=20 below. The arrangement of the statements in the = figure=20 and corresponding table does not imply any particular = order or=20 sequence in which they should be addressed.

Process Architecture for the=20 "Goal-Question-Metric Approach" Gold Practice

Note: In the table below, = specific Gold=20 Practices with strong relationships to = Goal-Question-Metric=20 are identified with italics.

Summary of Relationship = Factors

INPUTS TO THE=20 PRACTICE

Justify the need = for a=20 measurement program

There is=20 a cost associated with implementing any type of=20 measurement program. Without management=20 commitment, it is difficult to create or sustain = an=20 effective measurement program. Developing and = Maintaining a Life-Cycle Business Case, in = which you=20 communicate the alignment of the measurement = goals with=20 the business goals, is a key factor. Lack = of such=20 a business case sets the stage for = failure.

Formulate and align = goals

Since=20 goals drive the identification of metrics in the = GQM=20 paradigm, if the organization and project is = accustomed=20 to defining and communicating goals, then the = task of=20 identifying measurement goals is easier. = Practices=20 that support goal articulation at any level will = likely=20 be a positive influence on the GQM = practice. Thus,=20 Requirements Management and = Requirements=20 Trade-Off/Negotiations have a positive = impact=20 because they focus on documenting and = prioritizing=20 requirements. Many high level = requirements=20 are actually synonymous with project level = goals. =20 Getting the viewpoint of key people is one of = GQM=92s=20 success factors. Integrated Product and = Process=20 Development (IPPD) brings key stakeholders = together=20 throughout the process, specifically so that = varying=20 viewpoints are shared, thereby creating a = positive=20 environment for implementing GQM. = Establish=20 Clear Goals and Decision Points is a program = or=20 organizational level practice that establishes = overall=20 goals and a schedule for reviewing progress and=20 achievement in order to determine the future of = the=20 program. This provides clear guidance to = serve as=20 a basis for establishing the measurement = plan.

Treat measurement = as a=20 project

The=20 implementation of any metrics program needs to = be=20 addressed in much the same way as a software = development=20 project, although it is integrated within the = target=20 project(s). Significant planning is = required=20 initially; there are iterations or cycles of the = measurement process where plans are implemented, = feedback occurs, and then the plans are = revised. =20 Practices such as Formal Risk Management = and=20 People=96Aware Management Accountability, = which are=20 essentially project management activities, will = have the=20 same relative positive impact on the GQM = practice, as=20 they do on the software development effort = itself. =20 Capture Artifacts in Rigorous Model-Based=20 Notation focuses on organizing project = artifacts to=20 optimize usability. This same rationale, = when=20 applied to the artifacts of GQM, should = facilitate reuse=20 of measurement artifacts and, thus, reduce the = costs of=20 measurement across multiple = projects.

Integrate GQM = within an=20 improvement initiative

Since GQM=20 is, itself, an iterative process with a built in = improvement cycle, having an overall improvement = initiative within the organization or program = for which=20 GQM provides data has a positive impact on the = GQM=20 implementation. Requiring Structured=20 Development Methods (iterative processes) = optimizes=20 opportunities for using measurement=20 data.

OUTPUTS FROM THE=20 PRACTICE

Communicate project or = program=20 status

Since GQM=20 includes analysis and interpretation stages that = focus=20 on whether the goals have been addressed, data = is=20 organized and aggregated to optimize = interpretation and=20 use and to focus on what is deemed most = relevant. =20 This makes it easier to communicate = project/program=20 status in whatever perspectives are needed. = It is=20 scalable and flexible enough to support tracking = Earned Value, Defect Tracking Against Quality = Targets, Program-Wide Visibility of Progress vs. = Plan=20 and Binary Quality Gates at the = Inch-Pebble=20 Level.

Provide meaningful = measurement=20 results to decision-makers

A major=20 benefit of GQM is the relevance of the = measurement=20 results to key decision makers. GQM = therefore,=20 facilitates such practices as Metrics-Based=20 Scheduling and Management, Quantitative Progress = measurement, and Independent Expert = Reviews=20 (e.g., maturity assessments).

Assess=20 the value of processes, tools, techniques, = etc.

The=20 successful implementation of GQM establishes a=20 mindset and a structure for experimenting = with=20 various processes, tools and techniques to = determine=20 their value to the overall software development = process=20 or project. It provides both the planning = and=20 assessment framework in which to perform = evaluations to=20 determine the effectiveness of the process = within the=20 project or program environment. Organizations = have used=20 GQM to monitor and assess such practices as = Formal=20 Inspections, Leveraging COTS/NDI, Assessing = Reuse Risks=20 and Costs and Statistical Process=20 Control.

RESOURCES:

=A7 =20 Websites

0 = ;=20 NASA Software=20 Engineering Laboratory - created to = investigate the=20 effectiveness of software engineering technologies = when=20 applied to the development of applications software. = It=20 maintains an Experience Factory (experiences, packaged = into=20 useable formats such as downloadable handbooks etc.), = which=20 includes documents describing GQM.

http://sel.gsfc= .nasa.gov/website/exp-factory.htm

0 = ;=20 DACS Return-On-Investment (ROI)=20 Dashboard - presents benefits data from = software=20 technical and management improvements in a dynamic = graphic=20 display format. Additional data is being added to the=20 dashboard on a continuing basis. Benefits from=20 measurement programs is one of the optional = selections.

http://www.thedacs.com/dat= abases/roi/ =20

=20

0 = ;=20 The Goal/Question/Metric Method = website =96=20 These internet pages have been written and will be = maintained=20 to support the ideas published in the textbook = =91The=20 Goal/Question/Metric Method=92, written by Rini = Van Solingen=20 and Egon Berghout (McGraw-Hill, ISBN 007-709553-7,=20 1999).

http://www.gqm.nl/

0 = ;=20 PERFECT Project website =96 The objective of = PERFECT is=20 to assist European industry in measurement-based = improvement=20 of software processes. A set of techniques, = methods and=20 tools supporting the improvement activities have been=20 developed and are available in downloadable format = from this=20 site.

&= nbsp; =20 http://www.iese.fraunhofe= r.de/perfect/

0 = ;=20 History of Software Measurement by = Horst Zeus=20 =96 provides a comprehensive history of = measurement up to=20 1995, with links to the major=20 publications

http://= irb.cs.tu-berlin.de/~zuse/metrics/History_00.html

0 = ;=20 DACS Topic Area on Measurement = =96 updated=20 on a continuing basis, this site has links to = literature,=20 people, programs, organizations, tools and other = resources=20 relating to measurement in general, and related = subtopics such=20 as GQM.

http://= www.thedacs.com/databases/url/key.hts?keycode=3D3

=A7 =20 Tools=20 and Methods

A = multitude of=20 methods and tools may be useful in implementing and = improving=20 your measurement program. GQM is itself, a = method for=20 establishing a metrics program. As stated in the = description of this practice, the success of a GQM=20 implementation is dependent on establishing the = appropriate=20 infrastructure to support metric definition, data = collection=20 and analysis, and to capture for reuse the various = artifacts=20 of the metrics program. Having a database = management=20 system that is sufficiently flexible to allow ad hoc = query, as=20 well as dynamic reporting capabilities, is = essential. =20 Readers are encouraged to share information about = tools they=20 are using by participating in the Gold Practice=20 discussion forum for this practice. = Note: you=20 must be a Gold Practice member to contribute to the = discussion=20 forum. [Click=20 here] to = register for=20 free membership if you are not already a member. = To=20 date, there = are a few=20 references found in the literature to commercial tools = designed specifically to support the GQM process. = Most=20 organizations report developing their own GQM tool by=20 integrating aspects of generic commercial = products.

[The fact that=20 a tool or tool vendor is listed (or not listed) here = does not=20 constitute an evaluation or endorsement of the product = or=20 vendor by the DACS. These are simply links to = tools=20 currently available relative to the subject = practice.]

Tools That Specifically Address = GQM

Tool Name &=20 Vendor

Description and=20 URL

Technology Package for = the GQM=20 Paradigm,

Downloadable from the = website of=20 Christopher M Lott

This document offers a = concise=20 introduction to the Goal Question Metric = Paradigm (GQM=20 Paradigm), and surveys research on applying and=20 extending the GQM Paradigm. It describes = the GQM=20 Paradigm in terms of its basic principles, = techniques=20 for structuring GQM-related documents, and = methods for=20 performing tasks of planning and implementing a=20 measurement program based on GQM. It also = surveys=20 prototype software tools that support applying = the GQM=20 Paradigm in various ways. An annotated=20 bibliography lists sources that document = experience=20 gained while using the GQM Paradigm and offer = in-depth=20 information about the GQM Paradigm.

http://www.c= hris-lott.org/work/pubs/1996-gqm-tp.pdf

MetriFlame

VTT = Electronics

MetriFlame provides an = automated=20 means for tracking the software development = process=20 based on the GQM approach.

http://www.vtt.fi/ele/research/soh/products/metriflame/metriflame= 3.html

Measurement Tools = Website

The Univ. of Calgary = maintains a=20 Web page on GQM related Software Measurement=20 Tools. Several of the tools mentioned were = developed in the 1990s and are not being = marketed over=20 the web. However, you may be able to find = them=20 using the contacts from this website.

http://68.147.127.22/scot= t/seng621/webdoc/wd6.0.html#sec6.2

<= BR=20 clear=3Dall>

=A7 =20 Experts=20 and Contact Points

Basili,=20 Victor A professor at the University of = Maryland,=20 Dr. Basili is the developer of GQM and is a noted = advocate and=20 practitioner of quantitative methods in software = engineering.=20 He was a co-founder of NASA's Software = Engineering=20 Laboratory and has won numerous awards. His Web = page=20 provides pointers to the journals and organizations = with which=20 he is associated, as well as papers and other = descriptions of=20 his research interests.

http://www.cs.umd.edu/users/= basili/

basili@cs.umd.edu

Prof.=20 Rini van Solingen, Logica CMG, The Netherlands =96 = Professor=20 van Solingen has been involved with many = implementations of=20 GQM in conjunction with the PERFECT consortium, = a=20 European software improvement initiative active in the = late=20 1990s. He can provide real world experiences. = He=20 is also the Industry Chair for the PROFES (Product = Focused=20 Software Process Improvement) 2005 conference.
rini.van.solingen@logicac= mg.com

Luigi = Lavazza has written about = providing=20 automated support for the GQM approach and has real = world=20 experience in making GQM successful through automation = of data=20 collection.

lavazza@cefriel.it

=A7 =20 Training=20 Opportunities

The fact that=20 a course is listed (or not listed) here does not = constitute an=20 evaluation or endorsement of the course by the = DACS. =20 These are simply links to courses currently available = relative=20 to the subject practice. In the interest of = keeping the=20 document manageable, the following types of training = have not=20 been included:

=A7 =20 Pure = consulting=20 services

=A7 =20 Courses = focused on=20 broader areas, with GQM identified only as a=20 component

=A7 =20 Tool=96specific=20 training courses

Some vendors=20 offer a virtual classroom alternative to the = traditional=20 classroom training, and may offer on-site training=20 opportunities. Click on the links provided to = get this=20 type of information.

Courses on GQM

Offeror

Course = Description and=20 URL

Data and = Analysis Center=20 for Software (DACS)

Software Measurement Implementation = and=20 Practice

This course, taught by Dr. Victor = Basili, the=20 originator of GQM, will highlight the following = areas:=20 Measurement and Metrics; Measurement Paradigms; = and the=20 Experience Factory. This seminar will provide a = basic=20 understanding of measurement methods and = problems,=20 discuss the practical aspects of metric = collection, give=20 examples of metrics and management indicators, = discuss=20 measurement initiatives underway throughout the = world,=20 develop an understanding of measurement = techniques that=20 are utilized in practice, discuss new paradigms = for=20 measurement, and explain the concept of an = Experience=20 Factory.

http://www.dacsstore.com/category.p= hp?QCID=3D7261043b31ce55a
6cab08db2c89e466b&qcc=3D6&qpg=3D1

Software = Measurement=20 Services (SMS)

Using the=20 Goal-Question-[Indicator]-Measure Technique

Software = measurement=20 experts have used the GQM Method for many years. = It sounds simple=85 until you come to = apply it=20 yourself, in your own environment. Then = all sorts=20 of difficulties arise. Not the least of = which is,=20 what kind of =91indicators=92 do you use to = present the data=20 to different audiences? This course = addresses how=20 to gain practical advantage by using GQM, and = presents=20 the method as it has evolved under the = constraints=20 applied by the real business environment. = The text=20 used is 'The Goal Question Metric Method' by = Rini Van=20 Solingen and Egon Berghout. This course is = useful=20 for anyone who has to formulate, communicate and = work=20 toward clear goal statements.

http://www.gifpa.co.uk/training/apply/gqim2d.html##

<= /TD>

Software = Measurement=20 Services (SMS)

Goal-Question-Measure=20 Facilitated Workshop

A one-day = workshop=20 facilitated by an SMS consultant, designed for = Project=20 stakeholders, sponsors, project managers, team = members,=20 Software Engineering Process Groups (SEPG) and = Process=20 Action Team (PAT) members

http://www.gifpa= .co.uk/training/apply/gqmw.html

The Westfall = Team

Software Metrics Course

A two-day = course that=20 provides an overview of the 12 steps to = selecting,=20 designing and implementing software, combined = with a=20 survey of software metrics currently being used = in the=20 industry. The tools and practices needed = to use=20 quantitative data in the evaluation of software=20 projects, processes, products and services.

http://w= ww.westfallteam.com/Software_Metrics_Course.htm

The Westfall = Team

12 Steps to = Useful=20 Software Metrics

A 3-day = course designed=20 to provide an in-depth practicum on the 12 steps = to=20 selecting, designing and implementing software = metrics=20 that will be useful to your organization. = It=20 introduces the attendees to a practical process = for=20 establishing and tailoring a software metrics = program=20 that focuses on their goals and information = needs. The=20 objective of this course is to provide a = practical,=20 start-to-finish process of =93how to=94 select, = design and=20 implement metrics. Attendees will learn = how to=20 identify their software metrics customers and = utilize=20 the Goal/Question/Metric paradigm to select = metrics that=20 align with the organizational, project and = process goals=20 of those customers.

Note:=20 A=20 facilitated software metric definition workshop = of one=20 or more days can be added to either the Software = Metrics=20 or the 12 Steps to Useful Software Metrics=20 classes.

http://www.westfallteam.com/12_Steps_To_Useful_Software_Metric= s_Course.htm

Software = Engineering=20 Institute (SEI)

Implementing Goal-Driven Software=20 Measurement

This three-day course describes a = method for=20 identifying and defining indicators (graphical = displays)=20 and measures that directly support an = organization=92s=20 business goals related to product development, = process=20 improvement and project management. Using = lectures=20 and exercises, participants learn how to = determine=20 success, progress and analysis indicators that = show=20 traceability from an organization=92s high-level = business=20 goals, right down to the precise data = collected.

http://www.sei.cmu.edu/products/courses/implement.goal-driven.= sw.meas.html

Software = Engineering=20 Institute (SEI)

Developing Product Line Measures = Workshop

The Developing Product Line = Measures=20 Workshop is based on the SEI=92s = Implement=20 Goal-Driven Software Measurement course. The = workshop utilizes the same 10-step process as = the course=20 to guide participants to define meaningful = measures=20 aligned with their software product line = goals.

http://www.sei.cmu.edu/productlines/products_services/pl_meas= _workshop.html

=A7 =20 Bibliography

[Basili, = 1984]

Basili, Victor R. and Weiss, = D., =93A=20 Methodology for Collecting Valid Software = Engineering=20 Data=94, IEEE Transactions On Software = Engineering,=20 Nov. 1984,pp 728-738

[Basili,=20 1994]

Basili, Victor R.; Caldiera, = Gianluigi;=20 Rombach, H. Dieter, =93The Goal Question Metric = Approach=94,=20 1994

http://wwwagse.informatik.uni-kl.de/pubs/repository/basili9= 4b/encyclo.gqm.pdf

[Basili, = 1999]

Basili, Victor R., = =94Experimentation Using the=20 Goal/Question/Metric Paradigm: Studies on = Preventing and=20 Detecting Defects=94 , Draft of Chapter 4, = from=20 Basili notes at Univ. of Maryland, = 1999

http:/= /www.cs.umd.edu/users/mvz/mswe609/book/chapter4.pdf

[Basili, = 2002]

Basili, Victor R., =93Measurement=20 Frameworks=94, Briefing slides from = Course =20 CMSC 735, University of Maryland, Fall 2002

h= ttp://www.cs.umd.edu/class/fall2002/cmsc735/6measureframe.pdf

[Basili, = 2002a]

Basili, Victor R.; Caldiera, = Gianluigi;=20 Rombach, H. Dieter, =93The Goal Question Metric = Paradigm=94,=20 Encyclopedia of Software Engineering (Marciniak, = J.J.,=20 editor), Volume 1, John Wiley & Sons, 1994, = pp.=20 578-583

[Basili, = 2005]

Basili, Victor R., =93Using = Measurement to=20 Build Core Competencies in Software=94, Seminar = sponsored=20 by Data and Analysis Center for Software, = 2005

[Birk, = 1998a]

Birk, Andreas, Van Solingen, Rini; = Jarvinen,=20 Janne, =93Business Impact, Benefit, and Cost of = Applying=20 GQM in Industry: An In-Depth, Long-Term=20 Investigation at Schlumberger RPS=94, IEEE = Proceedings=20 from the Fifth International Symposium on = Software=20 Metrics, 1998

http://csdl.computer.org/comp/proceedings/metrics/1998/920= 1/00/92010093abs.htm

[Birk, = 1998b]

Birk, Andreas; Derks, Pieter; = Hamann, Dirk;=20 Hirvensalo, Jorma; Oivo, Markku; Rodenback, = Erik; van=20 Solingen, Rini; Taramaa, Jorma, =93Applications = of=20 Measurement in Product-focused Process = Improvement: A=20 Comparative Industrial Case Study=94, = Proceedings from=20 the 5^th International Symposium on = Software=20 Metrics, 1998

http://csdl.computer.org/comp/proceedings/metrics/1998/920= 1/00/92010105abs.htm

[Briand,=20 2002]

Briand, Lionel C.; Morascs, Sandro; = Basili,=20 Victor R., =93An Operational Process for = goal-Driven=20 Definition of Measures=94, IEEE Transactions = on=20 Software Engineering, Vol. 28, No. 12, = December=20 2002

http://= www.idi.ntnu.no/emner/empse/papers/briand2002.pdf

[Cantone,=20 1999]

Cantone, Giovanni and Donzelli, = Paolo,=20 =93Goal-Oriented Software Measurement Models=94, = Extracted=20 from Project Control for Software = Quality, Shaker=20 Publishing, 1999

http://www.iese.fraunhofer.de/network/ISERN/pub/technic= al_reports/isern-99-17.pdf

[CMU/SEI-2002-TR-011]

=93Capability Maturity=20 Model Integration (CMMI) for Systems=20 Engineering,Software Engineering, Integrated = Product and=20 Process Development, and Supplier Sourcing,=20 (CMMI-SE/SW/IPPD/SS, V1.1=94, Version 1.1 =96 = Continuous=20 Representation=94, Software Engineering = Institute,=20 CMU/SEI-2002-TR-011, March 2002

= http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr011.pdf

[CMU/SEI-2002-TR-012]

=93Capability Maturity Model = Integration=20 (CMMI^SM) for Systems=20 Engineering, Software Engineering, Integrated = Product=20 and Process Development, and Supplier Sourcing,=20 (CMMI-SE/SW/IPPD/SS, V1.1), Staged = Representation=94,=20 Software Engineering Institute, March 2002

= http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr012.pdf

[Conradi, = 2003]

Reidar = Conradi and Alf=20 Inge Wang (Eds.), =93Glossary of Common = Terminology for=20 Empirical Software Engineering=94, Empirical = Methods and=20 Studies in Software Engineering =96 Experiences = from=20 ESERNET, Springer Verlag LNCS 2765, ISBN = 3-540-40672-7,=20 Aug. 2003, 278 pages, pp. 274-278, Norwegian = University=20 of Science and Technology (NTNU), NO-7491 = Trondheim,=20 Norway

[DACS, = 2005]

=93The Return on Investment from = Software=20 Engineering Best Practices=94, Course Slides, = Data and=20 Analysis Center for Software, 2005

[Dekkers, = 2003]

Dekkers, Carol A., =93Combat = Resistance to=20 Software Measurement by Targeting Management=20 Expectations=94, Crosstalk, July 2003

http:= //www.stsc.hill.af.mil/crosstalk/2003/07/dekkers.html

[Differding, = 1996]

Differding, Christiane; = Joisl,=20 Barbara; Lott, Christopher M., =93Technology = Package for=20 the Goal Question Metric Paradigm=94, = Internal=20 Report 281/96, University of Kaiserslautern, = Germany,=20 April 1996

http://wwwagse.informatik.uni-kl.de/pubs/repository/differdi= ng96a/gqm-96.pdf

[ESPRIT, = 1996]

=93Measuring = Performance=20 from Software Process Improvement=94, Updated = Nov. 1996, a=20 web page found on the Community Research & = Development=20 Information Service (CORDIS) web site under Esprit, = The European=20 Union Information Technologies Program

http= ://www.cordis.lu/esprit/src/results/res_area/st/st1.htm=20

[Esteves, = 2001]

Esteves, Jose = Manuel=20 and Pastor, Joan, = =93Goals/Questions/Metrics Method=20 and SAP Implementation Projects=94, Technical = Research=20 Report, Universitat Polit=E8cnica de Catalunya, = Spain,=20 2001

http://www.lsi.upc.edu/dept/techreps/llistat_detallat.php?id=3D548=

[Esteves, = 2003]

Esteves, Jose = Manuel;=20 Pastor, Joan; Casanovas, Josep, =93A = Goal/Question/Metric=20 Research Proposal to Monitor User Involvement = and=20 Participation in ERM Implementation Projects=94, = Information Resources Management Conference = (IRMA),=20 2003

= http://peoplesoft.ittoolbox.com/documents/document.asp?i=3D1826

[GAO, = 2004]

=93DEFENSE ACQUISITIONS: Stronger = Management=20 Practices Are Needed to Improve DoD=92s = Software-Intensive=20 Weapon Acquisitions=94, GAO Report GAO-04-393, = March=20 2004
http://www.gao.gov/new.i= tems/d04393.pdf

[Gopal, 2002]

Gopal, = Anandasivam,=20 Krishnan, M. S., Mukhopadhyay, Tridas; = Goldenson,=20 Dennis R., =93Measurement Programs in Software=20 Development: Determinants of Success=94, IEEE = Transactions on Software Engineering, Vol. = 28 No. 9,=20 September 2002

h= ttp://ieeexplore.ieee.org/xpl/abs_free.jsp?arNumber=3D1033226

[Grable, = 1999]

Grable, Ross; = Jernigan,=20 Jacquelyn; Pogue, Casey; Divis, Dale, =93 = Metrics for=20 Small Projects: Experiences at the SED=94, = IEEE=20 Software, March/April 1999

http://ww= w.idi.ntnu.no/emner/it3605/artikler/s2021.pdf

[Gresse, = 1995]

Gresse, = Christiane,=20 Hoisl, Barbara; Wust, Jurgen, =93 A Process = Model for=20 GQM-Based Measurement=94, SoftwareTechnology = Transfer=20 Initiative (STTI), University of Kaiserslautern, = Germany, 1995

http://www.iese.fr= aunhofer.de/CEMP/index.html

[Hall, = 1997]

Hall, Tracy and Fenton, Norman, = =93Implementing=20 Effective Software Metrics Programs=94, IEEE=20 Software, 1997

http:= //csdl.computer.org/comp/mags/so/1997/02/s2055abs.htm

[Humphrey, = 1999]

Humphrey, Watts S., =93Pathways to = Process=20 Maturity: The Personal Software Process = and Team=20 Software Process=94, SEI Interactive, = June 1999

http://www.sei.cmu.edu/news-at-sei/features/1999/jun/Background= .jun99.pdf

[IEEE, 1990]

IEEE=20 Std=20 610.12-1990,=20 IEEE=20 Standard=20 Glossary of Software Engineering = Terminology,

http://ieeexplore.ieee.org/xpl/standardstoc.jsp?isnumbe= r=3D4148&isYear=3D1990

[Jarvinen, = 1999]

Jarvinen, Hanne; Hamann, Dirk; Van = Solingen,=20 Rini, =93 On Integrating Assessment and = Measurement: =20 Towards Continuous Assessment of Software = Engineering=20 Processes=94, Sixth International Symposium = on Software=20 Metrics, IEEE, 1999

http://csdl.computer.org/comp/proceedings/metrics/1999/040= 3/00/04030022abs.htm

[Kitchenham, = 1996]

Kitchenham, Barbara and Pfleeger, = Shari=20 Lawrence, =93Software Quality: the Elusive = Target=94,=20 IEEE Software, 1996

http:= //csdl.computer.org/comp/mags/so/1996/01/s1012abs.htm

[Komi-Sirvio,=20 2001]

Komi-Sirvio, Seija; Parviainen, = Paivi;=20 Ronkainen, Jussi, =93 Measurement Automation:=20 =93Methodological Background and Practical = Solutions =96 A=20 Multiple Case Study=94, Proceedings of the = Seventh=20 International Software Metrics Symposium = (METRICS=20 =9201), IEEE, 2001

http://csdl.computer.org/comp/proceedings/metrics/2001/104= 3/00/10430306abs.htm

[Kopanas, = 1997]

Kopanas, Vassilis; Sylaidis, = Vassilios;=20 Nanakis, Ioannis, =93GQM-Based Improvement of = Embedded,=20 Real-time Software Development Practices=94,=20 Proceedings of the 8^th = International=20 Workshop on Software Technology and Engineering = Practice=20 (STEP =9297), IEEE, 1997

http://ieeexplore.ieee.org/iel3/4852/13405/00615469.pdf?arnumber=3D= 615469

[Lassenius, = 1998]

Lassenius, Casper; Nissinen, = Maarit;=20 Rautiainen, Kristian; Sulonen, Reijo, =93The = Interactive=20 Goal Panel: A Methodology for Aligning R&D=20 Activities with Corporate Strategy=94, Helsinki = University=20 of Technology, Finland, 1998

http://ieeexplore.ieee.org/iel4/5884/15675/00727751.pdf?arnumber=3D= 727751

[Lavazza, = 2000]

Lavazza, Luigi, =93Providing = Automated Support=20 for the GQM Measurement Process=94, IEEE = Software,=20 May/June 2000

http:= //csdl.computer.org/comp/mags/so/2000/03/s3056abs.htm

[Linsdtrom, = 2004]

Lindstrom, Bjorn, =93A Software = Measurement=20 Case Study Using GQM=94, Masters Thesis, Lund = Institute of=20 Technology, Lund University, 2004

http://serg.telecom.lth.se/education/master_theses/docs= /62_Rep.5522.Lindstrom.pdf

[Maurice, = 1997]

Maurice, F.; Benzekri, A.; Raynaud, = Y.,=20 =93Evaluation and Improvement of Software = Products and=20 Processes based on Measurement=94, 1997 = High=20 Assurance Systems Engineering = Workshop, IEEE,=20 1997

http://csdl.computer.org/comp/proceedings/hase/1997/7971/00/7= 9710108abs.htm

[McGarry, = 2002]

McGarry, John; Card, David; Jones, = Cheryl;=20 Layman, Beth; Clark, Elizabeth; Dean, Joseph; = Hall,=20 Fred, =93 Practical Software Measurement: = Objective=20 Information for Decision Makers=94, ISBN = 0-201-71516-3,=20 Addison-Wesley, 2002

http://www.awprofessional.com/bookstore/product.asp?i= sbn=3D0201715163&redir=3D1

[Murine, = 1983]

Murine, G. = E.,=20 =93Improving Management Visibility Through the = Use of=20 Software Quality Metrics,=94 Proceedings from = IEEE=20 Computer Society=91s Seventh International = Computer=20 Software & Application Conference. New = York,=20 N.Y.: Institute of Electrical and Electronic = Engineers,=20 Inc., 1983

[Natwick, = 2003]

Natwick, Gary, =93Integrated = Metrics for CMMI=20 and SW-CMM: Lessons Learned=94, = Software-Engineer.Org A=20 Community for Software Engineers website, = October=20 2003

http://www.software-engineer.org/article_read.php?article_id=3D20= 000939

[Nick, = 1999]

Nick, Markus and Tautz, Carsten, = =93 Practical=20 Evaluation of an Organizational Memory Using the = Goal-Question-Metric Technique=94, = Proceedings of the=20 Fifth German Conference on Knowledge-Based = Systems,=20 1999

http://www.iese.fraunhofer.de/pdf_files/nick_pub/xps99om-publi= sh-crspr.pdf

[Oinas, = 2000]

Oinas, Arto,=20 =93Defining Goal-Driven Fault Management Metrics = in a Real=20 world Environment: a Case-Study from=20 Nokia=94, csmr, = p.=20 101, Conference on Software Maintenance = and=20 Reengineering, 2000.

http://csdl.computer.org/comp/proceedings/csmr/2000/0546/00/0= 5460101abs.htm

[Olsson, = 2000]

Olsson, Thomas and Runeson, Per, = =93V-GQM: A=20 Feed-Back Approach to Validation of a GQM = Study=94,=20 IEEE Software, 2000

http://csdl.computer.org/comp/proceedings/metrics/2001/104= 3/00/10430236abs.htm

[Park, 1996]

Park, Robert E.; Goethert, Wolfhart = B.;=20 Florac, William A., =93Goal-Driven Software = Measurement =96=20 A Guidebook=94, CMU/SEI-96-HB-002, Software = Engineering=20 Institute, 1996

http://www.sei.cmu.edu/pub/documents/96.reports/pdf/hb002.96.pdf

[PERFECT, = 1997]

=93Goal-Oriented Measurement Using = GQM: A=20 booklet from the Perfect ESPRIT Project 9090 = Handbook =94,=20 Copyright PERFECT Consortium 1997

htt= p://www.iese.fraunhofer.de/PERFECT/Handbook/handbook.html

[PROFES, = 1999]

PROFES Users Manual, 1999

http://www.vtt.fi/profes/|
The=20 Users Manual is downloadable from this=20 page.

[Putnam, = 2003]

Putnam, Lawrence H. and Myers, = Ware,=20 Five Core Metrics: The Intelligence = Behind=20 Successful Software Management, Dorset House = Publishing, 2003

[Ragland, = 1995]

Ragland, Bryce, =93Measure, Metric, = or=20 Indicator: What's the Difference?=94, = Crosstalk,=20 Software Technology Support Center, March = 1995

http:/= /www.stsc.hill.af.mil/crosstalk/1995/03/Measure.asp

[Ramos, = 2004]

Ramos, Cristiane S; Ltda, Politec; = Oliveira,=20 Kathia M; Anquetil, Nicolas, =93Legacy Software = Evaluation=20 Model for Outsourced Maintainer=94, = Proceedings of the=20 Eighth European Conference on Software = Maintenance and=20 Reengineering (CSMR=9204), = 1534-5351/04, IEEE,=20 2004

http://www.ucb.br/ucbtic/mgcti/paginapessoalprof/Nicolas= /Publicacoes/CSMR-04.pdf

[Schafer, = 2000]

Schafer, Linda, =93Goal Question = Metric=20 Paradigm: Where to Begin?=94, Slide Briefing = presented to=20 San Antonio SPIN, Athens Group, Inc. 2000

http://www.saspi= n.org/Saspin_Apr2000_Shafer.pdf

[Siviy, 2004]

Siviy,=20 Jeannine M.,=94 = The=20 Effective Application of Six Sigma in Software=20 Engineering=94, Software=20 Engineering Measurement & Analysis = Initiative,=20 Software Engineering Institute, Carnegie=20 Mellon University, 2004 (Sponsored by the U.S.=20 Department of Defense)

http://www.osqa.org/docu= ments/siviy.pdf

[Stark, = 1994]

Stark, George E. and Kern, Louise = C., =93A=20 Software Metric Set for Program Maintenance = Management=94,=20 1994

http://members.a= ol.com/geshome/ibelieve/sus.pdf

[Stoddard, = 1996]

Stoddard, Robert W., =94An = Extension of=20 goal-Question-Metric Paradigm For Software = Reliability=94,=20 Proceedings Annual Reliability and = Maintainability=20 Symposium, IEEE, 1996

http://ieeexplore.ieee.org/iel3/3581/10693/00500656.pdf?arnumber=3D= 500656

[Turner, = 2002]

Turner, R.G., =93Implementation of = Best=20 Practices in U.S. Department of Defense=20 Software-Intensive System Acquisitions=94, Ph.D. = Dissertation, George Washington University, 31 = January=20 2002

http://www.= goldpractices.com/survey/turner/index.php

[van Latum, = 1998]

van Latum, Frank; van Solingen, = Rini; Oivo,=20 Markku; Hoisl, Barbars; Rombach, Dieter; Ruhe, = Gunther,=20 =93Adopting GQM-Based Measurement in an = Industrial=20 Environment=94, pp. 78-86, IEEE Software, = Jan-Feb=20 1998

http:= //csdl.computer.org/comp/mags/so/1998/01/s1078abs.htm

[van = Solingen, 1999a=20 ]

van Solingen, Rini, =93Experiences = in Using the=20 Goal/Question/Metric Paradigm=94, briefing, The=20 Netherlands, 1999

http://sel.gsfc.nasa.gov/website/sew/1999/topics/solingen_SEW99= slides.PDF

[van Solingen, = 1999b]

van Solingen, Rini and Berghout, = Egon, The=20 Goal/Question/Metric Method: A Practical Guide = for=20 Quality Improvement of Software Development, = McGraw=20 Hill, 1999

http://www.mcg= raw-hill.co.uk/html/0077095537.html

[van = Solingen,=20 2001]

van Solingen, Rini and Berghout, = Egon,=20 =93Integrating Goal-Oriented Measurement in = Industrial=20 Software Engineering: Industrial = Experiences with=20 and Additions to the Goal/Question/Metric Method = (GQM)=94,=20 Proceedings from the Seventh International = Software=20 Metrics Symposium (METRICS 01), IEEE, = 2001

http://csdl.computer.org/comp/proceedings/metrics/1998/920= 1/00/92010093abs.htm

[van = Veenendaal,=20 1999]

van Veenendaal, Erik and van Der = Zwan, Mark,=20 =93GQM based Inspection=94, Improve Quality = Services BV, The=20 Netherlands,1999

http://www.improve= qs.nl/pdf/GqmInspecties.pdf

[Wong,2003]

Wong, Bernard, =93A Study of the = Metrics=20 Applied to the Software Evaluation Framework = =91SEF=92=94,=20 Proceedings of the Third International = Conference on=20 Quality Software (QSIC =9203), IEEE 2003

http://csdl.computer.org/comp/proceedings/qsic/2003/2015/00/2= 0150052abs.htm

APPENDICES

DEFINITIONS:

GQM=20 Paradigm

Goal=20 Question Metric (GQM): A method used to = define=20 measurement on a software project, process and = product. =20 GQM defines a measurement model on three levels: (1)=20 Conceptual level (goal): A goal is defined for = an=20 object, for a variety of reasons, with respect to = various=20 models of quality, from various points of view, and = relative=20 to a particular environment. (2) Operational = level=20 (question): A set of questions is used to define = models of the=20 object of study and then focuses on that object to=20 characterize the assessment or achievement of a = specific=20 goal. ( 3) Quantitative level (metric): A = set of=20 metrics, based on the models, is associated with every = question in order to answer it in a measurable = way.

A=20 collective term for the set of basic principles = concerning=20 goal-oriented measurement, the templates and = guidelines that=20 assists in defining goals, and methods of applying the = basic=20 principles.

[Differding,=20 1996]

GQM=20 Metric

This=20 describes something that we would like to measure. = Opinions=20 vary as to whether a GQM metric must be directly = collectible=20 (e.g., =93time in days=94) or whether a GQM metric may = be=20 something as complex as =93productivity.=94 The = argument for=20 directly collectible metrics is that all necessary = information=20 for refining a goal into collectible metrics should be = captured in the GQM plan. The argument for permitting = complex=20 metrics is that a metric need only state clearly what = a person=20 would like to know. The use of complex metrics = in a GQM=20 plan implies that some refinement of GQM metrics into = directly=20 collectible data items may be necessary before = beginning to=20 use a GQM plan.

Some hold the=20 opinion that a GQM metric need not even be collectible = at all;=20 in other words, no refinement may be necessary or even = possible. In the case of a non-collectible metric, = stating the=20 metric would merely serve to illustrate what is = desired, and=20 would thereby reveal the limitations of the metrics = for which=20 data are collectible.

[Differding,=20 1996]

SOURCES = (Origins of=20 the Practice):

The GQM=20 approach was originally developed by Victor Basili, = et. al. [Ba= sili=20 1984] at the University of Maryland, to evaluate = defects=20 for NASA projects. It was later expanded to = apply to a=20 larger context and now, often serves as a cornerstone = for many=20 process/product quality improvement initiatives. =

RECOMMENDING=20 SOURCES:

[This=20 section identifies segments from specific DoD, = industry, or=20 academic sources that recommend, specify or otherwise = support=20 adoption of the subject practice either directly, or = as part=20 of a broader best practice initiative, or a practice = with a=20 different name.]

=A7 =20 Capability=20 Maturity Model Integration (CMMI), Version 1.1, = Software=20 Engineering Institute, CMU/SEI-2002-TR-011, TR-012, = March=20 2002[C= MU/SEI-2002-TR-011],=20 [C= MU/SEI-2002-TR-012]=20

The CMMI = identifies the Measurement and Analysis key process = area (KPA)=20 as part of CMMI Level 2. The purpose of = this KPA=20 is to develop and sustain a measurement capability = that is=20 used to support management information needs. = The=20 following text is extracted directly from this = KPA. =20 Although there is no mention of GQM, the goals and = practices=20 described herein provide a strong parallel with most=20 descriptions of GQM. The KPA = involves:

=B7 =20 Specifying the=20 objectives of measurement and analysis such that they = are=20 aligned with identified information needs and=20 objectives

=B7 =20 Specifying the=20 measures, data collection and storage mechanisms, = analysis=20 techniques, and reporting and feedback = mechanisms

=B7 =20 Implementing the=20 collection, storage, analysis and reporting of the=20 data

=B7 =20 Providing=20 objective results that can be used in making informed=20 decisions and taking appropriate corrective = actions

=A7 =20 Report = of Defense=20 Science Board Task Force on Defense Software, November = 2000=20

Although = it does=20 not specifically mention GQM, the report refers to=20 establishing meaningful metrics as a commercial best = practice=20 that needs to be adopted by the DOD.

Specifically,=20 DSBTF (2000) found that one of several recommendations = from=20 the 1987 DSB Task Force on Military Software, namely, = =93that=20 the DoD should develop metrics and measuring = techniques for=20 software quality and completeness and incorporate = these=20 routinely in contracts=94, had not yet been = implemented in DoD=20 practice.

The Task = Force=20 also found that among numerous software development = best=20 practices utilized on commercial programs was the = practice of=20 =94incentivizing development teams based on = meaningful,=20 quantifiable metrics=94.

Although = the DoD=20 is making increased use of other commercial best = practices,=20 this practice is not well established in DOD=20 projects. =

=A7 =20 GAO = Report on=20 Defense Acquisitions, March 2004 [GAO,=20 2004]

The = consensus of=20 this report was that stronger management practices are = needed=20 to improve DOD=92s software-intensive weapons = programs. =20 The report goes on to state that the success (quality = software=20 product, on time and within budget) of leading = software=20 companies is attributable to creating a manageable=20 evolutionary development environment, using = disciplined=20 processes, and collecting and analyzing meaningful = metrics=20 to measure progress. The report does not=20 specifically mention use of the GQM approach but it is = generally accepted in the software community that GQM = provides=20 a way to establish meaningful metrics.

=A7 =20 Software = Program=20 Managers Network (SPMN) 16 Critical Software Practices = for=20 Performance Based Management

The SPMN = was=20 established in 1992 to identify proven industry and = government=20 software best practices and convey these practices to = managers=20 of large-scale DoD system acquisition programs. = The SPMN=20 16 = Critical=20 Software Practices^TM = specifically=20 address underlying cost and schedule drivers that have = caused=20 many software intensive systems to be delivered over = budget,=20 behind schedule and with significant performance = shortfalls.=20

=93Use=20 Metrics to Manage=94 is one of the 16 practices. = Although=20 it does not specifically mention the GQM approach, = many of the=20 elements of GQM are evident within the following list = of=20 essentials:

1. =20 All = programs=20 should have in place a metrics program to monitor = issues and=20 determine the likelihood of risks = occurring.

2. =20 Metrics = should be=20 defined as part of definition of process, = identification of=20 risks or issues, or determination of project success=20 factors.

3. =20 All = metrics=20 definition need to include description, quantitative = bounds,=20 and expected areas of application.

4. =20 All = programs need=20 to assign an organizational responsibility for = identification,=20 collection, analysis, and reporting of metrics = throughout the=20 program's life.

5. =20 Metrics=20 information should be used as one of the primary = inputs for=20 program decisions.

6. =20 The = metrics=20 program needs to be continuous.

GLOSSARY

Abstraction=20 Sheet

A tool used to capture the = perspective of individuals regarding the quality = focus,=20 variation factors, baseline hypothesis, and the = impact=20 of the variation factors on the quality = focus. =20 They are typically used as part of an interview = and show=20 differences of perspective of the = individuals=20 involved in a project.

[Gr= esse,=20 1995].

Analysis=20 Plan

Focuses on how to analyze = and=20 aggregate measurement data into presentation = formats=20 that others can easily interpret

Experience=20 Factory(EF)

Logical and/or physical = organization=20 for continuous learning from experience, = including an=20 experience base for the storage and reuse of=20 knowledge

[C= onradi,=20 2003]

Facilitation

The art of leading people = through=20 processes toward agreed-upon objectives in a = manner that=20 encourages participation, ownership, and = productivity=20 from all involved

GQM=20 Approach:

Synonymous with GQM=20 = Paradigm = ; = =20 [Differding,=20 1996]

GQM = Paradigm

Goal Question Metric = (GQM):=20 method used to define measurement on a software = project,=20 process and product. GQM defines a = measurement=20 model on three levels: (1) Conceptual level=20 (goal): A goal is defined for an object, = for a=20 variety of reasons, with respect to various = models of=20 quality, from various points of view, and = relative to a=20 particular environment. (2) Operational = level=20 (question): A set of questions is used to define = models=20 of the object of study and then focuses on that = object=20 to characterize the assessment or achievement of = a=20 specific goal. ( 3) Quantitative level=20 (metric): A set of metrics, based on the = models,=20 is associated with every question in order to = answer it=20 in a measurable way.

GQM = Plan

A single goal = plus the=20 sets of questions and metrics that provide an=20 operational definition of that goal. A GQM plan=20 documents the refinement of a precisely = specified=20 measurement goal via a set of questions into a = set of=20 metrics. Thus, a GQM plan documents which = metrics are=20 used to achieve a measurement goal and why these = are=20 used - the questions provide the rationale = underlying=20 the selection of the metrics. On the other hand, = the GQM=20 plan is used to guide analysis tasks because it=20 documents for which purpose the respective data = were=20 collected.

[Differding,=20 1996]

Indicator

A device or = variable that can be = set=20 to a = prescribed=20 state based on the results of a process = or the occurrence of a = specified=20 condition. For example, a flag or = semaphore.

[IEEE= ,=20 1990]

Measurement=20 Plan

When=20 coupled with a GQM Plan, a measurement plan = specifies=20 who collects the data required by the GQM Plan, = how the=20 data is collected, and when the data must be=20 collected. A measurement plan usually = includes the=20 data-collection forms as well as descriptions of = tools=20 that perform online data collection.

[Differding,=20 1996]

Metric

In=20 measurement theory, this is a mapping used to = assign a=20 value to some attribute of an entity.

[Differding,=20 1996]

Metric

A=20 quantitative measure of the degree to which = a=20 system, component, or process possesses a given=20 attribute.

[IEEE= ,=20 1990]

Quality=20 Metric

(1) = A=20 quantitative measure of the degree to which = an item=20 possesses a given quality attribute. (2) A=20 function whose inputs are software data and = whose=20 output is a single numerical value that can be=20 interpreted as the degree to which the software=20 possesses a given quality attribute.

[IEEE= ,=20 1990]

SEL

Software=20 Engineering=20 Laboratory

SPMN

Software=20 Program Managers=20 = Network

CASE STUDIES from the=20 literature:

A = Software=20 Measurement Case Study Using GQM

By = Bjorn Lindstrom,=20 Lund Institute of Technology, Lund University, Masters = Thesis,=20 2004

http://serg.telecom.lth.se/education/master_theses/docs/62_Rep.55= 22.Lindstrom.pdf

Description:

Software = measurement is about measuring attributes of the = process,=20 product and resources in software projects, and the = overall=20 objective for it is to get information about the work = to be=20 able to improve and make it predictable. = Software=20 measurement is far from a standard procedure in the = software=20 industry, mainly due to lack of immediate results. = HP=20 OpenView in Amsterdam, and the team Special Products=20 Engineering, is one example of this, where they have = not used=20 software measurement in their previous projects. = Hence,=20 the manager saw a need to evaluate the completed = projects with=20 measurement.

Measurement always=20 implies some effort and it is therefore important to = know why=20 you measure, for what purpose, and for whom. The = Goal=20 Question Metric (GQM) approach is a way to find out = why and=20 what to measure and this approach is applied. = The work=20 presented in this report is based on existing = information=20 only, which tightly limits the possible measurements.=20

Results:

This = statement is=20 extracted directly from Lindstron=92s thesis = [Lindstron,=20 2004]:

=93The = conclusion of=20 using GQM is that it is not suitable when the existing = data is=20 very limited because it results in too many questions = and=20 metrics that are not possible to answer and = measure. The=20 main conclusion of these measurements is that the way = the team=20 has been working is not sufficient for reliable = measurement.=20 Most measures are not possible to validate and = one=20 should therefore be careful to draw general = conclusions from=20 the result. Still the results indicate that the=20 organization is immature and the productivity = low.=94

What is = implied by=20 these conclusions is that the effort required to = implement GQM=20 may be perceived as wasted if the data needed to = establish a=20 baseline for many of the metrics could not be = validated. =20 He cautions against moving forward when existing data = is not=20 reliable. Using it to answer some of the = questions is=20 dangerous. Thus, in this scenario using GQM = would mean=20 abandoning existing data and starting from scratch to = collect=20 and validate data on a go forward basis =96 providing = a payoff=20 only at some future point in time for new = projects. The=20 study did give HP a much clearer impression of exactly = where=20 they were with respect to their measurement = capability,=20 however.

Integrated Metrics for CMMI = and=20 SW-CMM: Lessons Learned

By Gary = Natwick,=20 October 30, 2003

http://www.software-engineer.org/article_read.php?article_id=3D20= 000939

Description:

As=20 organizations move toward the Capability Maturity = Model=AE=20 Integration (CMMI), requiring the integration of = technical and=20 management processes across functional disciplines, = the tool=20 suites used to plan, manage, and monitor these = integrated=20 processes must also evolve to support them. One = example=20 of this is an integrated engineering metrics set to = reinforce=20 process deployment, provide effective management = oversight,=20 and ensure alignment with organizational business = goals.=20 Harris Corporation used a Goal-Question-Metric = approach=20 to develop an integrated metrics set for quantitative=20 management of performance, progress, cost, schedule, = and=20 resources across systems, software, and hardware = engineering=20 disciplines. Institutionalization of this = metrics=20 approach resulted in achieving CMM for Software Level=20 4.

Results:

Lessons learned=20 from implementing a metrics program and tool within an = integrated discipline work force are as follows:

=B7 =20 One metric does not tell the whole = story.=20 You need integrated, and many times, orthogonal = views of=20 metrics to get a complete picture; trending is key. =

=B7 =20 Project planning is key, and data = collection is=20 the hardest

=B7 =20 Having an organizational standard tool = is a must=20 for consistency; it should be user friendly with easy = access=20

=B7 =20 Cultural change is hard, so train = everyone about=20 the organizational metrics program and tool to = increase=20 acceptance and buy-in

Conclusion:
Integrated=20 engineering metrics are required to provide effective=20 management oversight and to ensure alignment with=20 organizational business goals. As organizations = move=20 toward the CMMI requiring the integration of technical = and=20 management processes across functional disciplines, = the tool=20 suites used to plan, manage, and monitor these = integrated=20 processes must also evolve to support them.

Validation of=20 an Approach for Improving Existing Measurement=20 Frameworks

By Manoel G. Mendonca and = Victor R.=20 Basili, Fellow, IEEE, Transactions on software = Engineering,=20 Vol. 26, No 6, 2000

Description:

Software=20 organizations are in need of methods to understand, = structure,=20 and improve the data they are collecting. We have = developed an=20 approach for use when a large number of diverse = metrics are=20 already being collected by a software organization = [1], [2].=20 The approach combines two methods. One looks at an=20 organization's measurement framework in a top-down=20 goal-oriented fashion and the other looks at it in a = bottom-up=20 data-driven fashion. The top-down method is based on a = measurement paradigm called Goal- Question-Metric = (GQM). The=20 bottom-up method is based on a data mining technique = called=20 Attribute Focusing (AF). A case study was executed to = validate=20 this approach and to assess its usefulness in an = industrial=20 environment. The top-down and bottom-up methods were = applied=20 in the customer satisfaction measurement framework at = the IBM=20 Toronto Laboratory. The top-down method was applied to = improve=20 the customer satisfaction (CUSTSAT) measurement from = the point=20 of view of three data user groups. The bottom-up = method was=20 used to gain new insights into the existing CUSTSAT = data. This=20 paper uses the case study and its results to = qualitatively=20 compare our approach against current ad hoc practices = used to=20 improve existing measurement frameworks = (MF).

Results:

The = top-down=20 approach identified several new metrics for the = interviewed=20 groups, and contributed to better understanding the = data user=20 needs. Unexpected associations between key = variables=20 prompted new business insights, and revealed problems = with the=20 process used to collect and analyze the CUSTSAT=20 data.

The case = study=20 tested the three parts of the new approach with = respect to ten=20 MF improvement objectives considered important or very = important by the Toronto Lab data manager. The case = study=20 showed that the new approach was effective in = achieving eight=20 of these ten objectives. The case study also = showed the=20 CUSTSAT MF is mature and has diverse mechanisms to = achieve=20 some of the improvement objectives. Nonetheless, even = in this=20 scenario, the AF and GQM-based methods contributed by=20 complementing and expanding the capabilities that = already=20 existed to improve the MF. This indicates that the new = approach acted in areas that were important but were = being=20 ignored in this and possibly in others MFs.

The=20 Relationship of GQM To Other Paradigms and=20 Frameworks

The principles on which GQM is = built=20 include [PERF= ECT=20 1997]:

=B7 =20 Set explicit measurement goals and = derive=20 appropriate metrics

=B7 =20 Consider the context of the object to be = measured

=B7 =20 Analyze data according to the goals

=B7 =20 Involve the people who are experts on = the object=20 to be measured

Several measurement = methodologies have emerged in recent years. Some = are=20 variants of the GQM approach. Some address the = same=20 elements but from a slightly different perspective, or = using=20 slightly different terminology. Some are = themselves=20 measurement frameworks and some are organizational = frameworks=20 that have an integrated measurement methodology = embedded=20 within them [Ba= sili=20 2005]. This section provides a brief = explanation of=20 some of these paradigms, and how they may or may not = relate to=20 GQM, The Figure below illustrates = the=20 relative chronology of these paradigms and methods and = relates=20 them to either the measurement or organizational = framework=20 depending on their scope.

Application of Metrics in Industry=20 (AMI): = The AMI=20 method is the outcome of the AMI project, initiated in = the=20 1990s by the AMI consortium, comprised of experts in = software=20 measurement and users from across Europe, applying=20 quantitative methods to the management of their=20 projects. The goal of the AMI project was to = provide a=20 practical and validated approach to implementing = quantitative=20 approaches to control and improve the software = process. =20 The project produced the AMI method and the AMI = Handbook,=20 which was launched in 1992 and published by = Addison-Wesley in=20 1995. AMI had a wide user base [ES= PRIT=20 1996].

This approach establishes an = integrated=20 framework for software process improvement using both = analysis=20 and benchmarking techniques to quantify software=20 projects. AMI consists of four primary = activities=20 - Assess,=20 Analyze,=20 Metricate,=20 Improve:

Assess the = project=20 environment (with its objectives and problems) to = define=20 primary goals for measurement

Analyze the primary goals to = derive=20 sub-goals and relevant metrics (make a measurement=20 plan)

Metricate =96 Implement the measurement=20 plan

Improve - Compare results with the = plan and=20 make adjustments as necessary

Although=20 its activities have different names and are grouped=20 differently, the AMI method embodies the same = principles as=20 GQM. However, GQM places greater emphasis on involving = the=20 stakeholders throughout the process.

Balanced Scorecard (BSC): = The=20 balanced scorecard is both a measurement system and a=20 management system that enables organizations to = clarify their=20 vision and strategy and translate them into = action. It=20 was developed in the early 1990's by Drs. Robert = Kaplan=20 (Harvard Business School) and David Norton. BSC = views=20 the organization from four perspectives and develops = metrics,=20 collects data and analyzes it relative to each of = these=20 perspectives. There must be a balance among the=20 perspectives. Figure 4 provides a visual synopsis of = this=20 approach.

Source: Balanced Scorecard = Institute web=20 site (see http://www.balancedscorecard.org/ =20 )

Figure 4: The Balanced = Scorecard=20 Approach to Performance Measurement for Strategic = Management=20

The=20 BSC methodology builds on some key concepts of = previous=20 management ideas such as Total Quality Management = (TQM),=20 including customer-defined quality, continuous = improvement,=20 employee empowerment, and -- primarily -- = measurement-based=20 management and feedback.

BSC is=20 focused on getting from the organization=92s vision to = specific=20 objectives for achieving that vision. It is = forward=20 reaching in that it assesses not only how you have = been doing=20 (=93lagging indicators=94), but how well you are = currently=20 doing (=93current indicators=94) and how well you can = expect to do=20 in the future (=93leading indicators=94). = Although BSC is=20 typically implemented at the organizational level and=20 encompasses organizational level goals being = identified in=20 addition to measurement defined, GQM can be used = within BSC to=20 fill the gap between these higher-level goals and the = precise=20 measures that will need to be collected. BSC = provides=20 guidance for the presentation of data and the areas of = measurement that need to be addressed, but GQM, or = other=20 specific measurement methodologies, can be used to = support the=20 implementation of BSC. BSC is focused on upper=20 management needs and essentially involves management=20 stakeholders, while GQM focuses on getting the right = people=20 involved at all levels to ensure the right goals and = metrics=20 are identified and get buy in from the = stakeholders. BSC=20 can incorporate the human focus of GQM, although this = is=20 seldom addressed in the literature about = BSC.

The=20 Department of Energy (DOE) Federal procurement system = has been=20 implementing the BSC approach in planning for FY04 and = FY05=20 and has some excellent implementation examples on its = website,=20 which include not only the planning but the reporting = of=20 results as well.

See http://professionals.pr.doe.gov/ma5/MA-5Web.nsf/= Business/Balanced+Scorecard?OpenDocument

Although=20 the visualization of the BSC model fits nicely on one = page=20 and, perhaps, suggests that the actual goals and = measurement=20 information fit into this visual space, the volume of=20 information typically needed necessitates a different = visual=20 structure. This is exemplified in the BSC documents = found on=20 the DOE website noted above.

In=20 some communities of interest the =93Balanced = Scorecard=94=20 terminology is used to reflect the structure of = management=20 reports that present several aspects of project data = on one=20 page so that a manager can quickly see the financial = status,=20 the technical status and any significant issues along = with a=20 brief statement of the project primary purpose. = These=20 are sometimes called QUAD charts. They are = monitoring=20 functions that attempt to capture the =93balancing=94 = notion of=20 BSC.; however, they could be part of a broader BSC, or = GQM,=20 initiative.

Capability=20 Maturity Model (CMM and CMMI): The CMM for = Software and=20 its successor, CMMI, provide a process benchmark, = which=20 organizations can use to assess their capabilities as = software=20 developers. The maturity level of an = organization, with=20 respect to its capability to develop quality software, = is=20 quantified on a scale from one to five, with five = signifying=20 the highest level. In order to achieve a high = overall=20 rating, the organization must demonstrate that it = implements=20 the processes defined in each of the process areas = (PAs) of=20 the Model. The model serves as a benchmark = because it=20 reflects processes that are evident in organizations = that have=20 repeatedly developed quality software on time and = within=20 budget. One of the KPAs is =93Measurement and=20 Analysis=94. The processes within this PA are = identified=20 in Figure 5. Although GQM is not specifically = mentioned,=20 the fundamentals of GQM are embodied within this = PA.

Source:=20 http://www.osqa.org/documents/siviy.pdf

Figure=20 5: CMMI Measurement and Analysis Key Process=20 Area

Goals Questions Indicators = Measures=20 (GQIM): GQIM, which emerged in the 1990s, = is a=20 10-step process developed by the Software Engineering=20 Institute (SEI) that helps organizations to identify = and=20 define software measures that directly support their = business,=20 process-improvement, and project goals. In GQIM, = the=20 user defines indicators and measures based on what is = needed=20 to manage the User=92s goals and decisions and = decision criteria=20 related to managing the user=92s goals [Siv= iy=20 2004]. GQIM differs from GQM in that GQIM = focuses on=20 identifying Indicators up front rather than=20 metrics. Indicators address the presentation of = data in=20 order to optimize what is communicated to the = recipient. =20 Indicators are typically high-level aggregates of = measurement=20 data, presented in graphical formats. Some = typical=20 indicators are trend lines and circle graphs of = normalized=20 data. In the SEI publication titled = =93Goal-Driven=20 Software Measurement - A Guidebook=94[Park = 1996], the authors define an Indicator as = =93a=20 display of one or more measurement results that is = designed to=20 communicate or explain the significance of those = results to a=20 reader.=94 GQIM asserts that seeing how = measurement data=20 will be displayed (focusing on indicators) helps point = to and=20 clarify exactly what we must measure, better = positioning us to=20 develop the procedures for the data to be collected [Park= =20 1996]. GQIM avoids use of the term = =93metric=94=20 because there is no standard meaning for it in the=20 industry. Figure 6 illustrates this = slightly=20 different focus of GQIM.

Figure 6: The GQIM Paradigm as = presented=20 in [Siv= iy=20 2004]

Base = measures are=20 needed in order to build the Indicators. GQM = also builds=20 indicators but they are often developed as part of the = analysis function, which is performed later in the = process=20 after the metrics have been identified. The two = methods=20 also differ in their stakeholder focus. As noted = in=20 Figure 6, GQIM is primarily oriented to Senior = Managers, while=20 GQM is focused on involving the right people in the=20 measurement process irrespective of the intended = recipients of=20 the measurement results.

Personal=20 Software Process (PSP) and Team Software Process=20 (TSP): The=20 PSP, introduced in the mid-1990s by the SEI, enables = software=20 engineers to plan their work based on personal data, = to=20 measure their work and use their results to = continually=20 improve, and to feel personally responsible for the = quality of=20 the products they produce. The SEI TSP enables = managers=20 to institute a team environment that supports PSP work = and to=20 build and maintain self-directed teams whose members=20 understand business and product goals and who produce = their=20 own plans to achieve those goals. The PSP and TSP are = powerful=20 tandem tools that foster the necessary skills, = discipline, and=20 commitment required for successful software = development. =20 Collectively, these processes assist organizations to = climb to=20 higher levels on the software capability ladder, as = defined in=20 the SEI CMM and CMMI.

The CMMI=20 identifies what processes are necessary ; PSP and TSP = focus on=20 how to implement those processes.

PSP was=20 designed to show software engineers how to apply = process=20 principles to their work by infusing them with = appropriate=20 experience and exposure to advanced software = engineering=20 methods in a controlled training environment. As = part of=20 their training, the engineers write a defined set of = ten=20 programming exercises and five reports, which = gradually=20 introduce the engineering methods, providing direct = exposure=20 to a new way of working. They measure their own = progress=20 and get to see the effect of these new methods on = their work=20 [H= umphrey=20 1999].

The focus=20 is on developing estimating, planning and tracking = skills and=20 on five basic metric concepts, considered to be = fundamental or=20 core metrics for software development, namely, = time/effort,=20 size, quality, schedule and productivity. = Further=20 details on PSP and TSP are available on the SEI = website (see=20 http://www.sei.cmu.edu/tsp/).

PSP and=20 GQM are not conflicting methodologies; they are = indirectly=20 related through their alignment with CMM and = CMMI. Their=20 purpose is different. PSP asserts what metrics are = essential=20 to the software development process and focuses on = individual=20 skill building, providing specific guidance on what = data to=20 gather and how it should be presented. PSP = focuses on=20 improving one=92s individual estimating skill with = respect to=20 time, effort, size and schedule, and developing = procedures for=20 deriving the estimates. With GQM, such estimates = would=20 be identified as metrics in response to = questions. Using=20 GQM, one would assume that those same metrics would be = the=20 natural result of the team=92s identification of = important=20 measurement goals for their project but the GQM = process itself=20 does not require identification of those = metrics. Both=20 GQM and PSP emphasize the importance of analysis of = the data=20 and presentation and communication of measurement=20 results. GQM is perhaps more closely related = with TSP=20 than PSP because of the GQM focus on involving the = right=20 stakeholders and sharing of perspectives and = perceptions among=20 the team. An individual could certainly apply = GQM at a=20 personal level, applying it to their specific tasks = within a=20 project, and GQM could be applied by a team trained in = PSP and=20 TSP.

Practical=20 Software Measurement (PSM): PSM is an=20 information-driven, issue-based approach to software=20 measurement for project management [M= cGarry=20 2002]. This measurement process initiative,=20 established in 1994, is funded by the DoD and is freely available at = http://www.psmsc.com./ =20 It has been = endorsed by=20 government Acquisition Executives, and is a key = element of the=20 OUSD (A&T) software initiative. The = methodology is=20 also being adopted by government and industry=20 organizations.

Highlights of the=20 PSM process are:

Its = focus is on=20 cost, schedule, and technical objectives=20
It = identifies=20 project-specific issues=20
Issues = are=20 grouped into common software issue areas=20
Measurement=20 categories correspond to issue areas=20
Each = measurement=20 category has a candidate set of proven = measures=20
Measures are=20 selected based on availability, environment, = and=20 other factors.

The PSM = process=20 defines and uses seven common information categories = to=20 facilitate the identification and prioritization of a=20 project=92s specific information needs [McGarry = 2002]. =20 They are:

Schedule and=20 Progress=20
Product Size and=20 Stability=20
Product=20 Quality=20
Process=20 Performance=20
Resources and=20 Cost=20
Technical=20 Effectiveness=20
Customer=20 Satisfaction

PSM=20 differs from GQM in the following ways:

=B7 =20 PSM is=20 information-driven while GQM is goal-driven. PSM = does=20 not ignore goals; rather it focuses on issues that = result when=20 striving to achieve a goal. Many of the broad = goals=20 relating to cost, schedule and quality are assumed.=20

=B7 =20 PSM = focuses on=20 using existing measures or metrics rather than = creating new=20 ones. It does this by providing a pick list of = measures=20 for each of the various information categories, which = is based=20 on best measurement practices of DoD, government and = industry=20 programs. GQM perceives each project as unique = and=20 therefore asserts that one should not arbitrarily = adopt=20 another project=92s plan or measures. PSM, = however,=20 advocates for tailoring the templates and plan to the = specific=20 environment.

=B7 =20 PSM=20 is primarily a tool for the project managers who work = with a=20 measurement specialist to establish the measurement = program=20 that meets their needs. Other stakeholders, such = as the=20 development team are not directly involved in the = planning=20 although they are called upon to provide data and = implement=20 data collection procedures. GQM seeks to involve = the=20 stakeholders in the entire process and focuses on the = human=20 and cultural issues involved in implementing a metrics = program, operating on the principle that involvement = at all=20 levels is critical to a successful measurement = program.=20

=B7 =20 PSM is = designed to=20 help put measurement into practice at the project = level,=20 thereby providing the data for addressing enterprise = level=20 performance, process improvement, and business-related = questions. GQM has a more generic application in = that it=20 can be used wherever there is a need to assess = satisfaction of=20 goals, irrespective of the subject domain or the=20 level.

Quality=20 Function Deployment(QFD): = QFD is=20 not a measurement program, but it is goal-driven as is = GQM.=20 QFD is a systematic process for motivating a = business to=20 focus on its customers. It is used by = cross-functional=20 teams to identify and resolve issues involved in = providing=20 products, processes, services and strategies that will = more=20 than satisfy their customers. It is an = organized,=20 disciplined process for determining the product or = service=20 requirements necessary to achieve customer-perceived = expressed=20 or unexpressed quality. It assists the developer = with=20 translating goals into measurable observable = product and=20 process requirements and it does have a data = collection=20 component and an interpretation = component.

Although QFD assists = decision-makers=20 by quantifying different strategies (using numerical = values=20 and weighted scores to help identify which = alternatives might=20 offer the best result), it is not a measurement = program. =20 It helps the developer to determine what to do, but it = does=20 not measure progress or the results of strategies=20 implemented. It is more of a planning and = estimating=20 tool than a measurement tool.

GQM and=20 QFD can be implemented together because they are = complementary=20 methods.. While GQM seeks to identify = measurement goals=20 and align them with pre-defined organizational goals, = QFD=20 focuses specifically on achieving the ultimate goal of = customer satisfaction by listening to the customer and = soliciting their input. In QFD, the customer = drives goal=20 setting; QFD provides the guidance for prioritization = of=20 customer satisfaction goals while GQM provides = guidance for=20 identifying appropriate measures and indicators of = progress=20 toward those goals. GQM focuses on how to assess = progress toward goals by addressing the questions used = to=20 identify the measures; QFD establishes performance = thresholds=20 necessary to reflect goal attainment.

Figure=20 7 presents the visual structure of the first phase of=20 QFD. When completed, it resembles a house = structure (or=20 series of houses) and is often referred to as the = =93House of=20 Quality=94. The House is divided into several = rooms. =20 Typically, there are customer requirements, design=20 considerations and design alternatives in a = 3-dimensional=20 matrix to which one can assign weighted scores based = on=20 customer information collected. = QFD=20 originated in Japan in the late 1960s as a quality = assurance=20 method that would design customer satisfaction into = the=20 product before it was manufactured. Prior = methods were=20 primarily aimed at fixing a problem during or after=20 manufacturing. The QFD methodology was embraced = by a=20 wide variety of industries in the U.S. and Western = Europe in=20 the 1980s because of its flexibility and = comprehensiveness,=20 and it continues to be used today. Although it = can be=20 applied to software, there are few instances of such=20 implementation in the open = literature.

Figure=20 7: Sample Structure of a House of Quality (QFD = Phase=20 1)

Cross-functional=20 teams typically participate in a four-phase process = that=20 consists of matrices that analyze data sets according = to the=20 objective of the QFD process. These matrices, = connected=20 together using priority ratings from the previous = matrix, are=20 used to translate higher level needs into lower level = hows,=20 product requirements or technical characteristics to = satisfy=20 the needs.

Further=20 details about QFD and its history are available on the = QFD=20 Institute web site (see http://www.qfdi.org/ = ). =20 See also the DACS Gold Practice titled =93Acquisition = Process=20 Improvement=94 (http://www.= goldpractices.com/practices/api/%23detail=20 ). The section about Acquisition Process = Improvement=20 Techniques describes how QFD=20 could be used to translate customer requirements into = a=20 successful acquisition.

Quality=20 Improvement Paradigm (QIP)

QIP originated at the NASA = Software=20 Engineering Laboratory in the late 70s. It = provides a=20 bottom-up approach for software improvement through=20 experimentation and analysis at the project and = corporate=20 levels. Lessons learned from QIP in the late 70s = demonstrated that data collection needed to be goal = driven and=20 provided the impetus for the development of GQM in the = 80s. Both GQM and QIP evolved throughout the = 1980s into=20 formalized paradigms. QIP is based on six = essential=20 activities [Bas= ili=20 2005]:

1. =20 Characterize the current project and its = environment with respect to models and metrics.

2. =20 Set quantifiable goals for=20 successful project performance and improvement.

3. =20 Choose the appropriate process = model and=20 supporting methods and tools for this project.

4. =20 Execute the processes, construct = the=20 products, collect, validate, and analyze the data to = provide=20 real-time feedback for corrective action.

5. =20 Analyze the data to evaluate the = current=20 practices, determine problems, record findings, and = make=20 recommendations for future project improvements.

6. =20 Package the experience in the = form of=20 updated and refined models and other forms of = structured=20 knowledge gained from this and prior projects and save = it in=20 an experience base to be accessed or reused on future=20 projects.

GQM is integrated within the QIP = in that=20 it provides the practical guidance for setting = measurement=20 goals, identifying the right metrics, and establishing = the=20 measurement program that is needed to assess goal=20 attainment. The same GQM principles are applied = both=20 within specific projects and at the organizational=20 level. Further description of GQM within QIP is = provided=20 in the body of this document.

Software Quality Metrics=20 (SQM): SQM is a method formalized by G. = Murine=20 [Mur= ine=20 1983] in the 1980s, based on earlier work by Boehm = [Boehm=20 et al. 1978] and McCall [McCall et al.=20 1977] Murine describes SQM as follows:

"A = method of=20 providing improved management visibility into software = development using a metric measurement approach. = ... Software is defined to include code, data, = and=20 documentation. Each of these parts is measurable using = a=20 methodology called Software Quality Metrics (SQM). SQM = defines=20 specific, high-order, management oriented = characteristics=20 (factors) of =91good=92 software which are measurable = in terms of=20 their components or =91criteria.=92 The criteria = scores, in=20 turn, are determined by measuring individual, = countable=20 occurrences called elements. This measurement process = may be=20 applied at each of the software=92s developmental=20 phases--Requirements Analysis, Design, Code, and = Test. =20 Product (software) status is reported at either the = factor or=20 criteria level. Discrepancy status reporting is=20 included."

SQM was=20 developed to allow the customer to assess the product = being=20 developed by a contractor. SQM instructs users = to=20 evaluate the quality of software by choosing = high-level=20 factors, refining those factors into criteria, and = refining=20 those criteria, in turn, into metrics. The SQM = approach=20 defines a fixed set of 12 factors, 23 criteria, and = over 400=20 metrics. Some emphasis is placed on scoring = techniques=20 that will yield a single number representing the = =93goodness=94=20 (author=92s terminology) of an artifact. Murine = states=20 that the data gathered for the metrics must be = interpreted in=20 light of the criteria and factors, an argument for = viewing SQM=20 as a top-down approach to measurement. However, = he also=20 states that the appropriate life-cycle phase and = documents=20 must be considered beyond the factors and criteria in = order to=20 choose metrics, which suggests that the factors and = criteria=20 alone are insufficient for the selection of=20 metrics.

The scope=20 of the SQM approach is limited to assessing the = conformance or=20 nonconformance of software products with respect to = the=20 predefined factors. The 12 factors are: =

Correctness &n= bsp; &nb= sp; =20 Reliability

Efficiency &nb= sp; &nbs= p; = ; Integrity

Reusability &n= bsp; &nb= sp; &nbs= p; =20 Usability

Maintainability &nbs= p; = ; =20 Testability

Flexibility &n= bsp; &nb= sp; &nbs= p; =20 Portability

Interoperability &nb= sp; &nbs= p; =20 = Intraoperability

SQM seeks=20 to produce a single score for each factor. = Factor scores=20 (e.g., 90% correct or 85% reliable) are reportable to=20 management and thus provide a basis for quantifiable = software=20 status.

Because=20 the set of factors is fixed, SQM does not support the=20 definition of user-specific measurement goals, as does = GQM. In addition, the predefined refinement of = the=20 factor, criteria, and metric hierarchy precludes any = tailoring=20 of the refinement hierarchy, which makes this method=20 significantly different from GQM.

Copyright =A9 2009, ITT = Corporation Privacy = Policy	webmaster@dacs.dtic.mil = 775=20 Daedalian Drive Rome, NY 13441 (800) 214-7921 Fax: 315-838-7130 =
This site is best viewed in = Firefox 1.0+=20 or IE 6.0+	=