The Software Capability Maturity Model

By

Pattie G. Dickerson

Introduction to CMM

Advantages for Small Organizations

Disadvantages for Small Organizations

Advantages for Large Organizations

Disadvantages for Large Organizations

Conclusion and Summary

Introduction to CMM

The Capability Maturity Model (CMM) for software, developed by the Software Engineering Institute (SEI), describes the principles and practices underlying software process maturity. Its purpose is to enable organizations to improve the maturity of their software processes. By applying CMM standards, software organization’s customers can  identify the strengths, weakness, and risks associated with their software suppliers. The maturity model is organized into five levels starting from the first level to the fifth level to indicate process capability.

These levels describe organizations ranging in ability from those with no guaranteed success in producing software (though might achieve success because of talented individuals), to those where the software process is not only well-defined and effective but which have also institutionalized an ongoing improvement program. To qualify for a given level an organization must demonstrate minimum competence in a number of Key Performance Areas (KPA), which contain detailed accounts of various software activities.

SEI's Fundamental Concepts

Software Process can be defined as a set of activities, methods, practices, and transformations that people use to develop and maintain software and the associated products (e.g., project plans, design documents, code. test cases, and user manuals). As an organization matures, the software process becomes better defined and more consistently implemented throughout the organization.

Software process capability describes the range of expected results that can be achieved by following a software process. The software process capability of an organization provides one means of predicting the most likely outcomes to be expected from the next software project the organization undertakes.

Software process performance represents the actual results achieved by following a software process. Thus, software process performance focuses on results expected.

Software process maturity is the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective. Maturity implies a potential for growth in capability and indicates both the richness of an organization's software process and the consistency with which it is applied in projects throughout the organization.

The practices in each key area are organized by features common to that topic. These common features contain key practices that describe infrastructure or activities. Based upon the results of an assessment survey, an organization falls into one of five software development process categories, or levels:

Information regarding an organization’s software development processes is obtained via assessments. Assessments are initiated by the organization to aid in the improvement of its software development practices. The assessment can be conducted by the organization itself or by an independent agency (SEI or SEI-licensed assessment vendor). The assessment provides feedback on the organization’s current software development capabilities and trains the organization on ways to improve its capabilities. The following are the six-phases of an assessment:

An assessment conducted by a SEI-certified organization is typically viewed to be more credible and objective than a self assessment. Currently, there are only a few organizations that have achieved a Level 4 or higher rating.

Back to top

Advantages for Small Organizations

The best way to start an improvement program is while an organization is a very small one, with only a few software engineers, and with a simple management structure. CMM provides a method by which all roles and responsibilities are clearly defined. Since CMM is a guideline, and not a rigorous requirement specification, it is flexible and easily tailored to any size organization. Functions and roles can be mapped to the number of employees available, with each employee taking on a number of responsibilities.

Software CMM is the most widely used software process improvement model in the United States. A large number of CMM projects have been implemented; the lessons learned and experiences of those projects are publicly available on the Web. A small organization can benefit directly from studying the results of these projects.

Many small organizations find it necessary to subcontract parts of their projects, or even entire projects to other companies. The levels of CMM may be beneficial in assessing which companies would be candidates for such outsourcing. If a subcontractor is identified by the CMM as a level one company, the risks involved with dealing with this company may be reduced by constant monitoring. Or, the contractor may decide to go with a level two or three company for their outsourcing if the project is a more demanding and critical one.

CMM is not a process description or a process model. It is a description of key process attributes that may be realized in many different ways. It is aimed at solving systematic problems in the way software projects are managed. CMM has a strong emphasis on documenting the process and implementing the process as documented. Understanding the appropriate degree of documentation that is critical to consistent, high-quality performance of the process is essential to defining usable processes regardless of the size of the organization.

Disadvantages for Small Organizations

The Software CMM is designed primarily for organizations that work on large government contracts. It is applicable to smaller companies, but an extensive and  reasonable interpretation of the practices for that industry needs to be taken for some of the model content.

CMM proposes more than 25 organizational roles with various tasks and responsibilities. In a small organization, there are not enough people to fill the roles proposed, neither is there any need of many of those roles. In small companies, for example, it is not generally possible to have dedicated process specialists, so every engineer must participate at least part time in process improvement.

Improvement requires change, and changing the behavior of software engineers is a nontrivial problem. Software engineers develop their personal practices when they first learn to write programs. Since they are given little or no professional guidance on how to do the work, most engineers start off with exceedingly poor personal practices. As they gain experience, some engineers may change and improve their practices, but many do not.

Apart from the concerns mentioned above, the most powerful argument against the CMM as an effective prescription for software processes is the many successful small companies that, according the CMM, should not exist. This point is most easily made against the backdrop of the Silicon Valley.

In the Silicon Valley, innovation, non-linearity, and revolution reign supreme, and it is from the vantage point of the innovator that the CMM seems most lost. Proponents of the CMM commonly mistake its critics as being anti-process, and some are. But a lot are process specialists who believe in the kinds of processes that support innovation. The emphasis is on systematic problem-solving leadership to enable innovation, rather than mere process control to enable cookie-cutter solutions.

Innovation per se does not appear in the CMM at all, and it is only suggested by level five. This is shocking, in that the most innovative firms in the software industry are small and score at level one. By contrast, large companies like IBM, which by all accounts has made a real mess of the Federal Aviation Administration’s Advanced Automation Project, score high in terms of maturity.

SEI argues that innovation is outside of its scope, and that the CMM merely establishes a framework within which innovation may more freely occur. According to the literature of innovation, however, nothing could be further from the truth. Preoccupied with predictability, the CMM is profoundly ignorant of the dynamics of innovation.

Where innovators advise companies to get flexible, the CMM advises them to get predictable. Where the innovators suggest pushing authority down in the organization, the CMM pushes it upward. Where the innovators recommend constant constructive innovation, the CMM mistakes it for chaos at level one. Where the innovators depend on a trail of learning experiences, the CMM depends on a trail of paper. Further, the CMM is described on a huge number of text pages; it is cumbersome and time consuming to even attempt to comprehend and apply.

Back to top

Advantages for Large Organizations

The SEI CMM  is a comprehensive model that serves as a basis for assessing and improving the effectiveness of software engineering organizations. It was initially developed to serve government purchasing agencies overseeing large, complex, third-party development projects, and the CMM has become the model of choice for commercial organizations attempting to improve their software development practices. 

The CMM is software-focused, and has become the standard set of process improvement requirements for many software organizations. Many organizations are adopting the model's practices as a way of not only cutting costs, but becoming a software provider of choice. This model for gradual process improvement provides software organizations the guidance they need to move toward best in class practices.

The capability model theoretically is voluntary, though certainly it is evident that the practices called out by the CMM have been imposed. U.S. government contracting for software has in some cases imposed a requirement that contractors demonstrate a minimum of Level 2 or Level 3 ranking in the CMM. Therefore, a large corporation benefits from attaining these levels in that they may be eligible to contract for government projects.

The CMM enables managers to measure their organization's capabilities against a recognized standard, and to determine their relative level of maturity. 

Large contractors have reported an increase in productivity due to the improvement of their software development process. Raytheon yielded a twofold increase in its productivity and a ratio of 7.7-to-1 return on its improvement expenditures, for a savings of $4.48 million during 1990 for a $0.58 million investment.  Various organizations have realized benefits from maturing from one level to the next. Productivities have increased from as little as 2.5 percent to as much as 130 percent. Published studies of software engineering improvements measured by the CMM indicate significant cost savings or profit return. This implies that software testing and maintenance costs were reduced, since the software better met verification and validation requirements. Some organizations showed a savings of $2 million to $3.4 million in project dollars. Contractors have also experienced a decrease in rework, code problems, and retesting costs.

It is generally accepted that higher CMM levels lead to better quality software products and therefore a better company reputation. CMM compliance may also change the manner in which a company interacts with its customers, because there are stringent requirements for maintaining a high maturity level. Highly rated organizations are more adept at handling quick demands by the customer. Fortunately, compliance leads to higher quality software at lower cost. Also compliance improves a company’s reputation, which should be a very potent ingredient for winning and maintaining contracts.

One organization cited that they went from delivering products on time 51 percent of the time to 94 percent of the time. Some organizations have experienced a savings as high as 20 percent in their schedule. Generally, the more mature an organization is in the way it does business, the more successful it will be in delivering a quality product within project constraints.

Participating companies are looking at meeting their quality goals, meeting their requirements, building a maintainable product, and seeking better and improved quality as well as stabilizing schedule, meeting commitments, and accelerating or reducing schedule. Several software organizations have experienced a reduction in defects that ranged from as low as 10 percent to as high as 80 percent. One organization reported a 45 percent decrease in its reduction error rate, while two more organizations’ product error rates decreased from 2.0 to 0.11 per thousand source lines of code and from 0.72 to 0.13 per thousand non-commented source statements.

Back to top

Disadvantages for Large Organizations

Some argue the CMM focuses too heavily on process or technology, not people. Furthermore, those organizations deemed mature indicated that their progression to this state required significant changes in managing people, and their continuing improvement in their organizational capability required them to address issues regarding their people assets and human resources management.

If companies are compared solely on their CMM stamp, it may not be a good enough reason to conclude that company X is better than company Y simply because company X is at a higher CMM level than company Y.

CMM is a framework for improving software process instead of a step-by-step recipe for being an effective software organization, an organization will tailor the framework to its specific needs and goals. Two level 3 organizations will likely have different process for addressing the same KPA. It is hard to conclude which organization is addressing the KPA more effectively without considering other issues like the domain and the organizational culture.

The CMM may not provide the perfect assessment tool for an outsider to assess the suitability of an organization bidding for a specific contract. Like cost estimation, expert judgment still plays an important role. In addition, external SEI assessments may not be a good benchmarking tool, due to objectivity.  Two assessors could assess an organization at different levels due to interpretation.

In order for a software organization to mature, there has to be capital to support the effort. Many organizations have expended large amounts of money and effort in support of their initiatives, yet they have little idea of what, if any, return they are accruing from their investments. Costs for CMM-based process improvement programs have shown increases in software and hardware, data collection, design defects repair, code defects repair, first-time testing and overhead costs. One nearly universal complaint is that moving from level to level can cost hundreds of thousands or even millions of dollars.

Improving the software development process also includes increased training time and less time for working on projects. An organization has to continue business as usual or make the sacrifice to improve the process which will result in higher quality products. Some organizations had difficulty finding the time to work on software-process improvement because they had extreme commitments to deliver customer products. Additionally, in order to mature from one level to the next, extensive training is required so that the organization understands the processes.

Back to top

Conclusion and Summary

The CMM standard categorizes various software organizations based on their process capabilities.  The degree of complexity of a software project can easily be associated with a reasonable level of process capability to which any successful organization must comply or be certified.

Emphasis on process improvement and change management is one aspect that can lead to development of quality, reliable and maintainable software. The CMM also places emphasis on how to sublet part of the contract to other specializing suppliers. It is a model that categorizes the very potential of various organizations based on their process capabilities rather than certifying organizations based on minimum requirements. Although the CMM provides a powerful improvement framework, its focus is on what organizations should do and not on how they can accomplish their goals.

In many business environments the software development process does not begin with the receipt of already-constructed formal requirements. Rather it begins with some sort of analysis process which results in the formulation and validation of the problem statement. In many cases, that analysis must be conducted in an environment of not only continuing change but seemingly change at an increasing rate. The software industry generally accepts the idea that the earlier an error is caught, the easier and less expensively it is to correct. CMM, however, appears to not offer much assistance in the task of analysis and problem formulation. That is, it does not consider an analysis defect which perhaps fails to identify the correct function in the first place. Indeed, the Guidelines contain this interesting caveat: "The CMM does not explicitly state that the customer should be satisfied (or delighted) with the software product". And correspondingly the Guidelines offer no real metrics to help in an area where failure occurs time after time -- identifying the right business problems to work on, and measuring whether the software solutions really address those business problems.

References

SEI's CMM and ISO 9001 for Software.   http://www.dci.com/onsite/seiiso/

Sheard, Sarah A. and Dr. Jerome (Jerry) G. Lake. Systems Engineering Standards and  Models Compared, http://www.software.org/pub/externalpapers/9804-2.html

Paulk, Mark C, Mary Beth Crissis. November 1999 High Maturity Workshop, http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00sr003.pdf

Marshall, Richard M. Manage development better with CMM, http://www.enterprisedev.com/upload/free/features/entdev/1999/05may99/rm0599/rm0599.asp

SECAT LLC. Why Would You Want to Use a Capability Maturity Model?, 1998 pdf file

OKE, Adeniyi Adebowale. SOFTWARE PROCESS STANDARDS: SUMMARY AND COMMENTS, http://www.cs.usask.ca/grads/aao804/pro-standard.html

http://www.sei.cmu.edu/pub/documents/94.reports/pdf/tr24.94.pdf

Paulk, Mark C. Using the Software CMM in Small Organizations, SEI Interactive, Carnegie Mellon University, 1998 pdf file

Houston, Dan and J. Bert Keats. Cost of Quality Decreases for Maturing Software Organizations Research Shows, http://www.utexas.edu/ftp/coe/sqi/newsletter/spring97.html

Paulk, Mark C, Bill Curtis, Mary Beth Crissis, Charles V. Weber. Capability Maturity Model for Software, Version 1.1, February 1993. pdf file

Peters, Tom. Thriving on Chaos: Handbook for a Management Revolution, HarperCollins, 1987

Paulk, Mark C. How ISO9001 Compares with the CMM, http://www.sei.cmu.edu/publications/documents/96.reports/96.ar.iso.vs.cmm.html

Paulk, Mark C. Practices of High Maturity Organizations, The 11th Software Engineering Process Group (SEPG) Conference, Atlanta, Georgia, 8-11 March 1999. http://www.sei.cmu.edu/pub/cmm/high-maturity/survey98.pdf