Assessing the maturity of project management processes

1. ABSTRACT

This work sums up a study conducted by researchers at the University of California, Berkeley, with the support and sponsorship of the PMI Education Foundation and the PMI Northern California Branch. The main objective of the study was to determine the financial and organizational impact of project management on the organization. This article is one of a series of articles summarizing this study.

The study was started with the development of a maturity model of project management processes and an analysis methodology to assess the maturity of these processes. This methodology is what we call a benchmarking tool. It consists of 148 questions, each of which offers several options for answers, of which one must be selected. These questions cover 8 Knowledge Areas and 6 Project Phases. The maturity model and methodology were then used in benchmarking for 38 different companies and government (government) agencies in 4 different industries.

The result of assessing the maturity of project management, averaged for all companies, was 3.26 on a five-point scale (1 point - the worst result, 5 points - the best result). In the engineering and construction industry, the overall result was the best, and in the information systems industry it was the worst. The results shown by individual companies vary considerably, so the averaging within the industry does not reflect the true state of affairs in the industry (and is confusing). Even those industries and companies that have performed best have significant room for improvement. Risk management and the project execution phase are weak points with low process maturity, while cost management and the project planning phase are strong points characterized by high maturity.

We conclude that the project management maturity assessment methodology allows for the study and comparative analysis of different approaches to project management for industries and for companies within the industry. The result is that this research methodology offers a set of tools that can be used by organizations in identifying those key aspects of their activities in which significant improvements in project management processes are possible.
 

2. INTRODUCTION

2.1. MAIN PROVISIONS OF THE STUDY

Many organizations present their operations and processes in the form of projects for the purpose of planning, managing and successfully completing them. As is well known, the reason for using such a view (in the form of projects) is the increasing pressure on managers to combine, plan and manage unusual and peculiar undertakings carried out within complex and rich schedules.

However, it is fair to say that in many organizations there is a misunderstanding, confusion and even misperception of the current state of affairs in the field of project management.

Moreover, investing in project management tools, approaches and processes can be very difficult to justify.

One of the reasons for these difficulties is the lack of universally accepted methodologies and well-defined processes for obtaining an objective assessment of the state of affairs in project management, both within a single organization and across the industry. 

An example is the PMSC standard, which, although it is a generally accepted official (authoritative) source of information on project management, nevertheless does not contain detailed information of this kind. The result of this state of affairs is that many companies face significant difficulties in implementing or improving project management processes. This is especially important for managers, who must scrupulously justify all the costs of implementing project management processes and techniques in their organizations.

Early studies have identified a number of benefits that project management offers, although they have been expressed in a qualitative (not quantitative) form and have been demonstrated mainly on plots and examples, rather than on strict formulations. Much of this research was based on questionnaires of theorists and practitioners working in a particular industry.

In order to focus our efforts on the shortcomings identified in the previous section, we began our research by formulating a goal: to determine the financial and organizational impact of project management. Of particular note, the Berkeley research team, in conjunction with the industry advisory group, concluded that this area of activity (project management) needs an analysis methodology that can be used to obtain a reliable quantitative assessment of the current state of affairs in project management in the organization. Ideally, this methodology should provide an opportunity to study and compare different approaches to project management, both between industries and within companies within the industry. It should also be able to provide support to any organization that wishes to assess its current state of affairs with a view to developing project management.

Thus, the key objective of the study was to develop and transfer to managers the procedure for measuring project management processes and the impact of the implementation of these processes in their business methods. This procedure is designed to help managers assess which project management techniques (if any) will be most useful to their organizations. In addition, this procedure should help in promoting improved project management practices in organizations.

3. RESEARCH METHODOLOGY

3.1. RESEARCH STEPS

The following are the steps that were performed during this study.

• Familiarize yourself with current and past work and literature on project management and use these materials in your work as appropriate.
• Development of a 5-level model of maturity of project management processes (PM)2 in order to determine the position of a particular organization regarding the levels of this model.
• Development based on the project management process maturity (PM) model2 detailed methodology to assess the current level of project management and the actual progress of project execution.
• Selection of industries and organizations to participate in the study.
• Measure the level of project management and the actual progress of projects in your organization with a 3-part questionnaire.
• Analysis of the collected information in order to perform evaluation and benchmarking of project management processes and methods in the organization, based on certain criteria.
• Identification (identification) of strengths and weaknesses of the organization in the processes and methods of project management.
• Find any "drivers of success" by checking for a correlation between the level of project management in an organization and the actual completion of project deadlines and cost.
• Development of a computational procedure for determining the order of the amount of return on invested funds for this organization (PM / ROI - Project Management / Return Of Investment). In other words, the development of techniques that allow you to approximate the return on investment in project management for the organization.
• Development of proposals and recommendations for continuous improvement of project management processes.

One of the advantages of this process is that it can be repeated regularly and thereby continuously evaluate and improve project management in the organization. More detailed description and consideration of the 5-level model of maturity of project management processes (PM)2 and procedures for determining PM/ROI (study steps ?1,8,9) are presented in the supplementary work. [Ibbs 98].

3.2. METHODOLOGY OF ASSESSMENT ACCORDING TO THE MODEL OF MATURITY OF MANAGEMENT PROCESSESPROJECTS (PM)2

A model estimation (PM) methodology was developed2, which uses a statistical approach in assessing the maturity of project management processes and methodologies within selected industries. The purpose of the methodology is to assess strengths and weaknesses and to provide an organization with the opportunity to compare itself with other similar organizations (with the so-called peer group). It (methodology) also examines the level of correlation between the level of project management processes of the organization and the actual execution of projects. As explained in the supplementary work [Ibbs 98], it allows you to derive and calculate the order of magnitude of the financial return on investment for a given organization (PM/ROI). Table 1 summarizes the characteristics of the model assessment methodology (MM)2.

Before we develop our Model Estimation (RM) methodology2, we studied the SEI capability maturity model, which is very popular among organizations dealing with information systems (IS). However, it seemed to us that it was not sufficient for our purposes, since it has a significant bias towards organizations dealing with information systems, and does not offer an easy way to measure the amount of financial return on investment.

When developing a model assessment (RM) methodology2 we used a detailed and point-by-point questionnaire designed to collect information. For this purpose, Integrated Project Systems in San Carlos, California, provided us with its benchmarking questionnaire, and California State University adapted the questionnaire to our study. As a result, a 3-part questionnaire was developed to measure the maturity of any organization's project management.

All this was done in order to assess the existing degree of maturity in project management in the organization and describe it in terms of a 5-level model. Using the 3-part Model Questionnaire (PM)2 allows you to achieve the following three goals:

• Benchmarking the maturity of project management in an organization can be done in various aspects. In particular, the strengths and weaknesses of project management are identified in 8 different areas of knowledge and in 6 different phases of the project.
• The order of return on investment (PM/ROI) can be calculated, using statistical relationships between the maturity of project management and the progress of project execution.
• This 3-part questionnaire can help improve an organization's project management processes in specific ways that are most useful and effective for that organization.
• Model Questionnaire (PM)2 divided into 3 main sections: General information about the organization (section 1), Assessment of the maturity of project management processes (section 2), Assessment of the actual progress of the project (section 3). The companies participating in the study are encrypted in order to maintain the confidentiality of information.

Project management process maturity levels are defined as points on the Likert Scale with values between 1 and 5. A value of 1 corresponds to the lowest level of maturity, and 5 to the highest level. The results obtained in answering each of the questions are averaged within each of the 8 areas of knowledge and each management process with a view to their subsequent analysis and obtaining a unified assessment of the level of maturity of project management in the organization. The results are then entered into a database for comparison with the results shown by other organizations and specific to the industry as a whole.

Section 2 of the questionnaire contains 148 questions, each of which offers several answer options, from which one must be selected. The team of researchers used the RMVOC Handbook [PMSC 94] as the main source of information. The team also decided to add another additional project management process, called the "project-oriented organization environment" and designed to maintain the viability of the project-oriented organization. The numbers in the cells of Table 2 show how these 148 questions are divided into 8 knowledge areas and 6 phases of project management.

The full questionnaire is too long to be published. The following is an example of one of the 148 questions:

Question "56. How the critical path of the project is determined"

1. No critical path calculations are performed. For each subproject, critical phases are defined independently and work priorities are set independently – 1
2. The critical path is determined based on the fixed (directive) dates of the control events. Critical path calculations (ITUC/CPM) are not performed or are performed only for individual subprojects – 2
3. Key critical tasks are determined on a qualitative (not quantitative) basis and are used in the calculation of the critical path – 3
4. The critical path is calculated using a single schedule, but only data on key control events are transmitted to sub projects – 4
5. All critical tasks in the schedule of each sub project are identified and shown. the critical path is calculated using a single schedule – 5

To compute an overall maturity estimate from a model (PM)2 the results of the answers to all 148 questions were averaged. To calculate the maturity score for each of the 8 knowledge areas or for each of the 6 phases of project management, the results were averaged for each row or column in Table 2. To calculate the maturity score for a specific area of knowledge for a particular industry, the results were averaged for the corresponding area within the same industry.

3.3. DATA COLLECTION METHODOLOGY

One of the most serious difficulties in this study was the collection of data. The main reason for these difficulties was that the project management methods and tools were relatively new to many of the organizations involved in our study, and that it was difficult to select organizations to participate in it. Invitation letters were sent to potential participants, announcements were made in various journals on project management issues, meetings were held to show the importance of this study. Then, those companies that had the opportunity to spend a large amount of time and make active efforts on participation in the study were carefully selected.

Next, a pilot test was conducted, in which two selected organizations participated. The purpose of the pilot test was to confirm the validity of the model assessment (RM) methodology.2. The results of this pilot test confirmed that the application of the Model Assessment (RM) methodology2 it is appropriate, and that the research methodology is generally sound.

Particular efforts have been made to collect objective data on project management processes and practices. In order to ensure the confidentiality of data, we have signed and presented to companies a non-proliferation agreement that does not give us the right to use in reports at conferences such as this, the original information obtained from companies.

Finally, all participating companies were presented in anonymous coded form and all of the companies' own information, including cost and time, was kept secret and used only in the process of data analysis. Only the main participants in the study knew the correspondence between the actual name and code name of the organizations.

3.4. INDUSTRIES FOR WHICH THE STUDY WAS CONDUCTED

4 different industries were selected for the study. In total, 38 large international companies took part in the study, among which were private and public organizations. So, research was conducted for the following industries: Engineering and Construction (E-C), Information Management and Movement (IMM, also known as telecommunications), information systems (Information System, IS, known as software development), high-tech manufacturing (Hi-Tech Manufacturing, HTM).

Project management in the civil engineering industry focuses on the processes of development (design), engineering, procurement and construction in the implementation of projects of buildings and structures in the residential, commercial and industrial sectors around the world. The industry of information management and transmission includes companies and individual specialists who focus their efforts on technologies designed primarily for the transmission of voice, data and images through various channels (analog, digital, terrestrial, satellite, microwave, fiber optic), as well as firms that provide certain services to consumers of these technologies, and firms - providers (providers) of network infrastructure.

The industry of information systems or software development deals with projects for the creation of various software products, the purpose of which (projects) is to supply the consumer with a high-quality product that meets the specifications in a timely manner and within budget. The high-tech manufacturing industry deals with the planning, development, management and execution of projects of high-tech products, processes and services. A high-tech product may refer to computer hardware, semiconductors, and related equipment and accessories.

3.5.LIMITATIONSEXPLORATION

A number of considerations related to the collection of information on project management techniques and tools should be mentioned. The main consideration in the collection was to ensure the accuracy of the information provided by the organizations. Ideally, the level of project management should be measured through a personal visit to each participating organization and a detailed survey of all project managers in each organization to determine the project management tools and methodologies used. However, such a path was not possible due to limited time and resources. Instead, copies of the Model Assessment Questionnaires (RM) were sent to the project managers of the participating organizations.2that they had to fill in.

In general, when filling out such questionnaires, people tend to overestimate their estimates. In order to collect more accurate data, 4 steps were taken. First, a preliminary sifting was carried out, during which companies were selected to participate in the study. The main selection criterion was the willingness and willingness to share data on their current state of affairs in the field of project management.

Secondly, we drew the special attention of each person responding to the questionnaire to the importance of being sincere and impartial. The main persuasive argument was that sincerity and impartiality serve their own interests, as it is the best way to find out what their project management methods and processes look like in comparison to other firms.

Third, any unclear question or concept was discussed with researchers at the request of the company. In compiling the questionnaire, some effort was made to make all 148 questions easily understood by respondents. The questionnaire was also accompanied by a glossary explaining all unfamiliar concepts or terms used in the questionnaire.

And finally, fourthly, researchers from Berkeley studied and analyzed part of the questionnaires filled out and sent by companies. As a rule, this was done during a telephone conversation. Thus, although the above 4 steps cannot guarantee perfection, nevertheless, the research team believes that sufficiently accurate information has been collected.

Another nuance is that the questionnaire also contained such questions, for the answer to which it was necessary to summarize the knowledge of several project managers. For example, it was encouraged when the same questions are answered by several people (if possible). This made it possible to get different answers and to learn different points of view, often criticizing each other, which, as a rule, was considered by the firm as a very valuable exercise. Typically, each organization was required to complete the questionnaire at a cost in the region of 6-8 person-hours.

4. EVALUATION RESULTS

4.1. DEMOGRAPHIC CHARACTERISTICS OF ORGANIZATIONS

The study took significantly longer than originally planned – almost two years. The main reason is that many companies did not understand the true significance of the study or felt reluctant to share the details of their project management techniques. Some experienced confusion in their current project management methods and processes. Others believed they were using something special and didn't want to share it with other companies.

As already mentioned, a total of 38 companies took part in the study: 15 in the engineering and construction industry, 10 in the information management and transmission industry, 10 in the information systems industry and 3 in the high-tech production industry.

First, organizations were classified according to their size. Thus, 48% of organizations had a staff of 1 to 100 people and 52% - more than 100 people.

Second, organizations were classified according to their project management experience. In general, the responses showed that the experience of using project management ranges from 1 to 50 years, and the average value is 15.4 years for the engineering and construction industry, 10.7 years for the high-tech manufacturing industry, 8.0 years for the information systems industry and 6.8 years for the information management and transmission industry. 

 

The average value for all companies is 10.7 years. Further, 61% of organizations have had less than 10 years of project management experience. It is quite clear that companies have only recently begun to apply project management in their activities, and not enough time has passed to see and realize all the benefits and advantages that it carries.

4.2. ANNUAL COST OF PROJECT MANAGEMENT

Another important question that the study had to answer was what are the costs of managing projects as a percentage of total costs. These expenses were defined as a percentage of revenue or sales for projects in the organization or department. We have developed a list of items of expenditure that have been classified by us as related to project management. 

 

Copies of this list have been provided to company managers to help them more accurately and holistically estimate the average cost of project management in their organizations. Of course, examples can be given when it is necessary to additionally include other items of expenditure. Obviously, it is difficult to answer this question exactly. Table 3 lists the cost items for project management.

The average cost of project management was 6%. This figure represents the amount of expenditure listed in table 3 divided by the final cost of the project. The result was higher than previously reported (reported) by other sources [Archibald 67].

The results obtained for the 20 organizations that provided the information. As the study showed, 80% of companies that answered this question indicated that they spend less than 10% of the total cost of projects on project management. For companies in the engineering and construction industry, this figure was generally higher: 9.3% against 2.0% for all other types of companies.