DIPSIT

Dipartimento di Studi per lImpresa e il Territorio

Objective and subjective measures of CEOs performance: can economic and managerial literatures be reconciled?
Anna Menozzi
anna.menozzi@eco.unipmn.it

Working paper n. 5, Ottobre 2005

Abstract Research on compensation practices of firms has traditionally focused on the objective criteria of performance evaluation, mainly because of data availability. Subjective criteria have received minor attention. Nonetheless, many workers are evaluated on the basis of subjective criteria, and the academic work on this topic has provided important insights into how incentives operate, how they translate into contracts, and how this relates to the overall firms performance. Both economic and managerial literatures have contributed to the topic, but findings are sometimes distant or difficult to reconcile. This paper offers a review of main, recent contributions in both areas and a guide to their reconciliation.

ANNA MENOZZI Facolt di Economia di Novara dellUniversit del Piemonte Orientale

OBJECTIVE AND SUBJECTIVE MEASURES OF CEOS PERFORMANCE: CAN ECONOMIC AND MANAGERIAL LITERATURES BE RECONCILED?
SUMMARY: 1. Introduction - 2. Measures of CEOs performance - 3. Objective and subjective criteria in performance evaluation - 4. Are objective and subjective measures of performance substitutes or complements? - 5. Conclusions.

1. Introduction
This study investigate economic and managerial literatures point of view about objective and subjective measures of performance in reward systems. Although there are multiple ways to partition performance measures, the most popular has been between objective and subjective measures. Objective measures are defined as direct measures of countable behaviors or outcomes, such as productivity, profit, return on assets, typically taken from externally recorded and audited accounts. Subjective measures consist of supervisor evaluations of employee performance or judgments on employees personal characteristics. Although these categories are somewhat arbitrary, they provide a useful distinction by which research may be organized and interpreted. There are good reasons why subjective measures of company performance have been employed (and they might continue to be). Economic and managerial literature agrees on the fact that subjective measures are cost effective, if compared with the objective ones: in principle, they can be collected through questionnaire or interview surveys, so that great numbers of individuals can be contacted at a time. In reality, social scientists know how difficult is to gather information from people or organizations: the database construction requires a patient dedication and a continuous follow up. In this respect, the definition of objective measures based on financial statements may be less costly and the preference toward subjective criteria is no more justified. Sometimes subjective measures are the only evaluation criterion. For many public service and voluntary sector organizations, or for small enterprises, there are no appropriate financial records. In other cases, subjective evaluations are the main influent factor, even when objective criteria are ready to use. In example, when reward takes the form of promotion, it is difficult to justify it simply on the basis of goals achieved or numbers obtained. There is no rule (no contract) specifying that to the attainment of a given result promotion automatically follows. Even in companies where the career path

is quite regularly scheduled over time (think in example in the early stages growth assured by the top consultancy companies, at least up to some years ago), subjective evaluations play a major role in the decision on who is eligible for promotion and when. If objective and subjective measures of performance are considered to both matter in the contractual terms definition, differences among contracts should reflect a diverse combination of objective and subjective evaluations of individuals. Reality tells a different history. Managers contracts are compressed into a few categories and less variable than actual performance. The implementation of a continuum of different contracts faces evident problems of cost feasibility. Even disregarding this aspect, there is another reason why contracts are smoother than individuals performance. Performance evaluations are unable to capture personal differences among managers, so that performances are actually different, but this distinction is lost in the appraisal process: managers appear to be more similar than what they really are when their contracts are signed or revised on the basis of their performance evaluations. The economic literature, in particular, is devoted to this aspect. The limited variability of performance evaluations is a constraint to the optimal contract definition, which is a common concern for labor and industrial economists, behavioral researchers and game theorists, among the others. On the other hand, the effectiveness of incentive contracting in organizations depends on a large set of social, psychological, and economic factors, only a few of which have been explored by economists. The managerial literature has concentrated on the measurement of company performance and the relationship between performance and compensation practices, strategic choices, total quality management, and so on. Managerial, psychological, sociological literatures have the merit of having introduced in the debate the nonexclusively economic factors that matter in the incentives mechanism. The basic motivation of the paper can be resumed in the following questions: 1. How do objective and subjective performance measures affect performance evaluation? 2. Are objective and subjective measures interchangeable? 3. Can economic and managerial literature be reconciled on the subject matter? The agenda of the paper is as follows. Section II defines performance measures contents. Section III reviews some economic and managerial articles illustrating objective and subjective measures of performance, then answering question 1. Section III addresses the question whether objective and subjective measures of performance are substitutes (question 2). The section IV drives the conclusions and answers question 3.

2. Measures of CEOs performance

The measures, objectives and targets used in the evaluation process are strictly connected to the set of performance dimensions to be evaluated. There are three generic classes of CEO performance: bottom-line impact, operational impact, and leadership effectiveness. An underlying assumption of almost all CEO evaluation and pay-for-performance plans is that the CEO has a direct and significant impact on corporate performance, i.e. on firms bottom line. Figure 1 shows the types of bottom-line metrics used to evaluate the effectiveness of one CEO.

FIGURE 1. BOTTOM-LINE IMPACT DIMESIONS USED IN EVALUATION OF CEO

Net Operating Revenues

Operating Cash Flow

Net Income

Earnings per share

Capital Expenditures

Share Price

These metrics are objective measures of CEO performance: they are derived from firms financial statements, internal reports, stock market. While CEOs know that it is critical to keep focused on corporate financial success, these bottom-line measures have evident severe deficiencies as sole indicators of CEO performance. As the person at the top, CEO recognizes that their ability to affect the organizations bottom line is not exactly direct and not always overwhelming. Operational impact refers to the CEOs effect on the companys operational and organizational effectiveness. Operational impact measures include indicators of organizational functioning (e.g., retention rates, employee satisfaction, scores),

operational effectiveness (e.g., quality, rating of products, time to market), and strategic implementation (e.g. number of acquisitions, total headcount reduction). The Conference Boards survey of corporate directors (2001) identifies the operational impact dimensions as the most important in evaluating CEO performance. Figure 2 shows these dimensions.

FIGURE 2. MOST IMPORTANT DIMENSIONS OF

OPERATIONAL IMPACT ACCORDING TO SURVEY OF CORPORATE DIRECTORS

Organizational flexibility and agility Company Morale Customer satisfaction

Company image

Research and development

While still subject to considerable external and internal forces outside of the CEOs immediate control, this type of performance is more closely related to the CEOs actions. Similarly to the bottom-line metrics, these indexes may be considered objective, as long as they imply quantitative measurements. Leadership effectiveness refers to personal behaviors, which are completely within the CEOs control. These behaviors include CEOs ability to carry out his or her responsibilities, such as identifying a successor, meeting with key customers, improving relationships with external stakeholders, energizing the organization, etc. Figure 3 identifies three key leadership effectiveness dimensions used to evaluate CEOs.

FIGURE 3. LEADERSHIP EFFECTIVENESS

DIMENSIONS

Strategic leadership Leads the development of appropriate strategies for the enterprise; achieves support and commitment for the strategies from management and the Board of directors Enterprise guardianship Guarantees enterprise reputation, ethics, legal compliance, customer relations. Board relationship Works collaboratively with Board members and committees, communicates information in a timely manner to ensure full and informed consent about matters of enterprise governance Source: The Conference Board (2001). These dimensions do not lend themselves to hard, quantitative measurement. Nevertheless, there are robust methods for reliably and validly measuring these variables as well. For instance, leadership behaviors can be measured through behavioral rating methods that ask Board of directors members to indicate the frequency with which the CEO engages in desired behaviors and to what perceived effect. The three categories just identified (bottom-line impact, operational impact, and leadership effectiveness) simply describe CEO performance in generic terms. The specific dimensions and objectives used in a particular evaluation process will vary for each company. Is appears clear that the scheme of measures and rewards chosen by a company will always include, to some extent, both objective (such as bottom-line) and subjective (e.g. leadership effectiveness) measures of performance. Financial

measures of corporate performance, while critical, capture only one aspect of CEO performance. To perform a more holistic evaluation and to compensate for some of the limitations of bottom-line measures, it is important to include objectives that relate to leadership behavior as well as the CEOs impact on the organizations operational effectiveness. While a comprehensive evaluation of CEOss performance seems so convenient, it has not always been considered as a fact in the literature. On the contrary, only in the last decade both economic and managerial literature have paid attention to subjective measures of performance and to their usefulness. In the next paragraph, we will review some of the topics addressed by these literatures.

3. Objective and subjective criteria in performance evaluation

3.1. The economic approach Economic literature has extensively treated the optimal contracting problem within the Principal-Agent theory1. This theory predicts that the optimal contract between the owner (the principal) of the firm and the manager (the agent) should be tailored on managers personal characteristics and behaviors, so that managers and owners incentives are consistent and the common goal (increase in shareholders value in example) is achieved. Any useful piece of information about managers should be used in defining its compensation. First, information about goals achieved by the manager and quantifiable in economic results of the area or function he heads can be obtained. Financial indicators are the classical example of objective measures of performance: they can be derived from the financial statement of the company and compared with the plan, the budget or the target the manager had been assigned. Managers also are evaluated on the basis of subjective criteria. In this case, the reward cannot be directly related to a specific indicator or index: other considerations concur in defining it. Economic literature on incentive contracts has traditionally focused on the problem of designing compensation schemes based upon verifiable measures of performance. In the last decade authors have begun to pay attention to the importance of subjective criteria, and to include them in their models to reflect principal and agents features. In the search for an answer to the incapacity of contracts to exactly adapt to individual characteristics, economists have taken different positions toward the role of objective and subjective measures of performance. MacLeod (2003) shows that the minor differentiation in contracts is a feature of the optimal contract between a risk-neutral principal and a risk-averse agent when rewards are based upon a subjective evaluation of performance. In the model, compensation
The exhaustive explanation of the Principal-Agent theory goes beyond the scope of this article. For further readings and for the analytical economic model see Holmstrom (1979).
1

only depends upon the principals evaluation. This is consistent with the general recommendation in the management literature against the use of self-evaluation to set compensation, supported by Milkovich and Newman (1999), among the others. If managers were evaluated on the basis of objective measures solely, the contract could be precisely designed in order to include (almost) all the possible outcomes and link them to the corresponding compensation2. Uncertainty increases when subjective evaluations are relevant to the compensations definition, and the design of an ad hoc contract becomes more difficult. The problem of the optimal contracts design becomes even tougher when no objective measures of performance can be applied. In fact, most managers cannot be evaluated on the basis of an objective criterion because the nature of their job or the task they cover is not suitable for being exactly judged in terms of numbers or measurable adjectives. The extent to which the principal is able to reward the agent as a function of a subjective evaluation depends upon the degree to which the agent agrees with these evaluations. Managers have their own opinions about their performance. In the performance assessment process, individuals are asked to judge themselves on the basis of non quantifiable criteria, and prior to the interview with their supervisor. When the principals and the agents subjective evaluations concur, then one can implement the optimal contract, just as if evaluations were objective and verifiable3. On the contrary, when the principals and agents signals are uncorrelated, the optimal contract compresses evaluation into two levels acceptable and unacceptable- with only the very worst performances receiving the unacceptable ranking. The reason of this result lays on the employees behavior when she feels that her evaluation is unfair. If the employee cannot justify the appraisal she is receiving on the basis of objective criteria, and does not accept that the non-coded subjective opinion of her supervisor is worsening the overall evaluations, she might react either by leaving the company, or by changing their attitude in a negative way. In both cases, the relationship between the principal and the agent is harmed and the benefit they get from their interaction diminishes. The consequence is that employers tend to avoid assigning their subordinate extreme (negative) evaluations. Take on example an appraisal in the rating format: employees are evaluated on some standard measured on a scale like well above average, above average, average, below average, well below average, but in case employer and employees subjective evaluation do not coincide, the well below average rate would never be used. Individuals in the upper two cases would be pooled together, and the majority of the employees would receive the highest rating, a result that is consistent with some of the evidence on performance rankings reported by Milkovich and
The problem of moral hazard cannot be eluded, so that even in presence of objective measures of performance only the optimal contract cannot be designed. 3 Here optimal contract is intended to be the best among all the possible contracts that can be signed in presence of asymmetric information.
2

Newman (1999). The less significant differentiation among performances causes the smoothing in compensations and the bias introduced in the contracts design. On the other hands, companies that have shared values of what constitutes good performance should show less pooling of evaluations, and more effective incentive pay. In the limit, when the beliefs of the principal and agent are perfectly correlated, there are no agency costs associated with the use of subjective evaluations: the employer agrees on the evaluations received and aligns its behavior with the principals expectations. McLeods analysis justifies the tendency of employees to agree with the views of their supervisors, and, on the other hand, the reluctance of supervisors to distinguish between employees, particularly when it affects compensation, as remarked by Prendergast (1999). Baker, Gibbons and Murphy (1994) view subjective assessments for performance as a means to mitigate the effect of distortionary objective measures. Compensation plans based on objective measures, like sales or orders, for example, can induce individuals to modify their behavior or to manipulate their numbers in order to get a higher bonus. Take in example a company selling medical equipments, in which salespeople have their variable part of the compensation linked to number of devices sold and installed at the customers site. Technicians in charge of installation may be tempted or induced to sign the technical report that guarantees that the equipment is up and running even when not all the conditions are met. In this case, the firm will probably incur future costs: an internal audit could detect the improper procedure and impose a sales reversal some period after. The cause of this dysfunctional behavior is not the compensation system per se, but rather the fact that the compensation system is based on an inappropriate performance measure. The point is made clear by Baker, Jensen and Murphy (1998). Subjective evaluations can help in easing the problem, if integrated with the objective ones. In our examples, salespeople can be judged and in most companies they really are, not only on the basis of the number of equipments sold but also for their contribution to customers satisfaction, to marketing, to subordinates growth, etc. The set of skills that can only be captured by subjective considerations includes many qualities, varying in a wide range of nuances each. The final appraisal should differentiate among them and define employees compensation accordingly, in order to induct the right incentives in any different person. The peculiar difference between Baker, Jensen and Murphys approach and MacLeods is that, according to Baker, Jensen and Murphy, the principals subjective evaluations end up in evaluations of agents subjective characteristics. Subjective evaluations are viewed as appraisals of non-objective factors, like employees dedication, passion, trustworthiness. On the other hands, just like in MacLeods model,

10

the problem of the correct definition of subjective evaluations in contractual terms arises. The employees contribution to firm value is difficult to compute, when objective measures are considered inadequate or they are not available. The subjective assessment is assigned to supervisors who are best placed to observe employees behavior and opportunities. Even if such subjective assessments are imperfect, they may complement or improve the available objective measures. Thus, an implicit contract based on subjective performance assessments may augment or replace an explicit contract based on objective performance measurements. The explicit contract is signed by parties and can eventually be enforced by a court, while the implicit contract cannot, so is vulnerable to reneging by the firm. Implicit contracts are sustained by trust among parties, and by firms concern for its reputation in the labor market in case of unfair treatment or discrimination against employees. Prendergast and Topel (1996) explicitly say that workers don't trust subjective performance evaluation when they feel that supervisors indulge in favoritism. This may happen when supervisor formulates her subjective performance assessment and does not report it truthfully. The latter result is consistent with the conclusion in MacLeod that the optimal contract in presence of subjective evaluations can be enforced as long as employer and firm agree on these evaluations. The economists interest in subjective evaluation has reduced the gap between economic and managerial literature. Managerial literature has always emphasized the importance of psychological factors, such as trust and fairness, in the definition of incentives. The importance granted to subjective measures of performance is the consequence of a natural evolution of the theory of compensation toward greater consideration for those psychological aspects. 3.2. The managerial approach Although the traditional management accounting literature advocates the use of financial performance measures, many writers attribute many problems to the use of financial performance measures. Ridgway (1956) presents many examples where a managers performance is maximised in the light of selected performance targets. However, such high performance does not contribute to the companys overall objectives. In addition, financial performance measures are frequently criticised on the grounds that they can lead to many behavioural problems including behavioural displacement, myopia (focusing on achieving results on the short term) and dysfunctional behaviour in terms of budgetary slack and data manipulation. The bias introduced in behaviour by performance measures is also described in economic terms by Holmstrom (1979). In addition, Eccles and Pyburn (1992) argue that financial performance measures are lagging indicators since they determine the outcomes of managements actions after a

11

time period. Therefore, it is difficult to establish a relationship between managers actions and the reported financial results. Also, any corrective action has nothing to do with the past and it will be difficult to identify which action leads to a particular result. During the early 90s much of the academic literature claims that traditional performance measurement is too financially biased, focusing only inside the organization on cost and budget variance data. The attack to the traditional performance measurements leads to supplement them with non-financial (and often intangible) measures. These measures may include measures for productivity market effectiveness, product leadership, personnel development, employees attitudes, public responsibility, market share, product development, quality, inventory costs and employee turnover. In addition, it has been argued that the intensity of competition leads to the use of sophisticated control and performance evaluation systems (Otley, 1994, among the others). In this context, researchers end up developing more rounded and comprehensive measurement frameworks, such as the balance scorecard (Kaplan and Norton, 1992). Such a methodology widens the concept of performance measurement by making executive look externally, at how customer and shareholders see the business, as well as internally at process performance and the source of innovation and learning. The balance scorecard initially includes four dimensions: the financial perspective (the drivers of shareholder value), the customer perspective (the differentiating value proposition), the internal perspective (how value is created and sustained) and the learning and growth perspective (role for intangible assets: people, systems, climate and culture). More recently, the concept of multiple stakeholders comes to the fore. Companies can no longer be satisfied with only considering shareholders and customers. Employees are also seen as important stakeholders, as are suppliers, regulators and the community at large and these stakeholders need to be incorporated into the performance measurement system. One of the changes is that companies move from purely internally focused performance measures through multiple dimensional frameworks to having measurement systems including in their focus the wants and needs of relevant stakeholders. In addition to that, the emphasis in practices is on the transformations rather than the individual stock measures. Examples of this second generation measurement frameworks (Neely et al., 2003) include strategy maps, developed by Kaplan and Norton (2000), success and risk maps (Neely, 2002), and the IC-Navigator model (Roos et al., 1997). The detailed explanation of these measures goes beyond the scope of the article. What is important to know is that all methodologies have a stronger focus on long-term value creation potential, thus emphasizing subjectivity. The weakness of the second generation measurement frameworks is their inability to link the business-orientated methodology to real free cash flow, which is the current cornerstone of market valuation. That is why there is a call, in management research, for

12

the venue of a new generation of performance measures, which would seek greater clarity about the linkages between the non-financial and intangible dimensions of organizational performance and the cash flow consequence of these. Neely et al. (2003) list the new performance measures challenges (p. 135): a) Models must reflect the static and dynamic realities of organizations but at the same time not lose appropriateness as a managerial tool; b) We must move from data to information and must provide rigorous information especially for the intangible value drivers in organizations; c) The models must be practical and aligned with other organisational processes in order to allow actions to be taken; d) We must seek increasingly robust ways of demonstrating the cash flow implications of the non-financial and intangible organisational value drivers. These measures may give a better indication of a companys underlying potential to create value, rather than looking to, for example, the immediate stock price, which is subject to market-wide volatility. The past obsession with pure financial performance is now decreasing and there may be a recognition that there is a trade off between hitting todays financial results and sustaining the capabilities and competences that allow companies to compete effectively in the future. Clearly identified measures for each objective greatly facilitate the sharing of performance expectations between CEO and stakeholders. Nevertheless, clarity on performance dimensions and objective helps, but does not guarantee, the implementation of an effective CEO evaluation. In the early stages of designing a CEO evaluation, a debate over the appropriate criteria for assessing performance is probable (and healthy), indicating that the relevant stakeholders are involved and are thinking carefully about the process. Before the process can be enacted however, the CEO and the Board in charge of CEOs evaluation must agree that the dimensions of performance and objectives used are the right ones. The importance of agreement on performance measures, so deeply stressed in the economic literature, is cited but not analytically treated in the managerial contributions. Managerial literature should borrow the detail of analysis and the results obtained from the economic literature, so that the managerial studies may gain in validity.

4. Are objective and subjective measures of performance substitutes or complements?

In the last ten years the dominant trends in compensation have centered on finding ways to enhance competencies in employees. A merit pay system links increases in base pay (called merit increases) to how highly employees are rated on a subjective performance evaluation. Merit increases may be linked to employees ability and willingness to demonstrate key competencies. In the previous paragraph we mentioned

13

salespeople contribution to customer satisfaction and marketing. An example of a possible set of subjective criteria for measuring competencies in customer care is illustrated in Table 1.

TABLE 1.

Competency: Customer Care

1. Follows through on commitments to customers in a timely manner 2. Defines and communicates customer requirements 3. Resolves customer issues in a timely manner 4. Demonstrates empathy for customer feelings 5. Presents a positive image to the customer 6. Displays a professional image at all times 7. Communicates a positive image of the company and individuals to customers Source: Milkovich and Newman (1999). More recently, while economic literature has renewed its interest in subjective measures of performance, managerial literature has started putting merit pay under attack. The main concern is in fact an old one: merit pay is unable to solve the problem of optimal contract definition and does not guarantee the achievement of the desired goal improving employees and firms performance. Already in 1992, Heneman had warned about merit pay limitations and advocated the improvement in accuracy of performance ratings, the allocation of merit money to truly reward performance and the guarantee that the size of the merit increase differentiates across performance levels. Even worse, incentive plans can lead to unexpected and undesired behaviors. We already provided an example of this conflict in the previous section. What matters remarking here is that employees and managers (or managers and firms) end up in disagreement because the incentive system focuses only on one part of what the company believes important to be successful. Employees, being rational, do more of what the incentive system pays for. Bommer et al. (1995) give an important imprimatur to the debate. They think that the objective/subjective distinction has been given too much attention, at the expense of examining the construct validity of performance measures. The risk is to consider objective and subjective criteria as substitutes before attesting their actual interchangeability. In their meta-analytic analysis, the authors consider managerial ratings as subjective measures on one hand, and objective employee performance measures, on the other.

14

The economic studies we reviewed in the previous paragraph predict that objective and subjective measures can indeed be considered substitutes when trust or shared values characterize the principal-agent relation. Bommer et al. suggest instead that the consequent reference to a broad performance construct might be misleading. If the population correlation between objective and subjective performance measures is of a magnitude suggesting convergent validity, then the measures may be combined largely without incident. The correlation between the two types of measures can vary accordingly to specific situations or in presence of peculiar factors (moderators). In example, the authors expect objective and subjective measures to be more strongly related in samples using salespeople than in samples using other functions employees. In sales samples objective performance measures are generally easily assessed and readily available, providing sales managers with the information necessary to evaluate their personnel. Salespeople are traditionally evaluated on sales output, so managers are likely to consider it when evaluating performance. Analogously, we might expect that objective and subjective indicators are more strongly related in samples using a quantitative objective measure than in samples utilizing qualitative measures. The objective measures content is related to job type, but it goes beyond a distinction based strictly on sales or non-sales samples. Production quantity information is likely to be more easily obtained than qualitative measures, and the higher frequency of the quantity-related behaviors should make them easier to observe. Bommer et al. meta-analysis results in an overall mean correlation between objective and subjective performance measures of 0.389, suggesting that the measures are not interchangeable. The second important result is that one moderator affects the strength of the relationship between objective and subjective measures: indeed, this connection is stronger when the objective measure assesses performance quantity, rather than performance quality. The third finding concerns the relationship between objective and subjective measures when they refer to a unique performance construct: when samples are restricted to meet this requirement, it appears that the measures are reasonably substitutable. The practical implication of Bommer et al. model is that, at the most basic level, subjective measures should not be used as proxies for objective measures, if objective performance is the behavior of interest. For example, if sales are the desired outcome, organizations should not reward employees based on a supervisors overall performance evaluation of that employee. Conversely, if a more broadly defined performance is considered more important, it is equally inappropriate to reward employees solely on gross sales. Although a universal substitutability is not advised on the basis of this model, a restricted substitutability is acceptable. In case of production quantity, the problems associated with using proxy measures may not be severe. This is also true for

15

other cases where it is the precise construct that is being measured through multiple means, i.e. through both objective and subjective measures. A very recent study from Wall et al. (2004) goes deeper in the analysis of performance indicators validity. Their primary concern is to address the threat to validity that comes from potential error in subjective measures. Two types of errors are particularly significant. First, if subjective performance measures contain random error, for example because respondents remember figures incorrectly or because they refer to the wrong period, then the effect will be to attenuate any real underlying relationship with associated variable of interest (Type II error, or false negative). More severe is the possibility of systematic bias creating relationships between practices and performance that do not really exist (Type I error, or false positive). Authors lament that although recent research has begun to address the question of the validity of subjective measures in the human resources management area, it has mainly focused on the measurement of practices. Validity requires not only objective and subjective measures of the same performance construct to be associated (convergent validity), but also that their association should be stronger than that between measure of related but different constructs using the same method (discriminant validity). For example, the relationship between objective and subjective measures of productivity, and between objective and subjective measure of profit, should be stronger than the relationships between subjective measures of productivity and profit, or between objective measures of productivity and profit. In other words, different ways of measuring the same construct should show stronger correspondence than the same way of measuring different constructs as advocated by Bommer et al.. A second way of increasing the understanding of subjective performance measures is to examine their construct validity. Even for a subjective performance measure with good convergent validity, the amount of unique variance in that measure leaves plenty of scope for relationships to be found with other variables. It is important to determine not only whether objective and subjective measures of performance are associated with one another, but also whether they are associated with other variables of interest in the same way. Our initial question Are objective and subjective measures interchangeable? can be reformulated as Does the use of a measure of subjective performance lead to different findings than those based on objective measures of performance? If there is no such difference, we can be more confident that the subjective performance does not lead to erroneous conclusions. On the basis of two samples comprising, on one hand, 80 single-site UK manufacturing companies and, on the other, both single- and multi-site publicly quoted UK companies with more than 50 employees, Wall et al. demonstrate that convergent, discriminant and construct validity are evident.

16

There are three main implications of Wall et al.s research. First, the degree of equivalence between the findings for objective and subjective measures supports the use of subjective performance measures in future studies where objective ones may not be feasible or for certain levels of analysis that do not have separate objective measures. The second implication is that there is a need to improve both types of measure. Objective measures can be ameliorated by investigating the accounting decisions in any given reporting period and eventually making convenient adjustments. Subjective measure can be systematically improved by, in example, asking specific questions that directly correspond to the objective performance index of interest, or investigating the relative validity of subjective measure of different kinds, or also by providing respondents with prior warning of the areas to be covered, or focusing on a particular category (finance managers or CEO, e.g.). The third implication is that, when possible, investigators should use both objective and subjective measure of performance within studies. Given each type of measure will contain its own error, more reliable estimate of performance may be obtained by combining them. At the same time, equivalent results from both types of measure would add credibility to any substantive findings, and differential findings alert investigators of underlying problems.

5. Conclusions
The results from Wall el al. (2004) show that the indicators used to measure a particular performance aspect suggest the types of error that would probably occur. Ratings are subject to numerous well-documented sources of systematic bias and random error. Subjective evaluations are non-contractable, then any distortion deliberately introduced by the evaluator may provoke a deviation from the optimal contract. Table 2 illustrates some of the recurrent errors in the appraisal process.

TABLE 2.
Halo error Horn error First impression error

Common errors in the appraisal process An appraiser giving favorable ratings to all job duties based on impressive performance in just one job function. The opposite of a halo error. Downgrading an employee across all performance dimensions exclusively because of poor performance on one dimension. Developing a negative or positive opinion of an employee early in the review period and allowing that to negatively or positively influence all later perceptions of performance.

17

Recency error

Leniency error Severity error Central tendency error Clone error

The opposite of first impression error. Allowing performance, wither good or bad, at the end of the review period to play too large a role in determining an employees rating for the entire period. Consistently rating someone higher than is deserved. The opposite of leniency error. Rating someone consistently lower than is deserved. Avoiding extremes in ratings across employees.

Giving better ratings to individuals who are like the rater in behavior and/or personality. Spillover error Continuing to downgrade an employee for performance errors in prior rating periods. Source: Milkovich and Newman (1999).

Objectives measures tend to be less prone to bias and random error, but they are not a panacea. Performance constructs that can be measured by objective means tend to be narrowly focused and are typically representative of low-order factor structures, as stated by Gibbs, Merchant, and Vargus (2004), so that both theoreticians and practitioners should not rely solely on objective measures for their supposedly superior measurement properties. Many firms are implementing compensation plans that supplement financial metrics with additional measures in order to assess performance dimensions that are not captured in short-term financial results. These additional measures can take a variety of forms, ranging from quantitative, nonfinancial metrics, such as employee and customer survey results, to qualitative assessments of performance by the managers superior. One critical implementation issue that arises when incorporating multiple performance measures in reward systems is the determination of relative weights to place on the various measures when determining compensation, as considered by Ittner et al. (2003). The other important question, which was more extensively treated here, is their contingent effect of different kind of performance measures on the evaluation and the successive rewards determination. Both economic and managerial literature has provided an answer to this concern. Economics, in particular, admits that subjective measures may substitute for the objective ones when agents personal beliefs agree on the appraisal given by the principal.

18

Managerial literature offers a more extensive answer to the dilemma. The authors we reviewed here study the interrelation between objective and subjective factors and give indications on the ideal conditions under which they can be considered interchangeable. In agreement with the economic literature, there is a role for the substitutability of objective and subjective measures of performance. Nevertheless, the general recommendation is the analysis of each contingent situation so that the validity of both the type of measures is not prevented or harmed. In line with the general practice, objective and subjective evaluations should both enter in the compensation definition. The close observation of results and the repetition of preventive experimental studies may help improving the capacity to detect errors and biases compromising objective and subjective measures validity. Researchers should remember that it is better to imperfectly measure relevant dimensions than to perfectly measure irrelevant ones.

19

Riferimenti bibliografici
Baker, J., Gibbons, R., Murphy, K.J., 1994. Subjective performance measures in optimal incentive contracts. Quarterly Journal of Economics, 109(439), 1125-56. Baker, J., Jensen, M.C., Murphy, K.J., 1998. Compensation and incentives: practice vs. theory. Foundations of Organizational Strategy, Michael C. Jensen, Harvard University Press. Bommer, W.H., Johnsons, J.L., Rich, G.A., et al., 1995. On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48, 587-605. The Conference Board, 2001. Compensation committee of the board: best practices for establishing executive compensation. Report No. R-1306-01-RR. Eccles, R., Pyburn, P.J., 1992. Creating a comprehensive system to measure performance. Management accounting (UK), 74(4), October, 41-44. Gibbons, R., Murphy, K.J., 1990. Relative performance evaluation for chief executive officers. Industrial and Labor Relations Review, 23(1), 116-155. Gibbs, M., Merchant, K.A., Vargus, M.E., 2004. Determinants and effects of subjectivity in incentives. The Accounting Review 79(2), 409-436. Heneman, R., 1992. Merit pay: Linking pay increases to performance ratings. Reading, MA: Addison-Wesley. Holmstrom, B., 1979. Moral hazard and observability. Bell Journal of Economics 10(Spring), 74-91. Ittner, C.D., Larcker, D.F., Meyer, M.W., 2003. Subjectivity and the weighting of performance measures: evidence from a balanced scorecard. The Accounting Review 78(3), 725-754. Kaplan, R.S., Norton, D.P., 1992. The balanced scorecard measures that drive performance. Harvard Business Review, September/October, pp. 167-176. Kaplan, R.S., Norton, D.P., 2000. The strategy focused organization: how balanced scorecard companies thrive in the new business environment, Boston 2000. MacLeod, W.B., 2003. Optimal contracting with subjective evaluation. American Economic Review 93(1), 216-240.

20

Milkovich, G.T., Newman, J.M., 1999. Compensation. Chicago: Irwin. Neely, A., Adams, C., Kennerley, M., 2002. The performance prism: the scorecard for measuring and managing business success. London. Neely, A., Marr, B., Goran, R., Pike, S., Gupta, O., 2003. Towards the third generation of performance measurement. Controlling, Heft 3/4, Marz/April 2003. Otley, D., 1994. Management control in contemporary organizations: towards a wider framework. Management Accounting Review, 5, 289-299. Prendergast, C. 1999. The provision of incentives in firms. Journal of Economic Literature XXXVII, 7-63. Prendergast, C., Topel, R.H. 1996. Favoritism in organizations. Journal of Political Economy, 104(5), 958-978. Ridgway, V.F., 1956. Dysfunctional consequences of performance measurements. Administrative Science Quarterly, 1, 240-247. Roos, J., Roos, G., Dragonetti, N.C., Edvinsson, L., 1997. Intellectual capital: navigating the new business landscape. London. Wall, T.D., Michie, J., Patterson, M., et al., 2004. On the validity of subjective measures of company performance. Personnel Psychology 57(1), 95-118.

21