Proceedings of the 5th International Conference RelStat05

Part 2

COMPARISON SEQUENTIAL TESTING FOR RELIABILITY: EXACT COMPUTATION OF OC AND ASN

Y. H. Michlin1, G. Grabarnik2
Technion Israel Institute of Technology Quality Assurance and Reliability, Technion City, Haifa, 32000, Israel Ph.: 972-4-8294380. E-mail: yefim@tx.technion.ac.il IBM T. J. Watson Research Center 19 Skyline Dr., P.O. B. 704, Hawthorne, NY 10532, USA E-mail: genady@us.ibm.com
2 1

The present study is concerned with simultaneous testing of two systems, referred to as "basic" (subscript b) and "new" (subscript n) respectively with an exponential distribution of their times between failures (TBF's). A hypothesis is checked whereby the mean TBFn/MTBFb ratio equals a given value, versus the alternative whereby it is smaller than the latter. The need for comparison testing is due to the fact that the MTBF of the basic system is often unknown. Moreover, simultaneous testing permits elimination of uncontrollable factors. The present study showed that this approach yields a binomial pattern for which a series of solutions is available via sequential testing. In this paper a recursive algorithm is presented for computation of the probability of a given combination of failure numbers for the two systems, such that the operational characteristic (OC) and that of the test duration can be determined quickly and exactly. This algorithm was applied to Wald's SPRT with a limit on the failure number. An optimisation program seeks the basic parameters for the test in question, ensuring given probabilities for errors of the first and second kinds and for the discrimination ratio.
Keywords: sequential testing, comparison testing, MTBF ratio

1. INTRODUCTION
The present study deals with comparison testing of two systems: one "basic" (subscript b) and the other "new" (subscript n), having an exponential TBF. The test consists in examination of the hypothesis that the ratio of MTBFn and MTBFb equals a prescribed value, versus the alternative that it is smaller than the latter. An exponential distribution of the TBF is characteristic of many complex systems and in electronics [14]. The need for comparison testing arises when the task is not determination of the absolute value of the MTBFn, but verification that its ratio to MTBFb (likewise unknown) is not less that a prescribed value, so that "n" is a suitable substitute for "b". In addition, simultaneous testing permits neutralization of unregulated or uncontrolled factors such as the environment or maintenance quality. Solutions reported in literature for this type of testing [5, 6] refer to the case where the number of failures for each system has been established in advance of the test (fixed-sample-size test FSST). This approach necessitates large numbers of failures and different test durations. Where the above hypothesis has to be checked, it is convenient to resort to sequential testing, by which means both the necessary numbers of failures and the test durations can be significantly reduced on the average. However, hardly any works on such tests could be found in literature, including indepth reviews on the subject [611]. Girshick [12] considered sequential testing for a class of problems involving comparison of the parameters of a pair of random quantities. A particular feature of these problems is that observations should take place pair wise, yielding one value per observation for each of the compared quantities. This requirement introduces an inconvenient restriction in examining the MTBF ratio. Uno [13] presents a method for a similar problem, namely, sequential estimation of the ratio of two exponential scale parameters. The proposed solution is approximate with a significant bias, and concerns cases where the number of failures for such systems is >2 and the cost per unit sample tends to zero. As in simultaneous pair wise testing, there is a very high probability of one of the systems failing two or less time, this solution is inapplicable in our case. 295

Transport and Telecommunication

Vol.7, No 2, 2006

Two papers dealing with comparative experimentation [14, 15] were published. The solution in the earlier work was limited to a particular case, while in the latter the solution domain was enlarged and the test characteristics were analysed. Both works adopted the lately common simulative approach, and the attendant excessive computer time requirement may be critical to the development of the test. In this paper the proposed solution is applied to binomial sequential testing, on which extensive literature is available permitting the following conclusions: The analytical dependences between the parameters that define the decision boundaries on the one hand, and the probabilities of first ()- and second ()-kind errors on the other are approximate [7, 16]. A numerical recursive method is available for determining the probability of a prescribed combination of successes and failures [17, 18] but is practically not reflected in the literature on sequential testing [16, 10]. From the viewpoint of the average test duration, the SPRT is optimal [7, 19, 20]. The objective of this work is development and study of the truncated SPRT in checking the hypothesis on the MTBF ratio of pair of systems with exponential TBF distribution.

2. DESCRIPTION OF TEST METHOD. PRESENTATION IN BINOMIAL FORM

In the proposed test, the two compared systems operate simultaneously. When one of them fails, it is immediately taken care of and a decision is taken on continuing the test or stopping with acceptance/rejection of the null hypothesis [14] ("final decision").
H 0 : 0 ; (for = 0 probability of accepting H 0 is Pa ( 0 ) = 1 ) H1 : < 0 ; (for = 1 probability of accepting H0 is Pa (1 ) = )

(1)

were

= n / b;
n and b being MTBFn and MTBFb respectively;

(2)

1 = 0/d;
d >1 is the discrimination ratio.
In this test, is evaluated as
= (T / r ) /(T / r ) = r / r , n b b n

(3)

(4)

rb and rn being the respective number of failures up to time T. can change only at the moment of a failure, decisions are also taken at such a moment. Since Thus the considered test is reduced to a binomial form. The test plane is illustrated by Fig. 1, from each point of which a step can be taken either upwards in the REJECT direction (a failure of n) or to the right (a failure of b)) Let us consider the time interval between two failures (irrespective of the system involved). The probability PR,t,k of the new system failing next (a step in the REJECT direction) in the short segment k between (k1)t and kt equals that of intersection of the two independent events:

( ( k 1) t < Tn k t ) (Tb > k t ) ,

where Tn and Tb are, respectively, the times from the first failure in the considered time interval till that of the corresponding system.
PR, .t ,k = exp [ (k 1)t / n ][1 exp(t / n )] exp(k t / b ) .

(5)

296

Proceedings of the 5th International Conference RelStat05

Part 2

Summing these probabilities over k = 1, ..., , we obtain:

PR , t = PR , t , k = [1 exp( t / n ) ] exp( t / b ) [1 exp( t (1/ n + 1/ b )) ]
k =1

and letting t0, we obtain the sought probability of the new system failing next, PR():
PR ( ) = (1/ n ) (1/ n + 1/ b ) = 1 (1 + ) .
16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8

(6)

New system number of failures, rn

RDP ADP accept line reject line test points

9 10 11 12 13 14 15 16

Basic system number of failures,

rb

Figure 1. Test plane for = = 0.1, d = 2 and 0 = 1

The given test point (rb, rn) in Fig. 1 can only be reached from its two adjacent points, with probability
Prb , rn ( ) = Prb , rn 1 ( ) PR ( ) + Prb 1, rn ( ) (1 PR ( ) ) .

(7)

With given we can, starting at point (0,0) where the probability is 1, determine the successive probabilities for all the subsequent points for any prescribed decision boundaries. The "accept" decision point (ADP) can be reached only on failure of the basic system, and its "reject" counterpart (RDP) only on failure of the new one. Correspondingly, we can determine the probabilities PADP(rn, ) and PRDP(rb, ) of reaching the given points ADP, RDP respectively. Combined, these two functions constitute the unified probability distribution function (PDF) of the number of failures to stopping of the test. They serve as a useful test characteristic for the user in deciding how to run it [21]. The probability Pa() is the sum of all the component PADP(rn, ) of reaching all ADP's, hence and are obtainable from the pair of expressions

= 1 Pa(0);

= Pa(1)

(8)

The operational characteristic (OC) represents the relationship Pa(). The average sample number (ASN) to stopping is obtainable as
ASN ( ) =
rbMax rb = 0

( rb + RDP ( rb ) ) PRDP ( rb , ) +

rnMax

rn = 0

( rn + ADP ( rn ) ) PADP ( rn , )

(9)

and the average test duration (ATD) as

ATD( ) = b ASN ( ) (1 + 1/ ) .

(10) 297

Transport and Telecommunication

Vol.7, No 2, 2006

A function WAS04 was developed in Matlab for all the above test characteristics according to prescribed boundaries. Its time requirement on a 3 GHz personal computer is 0.01 to 0.02 s, depending on the number of points in the test plane.

3. BOUNDARIES OF DECISION. USE OF SPRT METHOD

Being in mind (6), the set of hypotheses (1) can be formulated as:
H 0 : p p0 ; (for p = p0 Pa ( p0 ) = 1 ) H1 : p > p0 ; (for p = p1 Pa ( p1 ) = )

(11)

where

p = PR();

p0 = PR(0)=1/(1 + 0);

p1 = PR(1) = 1/(1 + 1)

(12)

For such a system an SPRT is available, where the decisions are demarcated as straight lines with parameters dependent on (p0, ) and (p1, ) [7]. As noted earlier, these relationships have shortcomings: To obtain the sought and , we have to substitute in the boundary formulae values of ' and ', whose exact dependences on , , p0 and p1 are unavailable [7, 19]. Exact dependences reflecting the effect of the site and form of truncation of the test on its OC and ASN are also unavailable. Values of ' and ' compatible with the prescribed and were determined with the aid of an optimised Matlab function, which searches for them so as to minimize the function
RD =

( ( real ) ) + ( ( real ) )
2

(13)

where real and real are the actual values computed by WAS04 for the prescribed stopping boundaries dictated by the values of ' and '. The possible / pairs do not form a continuous plane [15]; there are fairly large tracts for which a solution is moot (Fig. 2), which become still larger on truncation. This feature, described in detail in [15], raises problems in comparative study of the test characteristics and complications in automated design of the test.

4. TYPES OF TEST TRUNCATION AND THEIR EFFECT ON ITS CHARACTERISTICS

The theoretical SPRT can have an unlimited duration, which is impracticable for obvious reasons. Accordingly, various types of truncation are adopted [7, 9, 10, 16], of which the following are most common for the tests under consideration on (Fig. 3): (a) H0 is rejected on reaching rnMax. (b) H0 is accepted on reaching rbMax. (c) A diagonal is drawn through (0,0) parallel to the SPRT boundaries; a horizontal segment of REJECT is drawn at the rnMax level until it meets the diagonal, at which point a vertical segment of ACCEPT is drawn downwards. The above programs permit computation of the ' and ' values and the test characteristics for the different types of truncation. Figs. 4 and 5 show the OC and ASN for types (c) and (a) at different rnMax ( = = 0.1, d = 2 and 0 = 1). The characteristics of the theoretical (non-truncated) SPRT are included as reference. Data of the FSS test are rn = rb = 28. Analysis of the computation results, reflected in part in Figs. 4 and 5, shows that: (1) The non-truncated SPRT is naturally optimal. 298

Proceedings of the 5th International Conference RelStat05

Part 2

Figure 2. Points of possible / pairs for d = 2 Upper diagram truncation at rn = 5*ASN(0) Lower diagram truncation at rn = 2*ASN(0) (drastic)

299

Transport and Telecommunication

Vol.7, No 2, 2006

New system number of failures, rn

50 Reject 40 30 20 10 0 0 10 Accept

rnMax

20

30

40

Basic system number of failures, r b

(a)
60 New system number of failures, rn 50 40 30 20 10 0 0 10 20 30 40 Basic system number of failures, r b Reject Accept

rbMax

(b)
New system number of failures, rn 50 40 30 20 10 0 0 10 20 30 40 Basic system number of failures, r b Reject Accept Center line

rnMax

(c)
Figure 3. Test boundaries for different types of truncation, and for = = 0.1, d = 2 and 0 = 1. The rnMax and rbMax values are the smallest possible for the given type of truncation

300

Proceedings of the 5th International Conference RelStat05

Part 2

WAS03 50 40 30 ASN 20 10 0 0 0.5 1 1.5 2 2.5

MTBF ratio,

1.0 0.8 0.6 0.4 0.2 0.0 OC

ASN_Th OC_Th ASN_44 OC_44 ASN_50 OC_50

Figure 4. Characteristics of test truncated as per type (a) for d = 2, 0 = 1. ASN_Th ASN theoretical (non-truncated test); ASN_44 rnMax = 44, real = 0.099985, real = 0.099915 (' = 0.023993, ' = 0.12723); ASN_50 rnMax = 50, real = 0.099874, real=0.099948 (' = 0.075000, ' = 0.12125); ditto for OC

WAS04
50 1.0

40

0.8

30 ASN

0.6 OC

20

0.4

10

ASN_Th OC_Th

ASN_35 OC_35
1.5 2

ASN_44 OC_44

0.2

0 0 0.5 1 2.5 MTBF ratio,

0.0

Figure 5. Characteristics of test truncated as per type (c). ASN_Th ASN theoretical (non-truncated test); ASN_35 rnMax= 35, real = 0.099871, real = 0.099997 (' = 0.053179, ' = 0.062741); ASN_44 rnMax= 44, real = 0.099966, real = 0.099594 (' = 0.086563, ' = 0.10687); ditto for OC

301

Transport and Telecommunication

Vol.7, No 2, 2006

(2) The OC and ASN computed by the proposed method coincide with the theoretical [7] for boundaries according to ' and ' complying with the prescribed and , and for a sufficiently remote truncation boundary. (3) Truncation of type (c) is superior to the others because of the smaller increase of the ASN versus the non-truncated test. Such truncation permits more drastic reduction of the maximal number of failures. Thus, the smallest value of rnMax for which the prescribed = = 0.1 at d = 2 is 44 for truncation of type (a), and 35 for that of type (c).

5. USER'S PLANNING ALGORITHM FOR THE TEST

1. Choose desired values for the pair of determining points of the OC: (0,1) and (1, ). 2. Calculate p0 and p1 as per (12). 3. Construct the OC and ASN for a binomial SPRT acc. to [7], estimate the ATD as per (1). If the three results are unsatisfactory, go to point 1 for correction of the characteristics or stopping of the test planning; otherwise, go to point 4. 4. Implement an iterative process of establishing the truncation limits, if necessary. Determine ', ' and the OC, ASN, ATD and PDF for the truncated test using the proposed methodology. This iterative process completed, determine the ADP, RDP and final test characteristics.

6. CONCLUSION
A methodology was developed for comparison sequential testing with a view to checking a hypothesis on the MTBF ratio of a pair of systems with an exponential distribution. The test was reduced to a binomial form, thereby lending itself to application of the well-known SPRT theory. A recursive calculation program was worked out for the probability of a decision being taken at prescribed points of the test. It served as basis for exact determination of the OC, ASN, ATD and PDF of a truncated test. The small time requirement of the program (0.010.02 s) permits its incorporation in an optimisation program searching out the stopping boundaries of the test, such that the resulting OC would be closest to the prescribed one. From among the commonly used types of truncation, the one recommended does not affect the OC, least increases the ASN, and permits more drastic truncation of the test. The proposed test is recommended for incorporation in IEC 61650 [5].

ACKNOWLEDGMENTS
The authors are indebted to Mr. E. Goldberg for editorial assistance and to Ms. L. Leshchenko for carrying out the computations and preparing the drawings. The research project was supported by the Israel Ministry of Absorption and by the Planning and Budgeting Committee of the Israel Council for Higher Education.

References
[1] MIL-HDBK-781A: Reliability test methods, plans, and environments for engineering, development, qualification, and production, 1996. [2] IEC 61124 (1997) Reliability testing - Compliance tests for constant failure rate and constant failure intensity. [3] Kapur K.C., Lamberson L.R. Reliability in Engineering Design. NY: Wiley, 1977. [4] Kececioglu D. Reliability & Life Testing: Handbook. NJ: Prentice Hall, 1993, Vol.1. [5] IEC 61650 (1997) Reliability data analysis techniques - Procedures for comparison of two constant failure rates and two constant failure (event) intensities. [6] Mace A. E. Sample Size Determination. NY: Robert E. Krieger Publishing Company, 1974. [7] Wald A. Sequential Analysis. NY: John Wiley & Sons, 1947. [8] Basu A. P. Sequential Methods in Reliability and Life Testing. In: Handbook of Sequential Analysis / Ed. by B.K. Ghosh, P.K. Sen. New York: Marcel Dekker, 1991, pp. 581-592.

302

Proceedings of the 5th International Conference RelStat05

Part 2

[9] Epstein B., Sobel M. Sequential Life Test in the Exponential Case, Annals of Mathematical Statistics, Vol.26, 1955, pp. 82-93. [10] Siegmund D. Sequential Analysis: Tests and Confidence Intervals. New York: Springer, 1985. [11] Sun D. Exact Computation for Some Sequential Tests, Sequential Analysis, 17(2), 1998, pp.127-150. [12] Girshick M. A. Contributions to the Theory of Sequential Analysis, Annals of Mathematical Statistics, Vol. 17, No.2-3, 1946. [13] Chikara Uno. Sequential Estimation of the Ratio of Scale Parameters in the Exponential Two-Sample Problem, J. Japan Statistic. Soc., Vol. 33, No2, 2003, pp. 231-244. [14] Michlin Y. H., Migdali R. Test Duration in Choice of Helicopter Maintenance Policy, Reliability Engineering & System Safety, Elsevier, Vol. 86/3, Dec. 2004, pp. 317-321. [15] Michlin Y. H., Shai Y. Sequential Testing for MTBF Ratio in Comparative Reliability Evaluation. In: The 15th Internat. Conf. of Israel Society for Quality. Jerusalem, 2004, pp.264-268. [16] Wijsman R. A. Stopping Times: Termination, Moments, Distribution. In: Handbook of Sequential Analysis / Ed. by B.K. Ghosh, P.K. Sen. New York: Marcel Dekker; 1991, pp. 67-119. [17] Barnard G. A. Sequential Test in Industrial Statistics, J. R. Statist. Soc. Suppl., 8, (1946) pp.1-21. [18] Hill I. D. Algorithm AS 31: Operating Characteristic and Average Sample Size for Binomial Sequential Sampling, Applied Statistics, Vol. 19, No. 2 (1970), pp. 197-203. [19] Eisenberg B., Ghosh B. K. The Sequential Probability Ratio Test. In: Handbook of Sequential Analysis / Ed. by B.K. Ghosh, P.K. Sen. New York: Marcel Dekker; 1991, pp. 47-66. [20] Thornburg H. Introduction to Bayesian Statistics. Stanford, California: Stanford University, 2000. [21] Michlin Y. H. Sequential Evaluation of QNDE Devices for Underground Storage Tanks, Review of Progress in Quantitative Non-destructive Evaluation, Vol. 22, NY: AIP, 2002, pp. 907-914.

303