Design Of Experiments (Doe) And Regression
Errors from DOE Procedures:
- Investing in experimentation of any type is intrinsically risky. This follows because if the results were known in advance, experimentation would be unnecessary.
- The theory can also aid in the comparison of method alternatives. concepts associated with errors in testing hypotheses are described which are relevant to many design of experiments methods.
- These concepts are helpful for competent application and interpretation of results. They are also helpful for appreciating the benefits associated with standard screening using fractional factorials.
- The definition of these errors involves the concepts of a “true” difference and absence of the true difference in the natural system being studied. Typically, this difference relates to alternative averages in response values corresponding to alternative levels of an input factor.
- In real situations, the truth is generally unknown. Therefore, Type I and Type II errors are defined in relation to a theoretical construct. In each hypothesis test, the test either results in a declaration of significance or failure to find significance.
- Failure to find significance when no difference exists is only a “semi-success” because the declaration is indefinite.
- Implied in the declaration is that with more runs or slightly different levels, a difference might be found. Therefore, thedeclaration in the case of no true difference is not as desirable as it could be.
- for two-sample t-testing, the chance of errors depends on all of the following:
1. The sample sizes used, n1 and n2
2. The α used in the analysis of results
3. The magnitude of the actual difference in the system (if any)
4. The sizes of the random errors that influence the test data (caused by uncontrolled factors)
Like many testing procedures, the two-sample t-test method is designed to have the following property.
- For testing with chosen parameter α and any sample sizes, the chance of Type I error equals α. In one popular “frequentist” philosophy, this can be interpreted in the following way. If a large number of applications occurred, Type I errors would occur in α fraction of these cases.
- Therefore, fixing α determines the chance of Type I errors. At the same time, the chance of a Type II error can, in general, be reduced through increasing the sample sizes. Also, the larger the difference of interest, the smaller the chance of Type II error. In other words, if the tester is looking for large differences only, the chance of missing these distances and making a Type II error is small, in general.
Example (Testing a New Drug)
An inventor is interested in testing a new blood pressure medication that she believes will decrease average diastolic pressure by 5 mm Hg. She is required by the FDA to use α = 0.05. What advice can you give her?
a. Use a smaller α; the FDA will accept it, and the Type II error chance is lower.
b. Budgeting for the maximum possible sample size will likely help prove her case.
c. She has a larger chance of finding a smaller difference.
d. All of the above are correct.
e. All are correct except (b) and (d).
Answer: As noted previously, if something is proven using any given α, it is also proven with all higher levels of α. Therefore, the FDA would accept proof with a lower level of α. However, generally proving something for a lower α implies an
increased chance of Type II error. Generally, the more data, the more chance of proving something that is true. Also, finding smaller differences is generally less likely.
Therefore, the correct answer is (b).