Response Surface Methodology

Overview

A brief overview of response surface methodology (RSM) is given in the Experimental Design Lecture.

RSM basically consists of fitting a polynomial surface to a multi-input, multi-output function,

$ \boldsymbol{y} = f(\boldsymbol{x}) $

Types of Linear Models

An overview of different linear models is given at the Linear Models page. This describes how RSM fits into the big picture of linear models.

Strategy for Comprehensive Regression Analysis

From Mason Ch. 14:

Plan the data collection effort

Investigate the data, calculate relevant statistics, plot the data

Specify a functional form for each variable and formulate an initial model

Estimate the model parameters, and calculate statistics that quantify the goodness-of-fit

Assess the model, assess the model assumptions, look for things like collinearities or influential observations

Select statistically significant predictor variables

Some thoughts on this comprehensive strategy:

• It's difficult to plan data collection if you don't already have a model in mind (experimental design), unless you have a very cheap function
• This strategy seems best suited for univariate models, or cheap experiments/function evaluations
• When you have an expensive experiment/function, everything really hinges on the form of the assumed model.. so the fact that Mason doesn't include specify until step 3 indicates that this strategy probably doesn't apply well to those situations

An alternative proposal should follow more closely the validation strategy in the NISS paper

Steps 1-3 provide information going into the surrogate model construction process

• Determine important variables (prior steps)
• Specify a model form
• Design experiments to determine function samples
• Define selection criteria and comparison metrics
• Calculate comparison metrics, plot data
• Assess if the model is good enough, assess model assumptions, select if satisfied (otherwise proceed to next step)
• Specify a new model form (that can incorporate already-gathered information!) and repeat until criteria met

Selecting a Model

In order to select the model that is best for the indented use, two things must be done:

First, figure out what is wanted out of the model (the selection criteria).

Second, figure out how to select the model that is the best for that criteria (the comparison metrics).

Part of the difficulty in defining goals and selection criteria is that multivariate surfaces are very difficult to visualize in higher than 2 dimensions. Various selection criteria, i.e. numerical quantities related to error, curvature, best fit, etc., should be used to determine which surface is the best for the intended use.

Pre-Selection Step: Experimental Design

Before selecting the form of the surrogate model, you must first select your experimental design. Typically the experimental design is selected to regress some particular functional form (e.g. a polynomial).

A form of the model output(s) as a function of the model input(s) is assumed in order to sample the function as few number of times as possible.

If a Monte Carlo simulation is being run, the cost is very high, but the method is very flexible - any linear model from above may be selected and fit to the data (in this case, it is useful to explore different models of different forms and degrees).

For more information on the experimental design step, see the Experimental Design Lecture.

Selection Criteria

The most obvious criteria is minimization of error $ y - \hat{y} $

$ y $ = real function's response

$ \hat{y} $ = surrogate model response

What experimental design is trying to accomplish for simulations and experiments is similar:

• Simulations are trying to make a complex function evaluation very cheap, without losing too much information
• Experiments are trying to create a model for a complex physical process

However, the end use is often different:

• Simulations are trying to determine the values of input parameters that make the model match experimental data
• Experiments are trying to optimize a process and find minima/maxima of the physical process

Least Squares for Linear Regression

Using least squares for a linear regression model approximates the coefficients of the linear model by minimizing the sum of the squared error residuals (SSE),

$ SS_{E} = \sum r_i^2 = \sum \left( y_i - \hat{y} \right)^2 $

The estimated coefficients are then called least squares coefficient estimates

To do this, SSE equation (above) differentiated with respect to each of the parameters $ {b_0, b_1, b_2, \dots} $

All these derivatives are set equal to 0, and these equations are solved simultaneously.

Interpretation

Cannot necessarily interpret approximated coefficients as "amount of change in $ \hat{y} $ for a unit change in $ x_j $"... This assumes that the coefficients are completely independent

In order to determine how good this assumption is, regress the input/predictor variable $ x_j $ on all other variables (i.e. find $ x_j $ as a function of all other input/predictor variables + constant)

The residuals from this fit are $ r^{\star} = x_{ij} - \hat{x}_{ij} $

The coefficient estimate $ b_j $ measures change in response due to unit change in $ r^{\star} $, not in $ x_{ij} $

If $ x_j $ can't be predicted by other variables, then $ r^{\star} \approx x_{ij} - \overline{x}_{ij} $ (where overline = average value)

In this case, $ b_j $ can be interpreted in the way specified: i.e. measure of change in $ \hat{y} $ for unit change in $ x_j $

Significance

Coefficients cannot be used by themselves to determine relative significance of various terms in the linear model

To actually do this, you need to use normalized/weighted coefficients $ \hat{\beta}_j $

Defined as:

$ \hat{\beta}_j = b_j \left( \frac{s_{jj}}{s_{yy}} \right)^2 $

where

$ s_{jj} = (n-1)^{-1} \sum (x_{ij} - \overline{x}_j)^2 $

$ s_{yy} = (n-1)^{-1} \sum (y_i - \overline{y}_i)^2 $

Comparison Metrics

Analysis of Variance (ANOVA) Table

Mason ch. 6, 8

Contour Plots

Contour plots can be used to determine sensitivities: if the response $ y $ changes significantly in one parameter direction, it is sensitive to that parameter. If the contour shows a structure that is uniform in one parameter direction, the response is not sensitive to that parameter.

For multiple responses, a contour plot for each response can be made, infeasible regions shaded gray, and the plots overlaid to yield the feasible region.

Other Things to Look At

• Correlation between input variables
  • e.g. for time and temperature: $ {t, T, tT, t^2, T^2} $

Issues

Dealing with Multimodal Variables

Sometimes, when constructing response surfaces, modal variables appear. Modal variables are variables that have multiple modes, or distinct sets of values. There are two variations of modal variables:

1 uncertainty range (sampled with N parameter values)

These types of modal variables have a single range of uncertainty assigned to them, but the values within that range of uncertainty are discrete. In order to sample the parameter within the range of uncertainty, the parameter must be sampled at distinct, discrete values.

For example, if I am using the discrete ordinates model (DOM) for radiation calculations, the DOM requires a number of ordinate directions. This is a discrete value with distinct sets of values - e.g. 3, 6, 8, 24, etc.

Each discrete value in this case composes a single range of uncertainty. Using the DOM example, that range of uncertainty would be $ [3, 24] $.

N uncertainty ranges

The other type of modal variables have several ranges of uncertainty assigned to them, with no restriction on values within that range of uncertainty being discrete or distinct. Essentially this can be thought of as a bimodal uncertainty distribution, where the two modes are distinct. Each mode can be sampled as usual, the only sticking point is that there is more than 1, and that they are distinct.

This case provides an excellent example. The variable $ \dot{m} $ is a modal variable - the two modes are 1.0 and 2.0 - but each mode also has a range of uncertainty, namely $ 5% $ each.

How to Deal

Multimodal variables can be dealt with in two ways:

Method 1: Separate Response Surfaces for Each Mode

The first way is to create a separate response surface for each distinct mode. This method works for both types of modal variables (1 uncertainty range represented by N distinct values, and N uncertainty ranges). This method is illustrated in the figures below. Each distinct mode (gray region) has its own computed response surface (blue dotted line), distinct from the response surface of the other modes.

Of course, if the variable type is 1 uncertainty range represented by N distinct values, then there is no uncertainty range for each mode, and each gray region is, in fact, a delta function. As mentioned above, this means that the input variable is eliminated as a response surface parameter.

If the variable type is N uncertainty ranges, then each uncertainty range is sampled as usual, and each response surface is constructed as usual.

An example of a "true" response, which is unknown to the modeler.	The modeler is only interested in distinct regions of the input parameter $ x $ (shown in gray). The remaining regions are left out of the response surface.	The response surfaces actually obtained by the user (blue dotted line). There is a separate response surface obtained by the user (2 distinct blue lines) for each mode (gray region).

Method 2: Single Response Surface (Ignore Modes)

A second way is to create a single response surface. This is typically only possible with N uncertainty ranges type of problems, because the parameter value is continuous, but it is only certain regions that are of interest. This approach is illustrated below.

Essentially, this approach does away with any special treatment of modes.

(see above, image repeated for clarity)	(see above, image repeated for clarity)	An example of the second approach, in which the modeler constructs a single response surface, essentially ignoring the modes of the input parameter $ x $.

Analysis of Results and Construction of Response Surface

The ultimate reason for sampling the function is to construct a response surface, and in order to construct a response surface, some kind of generalized linear model will have to be used.

NOTE: there is a more general discussion of experiment design, of which the response surface methodology is only one of several, at the following page:

Experimental Design Lecture

See Also

• Experimental Design Lecture