Notes, STAT 621, MBA Program, Wharton

MODULE 4: THE MULTIPLE REGRESSION MODEL

MULTIPLE VS SIMPLE REGRESSION:
We know by now: multiple regression means more than one predictor.
We will move briskly through the basics of multiple regression,
emphasizing what is different from simple regression.
ROADMAP:
1. Intro to multiple regression with an example:
  data about fuel efficiency of cars
2. Fundamental interpretation of multiple regression slopes:
  the average increase in y per unit increase in the predictor,
  holding all other predictors fixed!!!!!!
3. Aspects in which simple and multiple regression are similar:
  - model with true intercept, slope(s), s, and normal errors
  - estimation with Least Squares: b₀, b₁, (b₂,...),
  - inference with SEs, CIs, t-statistics, p-values
  - s_e = RMSE as estimate of s
  - R² and adjusted R² as fraction of variation explained
  - prediction intervals for future observations
4. Model checking and leverage: some fundamental differences between simple and multiple regression
  - actual by predicted: y_i by yhat_i, instead of y_i by x_i
  - residual by predicted: e_i by yhat_i, instead of e_i by x_i
  - leverage plots, one per predictor
5. JMP skills:
  - "Fit Model"
  - interpretation of outputs
  - construct prediction intervals
BY WAY OF EXAMPLE
1. A business case: Fuel efficiency of cars under the CAFE legislation [slides 4-1...4-5]
  Background: In 1975, Congress reacted to the 1973 oil embargo imposed by the Organization of Petroleum Exporting Countries (OPEC) by establishing the Corporate Average Fuel Economy (CAFE) Program as part of the Energy Policy and Conservation Act. For the current political debate about CAFE, search the internet for "CAFE fuel efficieny".
  The automative industry as well as car buyers have an interest in understanding what factors in cars affect fuel efficiency. The industry can approach this problem with engineering knowledge, but the car-buying public and policy makers may want to collect publicly available data and perform an empirical analysis. Even car manufacturers may be interested in an empirical analysis that includes competitors' products.
2. Dataset: Fuel Efficiency Data
  - Choice of response variable:
    "MPG City"? (3rd column)
    On second thought, physics suggests that energy consumption should be measured by fuel spent per mile, not miles driven per gallon:
    Y = "GP1000M City" (right-most column)
    Use 1000 miles for fewer digits after the decimal dot.
    Intuition pump:
    10 miles per gallon =100 gallons per 1000 miles
    20 miles per gallon = 50 gallons per 1000 miles
    40 miles per gallon = 25 gallons per 1000 miles
    
    Predictor variables:
    There will be more than one factor affecting fuel efficiency.
    => multiple predictors = multiple regression
    Example: X₁="Weight(lb)", X₂="Horsepower" (4^t^h + 5^t^h column)
MULTIPLE REGRESSION IN JMP: [slide 4-5]
"Fit Model": Analyze > Fit Model > "GP1000M City":Y, "Weight(lb)":Add, "Horsepower":Add; Run Model
In the output focus on the sections "Summary of Fit" and "Parameter Estimates".
Ignore the other sections; you can close them by clicking their blue diamonds.
For the moment ignore also the plots at the top.

Organization of JMP:
"Fit Model" is used for multiple linear regression;
"Fit Y by X" is used for simple regression;
"Fit Model" has no "Fit Special" for transformation such as logs;
instead we have to form new columns with formulas.
INTERPRETATION OF REGRESSIONS WITH MULTIPLE PREDICTORS [slide 4-7]
1. New are two things:
  - More than one slope: each predictor gets its slope.
  - Each slope expresses the following:
    the average increase in y per unit increase in the predictor, holding all other predictors fixed!!!
2. Regression coefficients:
  - b₀ = 11.68 GP1000M
    = estimated intercept
    = estimated average GP1000M for zero Weight and zero Horsepower
    (As it is here, this is often a meaningless extrapolation.)
  - b₁ = 0.0089 GP1000M / lb
    = estimated slope of Weight(lb)
    = estimated average increase in GP1000M per additional lb of Weight,
    holding Horsepower fixed
  - b₂ = 0.0884 GP1000M / HP
    = estimated slope of Horsepower
    = estimated average increase in GP1000M per additional horsepower
    holding Weight fixed
3. Mean prediction:
  Estimated mean fuel consumption in GP1000M
  = 11.68 + 0.0089 * Weight(lb) + 0.0884 * Horsepower
  = 11.68 + 0.89 * Weight(100lb) + 0.0884 * Horsepower
PARTIAL VS MARGINAL REGRESSION COEFFICIENTS [slides 4-9, 4-10]
Compare the above regression onto Weight and Horsepower with the simple regression on Weight:
Estimated mean fuel consumption in GP1000M
= 9.43 + 0.0136 * Weight(lb)
These coefficients are different from those of the multiple regression:

Regression coefficients depend on what other predictors are present in the regression!!!

Reason: Each coefficient describes an average change in the response per unit change in the predictor, holding the other predictors fixed.
Example: If Horsepower is dropped from the regression, it is no longer held fixed, and the coefficient of Weight changes.
Jargon: The coefficients of a regression with some predictors dropped are called "marginal coefficients". The coefficients of a regression with the larger set of predictors are called "partial coefficients". Simple regression coefficients are always "marginal".
For a deeper discussion of partial vs. marginal coefficients: see [slide 4-10]
THE MULTIPLE REGRESSION MODEL (MRM) AND LS ESTIMATION [slides 4-6...4-8]

The above regression coefficients and mean predictions are of course estimates of a model:
y₁ = b₀ + b₁ * x₁₁ + b₂ * x₂₁ + e₁
y₂ = b₀ + b₁ * x₁₂ + b₂ * x₂₂ + e₂
.....
y_n = b₀ + b₁ * x₁_n + b₂ * x₂_n + e_n
In general:
y_i = b₀ + b₁ * x₁_i + b₂ * x₂_i + e_i,
e_i ~ i.i.d. N(0,s²)

The parameters are estimated by Least Squares:
Find b₀, b₁, b₂ that make the residual sum of squares, RSS,
as small as possible.

Def.: fits, yhat_i = b₀+b₁*x₁_i+b₂*x₂_i
Def.: residuals, e_i = y_i - yhat_i = estimated errors
Def.: RSS= e₁²+...+e_n²

If the model is a good fit, it can be used to fake data that look like real.
Whether the model IS a good fit, is seen with model checking.
FOR THE REMAINING MATERIAL OF THIS MODULE, SEE THE SLIDES.