Marginal Effects of Linear Models with Data Transformations

by Eric · Published May 7, 2019 · Updated August 22, 2023

Introduction

We use regression analysis to understand the relationships, patterns, and causalities in data. Often we are interested in understanding the impacts that changes in the dependent variables have on our outcome of interest.

Some models provide coefficients that can be directly interpreted as these marginal effects. The coefficients directly represent the predicted change in y caused by a unit change in x.

However, not all models provide such straightforward interpretations. Coefficients in more complex models may not always provide direct insights into the relationships we are interested in.

In this blog, we look more closely at the interpretation of marginal effects in three types of models:

• Purely linear models
• Models with transformations in independent variables
• Models with transformations of dependent variables

Purely linear models

Marginal effects measure the impact that an instantaneous change in one variable has on the outcome variable while all other variables are held constant.

In the simple OLS model with linear effects, estimated coefficients are always equal to marginal effects. To understand why let's consider the model:

$$ price = \beta_0 + \beta_1*weight $$

with estimated coefficients $\beta_0 = -6.707$ and $\beta_1 = 2.04$.

Consider what happens to our predicted outcome, price, as we incrementally increase the vehicle weight.

With each one unit increase in weight there is a $2.04 increase in price. There are two key things to note about the marginal effect of weight on price in this case:

• It is equal to the estimated coefficient, $\beta_1$.
• It is constant across all values of weight.

Transformed data

In the case of the purely linear model, the estimated coefficient is, conveniently, equal to the marginal effects. This result doesn't hold if we consider more complicated models with non-linearities, such as interactions terms, logarithmic terms, or power terms.

Interaction terms

For example, let's add an interaction term into the model

$$ price = \beta_0 + \beta_1*weight + \beta_2*weight*mpg $$

with estimated coefficients:

$$ \beta_0 = 2787.94\\ \beta_1 = 2.227\\ \beta_2 = -0.055 $$

Let's look at what happens to our predicted outcome, price, as we incrementally increase the vehicle weight while holding mpg constant at 16.

With each one unit increase in weight there is a $1.35 increase in price. In this case, the marginal effect of weight on price is no longer equal to the estimated coefficient, $\beta_1$. This is because weight also influences price through the interaction term, $\beta_2 * weight * mpg$.

$$ \begin{aligned} \beta_1 & + \beta_2 * mpg &= 1.345\\ 2.227 & - 0.0552 * 16 &= 1.345 \end{aligned} $$

Therefore, while the marginal effect of a change in the weight variable is still independent of the value of weight, it does depend on the value of mpg.

As we can see on the above graph, increasing mpg dampens the impact that increasing weight has on price. For example, at a mpg of 16, increasing weight increases price by $1.345 for every pound. However, if we increase mpg to 25 the marginal effect of increasing weight decreases to $0.849 for every pound.

In our example linear model with interaction terms, we have seen that the marginal effects:

• Are no longer equal to the estimated coefficient, $\beta_1$.
• Are independent of the value of weight but do depend on the value of mpg.
• Must account for the impact of weight on price which occurs through the interaction term.

Power terms

For our next example, we will look at a model that includes a quadratic term. For this example, we will examine the impact that weight has on mpg:

$$ mpg = \beta_0 + \beta_1*weight + \beta_2*weight^2 $$

with estimated coefficients:

$$ \beta_0 = 51.183\\ \beta_1 = -14.158\\ \beta_2 = 1.3244 $$

Now let's incrementally increase the vehicle weight and see what happens to predicted mpg.

In this case, the marginal effect of a change in the weight on mpg changes as we change the value of weight.

Analytical marginal effects
The table gives us a numerical estimate of the marginal effects. However, in this case we can easily compute the precise analytical marginal effect:

$$ \begin{aligned} &\beta_1 + 2*\beta_2*weight \:=\\ -&14.16 + 2.6488*weight \end{aligned} $$

Using this, let's find the analytical marginal effect when $weight = 2$:

$$ \begin{aligned} \beta_1 & + 2* \beta_2 * weight &= &\\ -14.16 & + 2 * 1.3244 * 2 &= &-8.8624 \end{aligned} $$

Right away we can see that our numeric estimate of the marginal effect when $weight = 2$ does not equal the analytical marginal effect.

Numerically approximated marginal effects
Why is this? Let's look more carefully at how we compute the marginal effects in our table.

We start by predicting the outcome mpg at each value of weight using our regression results:

$$ \widehat{mpg} = 51.183 - 14.16*weight + 1.3244*weight^2 $$

Once we have found these predicted prices we can determine the change as we increase weight by one:

$$ \text{Marginal effects} = \frac{\widehat{mpg}_{weight=3} - \widehat{mpg}_{weight=2}}{3-2}\\ \ \\ \text{Marginal effects} = \frac{20.622 - 28.160}{3-2} = -7.538 $$

In more general terms, we can numerically estimate the marginal effects using the numerical derivative

$$\lim_{\Delta x\to0} \frac{ f(x + \Delta x) - f(x)}{\Delta x} $$

This approximation is valid only as $\Delta x$ gets close to zero.

In our table we use $\Delta x = 1$ to approximate our marginal effects. Let's see what happens if we use $\Delta x = 0.001$ to approximate the marginal effects when $weight = 2$.

First, we find the predicted price when $weight = 2$

$$ \widehat{mpg}_{weight=2} = 51.18 - 14.158*2 + 1.324*2^2 = \$28.160$$

Next, we find the predicted price when $weight = 2 + 0.001$

$$ \widehat{price}_{weight=2.001} = 51.18 - 14.158*2.001 + 1.324*2.001^2 = \$28.151$$

Finally, we subtract the two and divide by the change in weight

$$\text{Marginal effects} = \frac{28.151 - 28.160}{0.001} = -8.861$$

By using a smaller change in weight to compute our numerical approximation we obtain a closer estimate of the analytical solution.

Transformed dependent variables

Sometimes we have models with transformed dependent variables such as log, logit, or probit regressions. In these models we relate a function of Y to our dependent variables:

$$ F(Y) = Y' = X\beta + \epsilon$$

The coefficients from these models do not always intuitively reflect the direct relationships between our observed outcomes and regressors. For this purpose, we can use the marginal effects to better interpret the relationships between our regressors and outcome.

Probit example

Let's consider a simple example of a probit model. The probit model transforms binary outcomes to continuous Y data using the cumulative normal distribution:

$$Y = \Phi (X\beta + \epsilon)$$ $$\Phi^{-1} Y = X\beta + \epsilon$$ $$ Y' = X\beta + \epsilon$$

In this model, we treat $X\beta$ as a z-score. Therefore, as $X\beta$ increases the event measured by Y is more likely to occur.

The estimated coefficient, $\beta$, reflects the increase in the z-score that occurs with an incremental increase in X. However, we may be more interested in knowing how a variable impacts the probability that the event will occur.

For this we use the marginal probability effect, which reflects two things :

• As X increases, the z-score increases through the term $X \beta$. As we noted earlier, the marginal impact of an increase of X on the z-score is equal to $\beta$.
• As the z-score increases, the probability of the event changes. This probability change is measured by the value of the standard normal p.d.f at $X \beta$, $\phi(X\beta)$.

The total marginal probability effect is equal to the combined effect of $\beta$ and $\phi(X\beta)$:

$$\beta * \phi(X\beta).$$

Note that the marginal probability effect is dependent on X. There are a number of ways to choose the most useful X values :

• We can use theoretically relevant X values.
• We can use the mean X values. This will yield the marginal effects at the mean (MEM).
• We can compute the marginal effects at all X values and take the average. This will yield the average marginal effects (AME).

Conclusion

In today's blog we discuss marginal effects. Here are a few key points to take away from this discussion:

• Marginal effects allow us to interpret the direct effects that changes in regressors have on our outcome variable.
• Marginal effects are equal to the estimated coefficients in only a few select cases.
• To understand the direct relationship between regressors and outcomes we need to properly compute the marginal effects based on the functional form of our regression.

Eric( Director of Applications and Training at Aptech Systems, Inc. )

Eric has been working to build, distribute, and strengthen the GAUSS universe since 2012. He is an economist skilled in data analysis and software development. He has earned a B.A. and MSc in economics and engineering and has over 18 years of combined industry and academic experience in data analysis and research.