Abstract

Sometimes you have a set of data in which each data point is not equivalently relevant. Here I will walk through a variant of the standard derivation for least squares regression that incorporates an arbitrary weight vector that will allow certain data points to have a greater or lesser effect on the model's fit.

Method

Note: for clarity, I will now be now be omitting the bounds of all sum notation. Just know that it will always be over $(1, n)$. \begin{align} \frac{\partial \ew}{\partial a} & = \mysum -2w_i (Y_i - a - bX_i) \linesp & = 2 \mysum w_i a + w_i b X_i - w_i X_i \linesp & = 2 (a \mysum w_i + b \mysum w_i X_i - \mysum w_i Y_i) \label{d} \tag{4} \end{align} Recalling that we are aiming for this derivative to be equal to $0$, \begin{align} 0 & = 2 (a \mysum w_i + b \mysum w_i X_i - \mysum w_i Y_i) \linesp a \mysum w_i & = \mysum w_i Y_i - b \mysum w_i X_i \linesp a & = \frac{\sum w_i Y_i - b \sum w_i X_i}{\sum w_i} \label{e} \tag{5} \end{align} That gives us our formula for the intercept $a$ which just leaves us with the slope $b$ for which we will follow a similar, yet slightly more involved strategy. \begin{align} \frac{\partial \ew}{\partial b} = 0 & = \mysum -2 w_i X_i (Y_i - a - bX_i) \linesp & = a \mysum w_i X_i + b \mysum w_i X_i^2 - \mysum w_i X_i Y_i \label{f} \tag{6} \end{align} From here we substitute our formula for $a$, \begin{align} \frac{\wx \wy - b(\wx)^2}{\w} + b\wxx - \wxy & = 0 \linesp \frac{\wx \wy + b (\w \wxx - \wx^2)}{\w} - \wxy & = \linesp b \frac{(\w \wxx - \wx^2)}{\w} & = \wxy - \frac{\wx \wy}{\w} \linesp \end{align} Finally, dividing to isolate $b$, $$b = \frac{\w \wxy - \wx \wy}{\w \wxx - \wx^2} \label{g} \tag{7}$$

So we have our final closed-form solution, $$a = \frac{\sum w_i Y_i - b \sum w_i X_i}{\sum w_i} \linesp b = \frac{\w \wxy - \wx \wy}{\w \wxx - \wx^2}$$

References

http://seismo.berkeley.edu/~kirchner/eps_120/Toolkits/Toolkit_10.pdf