Talk:Bayesian linear regression

Bayes or Empirical Bayes?

This isn't a description of Bayesian linear regression. It's a description of the Empirical Bayes approach to linear regression rather than a full Bayesian approach. (Empirical Bayes methods peek at the data to ease the computational burden.) Not only that, it assumes a natural conjugate prior, which is a reasonable approach in many cases, but it is a serious constraint. The fully Bayesian approach with non-conjugate priors is nowadays tractable in all but the largest models through the use of Markov chain Monte Carlo techniques. In my view, this article represents a particular and rather dated approach. Blaise 21:53, 10 April 2007 (UTC)

Can you fix it and check my rating? Thanks - Geometry guy 14:19, 13 May 2007 (UTC)

I don't agree with the first comment, in that it is a full Bayesian approach as opposed to a max likelihood approach. It's true that a conjugate prior is limiting but it is also an enabler for application to very high dimensional input spaces. Also, this same approach can be extended into a kind of adaptive-variance Kalman filter in cases where the model parameters are expected to change through time. i.e. I think the page is very valuable, but would also like to see how to derive matrix A.

Malc S, January 2009.

— Preceding unsigned comment added by 77.100.43.38 (talk • contribs) 12:02, 30 January 2009‎

Some questions

I've got some questions about this article. What this method gives you is a weighted combination of some prior slope, and a new slope estimate from new data.

Q1) the weights are determined by the A. In this one-dimensional case I presume this is just a variance. There are no details as to how this A would be calculated or estimated. Does anyone know?

Q2) I could set this problem up using classical statistics, I think. I'd say "let's make a prediction based on a weighted combination of the prior slope and the new slope". Then I'd do some algebra to derive an expression for the weight. Does anyone have any idea whether the final answer would be much, if any, different?

thanks

82.44.214.29 20:09, 14 October 2007 (UTC)

Computations

Maybe a section on computation would be helpful. When the amount of data is large, direct computation can be difficult. — Preceding unsigned comment added by 173.166.26.241 (talk • contribs) 14:00, 7 August 2011‎

I would suggest that the computations be kept separate from the analytic results using the conjugate prior. I'm doing research, and I come to this page, and all I want is the posterior. It is there, but it is buried in computations. I don't care about the computations; I already know how to do them. I'm not saying get rid of the computations, but not having the posterior, posterior mean, etc, all available quickly makes this article less useful. --68.101.66.185 (talk) 19:08, 9 September 2012 (UTC)

Suggested merge

I've added a suggested merge tag, suggesting the page be merged into ridge regression, since this is essentially a Bayesian interpretation of ridge regression, which has exactly the same mathematics. Jheald (talk) 17:39, 11 November 2012 (UTC)

Proposed merge with Bayesian multivariate linear regression

Single-variable linear regression is just a special case of multivariable. I see no reason to have two articles; if anything, it's confusing. InverseHypercube (talk) 07:11, 27 March 2016 (UTC)

Oppose merge; while it is true that the single-variable case is a special case of the multivariable one, it is helpful to keep them separate because those without a mathematical specialism may more easily understand the single-variable case. Klbrain (talk) 21:09, 8 January 2018 (UTC)

Closing, given lack of support. Klbrain (talk) 09:37, 15 February 2018 (UTC)

Assessment comment

The comment(s) below were originally left at Talk:Bayesian linear regression/Comments, and are posted here for posterity. Following several discussions in past years, these subpages are now deprecated. The comments may be irrelevant or outdated; if so, please feel free to remove this section.

Geometry guy 00:02, 22 May 2007 (UTC) There is a variable A introduced with no explanation. Later it is describes as coming from the Cholesky decomposistion (U). The comments do not seem to be in synch with the expresssions. —Preceding unsigned comment added by 209.6.230.65 (talk) 15:16, 28 March 2009 (UTC)

Last edited at 15:17, 28 March 2009 (UTC). Substituted at 19:49, 1 May 2016 (UTC)

Suggestion: Intro paragraph

A generic problem with science and math articles is approachability. Stats articles like this one would really benefit from a plain/simple english paragraph explaining what the topic means to a general audience. Keep the first more technical introduction paragraph, but have a second paragraph that explains it to your neighbor who wants to make a genuine attempt at understanding what the topic is about but isn't a stats professor.

However, the technical definition needs work too: In statistics , Bayesian linear regression is an approach to linear regression in which the statistical analysis is undertaken within the context of Bayesian inference.

When you say 'In statistics' in the sentence do you really have to say "the statistical analysis"? I mean, in statistics can't we infer that analysis is statistical? Precision and painfully descriptive language is necessary in academia, but isn't that just a tad excessive? Additionally, while maybe technically correct, that definition sounds very circular and not informative. Like surprise surprise Bayesian linear regression refers to is linear regression using Bayesian' methods. Okay... Does that actually tell me anything that I didn't already know? It sounds like you defined the term by repeating the term. Can this please be hashed out more?

What about Inference?

In linear regression, one of the most important results is a confidence interval on the parameter estimates. The bayesian analog is, I believe, the credibility interval. To most practitioners (of merit), a parameter estimate without an accompanying confidence interval is more or less useless. So why no section on inference/credibility intervals in this article? That is a big part of looking at results of models but there is no mention at all here, so I think the article is missing a crucial component. Chafe66 (talk) 20:45, 19 April 2018 (UTC)

Dependence on noise variance dropped?

The solutions given for the posterior appear to be somewhat wrong to me. The equations for the posterior mean should contain a dependence on the estimated noise variance, but it does not. The weight for the likelihood based inference should be ${\frac {1}{\sigma ^{2}}}X^{T}X$ not just $X^{T}X$ I guess. I cannot pinpoint immediately, where this dependence is lost, but the same mean independent of $\sigma$ is certainly wrong. Xenonoxid (talk) 14:13, 19 March 2024 (UTC)

I think I found where this comes from: It seems that the prior here is chosen such that this dependence drops out. This is highly unusual though: Why would we want to be less certain about the parameter values whenever the noise variance is larger? — Preceding unsigned comment added by Xenonoxid (talk • contribs) 14:19, 19 March 2024 (UTC)