I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons."
In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for Bayesianism on their head.
I'm actually neither Bayesian nor frequentist, or both, depending on how you look at it, but I think Bayesianism is being overhyped. Sure, the prior is part of the model, but if you can estimate something without adding extra baggage, why not?
Imagine doing a meta-analysis, and now having all the extra heterogeneity due to priors. Why add that?
My guess is a lot of the appeal of Bayesianism has to do with the success of the machinery surrounding it, like MCMC, which is sort of automatic and has certain other appeals. As people realize you can do stochastic optimization with raw ML inference, some of the appeal will probably dissipate a bit (although not entirely).
I recommend reading this[1] essay which explains exactly the difference between the two methods. (I'm linking to part 3 because it specifically answers your questions, but the entire series is worth reading)
Why does it matter? Frequentist and Bayesian methods offer different interpretations of what your result means.
... speaking broadly, frequentists consider model parameters
to be fixed and data to be random, while Bayesians consider
model parameters to be random and data to be fixed.
This distinction is important, because...
> the frequentist and Bayesian methods are the same
...this isn't always true. See [1] for the details.
Common question in our line of work: when should I end my A/B test? End too early and I have wrong conclusions, end too long and it's expensive. Classic exploration vs. exploitation.
Frequentist statistics will tell us to suck it up and just choose a sample size, or write down every time we look at the numbers to adjust for peeking effect, or assume the worst case that we always peek and wait a very long time.
Bayesian statistics tells us not to use a single termination condition, but adjust the samplings according to our continuously updating priors. (This is what Google Analytics does. https://support.google.com/analytics/answer/2844870)
In fact, this turns out to be the mathematically proveable optimal solution for maximizing payoff.
This latter approach is beyond the reach of frequentist methods because it's just a completely different paradigm.
The question of frequentist and bayesian not "which is more correct", but rather "which paradigm best matches the question I have?" If your testing paradigm is fixed sample sizes, choose frequentist. If it is iterative (like a lot of ML), choose bayesian.
Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectively!
Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling"
From your Wikipedia link:
"Probability matching strategies are also known as Thompson sampling or Bayesian Bandits, and surprisingly easy to implement if you can sample from the posterior for the mean value of each alternative."
Oh yeah there are a few Bayesian methods which work great, Bayes-UCB is another. Personally though I think KL-UCB or just plain old UCB would be the ones I'd choose. Like I said earlier, I think these techniques are like programming languages: choose the one you know well enough to get the job done with it.
If the inferential question you're interested in is, "given data X, what do I conclude about underlying cause/variable/parameter T?", then you are a Bayesian, like it or not.* Sure you can define a likelihood function p(X|T), but that doesn't give you p(T|X) unless you multiply by a prior p(T).
Now certainly p(T|X) is not always the question, but in the vast majority of cases people do want to use data to draw conclusions, and will misinterpret likelihood-based confidence intervals as posterior credible intervals because the latter are what they intuitively wanted. By doing so they implicitly assume a flat prior regardless of whether that that is reasonable or even mathematically coherent for the problem in question.
The Bayesian argument is not that you have to use an informative prior (though often this can be very helpful!), but that since some sort of prior is mathematically necessary to answer the questions that people intuitively want statistics to answer, we should make that explicit and try to understand how the prior affects our conclusions, not just sweep it under the rug.
* If the question you're interested in is, "if I run some method in many repeated trials, how often will it identify the true parameter?" then you are a frequentist. I think it's much rarer for this to genuinely be someone's intuitive question, but it's certainly valid. And of course it's valid to ask both questions at once, in which case you might end up analyzing the frequentist properties of a Bayes-derived method.
Supose you don't know what color a car is, so you look and see it's green and then you look again and still green. Now, what's the Bayesian calculation.
Well how accurate are your eyes, how stable is car color ect.
Frequencist methods do not remove the prior, they merely hide it, make it implicit. Usually the implicit prior is reasonable but being blind of your prior still risks making you trip on subtle trade-offs and biases for more complex problems.
This argument would seem to assume that a frequentist is required to use maximum likelihood estimation for all problems.
If you remove that assumption, the distinction between Bayesian and frequentist methods becomes murkier. If you ultimately want a point estimate, you don't particularly care whether a method constructs a posterior distribution as an intermediate step.
He explained stuff plainly, using references (sci-fi, japanese manga and anime) that are popular among a fairy restricted set of people. It's not academic, and it's not all-inclusive.
Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
Mostly that some of us have had very poor experiences with them while trying to mind our own business. It's like living next to a cult's compound, and therefore having more concrete bad experiences with them than someone in the next city.
In my case, an IRC channel that I previously enjoyed (relationship-related) was unofficially absorbed into LessWrong's network of channels. This resulted in people treating it as a meat market despite being asked not to, then a debate platform despite being asked not to, some refused the existence of trans people to their faces because they couldn't logic it out, we had one argue in favour of cheating in a channel that explicitly mentioned openness and honesty in the topic, they damn near had a riot if we ever dared ban one of them for being spectacularly awful, and they generally put the value of other people's emotions at close to 0 in their messed-up logic for interacting with the world. This carried on for months. Apparently after I left most of them were finally banned, though not before a good many of the less awful people in the channel left for good. This was the first major issue the channel had in four years of its existence.
More generally, many people don't enjoy conflict, and the LessWrong way of discussing anything seems to be to turn it into a two-sided debate. Even when you're trying to discuss who you think you are, turning it into you defending yourself from attack. The concept of having a constructive conversation, seems to have entirely slipped past them, never mind the idea that not everything has to be debated right then and there.
Unfortunately, this isn't just my experience - various others I've spoken to have come across exactly the same sort of thing. Hence the extreme dislike from some people. I'm not sure whether the community explicitly pushes this way of interacting with the world, or whether it just attracts assholes and gives them a "logical" reason to be assholes, though.
Well… I've mainly limited myself with LessWrong top-level posts so far, so I haven't observed those problems first hand.
I may even have been part of the problem for a while, posting semi-relevant LW links all over the place. I have since learned to focus my arguments into something that doesn't require having read the sequences. Maybe no longer frequenting LW helped.
Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a philosophical position dictate their approach rather than best tools for the job.
Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather I am a more well-rounded, and thus more capable technician because I know the benefits and weaknesses of each approach.
If you have small-to-medium sample sizes, then clearly Fisher style statistics will not work very well. On the other hand, even if you have a small sample size, if you don't have some general knowledge to guide your priors, you may very well end up with garbage in a Bayesian approach.
How do you know the solution is any good? I'm not going to go all Socrates on you so I'll jump to my point: when discussing bayesian statistics, we're touching something very profound about epistemology that can't be swept under the rug in the name of pragmatism. We're dealing with the core philosophical underpinnings of what it means to "know" something.
This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest.
Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer.
The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation.
Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?).
To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.
Your different conditions simply represent the probability of the blue line crossing two different other lines.
Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.
> Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently.
That's exactly what I had in mind. It amounts to the same, really.
Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does.
Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid.
> Frequentism isn't stupid, but you do need to be really smart to use them correctly.
I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste.
When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them.
I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think?
> [VWO slides]
Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.
Many people learn about the philosophy of the Bayesian estimation and fall in love with it, or at least that happened to me.
I thought that only in the Bayesian formulation of statistics the estimated parameters (mean, standard deviation, kurtosis, percentiles, whatever) remain uncertain after the estimation, and therefore they have a distribution; I didn't know that in the frequentist interpretation what you calculate are actually estimators, and they are random variables and therefore have uncertainty in them. Silly me.
One day, reading about estimators on stackexchange, I was led to a quote from the "Elements of Statistical Learning" ([1], p. 272): "we might think of the bootstrap distribution as a poor man's Bayesian posterior".
So, I put off learning all the details of the Bayesian estimation, and I started using the bootstrap to deal with my problems. I recommend everyone to give the bootstrap method a try before they go all in for Bayesian estimation.
Once you get familiar with the bootstrap, you might feel that actually Bayesian estimation might be overkill.
But you shouldn't stop there. With a little more contemplation, you might start doubting the whole Bayesian edifice. Let me tell you why.
Here's a quote from a book [2] "Bayesian Risk Management" that, unsurprisingly, given the title, extols the virtues of the Bayesian framework: "If the data are consistent with our prior estimates, the location of the parameters will be little changed and the variance of the posterior distribution will shrink. If the data are surprising given our prior estimates, the variance will increase and the location will migrate" (p. 11). This is the common view of the Bayesian estimation, and it's wrong. To give you an example, if you do Bayesian estimation for the mean of a random sample assumed to come from a normal distribution with a given standard deviation, then the variance of that mean keeps going down with each observation regardless of the value of the observation (check first entry in [3])
But that shouldn't shatter your hope for a better world. The Bayesian estimation is bound to produce the wrong result if you assume the wrong model, regardless of what prior you use. But now you discover that the Bayesian estimation doesn't absolve you of the responsibility of choosing a good model (which is another commonly held believe). But then, if you do need to carefully choose a model, why do you need Bayesian after all.
I am not sure. Let me kick Bayesian while it's down a bit more, before I start defending it.
If you ever get curious about Kalman filter estimation, a good book to use is Durbin and Koopman [4]. In the first 20 pages you will learn that in the simplest setting (local level model), the Kalman filter gives exactly the same results in the frequentist and bayesian interpretation. Food for thought.
Here's a quote from Efron&Hastie "Computer Age Statistical Inference": "Computer-age statistical inference at its most successful combines elements of the two philosophies, as for instance the empirical Bayes methods in Chapter 6, and the lasso in Chapter 16. There are two arrows in the statistician's philosophical quiver, and faces, say, with 1000 parameters and 1,000,000 data points, there's no need to go hunting armed with just one of them."
Now I did say I'll come to the defense of Bayesian. To my knowledge the Markov Chain Monte Carlo method was developed only in the Bayesian setting. I do believe a frequentist interpretation is entirely possible, but so far nobody offered it. But until someone needs to use MCMC, I don't really see a need to go Bayesian, when bootstrap works perfectly fine.
What are the Bayesian methods I would use if I have multiple related outcomes and multiple predictors...you know...the situation in which many scientists find themselves with longitudinal data and typically turn to repeated measures and random effect general linear models?
38 comments
[ 5.5 ms ] story [ 84.5 ms ] threadIn large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for Bayesianism on their head.
I'm actually neither Bayesian nor frequentist, or both, depending on how you look at it, but I think Bayesianism is being overhyped. Sure, the prior is part of the model, but if you can estimate something without adding extra baggage, why not?
Imagine doing a meta-analysis, and now having all the extra heterogeneity due to priors. Why add that?
My guess is a lot of the appeal of Bayesianism has to do with the success of the machinery surrounding it, like MCMC, which is sort of automatic and has certain other appeals. As people realize you can do stochastic optimization with raw ML inference, some of the appeal will probably dissipate a bit (although not entirely).
I recommend reading this[1] essay which explains exactly the difference between the two methods. (I'm linking to part 3 because it specifically answers your questions, but the entire series is worth reading)
Why does it matter? Frequentist and Bayesian methods offer different interpretations of what your result means.
This distinction is important, because...> the frequentist and Bayesian methods are the same
...this isn't always true. See [1] for the details.
[1] http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
Frequentist and bayesian statistics are different paradigms, not different number crunchers.
---
I talk about this with graphs and such in my blog post (https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...) but here's the short version:
Common question in our line of work: when should I end my A/B test? End too early and I have wrong conclusions, end too long and it's expensive. Classic exploration vs. exploitation.
Frequentist statistics will tell us to suck it up and just choose a sample size, or write down every time we look at the numbers to adjust for peeking effect, or assume the worst case that we always peek and wait a very long time.
Bayesian statistics tells us not to use a single termination condition, but adjust the samplings according to our continuously updating priors. (This is what Google Analytics does. https://support.google.com/analytics/answer/2844870) In fact, this turns out to be the mathematically proveable optimal solution for maximizing payoff. This latter approach is beyond the reach of frequentist methods because it's just a completely different paradigm.
The question of frequentist and bayesian not "which is more correct", but rather "which paradigm best matches the question I have?" If your testing paradigm is fixed sample sizes, choose frequentist. If it is iterative (like a lot of ML), choose bayesian.
[1]: https://en.m.wikipedia.org/wiki/Sequential_analysis
[2]: https://en.m.wikipedia.org/wiki/Multi-armed_bandit
And, as he said, you have to make adjustments to account for these interim analysis.
---
Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling"
From your Wikipedia link:
"Probability matching strategies are also known as Thompson sampling or Bayesian Bandits, and surprisingly easy to implement if you can sample from the posterior for the mean value of each alternative."
Now certainly p(T|X) is not always the question, but in the vast majority of cases people do want to use data to draw conclusions, and will misinterpret likelihood-based confidence intervals as posterior credible intervals because the latter are what they intuitively wanted. By doing so they implicitly assume a flat prior regardless of whether that that is reasonable or even mathematically coherent for the problem in question.
The Bayesian argument is not that you have to use an informative prior (though often this can be very helpful!), but that since some sort of prior is mathematically necessary to answer the questions that people intuitively want statistics to answer, we should make that explicit and try to understand how the prior affects our conclusions, not just sweep it under the rug.
* If the question you're interested in is, "if I run some method in many repeated trials, how often will it identify the true parameter?" then you are a frequentist. I think it's much rarer for this to genuinely be someone's intuitive question, but it's certainly valid. And of course it's valid to ask both questions at once, in which case you might end up analyzing the frequentist properties of a Bayes-derived method.
Supose you don't know what color a car is, so you look and see it's green and then you look again and still green. Now, what's the Bayesian calculation.
Well how accurate are your eyes, how stable is car color ect.
Although it is so esoteric that one might argue that it is more the field of algorithmic statistics which merely employs Bayesian.
https://en.m.wikipedia.org/wiki/Algorithmic_probability https://en.m.wikipedia.org/wiki/Solomonoff%27s_theory_of_ind...
If you remove that assumption, the distinction between Bayesian and frequentist methods becomes murkier. If you ultimately want a point estimate, you don't particularly care whether a method constructs a posterior distribution as an intermediate step.
Of course it will sound like a weird cult. I'm not sure why anybody would have any problem with that.
In my case, an IRC channel that I previously enjoyed (relationship-related) was unofficially absorbed into LessWrong's network of channels. This resulted in people treating it as a meat market despite being asked not to, then a debate platform despite being asked not to, some refused the existence of trans people to their faces because they couldn't logic it out, we had one argue in favour of cheating in a channel that explicitly mentioned openness and honesty in the topic, they damn near had a riot if we ever dared ban one of them for being spectacularly awful, and they generally put the value of other people's emotions at close to 0 in their messed-up logic for interacting with the world. This carried on for months. Apparently after I left most of them were finally banned, though not before a good many of the less awful people in the channel left for good. This was the first major issue the channel had in four years of its existence.
More generally, many people don't enjoy conflict, and the LessWrong way of discussing anything seems to be to turn it into a two-sided debate. Even when you're trying to discuss who you think you are, turning it into you defending yourself from attack. The concept of having a constructive conversation, seems to have entirely slipped past them, never mind the idea that not everything has to be debated right then and there.
Unfortunately, this isn't just my experience - various others I've spoken to have come across exactly the same sort of thing. Hence the extreme dislike from some people. I'm not sure whether the community explicitly pushes this way of interacting with the world, or whether it just attracts assholes and gives them a "logical" reason to be assholes, though.
I may even have been part of the problem for a while, posting semi-relevant LW links all over the place. I have since learned to focus my arguments into something that doesn't require having read the sequences. Maybe no longer frequenting LW helped.
Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather I am a more well-rounded, and thus more capable technician because I know the benefits and weaknesses of each approach.
If you have small-to-medium sample sizes, then clearly Fisher style statistics will not work very well. On the other hand, even if you have a small sample size, if you don't have some general knowledge to guide your priors, you may very well end up with garbage in a Bayesian approach.
Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer.
The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation.
Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?).
To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.
I've got slides that illustrate exactly your example. Look at the graph below:
https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2...
Your different conditions simply represent the probability of the blue line crossing two different other lines.
Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.
That's exactly what I had in mind. It amounts to the same, really.
Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does.
Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid.
> Frequentism isn't stupid, but you do need to be really smart to use them correctly.
I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste.
When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them.
I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think?
> [VWO slides]
Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.
I thought that only in the Bayesian formulation of statistics the estimated parameters (mean, standard deviation, kurtosis, percentiles, whatever) remain uncertain after the estimation, and therefore they have a distribution; I didn't know that in the frequentist interpretation what you calculate are actually estimators, and they are random variables and therefore have uncertainty in them. Silly me.
One day, reading about estimators on stackexchange, I was led to a quote from the "Elements of Statistical Learning" ([1], p. 272): "we might think of the bootstrap distribution as a poor man's Bayesian posterior".
So, I put off learning all the details of the Bayesian estimation, and I started using the bootstrap to deal with my problems. I recommend everyone to give the bootstrap method a try before they go all in for Bayesian estimation.
Once you get familiar with the bootstrap, you might feel that actually Bayesian estimation might be overkill.
But you shouldn't stop there. With a little more contemplation, you might start doubting the whole Bayesian edifice. Let me tell you why.
Here's a quote from a book [2] "Bayesian Risk Management" that, unsurprisingly, given the title, extols the virtues of the Bayesian framework: "If the data are consistent with our prior estimates, the location of the parameters will be little changed and the variance of the posterior distribution will shrink. If the data are surprising given our prior estimates, the variance will increase and the location will migrate" (p. 11). This is the common view of the Bayesian estimation, and it's wrong. To give you an example, if you do Bayesian estimation for the mean of a random sample assumed to come from a normal distribution with a given standard deviation, then the variance of that mean keeps going down with each observation regardless of the value of the observation (check first entry in [3])
But that shouldn't shatter your hope for a better world. The Bayesian estimation is bound to produce the wrong result if you assume the wrong model, regardless of what prior you use. But now you discover that the Bayesian estimation doesn't absolve you of the responsibility of choosing a good model (which is another commonly held believe). But then, if you do need to carefully choose a model, why do you need Bayesian after all.
I am not sure. Let me kick Bayesian while it's down a bit more, before I start defending it.
If you ever get curious about Kalman filter estimation, a good book to use is Durbin and Koopman [4]. In the first 20 pages you will learn that in the simplest setting (local level model), the Kalman filter gives exactly the same results in the frequentist and bayesian interpretation. Food for thought.
Here's a quote from Efron&Hastie "Computer Age Statistical Inference": "Computer-age statistical inference at its most successful combines elements of the two philosophies, as for instance the empirical Bayes methods in Chapter 6, and the lasso in Chapter 16. There are two arrows in the statistician's philosophical quiver, and faces, say, with 1000 parameters and 1,000,000 data points, there's no need to go hunting armed with just one of them."
Now I did say I'll come to the defense of Bayesian. To my knowledge the Markov Chain Monte Carlo method was developed only in the Bayesian setting. I do believe a frequentist interpretation is entirely possible, but so far nobody offered it. But until someone needs to use MCMC, I don't really see a need to go Bayesian, when bootstrap works perfectly fine.
[1] http://statweb.stanford.edu/~tibs/ElemStatLearn/ [2] SubiculumCode ↗ What are the Bayesian methods I would use if I have multiple related outcomes and multiple predictors...you know...the situation in which many scientists find themselves with longitudinal data and typically turn to repeated measures and random effect general linear models? kgwgk ↗ Maybe this helps: https://arxiv.org/abs/1506.06201 SubiculumCode ↗ Thanks. Your the first to ever reply to that question when I pose it.