> the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs.
That assumes that the efficient-market hypothesis holds true, but it has yet to be thoroughly proven or disproven... (and funds like Medallion would strongly suggest otherwise for the medium term: https://www.bloomberg.com/news/articles/2016-11-21/how-renai...)
It doesn't assume the Effiecient Market hypothesis - empirical studies of returns support random returns without the imposing a model (non-parametric tests).
That's not to say returns are actually random, but in any given time range, it appears to be.
Buying the S&P 500 in 1950 and holding 67 years does.
One sample tells you nothing about randomness. What if you buy in August 1929? What if you hold for a more realistic 20 or 30 years from peak earning years to retirement?
Annual Total Return: 9.1%
Annual Real Total Return: 5.9%
Bought in January 1987, held for a realistic 30 years:
Annual Total Return: 9.8%
Annual Real Total Return: 7.0%
There's always going to be some deviation, but over any given multi-decade holding period, you will generally end up with a predictable 5-9% annualized (inflation-adjusted) return. That is more than zero. My point stands: long-term investment in the S&P 500 can be reasonably expected to gain value faster than inflation.
If you're interested, here's a simulator that looks at historic market data. You'll note that even the lowest possible percentile of 30-year holding periods will still yield a 3.43% inflation-adjusted total return: https://dqydj.com/sp-500-historical-return-calculator-popout...
Let's buy in August 1929 at 5338.69, and sell 20 years later, in August 1949, at 1822.87 (inflation-adjusted). Congratulations, you lost two thirds of your money.
Sell 30 years later instead? August 1959, at 5525.23. Wow, after 30 years you're up almost 3.5%!
> So...How much will this do at forecasting stock prices? =)
Probably quite poorly (due to stocks appearing "random" at scale), especially for indexes, which are a sum of their parts.
On the other hand, this would probably be quite useful for things that have non-random trends (like the Global Energy Forecasting Competition: http://www.drhongtao.com/gefcom)
It would probably perform pretty poorly as other has suggested. This is mainly due to the fact that stock prices by itself is a pretty non-stationary dataset/measurement. Most of these probabilistic models are poorly equipped to make accurate predictions for non-stationary data since it's trends are seemingly similar to noise.
Faced with phenomena I view as self-affine, other students take an extremely different tack. Most economists, scientists and engineers from diverse fields begin by subdividing time into alternating periods of quiescence and activity. Examples are provided by the following contrasts:
between turbulent flow and its laminar inserts, between error-prone periods in communication and error-free periods, and between periods of orderly and agitated ("quiet" and "turbulent") Stock Market activity. Such subdivisions must be natural to human thinking, since they
are widely accepted with no obvious mutual consultation. Rene Descartes endorsed them by recommending that every difficulty be decomposed into parts to be handled separately. Such subdivisions were very successful in the past, but this does not guarantee their continuing success. Past investigations only tackled variability and randomness that are mild, hence, local. In every field where variability / randomness is wild, my view is that such subdivisions are powerless. They can only hide the important facts, and cannot provide understanding. My alternative is to move to the above-mentioned apparatus centered on scaling.
-Mandelbrot, in the foreward to Multifractals and 1/f Noise.
it's worth saying that Mandelbrot was apparently a large influence to E Fama, who proposed the efficient market hypothesis in the first place.
That's very cool, congrats and thank you to the Facebook guys!
A few days ago I was asked to do some forecasting with a daily revenue series for a client. Due to her business' nature the series was really tricky with weekdays and months/semesters having some specific effects on the data. I as many use Hyndman's forecast package, but I threw this data at prophet and it delivered a nice plot with the (correct) overall trend and seasonalities. Very cool and easy to do something.
Statsmodels is a grab-bag of various statistical models from linear regression upwards. This is an opinionated library for (some relevant parts of) econometrics.
FIS Prophet is targeted at actuaries, and really no-one else so I don't know if anyone will care. They have had the name a lot longer than Facebook though!
This would be non-trivial. Consider this paper on marijuana usage where the researchers had to group statistics by adjacent counties in Oregon and Washington in order to control the tests.
Worth noting Prophet is R/Python wrappers to some models with reasonable defaults, written in and fit by Stan, a probabilistic programming language, and Bayesian estimation framework.
Stan is amazing in that you can fit pretty much any model you can describe in an equation (given enough time and compute, of course)!
You are right. The docs have been due to be written for many months now, and that is the main reason the library has not been released yet. On the other hand, the test folder contains many tests, among them full examples from many chapters from the book Doing Bayesian Dara Analysis, recommended above.
I'm pretty interested in this as I do most of my work on the JVM and I love trying this out on our stream processor at work.
Cloned and tried to build it but I'm getting an error regarding uncomplicate:commons:0.3.0-SNAPSHOT being unavailable on clojars. Is that something you currently have installed to your local maven repo? I don't see it here: https://clojars.org/repo/uncomplicate/commons/
I can get it to build with 0.2.2 but it is missing the "releaseable?" function.
In any case this looks awesome and I'll be keeping an eye on it / playing with it for some new projects.
EDIT: I was able to get it building by cloning your commons library and running "lein install". :)
Do you know of any good beginner tutorials for Stan or probabilistic programming in general? All the examples that I found seemed quite complex and I was a bit overwhelmed by all the math. Which might also be a sign that I should brush up my math skills. What kind of math/stats should I revise to be able to better understand probabilistic programming?
That's the foundation. The way you set up your model is by nodes and edges that specify the flow of influence (directed or undirected). Then it seems that there are general methods for inference and learning on any kind of graph one might pose.
For simple graphs (and simple is something one might want when modelling) the methods should be fairly effective.
Unfortunately, the biggest book on the subject that I know (Koller & Friedman) isn't accessible. Koller's course is also not that accessible.
Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan. It is very approachable and also has lots of practice problems. It's not a math-heavy book at all.
Edit: I wouldn't recommend Probabilistic Programming and Bayesian Methods for Hackers. When I tried using it, I felt that too much was glossed over. The book that I recommend excels at conveying a strong intuition for how these various techniques work.
Probabilistic Programming & Bayesian Methods for Hackers [1] by Cameron Davidson-Pilon is exactly what you want, starting from a computational-first perspective, then introducing the maths later, although it uses PyMC rather than Stan. It's freely available as a set of Jupyter notebooks, as well as a printed edition.
Interesting definition of "scale" in this context, as it does not imply "big data" like every other usage of the word scale in data science. The tool works on, and is optimized, for day-to-day, mundane data.
Thanks, that's a good resource. I'm surprised though. It seems that Top-1000 articles by monthly views are 90% about celebrities and movies. I think tags or categories would be most useful.
We at https://yoghurt.io/ have been working towards similar forecasting solution. So far the feedback has been that automated solutions can also bring good results at a far lesser cost compared to hiring an expert analyst.
Its a completely managed solution. No need to setup anything yourself.Just upload the data and predict next week's data, today itself. There is a free trial and if anyone here is looking for an extended trial, they can reach out to me.
Example: Like you want to predict the app downloads of your website coming week. Just upload the data in time series format against the date and app downloads from last 30 weeks. It will return the next 7 days predicted app downloads along with the analytical confidence.
It can predict any KPI like visitors, app downloads, conversion etc. Just signup and start predicting.
Can you please try uploading XLS or XLSX format.
Normally, it should show error message in this case.We are going to fix it soon.
CSV and other formats support would be coming soon.
You can try https://yoghurt.io/. Its fully managed platform and no need to setup anything yourself.
Example: Like you want to predict the app downloads of your website coming week. Just upload the data in time series format against the date and app downloads from last 30 weeks. It will return the next 7 days predicted app downloads along with the analytical confidence. It can predict any KPI like visitors, app downloads, conversion etc. Just signup and start predicting.
Is it possible for example to send you monthly revenue numbers for my startup for the last two years (24 data points) and have yoghurt predict the next two years of monthly revenue?
If the model is autoregressive you can only forecast N steps ahead. Any further forecasting will be based on these generated near-future forecasts. In English, no. See https://www.youtube.com/watch?v=tJ-O3hk1vRw#t=01h16m
Its very simple to use Yoghurt, just upload the data and rest it does automatically.
24 data points is less to make any accurate prediction. You need more data points.
However, Yoghurt currently supports 1 week prediction only and very soon we will be adding prediction upto 1 Month and plus.
Excuse my ignorance, but how does 1 week fit into the equation? Why does the time scale (x-axis) matter? I.E. if I pass 180 points of revenue (y-axis) does it matter if they were sampled each day or each hour in terms of forecasting?
They probably take into account day-specific trends , such as if the data shows sales are usually lower on a Monday than a Tuesday, they would take that into account in the forecast. This is as far as I understand.
So, assuming they are doing this, the time scale does matter. What I am trying to say is that these solutions (like prophet) are opinionated and that is why they can get accurate, as they are taking into account these time-scale specific trends.
But being opinionated means that they are assuming stuff about your data. For example saying that the number of sales you make in a day is a function of or correlated to the day of the week is probably a reasonable statement. However if you move away from sales and marketing, and try to forecast say the number of seismic events in a day, nature doesn't care if it's a Monday or Tuesday or holiday. So any such correlation the program is able to find out and use in forecasting would be incorrect. Like maybe there are more earthquakes on Monday than any other day in a particular dataset, but that would just be incidental and doesn't mean earthquakes are more likely to occur in future on Mondays. It's not a good example but there could be other such cases where such assumptions could be wrong.
Yes it would matter. Our algorithm(SandDune)is built around measuring data on a daily basis at this stage. It takes daily input data and predicts the next week's data on a daily basis.
If you give it 180 daily data points, it will predict next 7 data points.
I'm curious...are you worried about this release? Seems like all I'd need to compete with you (vastly simplified but for arguments sake) is hack together a simple webpage with a submit button that uses Prophet. Assuming both models yield reasonably useful results (obviously you could compete on accuracy or ease of use where you're currently ahead for business-y customers).
My guess would be yes. I'm thinking this could be used to find out how effective a particular marketing campaign was. Just compare the forecast with actuals and the difference would be the number of sales/clicks you got from that campaign.
>Could this be re-purposed for detecting anomalies/outliers in time series data?
If you define anomaly as something unexpected then yes. In this case, if the reality differs significantly from the forecast (=expectation) then it is an anomaly (according to our definition). In numeric univariate case, there could be positive anomalies where you get more than expected, and negative anomaly where you get less than expected.
It seems that they have developed a model for only univariate forecasts and only numeric regular time series which is a classical use case in statistics. Yet, most data sources have many dimensions (for example, energy consumption, temperature, humidity etc.) as well as categorical data like current state (On, Off). The situation is even more difficult if the data is not a regular time series but is more like asynchronous event stream. It would be interesting to find a good forecasting model for some of these use cases. In particular, it is interesting if this Prophet model can be generalized and applied to multivariate data.
> most data sources have many dimensions (for example, energy consumption, temperature, humidity etc.) as well as categorical data like current state (On, Off). The situation is even more difficult if the data is not a regular time series but is more like asynchronous event stream. It would be interesting to find a good forecasting model for some of these use cases.
I'm guessing you already know about this based on the way you described the situation, but the Hyndman Forecasting book [1] discusses various models at length for doing multivariate forecasting models. It's loaded with code and samples in R.
We're planning to add forecasting to our SaaS analytics product (https://chartmogul.com) later this year, I'm going to look and see if we can use this in our product now.
I was trying to sort out whether adding this to an existing charting/analytics product makes sense but it looks like you've checked it out and think it does. I couldn't tell only because it seems to be built to do the charting/plotting itself, but I guess you can just use the data/API to get the forecasts then plot them yourself yes?
I may do a test implementation into Airbnb Superset actually to see how it flies.
I have been working for a few years on a similar project using evolutionary algorithms on top of other models (linear / ann). It works quite well (e.g., for equidistant energy demand / supply forecasts) but there's still lots of stuff to do.
It's major benefit is that it figures out relationship to the target time series by itself, so you can just throw in all time series and see what comes out.
Language is Clojure, 20kloc, incanter, encog. If anyone is interested in working for/with it, let me know. I currently develop a Rest Api for it and plan to release it as open source once the major code smells are dealt with.
I'd like to have a tested use case that mostly and simply works. Something to put in readme.md that shows how it works and that it works. Almost there...
/please ignore:
Oracle & Prophet. Oracle sifts through signs but Prophet has a line to the larger picture. I suppose the next 'product' will be called Messiah to complete the picture.
113 comments
[ 2.7 ms ] story [ 228 ms ] threadVery cool though --- I would be interested to dive into the methods they've implemented sometime in the near future!
That assumes that the efficient-market hypothesis holds true, but it has yet to be thoroughly proven or disproven... (and funds like Medallion would strongly suggest otherwise for the medium term: https://www.bloomberg.com/news/articles/2016-11-21/how-renai...)
That's not to say returns are actually random, but in any given time range, it appears to be.
Or are you saying that movements aren't actually random, and only appear to be?
That doesn't sound right. Let me clear that up for you. Since 1950:
Buying the straight S&P 500 beats inflation by seven percent, on average, every year. You're welcome!One sample tells you nothing about randomness. What if you buy in August 1929? What if you hold for a more realistic 20 or 30 years from peak earning years to retirement?
If you're interested, here's a simulator that looks at historic market data. You'll note that even the lowest possible percentile of 30-year holding periods will still yield a 3.43% inflation-adjusted total return: https://dqydj.com/sp-500-historical-return-calculator-popout...
If not, look at http://www.macrotrends.net/1319/dow-jones-100-year-historica...
Let's buy in August 1929 at 5338.69, and sell 20 years later, in August 1949, at 1822.87 (inflation-adjusted). Congratulations, you lost two thirds of your money.
Sell 30 years later instead? August 1959, at 5525.23. Wow, after 30 years you're up almost 3.5%!
Probably quite poorly (due to stocks appearing "random" at scale), especially for indexes, which are a sum of their parts.
On the other hand, this would probably be quite useful for things that have non-random trends (like the Global Energy Forecasting Competition: http://www.drhongtao.com/gefcom)
-Mandelbrot, in the foreward to Multifractals and 1/f Noise.
it's worth saying that Mandelbrot was apparently a large influence to E Fama, who proposed the efficient market hypothesis in the first place.
A few days ago I was asked to do some forecasting with a daily revenue series for a client. Due to her business' nature the series was really tricky with weekdays and months/semesters having some specific effects on the data. I as many use Hyndman's forecast package, but I threw this data at prophet and it delivered a nice plot with the (correct) overall trend and seasonalities. Very cool and easy to do something.
https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2841267
All my attempts thus far have pointed me to something called Gaussian Proceeses that I am still working through grokking.
Stan is amazing in that you can fit pretty much any model you can describe in an equation (given enough time and compute, of course)!
More on Stan here: http://mc-stan.org/
http://github.com/uncomplicate/bayadera
And I wait with bated breath.
Cloned and tried to build it but I'm getting an error regarding uncomplicate:commons:0.3.0-SNAPSHOT being unavailable on clojars. Is that something you currently have installed to your local maven repo? I don't see it here: https://clojars.org/repo/uncomplicate/commons/
I can get it to build with 0.2.2 but it is missing the "releaseable?" function.
In any case this looks awesome and I'll be keeping an eye on it / playing with it for some new projects.
EDIT: I was able to get it building by cloning your commons library and running "lein install". :)
That's the foundation. The way you set up your model is by nodes and edges that specify the flow of influence (directed or undirected). Then it seems that there are general methods for inference and learning on any kind of graph one might pose.
For simple graphs (and simple is something one might want when modelling) the methods should be fairly effective.
Unfortunately, the biggest book on the subject that I know (Koller & Friedman) isn't accessible. Koller's course is also not that accessible.
Edit: I wouldn't recommend Probabilistic Programming and Bayesian Methods for Hackers. When I tried using it, I felt that too much was glossed over. The book that I recommend excels at conveying a strong intuition for how these various techniques work.
Free is hard to beat, especially for someone just testing the waters.
[1] http://camdavidsonpilon.github.io/Probabilistic-Programming-...
I am puzzled how they managed to release Prophet under BSD with such a dependency.
See also the R vignette, which shows that the data is returned per-column which gives it a lot of flexibility if you only want certain values: https://cran.r-project.org/web/packages/prophet/vignettes/qu...
The wikipediatrend R package relies on http://stats.grok.se/, which in turn relies on https://dumps.wikimedia.org/other/pagecounts-raw/ which has been deprecated.
The new dump is located at https://dumps.wikimedia.org/other/pageviews/
Data is available in hourly intervals.
* pageviews-20170227-050000
[edit] There is a wikipedia-hosted OSS viewer for these logs, e.g. Swedish crime stats:https://tools.wmflabs.org/pageviews/?project=en.wikipedia.or...
https://tools.wmflabs.org/pageviews/?project=en.wikipedia.or...
What's up with Java? (Set "logarithmic scale" to improve the visualization)
An intro by Felipe Hoffa (Google): https://www.reddit.com/r/bigquery/comments/3dg9le/analyzing_...
https://wikitech.wikimedia.org/wiki/Analytics/PageviewAPI
https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.w...
Its a completely managed solution. No need to setup anything yourself.Just upload the data and predict next week's data, today itself. There is a free trial and if anyone here is looking for an extended trial, they can reach out to me.
Between this and Stan I think my free time for the next week is gone.
So, assuming they are doing this, the time scale does matter. What I am trying to say is that these solutions (like prophet) are opinionated and that is why they can get accurate, as they are taking into account these time-scale specific trends.
But being opinionated means that they are assuming stuff about your data. For example saying that the number of sales you make in a day is a function of or correlated to the day of the week is probably a reasonable statement. However if you move away from sales and marketing, and try to forecast say the number of seismic events in a day, nature doesn't care if it's a Monday or Tuesday or holiday. So any such correlation the program is able to find out and use in forecasting would be incorrect. Like maybe there are more earthquakes on Monday than any other day in a particular dataset, but that would just be incidental and doesn't mean earthquakes are more likely to occur in future on Mondays. It's not a good example but there could be other such cases where such assumptions could be wrong.
I've been using CasualImpact by Google [0] for months. This seems pretty straightforward.
[0] https://google.github.io/CausalImpact/CausalImpact.html
If you define anomaly as something unexpected then yes. In this case, if the reality differs significantly from the forecast (=expectation) then it is an anomaly (according to our definition). In numeric univariate case, there could be positive anomalies where you get more than expected, and negative anomaly where you get less than expected.
I'm guessing you already know about this based on the way you described the situation, but the Hyndman Forecasting book [1] discusses various models at length for doing multivariate forecasting models. It's loaded with code and samples in R.
1. https://www.otexts.org/fpp
We're planning to add forecasting to our SaaS analytics product (https://chartmogul.com) later this year, I'm going to look and see if we can use this in our product now.
I may do a test implementation into Airbnb Superset actually to see how it flies.
It's major benefit is that it figures out relationship to the target time series by itself, so you can just throw in all time series and see what comes out.
Language is Clojure, 20kloc, incanter, encog. If anyone is interested in working for/with it, let me know. I currently develop a Rest Api for it and plan to release it as open source once the major code smells are dealt with.
> df['y'] = np.log(df['y'])
https://gist.github.com/llimllib/385230f38c3f9b70c3e46158e60...