It doesn't specifically say as far as I can see, but I'm highly doubtful that the prediction was made at time 0 (i.e. with no seed data). That would literally be a lucky guess.
Many data science thought pieces don't quantify the results, and it's suspicious that they're not included in detail (typically, the R^2 of a simple stock market prediction model is super low, making it impractical putting actual money on the line as it's barely better than guessing randomly: https://twitter.com/minimaxir/status/1021885939361042432 )
In the stock market something that is consistently better than guessing randomly is very valuable! You don't need much of an edge to make a lot of money.
(I'm very skeptical about approaches like the OP though)
I recommend doing something similar to the original post. Neural networks tend to produce valid looking output for a stock price easily, a random walk does too. You have to find correlations, causations, and then review the results carefully.
For reference, I wrote my own financial advisor (not directly utilizing deep learning, that functions relatively well):
These types of experiments pop up from time to time. You have to compare the performance relative to the single lag error. So the baseline is to use the previous time step to predict the current time step. Ultimately, it's not worthwhile to attempt to predict stock timeseries the SNR is far too low. It's more valuable to attempt to predict a trend, e.g., +1 is there will be upward movement in time window x, -1 if downward, 0 is within some epsilon of the current price.
I preliminarily did the same thing as in this article a while back (with crypto data, since it's easily accessible), just as a way to learn keras.
Somehow my network always learned to output a delay of the input no matter how hard I tried to shape it. I've searched through literature briefly and some examples. Some blog posts even claimed they had good time series prediction when I clearly saw they were having the same problem.
What I mean to say is, be critical of what you find online, and be critical of your results. This is definitely not an easy problem, and some argue that it's unsolvable all together :)
You are not alone. With time series that is the easiest solution for any model. That is why you need a lot of work creating your datasets and the experiment setup with time series. To avoid the easy solution and obtain something useful.
Nobody's gonna do much better than lagging the market price using a model based purely on historical price data, markets aren't nearly that predictable.
I'm skeptical of any system that proves it can predict stock prices vaguely.
I would be less skeptical if it was more humble, and specific in its constraints. For example, could it predict the % probability a stock would up X% if it had N consecutive negative days? Or, how many minutes/hours after earnings are announced is the after-hours price usually indicative of what the price will be the next day?
You can't predict everything, but you might be able to have a probabilistic prediction of specific scenarios. though even then I'll still be skeptical.. daily stock movements are totally random.
I wonder if someone could comment on how AI is used at big investment banks? I assume its much more than technical analysis (like this model). I imagine that NLP might be helpful for quickly ingesting a news feed and then making buy/sell decisions based on the news in less time than humans can read and react?
As you mentioned, one way is to quickly ingest earnings announcements from press wires/company websites. As soon as the earnings report hits the wire, they want to immediately known how big a miss/beat was it, and whether they should immediately dump it or buy more.
Other examples include analyzing the words/sentiments in earning calls, to see if certain words indicate a bearish or bullish signal. For example if a CEO mentions the word "headwinds", does that historically lead to a much better quarter next time?
Some other companies do analysis with public/private data. For example, you can use a weather dataset to calculate the average temperature for all cities where there are restaurants owned by Cheesecake Factory in a quarter, and see if it will negatively or positively impact sales.
I don't think you need AI for that, though. There are news feeds that are specifically designed to be machine readable. All you need to do is parse it and compare the actual numbers to the expected ones, then calculate the delta for your pricing model. All of this is pretty simple. The real challenge is being faster with this than the competition.
I saw some guys present who did this with currency trading. They had to retrain the model every night and it didn't trade for the first four hours after the market opened.
Signal/noise on the market is extremely low, and determining whether you’re predicting something or just data snooping your way to a function that looks like it predicts something is a serious competitive advantage in itself (... because all you need then is novel functions to try).
Working out a function that passes a split test is inevitable and easy tbh. That function then making money is highly unlikely. This is especially true if your data sources are public.
Well, I'm only on mobile. But the problem with stock prices is often that you don't have good data, unless you are working for the big four or whatever. And even then, they are apparently still working on integrating external data in real time.
Imagine having the whole Twitter stream, maybe categorized by some network in an automatic way.
Regardless of whether or not this would make anyone money, it's a really nice introduction to forecasting time series using LSTMs. Thanks for the post!
11:15, restate my assumptions: 1. Mathematics is the language of nature. 2. Everything around us can be represented and understood through numbers. 3. If you graph these numbers, patterns emerge. Therefore: There are patterns everywhere in nature.
> Everything around us can be represented and understood through numbers.
One of the biggest problems facing someone trying to assemble a numerical model of some thing or process is whether or not their model is adequate to describe the phenomena they are observing.
This is why psychology remains a statistically modeled science, largely relegated to anecdotal research.
(Note that my intent is not to speak ill of psychology and its cohort, only illustrate that just because you can slap numbers on a thing doesn't mean you've described it well.
Look at the transition of understanding from Newtonian mechanics to relativity and quantum mechanics. Newtonian mechanics was enough until someone saw Mercury was doing weird things it shouldn't.
And before Newtonian mechanics we had a broken description of the solar system with more exceptions than rules. We were throwing numbers at a system and failing, in some part, to describe it with consistency.)
Total noob question in this space (algotrading with DL algorithms).
What DL is good at, is to automate tasks that are easy for humans (e.g. telling cat from dog, understand a sentence from sound wave, translation, etc) but hard for machines without DL.
Now, if a task is even hard/borderline impossible for humans to achieve (e.g. predicting stock market with higher accuracy and consistency), why would we believe DL could do a better job than humans?
DL isn’t just good at automating human tasks (see AlphaZero and AlphaGo).
It can be used to recognize patterns and train to solve problems better than humans. Though in this case since the market is people making predictions about a prediction I’m not sure how much it’d help.
There’s probably something about humans using a rough estimate of what the stock has been historically to inform where it could go though so it’s value is probably non-zero.
By DL, i meant generic DL techniques like CNN, LSTM (used in this article), which relies on large amount of labeled data to train, and predict on similar data, thus my comment about automation. Alpha Go (Zero) is very specialized for the game of go, not sure how much of its specialized algorithm could transfer to other generic use cases.
Good to know it generalized, but seems it's only generalized on board games like problems where you have a problem space to search through, is that understanding correct? If so, it probably won't help in the use cases we are talking about here, right?
Yeah it's not an AGI, but I think the response was more that DL isn't necessarily limited to automating tasks that are easy for humans but hard for machines without DL (which I think was your original statement).
DL could potentially do a better job by recognizing patterns in the dataset that lead towards winning (making more money) that humans might miss. Like how AlphaZero can recognize moves in Go or Chess that humans don't understand are the best moves to make.
It's not obvious to me how this would be done, but I think it's plausible that some clever implementation could help.
I didn't mean it's a requirement, but a fact that what ever DL could do is based on its input (labeled data), and that input has to come from human, or the task underneath is possible for human.
I think relating DL with human skills is confusing you. DL already beats humans in recognizing dog breeds. DL could always do a better job, because it looks at more patterns than humanly possible.
The reason I relate human skill with DL, is because for tasks where there are real patterns that humans can consistently recognize, DL is applicable, because it relies on human labeled inputs. For patterns, even humans can't consistently recognize (or if there are patterns at all), there's no way for DL to be better than humans, simply because the inputs from human can't be trusted. The example of DL does better job at e.g. recognizing dog breeds, recognizing cancers, whatever, is because those jobs are possible for humans to begin with. The only difference there would be DL could work on more human labeled data than any real human, so that they can recognize more patterns. But if those human labeled data/patterns dont exist in the first place, why we would believe DL could do a better job?
I'm not a purist believer in the efficient market hypothesis. BUT, I doubt there's much alpha to be gained simply from looking at price data which is widely and publicly available. Also keep in mind that markets are dynamic feedback loops so even if this model had an edge, the act of publishing this article would work against you to neutralize that edge in the future.
There's a good reason the most successful 'quantitative' trading fund, Rennaisance Technologies is so secretive and subjects its employees to a lifetime NDA/non-compete.
If there is edge in the market (and I think there are anomalies that can be traded on), I would assume it to come from correlating proprietary datasets with price. Or datasets that have a high barrier to access/analyze. One example that I'd bet still works in some industries would be counting trucks from a supplier to estimate product demand before the companies announces earnings ;)
Rennaisance is one large quantitative trading firm, there are other bigger and more successful ones however. I happen to work for another very large and successful quant trading firm.
Most of them aren't hedge funds, they're private (HFT, Electronic Trading, Botique Firms, Proprietary Trading are all terms you'll hear) so they don't have public data. It is literally impossible to prove without insider information.
However the Virtu Financials, the Citadels, etc, will always exist, and will be doing exceptionally well whether people realize it or not.
Yeah, a better name for these kinds of funds is "independent research groups." Edgestream is an example of another. Renaissance is the most (in)famous one, but many others exist which profitably manage tens to low hundreds of billions while employing only 10 - 100 people.
I realize this doesn't actually matter and I feel weird defending Renaissance... but Virtu is a public company and hasn't been doing that well, and returns from Citadel's various funds are not hard to find. Also, I don't know how to compare a fund's returns to that of a private business. But Medallion has been around since before HFT was really a thing, and I'm not aware of any HFT places that have been growing at 70% a year for 20 years.
These are wildly different businesses. Virtu is a global electronic market maker which in reality is more like a mature technology company than a risk taking hedge fund. Renaissance (Medallion specifically) is a weird fund that manages employee money. Citadel is a large, diversified financial services company that happens to have overlap with both of the previously mentioned firms. They have a large market making operation and offer a variety of different hedge funds that they market
It's not unheard of for market makers to have returns well above 100%, and one you get into certain latency arbitrage strategies the returns can grow significantly.
The issue is that these business are capital limited, so you can't reinvest your massive returns to compound them.
These predictions can also be self-reenforcing. If enough money believes the model is accurate, it will create it's own market conditions. In a distopian AI stock prediction world the best model will be the one with the widest publicity and adoption, not the best data points.
The problem I see with that hypothesis is that the asset pricing must eventually be tied to actual performance--while it may be self-reinforcing to some extend, if it _is_ fundamentally wrong, there will be a reckoning and an adjustment after a high value stock goes bankrupt, for example.
But the stock market isn’t just a bunch of people placing speculative bets about company performance; it’s a bunch of people placing speculative bets about company performance by buying shares of the companies. If the market is irrational, it can actually prop up companies that would otherwise go bankrupt.
I suppose they company could continuously issue stock in this case.
But in this world, it actually makes sense for those companies to stop doing their normal business and just go into the business of selling their shares.
This reality sounds absurd, but you could argue that BTC market is there. Enough of the market thinks that "always buy" is a good investment, regardless of the real-life value of the asset. Or maybe, less controversially, gold is that market. Any asset that is always increasing in value and not related to the real value of that asset is just a store of value.
Both of those things have valid services that I pay for.
BTC provides a number of services. From money changing and international money transfers to actual investment brokerage. Granted, the number of securities available in BTC is less than spectacular, but it's not zero. I pay for both those services. Now you may argue that those are unregulated services and therefore have trust issues, but one might argue that all markets have trust issues, and the only difference is the level. BTC, so far, seems to be more trustworthy than, for instance, the ECB (e.g. the Greek payment limits and the Cypriot bail in, one of which affected me, and both of them used MY money to achieve political aims, without my approval).
Gold provides a store of value, with a good story behind it. I pay my bank, I believe, around $40 per year for that same service. With frankly, not as good a story behind it (as I trust my bank less than I'd trust a bar of gold under my pillow when it comes to still having value tomorrow. Not that I have the kind of spare change to make that a pressing issue, but ...)
So given that both BTC and Gold provide services that clearly people are willing to pay for, who's to say they shouldn't have a valuation based on that income like every other financial service provider in the world ?
Generally when talking about price movements the opposite is true. That is a prediction that the market will move at some point in the future will cause that move to happen right now instead, but only if that prediction is believed.
And if you have a model that people believe which predicts the movement of stocks but doesn't have any good reason for those predictions then it may very well move the price of those stocks in the ways that it predicts. But in doing so the money of the people who believe its predictions will be transferred to the people who don't until it stops having an effect.
If enough people believe a given model and use it, it would actually tend to make the model over and under-predict. If a lot of people think the market will go up, it's race to buy before the others and eventually a race to sell before the others when either expectation shift or all the buyers run out.
Which is say, belief in a direction can make the direction happen but belief in a particularly shaped curve won't make the market resemble that curve.
I don't know anything about the finance industry, but it seems to me that there are a number of industries where the non-compete term is irrelevant, as you can still become persona non grata. For example Companies B, C, and D know that if they hire someone from Company A, they will get sued. The merits are irrelevant if Company A can outspend them relative to the value of the prospective employee.
My understanding is that legally, the contract says something reasonable, like a year. But the firm will use it's considerable resources to impede your career if you attempt to work in mathematical trading after that year's up.
On the flip side, employees are compensated very well, and the turnover is very low.
I'd assume turn over is low if by leaving they now have to change entire career paths. I'd put up with a lot of shit to not have to learn a brand new trade.
Well, being an employee gives you exclusive access to the best fund in existence by far (35% return every year for decades). Nobody is going to want to kill their golden goose.
Rentech and the other more secretive top quant firms such as TGS and PDT use nearly exclusively public data.
Funds like Two Sigma that haven't had as good (or scalable) returns are actually the ones that focus on novel data sources.
In reality, everyone in this industry is drowning in data, and the real edge comes from learning how to more efficiently parse and analyze data rather than acquiring more of it. The top performing firms work at a level where the data is merely an abstraction; above-market returns can be achieved from nothing but public data sources, if you've automated the process of extracting signals.
These are bold claims with nothing to support them. Given the claimed secrecy, I don't see how you could possibly know all that, unless you are (or have been) working for Rentech, or one of the others. Is that the case?
Funny things I've heard about Two Sigma. Everyone on the outside: Two Sigma is the best. People at the top of the financial industry: Two Sigma is cheesy.
I'm pretty sure that lifetime NDA is a myth. It gets repeated very often (like a lot of the Renaissance mythos) but I haven't seen evidence for it, and on its face it sounds flagrantly unenforceable. It could be that the compensation alone substantially reduces turnover. Likewise once they've hit "their number", employees may decide to go into tech or philanthropy instead of a competing firm.
For what it's worth, Renaissance (remarkably, in my opinion) advertised a role in the most recent Who's Hiring thread.
The NDA/non-compete may not enforceable, but if Renaissance believes employees with valuable IP are being poached, they will sue and make it very expensive to hire, then defend, ex-rentec employees.
They’re totally up front about this requirement, too, when making offers. (Source: I turned one down.) You are to move to Long Island, buy a house, buy into the employees-only fund and work there for the rest of your financial career.
(On the other side of the coin, they offer like a month of paid vacation a year, plus longer sabbaticals every few years, plus tons of benefits...if Long Island and math are your thing, it's a very nice place to work.)
I 100% agree with this as a Daily Fantasy Sports player who has published some blog posts and code related to strategy. Any success I've had has been in sports and formats that are not popular and I purposely do not write or open-source code on. Edge can go away almost immediately; I saw this firsthand in DFS when a number of websites came in with free tools that pros had been using for years.
I play in the stock market a little and my best wins have been on small cap stuff where I've had some edge with unique knowledge of the industry and have taken the time to read SEC filings / keep tabs on earnings reports / closely watch competitors, acquisitions, etc.
To be clear, I still think this article is great from a learning about Keras perspective. That said, to anyone who thinks building some ML models and outperforming the market is easy, remember that edge is only as good as the number of participants in the market who don't have it.
I'm very interested to read some of your blog posts related to strategy/code around DFS, but I don't know what your blog is. Would you mind posting a link to one of these posts?
"A research scientist and senior level employee who worked out of his Pennsylvania home […] His noncompetition agreement prevented Magerman from working for one year after leaving Renaissance for any firm engaged in the business of mathematically-based trading of futures and securities." [1]
Not quite lifetime, unless they expected him to drop dead within a year.
Do not minmax scale the data. New data outside ranges seen before will be chopped off. Do make relative (high/low), so patterns you find generalize to other domains. Don't fit complex LSTM models on tiny datasets. Somehow this is the shiny thing all starters jump on, while the most simple models are the most proven and robust. Don't just share such a model without giving a disclaimer ("I never tried this out on real data, and would not invest my own money in this solution"). This is real money, real economies, and you could horribly crash these. That is also why you don't write with authority about how to build a bridge (while you never crossed a bridge you've build before). The only reason for the "clearly showing how powerful LSTM models are" is it responding to test set samples it got wrong, and correcting with a lag. How about some evaluation, preferably on another stock than trained on?
>This is real money, real economies, and you could horribly crash these. That is also why you don't write with authority about how to build a bridge ...
Setting aside that first bit of hyperbole, I don't see the danger in the bridge example. I mean, if engineers are building bridges based on information they've gleaned from internet articles, there are larger issues at play.
As far as markets, retail traders basically exist and have existed within a massive bubble of misinformation since forever. There's a reason order flow is so valuable.
ML is dangerously overhyped (that is an issue at play). This article uses min max scaling, so it will treat a 30$ stock as if it were 20$. You want someone like that managing your pension fund?
I refer to engineering code, because it is clear to me we don't want our bridges build by those not skilled enough to make safe bridges. Yet, it is not so clear with ML/AI, while the potential for damage may be even greater.
Of which almost all is HFT. And by definition HFT has to be fully automated. I'm not sure if there's any data on the kind of algorithmic trading that the article talks about.
Simple linear regression would work just fine for something rudimentary like this. This using neural nets stuff for everything lately is getting ridiculous.
People in finance are much more advanced in terms of mathematical statistics and numerical methods, they use time series analysis, stochastic calculus, etc.
Keras is a high level framework for lazy people or for prototyping maybe, this is for trying at home.
These predictions and models work until they don't. And when they don't, they blow up in spectacular fashion.
Look at Long Term Capital Management. It had big names behind it and the model/strategy made a lot of money, until I believe the Russian bond crisis caused the markets to act irrationally. Then everything they had invested in quickly went down the toilet, and it was big enough to almost cause a financial crisis.
Is this graph at the end a one-step-ahead prediction? Because if it is, then it looks very much like a 50/50 success/failure ratio. Which is what I would expect, because otherwise you could extract a signal from it.
This doesn't take away from the exercise, but I notice the model seems to "predict" the movement just after it occurs.
It just lags the current price. If there is no alpha in the price signal, then the rational thing to do is to just stay at the current price, which is what the algo seems to have learned to do.
Whereas the author of this piece says the following:
From the plot we can see that the real stock price went up while our model also predicted that the price of the stock will go up. This clearly shows how powerful LSTMs are for analyzing time series and sequential data.
>From the plot we can see that the real stock price went up while our model also predicted that the price of the stock will go up. This clearly shows how powerful LSTMs are for analyzing time series and sequential data.
Yes I've noticed on HN recently ML and data science have become popular topics but I'm surprised a post like this has so many votes.
I made this code https://github.com/mouradmourafiq/tensorflow-lstm-regression at least 6 months earlier than the first push in the github repo you mentioned, and I am sure many people did some version before that. A lot of people sometimes just need a reason to play with some technology. I assume that OP used stock market to learn about LSTMs.
177 comments
[ 2.8 ms ] story [ 306 ms ] threadThat’s not playing catch up, if the prediction was made at time 0.
Many data science thought pieces don't quantify the results, and it's suspicious that they're not included in detail (typically, the R^2 of a simple stock market prediction model is super low, making it impractical putting actual money on the line as it's barely better than guessing randomly: https://twitter.com/minimaxir/status/1021885939361042432 )
In the stock market something that is consistently better than guessing randomly is very valuable! You don't need much of an edge to make a lot of money.
(I'm very skeptical about approaches like the OP though)
https://blog.projectpiglet.com/2018/01/perils-of-backtesting...
I recommend doing something similar to the original post. Neural networks tend to produce valid looking output for a stock price easily, a random walk does too. You have to find correlations, causations, and then review the results carefully.
For reference, I wrote my own financial advisor (not directly utilizing deep learning, that functions relatively well):
http://projectpiglet.com/
It works relatively well when checking the "causality" which also has it's limits: https://blog.projectpiglet.com/2018/01/causality-in-cryptoma...
EDIT: Added some evidence:
* 2016 stocks: https://imgur.com/a/j8YWR
* Early 2018 crypto: https://blog.projectpiglet.com/2018/01/30-weekly-returns-usi...
* Early 2018 stocks: https://twitter.com/AustinGWalters/status/976347632439209985
* All 2018 on Robinhood only: https://imgur.com/a/2CxEFqI
* Also (I am lettergram on HN), created: https://hnprofile.com and https://redditprofile.com
I do not see enough data to backup the claims of how well it works. And if you can't provide that - what good is it as a tool?
Somehow my network always learned to output a delay of the input no matter how hard I tried to shape it. I've searched through literature briefly and some examples. Some blog posts even claimed they had good time series prediction when I clearly saw they were having the same problem.
What I mean to say is, be critical of what you find online, and be critical of your results. This is definitely not an easy problem, and some argue that it's unsolvable all together :)
I would be less skeptical if it was more humble, and specific in its constraints. For example, could it predict the % probability a stock would up X% if it had N consecutive negative days? Or, how many minutes/hours after earnings are announced is the after-hours price usually indicative of what the price will be the next day?
You can't predict everything, but you might be able to have a probabilistic prediction of specific scenarios. though even then I'll still be skeptical.. daily stock movements are totally random.
Other examples include analyzing the words/sentiments in earning calls, to see if certain words indicate a bearish or bullish signal. For example if a CEO mentions the word "headwinds", does that historically lead to a much better quarter next time?
Some other companies do analysis with public/private data. For example, you can use a weather dataset to calculate the average temperature for all cities where there are restaurants owned by Cheesecake Factory in a quarter, and see if it will negatively or positively impact sales.
Working out a function that passes a split test is inevitable and easy tbh. That function then making money is highly unlikely. This is especially true if your data sources are public.
Imagine having the whole Twitter stream, maybe categorized by some network in an automatic way.
Also prequential evaluation.
In addition, major factors are external and trying to predict further price only based on past price is bound to disappoint.
One of the biggest problems facing someone trying to assemble a numerical model of some thing or process is whether or not their model is adequate to describe the phenomena they are observing.
This is why psychology remains a statistically modeled science, largely relegated to anecdotal research.
(Note that my intent is not to speak ill of psychology and its cohort, only illustrate that just because you can slap numbers on a thing doesn't mean you've described it well.
Look at the transition of understanding from Newtonian mechanics to relativity and quantum mechanics. Newtonian mechanics was enough until someone saw Mercury was doing weird things it shouldn't.
And before Newtonian mechanics we had a broken description of the solar system with more exceptions than rules. We were throwing numbers at a system and failing, in some part, to describe it with consistency.)
(For anyone who didn't get the reference, you have an excellent movie & coding soundtrack to catch up on: https://www.youtube.com/watch?v=ShdmErv5jvs)
What DL is good at, is to automate tasks that are easy for humans (e.g. telling cat from dog, understand a sentence from sound wave, translation, etc) but hard for machines without DL.
Now, if a task is even hard/borderline impossible for humans to achieve (e.g. predicting stock market with higher accuracy and consistency), why would we believe DL could do a better job than humans?
It can be used to recognize patterns and train to solve problems better than humans. Though in this case since the market is people making predictions about a prediction I’m not sure how much it’d help.
There’s probably something about humans using a rough estimate of what the stock has been historically to inform where it could go though so it’s value is probably non-zero.
AlphaZero is a generalized successor, and it does just that:
https://en.m.wikipedia.org/wiki/AlphaZero
DL could potentially do a better job by recognizing patterns in the dataset that lead towards winning (making more money) that humans might miss. Like how AlphaZero can recognize moves in Go or Chess that humans don't understand are the best moves to make.
It's not obvious to me how this would be done, but I think it's plausible that some clever implementation could help.
That said, the pitfalls of using ML in trading systems are numerous.
There's a good reason the most successful 'quantitative' trading fund, Rennaisance Technologies is so secretive and subjects its employees to a lifetime NDA/non-compete.
If there is edge in the market (and I think there are anomalies that can be traded on), I would assume it to come from correlating proprietary datasets with price. Or datasets that have a high barrier to access/analyze. One example that I'd bet still works in some industries would be counting trucks from a supplier to estimate product demand before the companies announces earnings ;)
However the Virtu Financials, the Citadels, etc, will always exist, and will be doing exceptionally well whether people realize it or not.
The issue is that these business are capital limited, so you can't reinvest your massive returns to compound them.
But in this world, it actually makes sense for those companies to stop doing their normal business and just go into the business of selling their shares.
This reality sounds absurd, but you could argue that BTC market is there. Enough of the market thinks that "always buy" is a good investment, regardless of the real-life value of the asset. Or maybe, less controversially, gold is that market. Any asset that is always increasing in value and not related to the real value of that asset is just a store of value.
BTC provides a number of services. From money changing and international money transfers to actual investment brokerage. Granted, the number of securities available in BTC is less than spectacular, but it's not zero. I pay for both those services. Now you may argue that those are unregulated services and therefore have trust issues, but one might argue that all markets have trust issues, and the only difference is the level. BTC, so far, seems to be more trustworthy than, for instance, the ECB (e.g. the Greek payment limits and the Cypriot bail in, one of which affected me, and both of them used MY money to achieve political aims, without my approval).
Gold provides a store of value, with a good story behind it. I pay my bank, I believe, around $40 per year for that same service. With frankly, not as good a story behind it (as I trust my bank less than I'd trust a bar of gold under my pillow when it comes to still having value tomorrow. Not that I have the kind of spare change to make that a pressing issue, but ...)
So given that both BTC and Gold provide services that clearly people are willing to pay for, who's to say they shouldn't have a valuation based on that income like every other financial service provider in the world ?
And if you have a model that people believe which predicts the movement of stocks but doesn't have any good reason for those predictions then it may very well move the price of those stocks in the ways that it predicts. But in doing so the money of the people who believe its predictions will be transferred to the people who don't until it stops having an effect.
Which is say, belief in a direction can make the direction happen but belief in a particularly shaped curve won't make the market resemble that curve.
if you were paid
On the flip side, employees are compensated very well, and the turnover is very low.
Most of them are also qualified for general software engineering roles; some of the people with more theoretical backgrounds might not be.
Funds like Two Sigma that haven't had as good (or scalable) returns are actually the ones that focus on novel data sources.
In reality, everyone in this industry is drowning in data, and the real edge comes from learning how to more efficiently parse and analyze data rather than acquiring more of it. The top performing firms work at a level where the data is merely an abstraction; above-market returns can be achieved from nothing but public data sources, if you've automated the process of extracting signals.
For what it's worth, Renaissance (remarkably, in my opinion) advertised a role in the most recent Who's Hiring thread.
https://www.marketwatch.com/story/renaissance-millennium-set...
How do you even enforce that? If I get a job somewhere else, there's no reason for me to disclose where I'm going.
By suing your new employer for $20 million [1].
(On the other side of the coin, they offer like a month of paid vacation a year, plus longer sabbaticals every few years, plus tons of benefits...if Long Island and math are your thing, it's a very nice place to work.)
[1] https://www.marketwatch.com/story/renaissance-millennium-set...
Medallion fund is profit maker but their strategy is obviously very size limited. Their strategy does not scale.
Betting syndicates can make similarly massive ROI, but they are very growth limited.
I play in the stock market a little and my best wins have been on small cap stuff where I've had some edge with unique knowledge of the industry and have taken the time to read SEC filings / keep tabs on earnings reports / closely watch competitors, acquisitions, etc.
To be clear, I still think this article is great from a learning about Keras perspective. That said, to anyone who thinks building some ML models and outperforming the market is easy, remember that edge is only as good as the number of participants in the market who don't have it.
- Pick Em strategies on DraftKings - https://medium.com/draftfast/evaluating-possible-strategies-...
- Thinking in multiples - https://medium.com/draftfast/thinking-in-multiples-7e7c76ee2...
Not quite lifetime, unless they expected him to drop dead within a year.
[1] https://www.forbes.com/sites/nathanvardi/2017/05/08/inside-t...
Setting aside that first bit of hyperbole, I don't see the danger in the bridge example. I mean, if engineers are building bridges based on information they've gleaned from internet articles, there are larger issues at play.
As far as markets, retail traders basically exist and have existed within a massive bubble of misinformation since forever. There's a reason order flow is so valuable.
I refer to engineering code, because it is clear to me we don't want our bridges build by those not skilled enough to make safe bridges. Yet, it is not so clear with ML/AI, while the potential for damage may be even greater.
https://en.m.wikipedia.org/wiki/Financial_Modelers%27_Manife...
Seriously tho', articles like this need to come with a big banner at the top saying warning: do not try this at home.
People in finance are much more advanced in terms of mathematical statistics and numerical methods, they use time series analysis, stochastic calculus, etc.
Keras is a high level framework for lazy people or for prototyping maybe, this is for trying at home.
Look at Long Term Capital Management. It had big names behind it and the model/strategy made a lot of money, until I believe the Russian bond crisis caused the markets to act irrationally. Then everything they had invested in quickly went down the toilet, and it was big enough to almost cause a financial crisis.
It just lags the current price. If there is no alpha in the price signal, then the rational thing to do is to just stay at the current price, which is what the algo seems to have learned to do.
Before clicking I thought "I bet they do stacked LSTM to predict the _next_ time step" :|
-- Yogi Berra
[0] https://en.wikiquote.org/wiki/Niels_Bohr
[1] https://quoteinvestigator.com/2013/10/20/no-predict/
https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Se...
Except Jakob explains the folly of the method.
Whereas the author of this piece says the following:
From the plot we can see that the real stock price went up while our model also predicted that the price of the stock will go up. This clearly shows how powerful LSTMs are for analyzing time series and sequential data.
Yes I've noticed on HN recently ML and data science have become popular topics but I'm surprised a post like this has so many votes.
I am sure plenty of people have done lstm on the stock market but the form of the code looked similar between the two.