I have my hand at this now with http://www.algxchange.com - its really cool, but training is a nightmare. I have 4 situations that required training, and an actual decision model that needed training on what situation model to use.
Having tried to do this sort of thing in the past, I'm fairly skeptical. Maybe if you're trading in large volumes over short spaces of time you can make money with AI, but at least in my simulated runs with small scale investment the results didn't look encouraging.
Also it should be borne in mind that this kind of activity isn't really making any constructive or lasting contribution to society. It's really just moving money around slightly more efficiently than previously. If you really want to change the world, do something else.
Also it should be borne in mind that this kind of activity isn't really making any constructive or lasting contribution to society. It's really just moving money around slightly more efficiently than previously.
I disagree. Efficient allocation of capital is what allowed modern society to advance as far as it did. Capital markets are a tremendous innovation and add tons of value to society. Almost every startup turned major corporation out there was capitalized by the financial markets. You trivialize this value when you say investing is simply "moving money around more efficiently". That money NEEDS to be moved around more efficiently so the most productive members and organizations in society are (a) given the resources to expand their business and (b) rewarded for the value they add to society.
Perhaps, but most of those funding decisions were not made in the public markets. Those capital allocations were made by angels, VCs, and the people who committed to investing in the IPO of the company. The only thing that the overall stock market directly controls for capital allocation is secondary offerings, which are a very small part of the allocation of productive capital. The stock market just serves to trade pieces of capital that already exist. Having the stock market be a bit more accurate, particularly on short timescales, doesn't really affect capital allocation.
He isn't disputing the fact that it has net-positive value creation. He's just saying that it's a tiny lever in terms of impact. You'll immeasurably increase market efficiency, but you'll still get a lot of money because finance is an area with a very high money-to-value ratio.
Yes, increased liquidity has allowed us to more efficiently raise capital, but beware the liquidity dogma.
Increased liquidity has been good in the past, so anything that increases liquidity must be good.
Not necessarily. What are the other effects? How much do we value those? Can we even estimate them a priori?
OTOH, I would argue that most web apps proposed on this site aren't really making any lasting or constructive contribution to society.
The one thing that can be said is that web-startup founders as a whole are less powerful and perhaps less cynically self-serving than financiers, so they distort government policy less.
"isn't really making any constructive or lasting contribution to society"
The common argument is that if you are increasing overall efficiency, that means that capital can be more readily available for real businesses which do make lasting contributions (often in the form of polluting the environment, but that's another question).
It seems your argument is a bit simplistic. Couldn't the same logic be applied to the distribution of goods, middlemen, marketplaces, banking, etc. Are you dismissing all those?
This is true only if we accept that AI trading is actually making the market more efficient. The 'wisdom' of crowds is often foolishness, and this could be even more true for a crowd of computers.
It is very possible that by using AI that looks mostly at numbers for trading is just destabilizing by relying too heavily on specific variables (the ones the algorithms can actually use as inputs) , and reduces efficient allocation of capital.
This is false. You can't make money by being consistently wrong about the "true" value of an asset. Then, the only way you can stay in business is to sell the asset when it's too high, and buy it when it's too low. Notice how doing this actually decreases the amplitude of the oscillations. If you're selling at a high point, you're pushing the price back down. Likewise, if you're buying at a low point, you're pushing the price back up (both times, towards its "true" value).
The reason I use "true" in quotes is because no one really knows the value of an asset. The only real estimate we have is the market estimate. Let me put it another way: if you don't believe markets efficiently allocate capital, then how would you propose we do it? By fiat?
Once again, I want to emphasize that you cannot be wrong about the value of an asset and consistently make money. If these AI algorithms are consistently making money, then they are, more often than not, accurately predicting the value of that asset (where value is defined as the future market price). Of course, my argument hinges on the market price being "right", but like I said, markets are the best way we know how to value things.
I'm not even talking about investors. I'm talking about how to price an apple. Do you think there's a better way to price it than by the free market?
Note that I'm not advocating getting rid of regulation -- I think that's very important, since most businesses don't correctly price in externalities, a la BP and the 2008 crash. To answer your question, though, he point I'm making is that no one knows the true value of an asset. That's why we have markets: to help us price them. Unless we have a better mechanism for pricing assets (anything, not just stocks), we should take the market price as close to fair.
I tend to agree. One of the difficult problems with using AI for market predictions is that the market includes human psychology in the loop, which is not always rational and there are factors which are difficult to include within the model. When people are making buying or selling decisions they're often factoring in imponderables such as gossip, personal prejudices and vague hunches. These hard to simulate decision processes are then greatly amplified by positive feedback - which includes automated trading systems.
Yes, but does it really matter if stocks are priced marginally more accurately? How does that significantly increase the productive output of the economy?
Suppose a pension fund wants to buy 10 million shares of MSFT. Suppose the "real" price of MSFT is $22.00 and the market price is $22.05. That amounts to roughly $500,000 in extra costs for its clients (may be more or less, because the transaction will have price impact and probably rip through a few levels of the bid/ask book). These costs are forwarded to you and me, the customers of the pension fund. Multiply this by the trillions of shares being traded every year, and that adds up to a huge difference.
Yes, but in that case, they're probably buying from another pension fund, or a mutual fund, or someone. If they pay 500k less, that's 500k less the other pension fund doesn't get. No value is created or destroyed by people trading at marginally wrong prices. Furthermore, as long as it's the same people trading with each other at randomly wrong prices, things even out in the wash. These days though, you're dealing with PhDs doing nothing but programming computers to trade, and extracting the money from these mistakes. This makes it not even out in the wash anymore, and they're being paid billions of dollars to solve a problem that never existed. I would much rather have those PhDs figure out how to program a computer to make an electric car that is 5% more efficient, or a wind turbine that is 5% more efficient, or a search engine that is 5% more accurate, or even figure out how to make a physical constant known to 5% greater accuracy. All of those things increase the total value in the world. Instead, they're all being siphoned off by trading companies to do things that make them money, but don't create much if any value.
'Also it should be borne in mind that this kind of activity isn't really making any constructive or lasting contribution to society. It's really just moving money around slightly more efficiently than previously. If you really want to change the world, do something else.'
I agree completely. While have reasonably efficient capital markets is necessary for society, its very hard to show that the marginal improvement offered by this kind of activity justifies the massive amounts of effort that go into it, just because:
1. It's cool.
2. It can make a lot of money.
But unlike making money out of making something people want (e.g. providing value), here you're making money (if you succeed) out of very small market inefficiencies that are probably not hurting anyone, or even just by being 1ms faster then the other guys who were already catching them anyway,
If human beings can’t understand the rationale that an AI-driven hedge fund is using to make its trades, then how can humans tell the difference between an AI that has actually figured out a strategy to beat the market and an AI that is just having a run of good luck?
Isn't the underlying assumption behind all these machine learning models that an incomprehensibly complex system can be modeled with a far simpler approximation of that system? I only have limited experience with ML techniques but my impression was that they're all essentially statistical and the essence of a statistical approach is that you don't really model the underlying mechanics.
I'm not saying that this approach can't work, but just that an insanely complex macro-system like the market is never really going to be reducible to a tractable statistical model.
The complexity of your statistical model is arbitrary. For example agent-based models can consist of hundreds of thousands of heterogenous agents interacting in an economy (fitted to real data). The only requirement for a predictive model is it computes faster than reality.
Sure but that's still a fairly coarse approximation of the actual problem domain in the case of the market, right? You're making an educated guess about how much of that raw information you can ignore. You probably can ignore a lot of it but I wouldn't be surprised if a lot of the serious left-turns in the market in the last century wouldn't have been predicted by the same models that could track day-to-day trends well.
Oh, great. AI will be over-hyped again, just like in the 80's, and we'll have to call it something else for a decade when our families ask us what we're doing lately.
Thinking about it, if an investment fund has a strategy, maybe it should really be possible to put it into code after all. If not, then what is the basis of the investment fund - the gut feeling of it's managers?
You could also still add the gut feeling of the managers as one input variable for the system. Probably even then there would be lots of things left to automate. Just because it is AI doesn't mean it has to do everything by itself.
Yes in principle. Except that those who have a gut feeling for markets and economics usually are not the same as those with technical acumen.
Usually, those with technical acumen find manipulating economic and financial data by hand, and doing so repeatedly, boring. Which, I think, is necessary to gain intuition. On the other hand, those who are willing to do this and perhaps even enjoy studying balance sheets and cash flow statements and product markets spend so much time doing this that they wouldn't have time to learn math and science and AI.
In other words, you will always have either/or guys. Those who understand the algos and those who understand the markets.
I've always wondered if there was a way to make some money in the market by performing lexical analysis of news articles about stocks and gauging the performance of those stocks based on the contents of the article.
So, you'd train your software by analyzing every news article you could find over the past N years for every company in the market you're targeting. It would look for words or phrases that occurred with higher frequency a few days before stocks made significant moves in either direction.
I suppose you'd probably want to weight the source as well, articles sourced from the WSJ might earn a better reputation than ones from the NY Post for example.
At some point there would hopefully be enough confidence in the data that you could test it out with $1000 or something small; news-driven automation... There are enough people who buy and sell on news and opinion pieces that there may be enough fat there to make some profits.
There was actually an interesting study done on quarterly earnings releases that had some interesting implications as far as "soft data" versus hard numbers:
http://ideas.repec.org/p/fip/fedgif/951.html
There was a company "Quant the News" that launched a product that did sentiment analysis on company related news, but their website no longer exists. http://www.crunchbase.com/company/quant-the-news
Market prediction is one of the hardest areas for machine learning/statistics/AI to prove their value. What the system can do is monitor more inputs than a single trader, so the screening process can be scaled up--provided you have relevant inputs. With more signals becoming available electronically it is possible to automatically incorporate non-market prices into trading decisions. All this will require lots of non-ML engineering to supply the data (just like Google search). When monthly-issued government economic statistic stop moving the market you'll know we are there.
The reason for this may be that the market is made up of interacting learning machines (if you subsume humans among them). Thus, every algorithm introduced changes the basis on whcih the algorithm was selected. Talk about moving targets...
I agree, and the word I would use is competing machines. Every time me (or my algo) bid up a security price I remove that "cheapness" signal from a competitor, increasing the noise they have to deal with and vice versa. ML hates noisy data. Returns are very noisy.
ML has a much easier time with data generated from situations that are lacking competition, such as data from any process inside a company, e.g. consumer behavior. These are problems that want to be solved.
it seems people call these techniques machine learning/AI simply because it sounds cooler: a lot of the techniques have a 30+ year history in statistics. this is the same story as neural networks: neural networks are just nested logistic regression, a technique with much history in statistics.
a lot of high frequency trading is simply linear/logistic regression (on the right features of course). anything more complicated is too slow.
39 comments
[ 3.2 ms ] story [ 103 ms ] threadSo many factors, Bear market, Bull market, etc...
Also it should be borne in mind that this kind of activity isn't really making any constructive or lasting contribution to society. It's really just moving money around slightly more efficiently than previously. If you really want to change the world, do something else.
There was an interesting discussion about this on quora: http://www.quora.com/Is-high-frequency-trading-good-for-the-...
I disagree. Efficient allocation of capital is what allowed modern society to advance as far as it did. Capital markets are a tremendous innovation and add tons of value to society. Almost every startup turned major corporation out there was capitalized by the financial markets. You trivialize this value when you say investing is simply "moving money around more efficiently". That money NEEDS to be moved around more efficiently so the most productive members and organizations in society are (a) given the resources to expand their business and (b) rewarded for the value they add to society.
Increased liquidity has been good in the past, so anything that increases liquidity must be good.
Not necessarily. What are the other effects? How much do we value those? Can we even estimate them a priori?
OTOH, I would argue that most web apps proposed on this site aren't really making any lasting or constructive contribution to society.
The one thing that can be said is that web-startup founders as a whole are less powerful and perhaps less cynically self-serving than financiers, so they distort government policy less.
The common argument is that if you are increasing overall efficiency, that means that capital can be more readily available for real businesses which do make lasting contributions (often in the form of polluting the environment, but that's another question).
It seems your argument is a bit simplistic. Couldn't the same logic be applied to the distribution of goods, middlemen, marketplaces, banking, etc. Are you dismissing all those?
It is very possible that by using AI that looks mostly at numbers for trading is just destabilizing by relying too heavily on specific variables (the ones the algorithms can actually use as inputs) , and reduces efficient allocation of capital.
The reason I use "true" in quotes is because no one really knows the value of an asset. The only real estimate we have is the market estimate. Let me put it another way: if you don't believe markets efficiently allocate capital, then how would you propose we do it? By fiat?
Once again, I want to emphasize that you cannot be wrong about the value of an asset and consistently make money. If these AI algorithms are consistently making money, then they are, more often than not, accurately predicting the value of that asset (where value is defined as the future market price). Of course, my argument hinges on the market price being "right", but like I said, markets are the best way we know how to value things.
There are many theories on this, and "what someone else will pay for it" is just one. Investors are not rational!
Note that I'm not advocating getting rid of regulation -- I think that's very important, since most businesses don't correctly price in externalities, a la BP and the 2008 crash. To answer your question, though, he point I'm making is that no one knows the true value of an asset. That's why we have markets: to help us price them. Unless we have a better mechanism for pricing assets (anything, not just stocks), we should take the market price as close to fair.
I agree completely. While have reasonably efficient capital markets is necessary for society, its very hard to show that the marginal improvement offered by this kind of activity justifies the massive amounts of effort that go into it, just because: 1. It's cool. 2. It can make a lot of money.
But unlike making money out of making something people want (e.g. providing value), here you're making money (if you succeed) out of very small market inefficiencies that are probably not hurting anyone, or even just by being 1ms faster then the other guys who were already catching them anyway,
I'm not saying that this approach can't work, but just that an insanely complex macro-system like the market is never really going to be reducible to a tractable statistical model.
1) there will be a renewed interest in the science with more funding leading to more discoveries which could, potentially benefit man. Who knows?
2) it may also be derided much like in the 80's when there were many funds employing the same "fail safe" measures to guard against loss.
You could also still add the gut feeling of the managers as one input variable for the system. Probably even then there would be lots of things left to automate. Just because it is AI doesn't mean it has to do everything by itself.
Usually, those with technical acumen find manipulating economic and financial data by hand, and doing so repeatedly, boring. Which, I think, is necessary to gain intuition. On the other hand, those who are willing to do this and perhaps even enjoy studying balance sheets and cash flow statements and product markets spend so much time doing this that they wouldn't have time to learn math and science and AI.
In other words, you will always have either/or guys. Those who understand the algos and those who understand the markets.
Which, by the way, doesn't keep me from trying ;)
The focus on number crunching seems contrary to advice given by Warren Buffet and other successful investors about investing in what you know.
i.e. "I think the market is under-estimating the impact of Product X, I'll invest in that company" versus "Analyze market cap, yearly revenue...."
So, you'd train your software by analyzing every news article you could find over the past N years for every company in the market you're targeting. It would look for words or phrases that occurred with higher frequency a few days before stocks made significant moves in either direction.
I suppose you'd probably want to weight the source as well, articles sourced from the WSJ might earn a better reputation than ones from the NY Post for example.
At some point there would hopefully be enough confidence in the data that you could test it out with $1000 or something small; news-driven automation... There are enough people who buy and sell on news and opinion pieces that there may be enough fat there to make some profits.
There was a company "Quant the News" that launched a product that did sentiment analysis on company related news, but their website no longer exists. http://www.crunchbase.com/company/quant-the-news
ML has a much easier time with data generated from situations that are lacking competition, such as data from any process inside a company, e.g. consumer behavior. These are problems that want to be solved.
a lot of high frequency trading is simply linear/logistic regression (on the right features of course). anything more complicated is too slow.
Note entries for "large grant" and "nice place to have a meeting."
http://www.paulgraham.com/submarine.html