Can AI Predict the Stock Market? No, But the Attempt Was Interesting (blogspot.com) 87
"We all want to be rich by having a computer just generate piles of money for us," writes long-time Slashdot reader TekBoy. "Here's one man's attempt at using AI to predict the market.
From the article (by tinkerer/writer/network guy Jason Bowling): Models that did great during their initial training and validation runs might do ok during runs on later data, but could also fail spectacularly and burn all the seed money. Half the time the simulation would make money, and half of the time it would go broke. Sometimes it would be just a few percentage points better than a coin toss, and other times it would be far worse. What had happened? It had looked so promising. It finally dawned on me what I had done.
The results cycling around 50% was exactly what you'd expect if the stock price was a random walk. By letting my program hunt through hundreds of stocks to find ones it did well on, it did stumble across some stocks that it happened to predict well for the validation time frame. However, just a few weeks or months later, during a different slice of the random walk, it failed. There was no subtle underlying pattern. The model had simply gotten lucky a few times by sheer chance, and I had cherry picked those instances. It was not repeatable.
Thus, it was driven home -- machine learning is not magic. It can't predict a random sequence, and you have to be very careful of your own biases when training models. Careful validation is critical.
I am sure I will not be the last to fall victim to the call of the old treasure map in the attic, but exercise caution. There are far less random time series to play with if you are looking to learn. Simulate, validate carefully, and be aware of your own biases.
From the article (by tinkerer/writer/network guy Jason Bowling): Models that did great during their initial training and validation runs might do ok during runs on later data, but could also fail spectacularly and burn all the seed money. Half the time the simulation would make money, and half of the time it would go broke. Sometimes it would be just a few percentage points better than a coin toss, and other times it would be far worse. What had happened? It had looked so promising. It finally dawned on me what I had done.
The results cycling around 50% was exactly what you'd expect if the stock price was a random walk. By letting my program hunt through hundreds of stocks to find ones it did well on, it did stumble across some stocks that it happened to predict well for the validation time frame. However, just a few weeks or months later, during a different slice of the random walk, it failed. There was no subtle underlying pattern. The model had simply gotten lucky a few times by sheer chance, and I had cherry picked those instances. It was not repeatable.
Thus, it was driven home -- machine learning is not magic. It can't predict a random sequence, and you have to be very careful of your own biases when training models. Careful validation is critical.
I am sure I will not be the last to fall victim to the call of the old treasure map in the attic, but exercise caution. There are far less random time series to play with if you are looking to learn. Simulate, validate carefully, and be aware of your own biases.
AI? (Score:5, Funny)
Just my 2 cents
Re: (Score:2)
Hey Slashdot... we've got a loser... time to IP ban!
Hey Trump, I can almost write a speech on how many bad attempts to campaign in your name are going on.
Absence of INFORMATION (Score:3, Insightful)
No information == no insight == throwing darts blindfolded == sh!t outta luck.
PS: The best source of high quality information will almost always come from INSIDERS, and trading on those kinds of tips tends to be a felony in the USA [although, curiously, it's not a felony if you're a third party to the information, but lying to the Feds about [wikipedia.org]
Publicly traded companies make money from products (Score:2)
You said "flipping stocks", so that's true if you're talking about day trading and that sort of thing.
Also, companies like General Mills, Johnson & Johnson, and Walmart make money consistently by providing products people want, at prices customers are happy with. You don't have to beat anyone to just choose companies that have good products ans competent management.
Re: (Score:2)
If people were perfectly rational and knowledgeabl (Score:2)
If people were omniscient and perfectly logical, the price would indeed reflect the quality of the product and all of the other relevant factors. They'd all have the same risk-adjusted PE ratio and therefore the same return.
What if I told you that by stock price, the largest homebuilder in the world is one that actually builds and sells less than 1% as many homes as any of the major builders? If the "largest" (by stock price) was number 22 by revenue? And they make no money - they've lost money almost eve
Re: (Score:2)
It's worse than that. This guy isn't the only one using technical methods to look for patterns in the stock prices, do any pattern that shows up will be priced into the stock immediately.
Re: (Score:2)
As far as dollar volume of cars, there are certainly bigger players than Tesla,
but what is their profit margin? It's like Android phones vs. Apple, or
Windows laptops vs. Macs, wherein the rewards go not to the highest volume
producers, but to the ones that can sustain high-margin profits.
Also, Tesla is not just a car company, just as Apple is not just a
cell handset provider.
Positive, instead of negative? (Score:2)
> As far as dollar volume of cars, there are certainly bigger players than Tesla, but what is their profit margin?
Let's have a look at that:
https://wolfstreet.com/wp-cont... [wolfstreet.com]
https://www.toyota-global.com/... [toyota-global.com]
Hmm, one company is consistently making billions in profit, the other losing billions.
I get it you LIKE Elon Musk. He's quite the personality. Reminds of of one of the most famous people of all time - PT Barnum. But to pretend Tesla is the largest and most successful car company of all time is utter
Re: (Score:3)
"Half the time the simulation would make money, and half of the time it would go broke" and they call this AI?
He is just some random stupid guy who thinks that if he fails at something, then nobody can do it.
Meanwhile, Jim Simons [wikipedia.org], the world's richest mathematician, has made over $21B for himself, and much more for his clients, but applying AI to the stock market.
Re:AI? (Score:5, Informative)
If the stock market is truly random, you can safely predict that a few outlier investors will be lucky for a long time.
Re: (Score:2)
If the stock market is truly random, you can safely predict that a few outlier investors will be lucky for a long time.
Not at all. If a coin is randomly flipped, that chance of 10 heads in a row is less than one-in-a-thousand. The chance of 50 heads in a row is less than one in a quadrillion. That isn't going to happen by chance.
That a college math professor was able to make $21 billion (and another $100B for his investors) wasn't because he was lucky. It was because his techniques worked.
Re: (Score:3)
If a coin is randomly flipped, that chance of 10 heads in a row is less than one-in-a-thousand. The chance of 50 heads in a row is less than one in a quadrillion. That isn't going to happen by chance.
Over how many flips?
Remember the Shakespeare-typing monkeys.
Re: (Score:2)
Not at all. If a coin is randomly flipped, that chance of 10 heads in a row is less than one-in-a-thousand.
If you start with a thousand people doing this, one will be left as a winner, and you'll think he's a genius.
It was because his techniques worked.
Or maybe he's very good at getting insider information, or running clever pyramid scheme...
Re: (Score:1)
Re: (Score:2)
It isn't. You just need to find the right data inputs to train your model.
Re: AI? (Score:2)
If the stock market is truly random...
If AI truly controlled the stock market, then its predictions would be spot on.
Re: (Score:2)
Meanwhile, Jim Simons, the world's richest mathematician, has made over $21B for himself, and much more for his clients, but applying AI to the stock market.
According to his interview with Numberphile, he got his start applying algorithms to commodities markets, not equities markets. And those are two very different markets indeed. Judging by the Wikipedia page, his shot at equities isn't going so well either.
Re: (Score:2)
Market is a "player advanatge" game that sometimes goes down, but in recent years seems to be doubling the 10% historical returns... 20-25% on this year's news... I'd like to see how much we make in a good year!
Re: (Score:2)
There's a very good chance that this is the run-up before a major crash. There are lots of warning signs of global recession, and the stock market often jumps to new heights right before it crashes.
Re: (Score:3)
No, in gambling, you win 49% of the time, and lose 51% of the time. The mob doesn't rush to put up casinos to lost money on a crapshoot.
Re: (Score:2)
Ironically, craps has the lowest house odds. So in a way, you're kind of wrong -- the mob does put up casinos and lose money in craps shooting.
But really, you're right, because craps is hard and a lot of people don't know how to play right and make low-odds bets.
Re: AI? (Score:1)
Re: (Score:2)
The odds seem low enough that a skilled person who knows when to quit when they're ahead may wind up on the positive side. There are a fair amount of bets with basically no house advantage.
The advantage to the house in craps seems more from the complexity of the game than a giant advantage in overall odds, meaning that an experienced player may wind up winning occasionally where a tourist would lose consistently.
Stock market is nothing but gambling (Score:2, Funny)
No perfect knowledge. Stock prices are a matter of opinion. Some human opinions, some computers guessing against other computers.
My latest theory is that the insanely low interest rates on safe investments are keeping money in the stock market, thus driving up the prices of securities. The money doesn't actually exist, because it's just a matter of opinion what the stock is worth, and essentially they are borrowing more and more money on the speculation that the prices will continue to go up so that they ca
Re: (Score:3)
Re: (Score:2)
Technical analysis is BS.
Re: (Score:2)
Technical analysis is BS.
That may be, but what other analysis is there? Tea leaves?
Sentiment analysis of humans is still damn near impossible, even in the aggregate. Isaac Asimov's dreams remain unfulfilled. And that's probably a good thing.
Re: (Score:2)
It's called fundamental analysis. Are you sincerely that ignorant? Or just playing up to your handle?
Re: (Score:2)
That's not your theory. One of the basic bits of macroeconomics is that if you lower interest rates you provide more incentive for people to spend money (buying stocks is giving your money to someone else to spend for you) instead of save it. Since interest rates are low, a lot of that money is imaginary money, called debt. That all works wonderfully except that if money is too available people spend it on stupid things. Eventually it catches up, and everything crashes.
What you're supposed t
Re: (Score:2)
I cannot tell from your response if you believe in conventional economic theories or you're just recapping the standard version.
I believe that we should approach things from a time-based perspective, essentially balancing recreation against investment to sustain a reasonable rate of growth. One of the ramifications could be a 3-level VAT, with essentials at the lowest rate, recreation at the highest rate, and investment-related goods and services in between.
Re: (Score:1)
Re: (Score:2)
How on earth is it a surprise that a failed investment strategy is no better than break even? OP is a moron.
Re: (Score:2)
Re: (Score:2)
Dunno, they claim ppl who gamble are intelligent too :)
Re: (Score:1)
AI, stock predicting and blockchain (Score:5, Insightful)
Re: (Score:3)
Yep, market is chaos, not a random coin flip.
Re:AI, stock predicting and blockchain (Score:5, Informative)
Much of the issue with this "study" centers on the author not having a proper machine-learning background.
For instance, letting the model choose, during training, which of the 32,000 features are useful, with what likely amounted to a limited set of data, is a very poor decision. The author could have done either supervised or unsupervised feature selection to cull features. This would have sped up training. It also would have likely made the resulting model a bit more robust, assuming that a proper modeling framework was chosen.
Speaking of models, the author relied on gradient-boosted decision trees for classification. Attempting to characterize a chaotic system, like the stock market, with a non-temporal model like decision trees is going to be rife with disaster. The author should have used more pertinent models like memory-based recurrent networks.
Re:AI, stock predicting and blockchain (Score:4, Insightful)
New business will be unpredictable, subject to C suite decision making. You can't guess what will happen.
The best investment I made was Chipotle at $45 on IPO day. I ate there and knew it wasn't going anywhere but up. Try predicting whether Uber would lose its London license.
You need piles of data, most of which would be internal to the company, to predict one particular business. And then do that for a bunch of companies.
Otherwise you're predicting the general economy, DJIA or NASDAQ. That's a different problem, and requires general news understanding.
The best you can do for a good long time will be marking the overall value as over or under valued. Take the emotion out of impulse buys or sells. I've been watching the biggest losers and buying when they are down. Works if you understand the business, I don't get involved if I'm not a customer of the loser or a competitor. That level of intimacy won't be in AI.
Re: (Score:3)
That and it's very difficult for an ML algorithm to understand how much future growth/margin is already priced into the stock. You don't have to be a rocket scientist to figure out there'll be more EVs sold this year than last year. Does that mean investing in Tesla is a good idea? Whooooooooole other question. Also if you genuinely believe the market is wrong about something it's not easy to predict how long it'll take for the market to realize you were right. Like I remember reading about someone who pred
Re: (Score:1)
Re: AI, stock predicting and blockchain (Score:2)
Re: (Score:2)
"The market -may- still prove to be predictable but requires proper input."
The "proper input" will never be constant. It will have to include all the other AI's and changing algorithms trading, which will continuously keep changing to adjust to rival automated trading.
Re: (Score:2)
The market -may- still prove to be predictable but requires proper input. He did GIGO.
Making money in the stock market isn't really about predictability, at the end of the day. It's about finding other people's mistakes, where something is priced wrong, and exploiting that mistake before the market catches up and corrects it. You make money when everybody else jumps in to make the correction. Once everybody else does figure it out, though, the market behavior has now changed, the mistake no longer exists, and your method for predicting prices doesn't work anymore.
So, the very best you can ho
random based on selected "input" criteria (Score:3)
i do a lot of black-box reverse-engineering and zero-prior-knowledge derivation. when doing pattern-recognition and pattern-searching, if your predictions come up as indistinguishable from random noise, it doesn't mean that you did something wrong with the prediction algorithm: what it means is: *you didn't select the right input correlation sources*.
so, for example, the most obvious thing that you didn't take into account was: *insider trading*. you also didn't take into account stock-market manipulation, and by the sounds of it you probably didn't take into account the timing and release of external factors such as news reports, quarterly reports, twitter and other social media, and so on. these things are much more likely to affect the price than anything else.
what *would* be interesting would be to isolate those stocks which *DO* have correlations that can be predicted. finding those stocks which, if one (or more) change - or do not change - then *others* go up (or down, or stay the same). however i would not expect an AI - not unless it's some seeeerious computational power behind it - to spot these kinds of trends, if they in fact exist. raw "Big Data" analysis might do it - a FULL cross-sectional correlation, not "arbitrarily something an AI *might* spot".
Good products, competent management, risk level (Score:3)
When I'm evaluating the likelihood that company will do well, I'm much more interested in whether they have a good product than the slim chance of insider trading. Companies that consistently make good products, with management that consistently make wise decisions, tens to do well.
Risk is another thing. Johnson & Johnson is toward one end of the spectrum - it's not likely to double in size over the next three years, or fail spectacularly. Johnson & Johnson will probably keep doing what they've be
Re: (Score:2)
Isn't there a whole class of investing called "value investing" where people look at more than just financial report numbers and then buy and hold the stock for some period of time? I think it also involves biasing towards dividend-paying stocks, since dividends allow you to make money other than simply selling based on stock price.
It makes me wonder if you could combine the numerical analysis of not just "stock market" numbers but also market analysis of market segments. It would seem like knowing what c
CNBC Bullseye Show (Score:4, Interesting)
There was a low-rated CNBC show years ago called Bullseye. Their theory was that all stocks walk up at 10% per anum each of the 250 or so trading days per year unless unexpected breaking news interferes. It worked and allowed for some Call The Close (predict tomorrows Dow/S&P values before the open) formulas to hit the jackpot.
Buy/Sell is just plain not a 50% coin toss. If these models are acting that way, they're doing worse than chance. This sounds like anti-market rhetoric and not a serious attempt at predicting the market, but designed to scare kids out of the market... try posting this story at 4pm on a trading day and watch the reaction of market traders who aren't on Slashdot at the moment.
Training (Score:2)
"Experience is like a lantern carried on one's back: it only lights the path behind you, not the path ahead".
Wish I knew who to attribute that quote to... But it's true: the numbers only matter if you understand what they represent. If there's a storm brewing (that you're aware of) which isn't reflected in recent numbers, basing any judgement on those numbers will miss the point. AI or otherwise. Stocks tanking or rising has more causes than the stocks history or 'stock market weather report'.
The headline is extra awful (Score:4, Funny)
Can AI predict the stock market? It didn't work for one guy who immediately says he has no financial background and tried once on a weekend, so let's say no.
Interesting try, but not worth the effort required (Score:2)
Make members of Congress disclose all buys (Score:3)
NextState = PresentState + Input (Score:2)
NextState =PresentState + Input
The reason they failed to predict the NextState is probably the lack of the a workable Input. With the stock market being driven by rumors and "whisper numbers" https://www.investopedia.com/t... [investopedia.com] , it's going to take nothing short of Input magic to get a working state machine.
Yawn (Score:2, Insightful)
This is not AI and it certainly is not ML. Its someone who does not understand the underlying model trying to predict behavior using time series. The result is 100% predictable.
The gall of this guy (Score:2)
To do one experiment and basically declare failure for everyone on the planet.
ML is used a LOT by finance and investing folks. This guy drew a sweeping conclusion based on a sample size of one.
Sorry it didn't work out for him, but he has to have an ego the size of Jupiter to make his stupendously broad claim.
Of COURSE it failed (Score:2)
We’ve still got many millennia to go before Hari Seldon is born.
Doing it wrong (Score:2)
If you want to know the future then you use entangled photons in the double slit experiment with delayed choice quantum eraser.
Re: (Score:2)
Nice try, but it won't work. Explanation here: https://www.youtube.com/watch?... [youtube.com]
Of course there might be ways that do work; quantum physics is weird.
Re: (Score:2)
I've seen it and he made a silly mistake - assuming only one setup would ever be used and then creating a straw man argument saying you can't use that apparatus to send more than one bit of information.
I don't disagree with him. But you can use the equipment to send 1 bit of information back - a one or a zero, a yes or a no. You can collapse the wave with that wave collapse happening earlier in time too.
This is enough for the stock market, the signal is simple - do I buy or do I not buy? If you did this eve
# not all algoritms (Score:2)
This guy wrote something that duplicated random behavior, it does not mean other's can't write something better. Many financial companies have done it successfully.
But yes, humans do have a huge tendency to see patterns where they don't exist.
It's why prejudice exists, it's why conspiracy theories exist, why people see faces and pyramids on Mars, why Astrology exists, etc.
Read The Money Game by Adam Smith (Score:1)
During the 2008 financial crisis I read this book, written by a wall street insider in the 1960s (Adam Smith is a pseudonym).
In it he explains how all of Wall Street's prediction methods are completely fucking useless. He explains how every established method fails the dartboard test - does it outperform stocks chosen at random via dartboard.
Computers were just starting to be a thing, and he hilariously explains how they are being used to quickly and efficiently produce the same wrong answers, with an ele
Re: (Score:2)
Computers have been used increasingly to attempt to predict the market, and over the approximate period of 1995 to 2005 the use of computers changed the way that the market behaves. Some algorithms that would have worked well to predict market and individual stock trends before 1995 have stopped working because computers have taken the profit out before the general public can make good use of the computerized predictions.
There are some techniques that allow you to beat the general market, whether you choose
Comment removed (Score:3)
Not enough inputs (Score:2)
Re: Can an AI predict the stock market? (Score:1)
Wrong focus (Score:1)
Re: (Score:2)
He forgot to include identifying thinly traded stocks and using real money to manipulate them.
IOW the OP is a moron.
Jason Bowling is an absolute moron (Score:3)
Thus, it was driven home -- machine learning is not magic. It can't predict a random sequence
This chode thinks that the stock market is random? A stock's price isn't based on any factors? It's just random? Why is the clown allowed to share his opinion about anything?
Stock movements do not predict themselves. (Score:2)
I saw this article soon after it came out, and AFAICT it mostly proves that past stock movements do not predict future stock movements.
Here's the thing, though. In a gas, past molecular movements do not predict future molecular movements, BUT if you know that someone is heating the container, you can make predictions about the speed of the future movements. If you open one end of the container, you can make some predictions about the general direction of future movements of the molecules. You just cannot
Re: (Score:2)
AInfinite recursion (Score:2)
They never learn (Score:2)
Not a really smart attempt (Score:3)
It's rather futile to day trade with an AI, if all it knows is the movement of the stock market (even from a longer time frame). There's no pattern. Maybe in crypto you might get more volatile and playable market in the weekend when enough 'traders' are drunk enough.
What AI should know are all the macroeconomic indicators, companies' news streams, general news streams related to all of the fields, managers' transactions - history of the stock market and trading reactions to various incidents and so on - all the relevant data of all time; even reddit, facebook and twitter posts and all that. That is the only way to try to "beat the market". It's all about information and the patterns are _there_. Stock market is a derivative of that information.
You could then have the AI play the market at an allowed risk factor and an allowed signal strengths of the future trends and events etc, accounting for the human psychology too...
Random? (Score:1)
Random walk (Score:2)
Or a chaotic system?
For the latter, there are some computational techniques that can identify some attractors [wikipedia.org] in the price state space. But not all. You can't predict price movements out into the future. But you can observe price movements, compare them to a current attractor model and detect when those points move or change due to underlying factors or manipulation of the market.
It's an interesting post (Score:1)
It's seems to me this is just some random persons blog post about their personal experiments on learning and trying this stuff. Interesting to see, even if nothing very advanced or special.
I tried to play a bit with this type of data in a recent Kaggle competition. Using LSTM's, gradient boosted trees, etc. News headlines, social media data, price history, ... It all gets quite complicated. Although I suppose you don't really have to beat the market by more than a small margin and then make a good trading s
Some basic research would have helped. (Score:2)
There were some thorough examinations of stock market trends and the ability to do technical trading.
The maximum time in the future you can make inferences from technical data is about fifteen minutes. So, if your trading horizon is longer than that, there is nothing to analyze. It's just noise. It is also why high frequency trading is the way that trading programs work.
Now, if he had been analyzing other traders and learning from their tactics, that might have been interesting, and I wouldn't be surprised
Only as good as the model (Score:2)
"AI" can only work with the inputs it is provided. It's predictions will be based only on that data. The problem is, the real world is so full of inputs that no one could anticipate, that reliable predictions become impossible.
Not random, has randomness (Score:3)
At the moment, I highly doubt any machine learning model can do a better job observing and understanding the market than even someone with almost no familiarity with it.