Stock Market Predictions with Markov Chains and Python
ฝัง
- เผยแพร่เมื่อ 3 มิ.ย. 2024
- Let’s create a multi-feature binary classification model. This is based on Pranab Gosh excellent post titled 'Customer Conversion Prediction with Markov Chain’ and we'll implement it based on his pseudo code in Python.
MORE:
Blog or code: www.viralml.com/video-content....
Signup for my newsletter and more: www.viralml.com
Connect on Twitter: / amunategui
Check out my book on Amazon "The Little Book of Fundamental Market Indicators"
amzn.to/2DERG3d
Pranab's post:
pkghosh.wordpress.com/2015/07...
Stock data:
finance.yahoo.com/quote/%5EGS...^GSPC
Transcript
Let's run some Stock Market predictions with Markov Chains and Python
I am basing this video from a great post by Pranab Gosh titled 'Customer Conversion Prediction with Markov Chain Classifier'
He lays out his approach using easy to understand pseudo-code so I recommend reading to understand the theory of the approach
He is applying it obviously to customer conversion data but that data isn't as easy to get a stock market data. Also, this is just my interpretation of his pseudo code as there are many ways of slicing and dicing this. But what I like about his approach is that his clever way of doing binary classification with by creating two transition matrices - a positive one and a negative one. Let's dig in.
Markov Chains
A Markov Chain offers a probabilistic way of predicting the likelihood of an event based on prior behavior or prior events. If you look at the drawing of Andrey Markov my son did, we surrounded him with dollar chains, each dollar is an event, and
Welcome to ViralML, my name is Manuel Amuantegui and am the author of Monetizing ML and other books that you can find on Amazon.
First-Order Transition Matrix
A transition matrix is the probability matrix from the Markov Chain. In its simplest form, you read it by choosing the current event on the y-axis and look for the probability of the next event off the x-axis. In the below image from Wikipedia, you see that the highest probability for the next note after A is C#.
In our case, we will analyze each event pair in a sequence and catalog the market behavior. We then tally all the matching moves and create two data sets for volume action, one for up moves and another for down moves. New stock market events are then broken down into sequential pairs and tallied for both positive and negative outcomes - biggest moves win (there is a little more to this in the code, but that’s it in a nutshell).
CATEGORY:DataScience
HASCODE:Predict-Stock-Market-With-Markov-Chains-and-Python.html
SPECIALFRAME:True - วิทยาศาสตร์และเทคโนโลยี
![[TH] 2024 PMSL SEA GF D3 | Summer | มาร่วมเป็นพยาน กับวินาทีแชมป์!](http://i.ytimg.com/vi/JCYvEsFlwx8/mqdefault.jpg)
There are forward biases in two different places in your algorithm. First, you are taking the mean of 'Outcome_Next_Day_Direction' as your label and assigning all features within the sequence ID to that label. The label contains future information. Second, you are using the patterns[id+1] to calculate your log prob. Basically, you need knowledge of the next day's transition to know what is the appropriate log prob... What your algorithm should do is not use the mean, keep each day's label as it is, and then use the patterns[id+2] after a transition from patterns[id] to patterns[id+1] has been observed.
nerd
just started learning Markov chains. complex yet intriguing. Excellent video Manuel!
Awesome video! I did have a question as I wrap my head around this code in terms of the predicted value for the next day. Is there a specific variable that gives the predicted binary outcome for the next day? I see how its contrasted against actual data in the training vs test set but I would like to be able to isolate the next days prediction value by itself as a forward looking indicator.
Hey Manuel, it is a great video but I have a question about the grid matrix, I learnt it with the name called the "Emission" matrix and as far as I know it gives the probabilites of observing that sequence given the state, so each rows represent a marginal probability distribution of the sequences, then shouldn't the rows' summation be equal to 1 ? In this example I see some rows whose sum exceed 1. Am I missing something ?
Brilliant Video and tutorial thanks for that, a topic that seems to be missing is the feature selection part in order to remove the noisy signals out, would you know of any good source in Python to do that?
Best value on the internet. Thank you for the workbook! Btw you can plug your random walk loop into a function and use multiprocessor for a huge speed-up.
That sounds super interesting. Have you tried coding that?
Thanks for the awesome video on an awesome topic. I didn't come here looking for stock specific ideas but rather to learn more about markov chains. After watching the video I am a bit interested in trying neural network embeddings using the vocabulary you generate here.
The problem I am working on is essientally trying to predict when some other model(which I don't have access to) is likely to produce a false positive, true positive, false negative or true negative. The original model has roughly a 69% overall accuracy and is decently calibrated(0.205 brier score) but the false positives and false negatives are an issue. Would this method scale well to more than two outputs(like 4 in my use case)?
Hi Manuel .. at 10:50 i see that you did pd.concat(new_set) .. and then new_set_df.shape you get 1998581, how did you get that number and it seems like you didnt concat the new_set with any other frames. Thanks!!
Great video! That’s a very elegant model. One mistake though, in your transition matrix, you never actually calculate the probability of volume going up/down given that a sequence appears. You instead calculate the probability that the sequence appears, given that the volume goes up/down. Due to bayes rule, the two probabilities are not equal.
You can't exchange integration (prob density) with summation (time-based chain of ups/downs) abitrarily for a stochastic function, only one that's montonic (deterministic), is this close to what you're pointing out? I remember that Bayes has to do the CLT.
I'm extremely late but currently trying to work through a project involving a markov chain model to predict stock action. Is there any easy way to find the probability value of stock action going up or down?
Thank you for this awesome video with detailed explanations. I just learned about Markov Chains this week and this video really helped graps the concepts a bit better.
Glad it helped!!
Hi, amazing work.
Is there any chance of implementing a fractal in your system ?
Many thanks 🙏
Great video. Thank you very much.
Welcome, Fernando!
Very neat. Thank you
8:30 You mention it is slow. You've made a nice learning experience code sample there. Well done. It can be sped up (although this is a basic concept, not optimised), instead of using that For Loop. So, you have an Object Oriented Programming language there. You could create a dynamically linked list of objects, differentiating between pass by address and pass by reference. You could also break the OOP rules for better performance in terms of the private attributes so that they are not hidden from the complier by virtual environments. Array indexing in R programming but considering the nature of the data you're interrogating, I'd anticipate Index matrices are what you'd be after. By then though, for handling Discrete Markov Chains, you'll want to at least consider the R (cran) Markovchain package. You'd even be able to spread the matrices over multiple processor cores and sockets.
My comment has no hate in it and I do no harm. I am not appalled or afraid, boasting or envying or complaining... Just saying. Psalms23: Giving thanks and praise to the Lord and peace and love. Also, I'd say Matthew6.
Are we binning because we are using discrete Markov chains?
Hi Manuel, nice job ... thank you for sharing. Can I ask you what you think about Particle Filters for market prediction? Would be interesting if you do a video on it, since doesn't exist any material on TH-cam about it. Best regards, Daniel.
Thanks Daniel for the kind words and the suggestion - I'll keep that in mind though I am not familiar with this approach. Do you have any links?
Hi!! Really interesting.. so right now I'm working really hard to find an equation with extensions and just one fractal using Markov's method .. also there is a range of market cap and a range of liquidity.. any ideas? Could you help me?
Manuel, I have a question. As I make an attempt to write the code in Mathematica, I am wondering a couple of things: In your example, you end up with two matrices of dimension 5 x 26. Why is that so? I know there are 3^3 = 27 ways to generate different patterns of "L", "M" and "H", allowing repetitions of course, but why do you have only the five rows "HHH", "HHM", "HLH", "HLL" and "HLM"? Are these just the options that begin with H? What happened to "HML"? I guess I still do not understand completely how to generate the Markov matrix...
@Miguel Bayona : I just went through the code and the matrices are actually 26x26 (or 24x24 with the data that Manuel had back when he posted this video). I think your confusion comes from the fact that, in the code, Manuel just prints the .head() of the matrices that, by default only shows the first 5 rows of the dataframe...
@Manuel Amunategui : Thanks for this and many other ineteresting videos you post!!
Why do you use a set of random-walks instead of taking the actual probability of the events in the set and computing the long-run probability by raising the transition matrix to a power?
Thank you for your video, and how can we estimate probability element in the Markov Matrix? Could you give me any hints? Thank you!
The transition probabilities between two states equals the number of sequences from the data set containing the from/to states, divided by the number of sequences in the dataset. I think.
Be careful though, the probabilities do not represent the probability that volume goes up/down given that the sequence appears. They instead represent the probability that the sequence appears given that the volume goes up/down. Due to bayes rule, these values are not equivalent.
The probability for state transition A -> B is the number of transitions A -> B divided by the total number of transitions A -> X where X is every possible state [A, B, ... Z]. In a practical implementation just just count every transition and calculate the probabilities from the counts in an after-pass.
It’s an interesting topic Manual. One bias you are introducing (at line 735) is filtering out all the events which result in small volumes. You say this confuses the model, but perhaps another way of saying it is that including it reveals that the method actually doesn’t have a high predictive power. Suppose we have a series of markov chains and assign a stochastic random variable as the target and filter those chains out which have a target value less than some threshold. We would see fewer chains and must be biased to higher target values. This would appear to improve the model, but it’s actually a bias. I’d like to hear your thoughts on that.
Abs. correct, snivesz32. That's why boosted classifiers like xgboost and catboost work so well, they break down features into groups and will train weaker ones more than stronger ones. Many ways of slicing and dicing this but your thinking this way will help you improve your models.
Best!
www.viralml.com
cant get the code to work? Does anyone have any suggestions?
Really interesting video. I am going to do the code w/your tutorial and I got the book. Looking forward to more videos as soon as you can.
Þþţ
Manuel, what a great video to show the Markov Chain in action. I would love to run the Python code on my machine. I am also very tempted to do this in Mathematica. Where can I get your Python code? I am also going to check out your book "Monetizing Machine Learning"...
Code links are always in the description section - and definitely, go to town with this stuff! That's what I did when I took a stab at it :-)
Manuel, I have a question. As I make an attempt to write the code in Mathematica, I am wondering a couple of things: In your example, you end up with two matrices of dimension 5 x 26. Why is that so? I know there are 3^3 = 27 ways to generate different patterns of "L", "M" and "H", allowing repetitions of course, but why do you have only the five rows "HHH", "HHM", "HLH", "HLL" and "HLM"? Are these just the options that begin with H? What happened to "HML"? I guess I still do not understand completely how to generate the Markov matrix...
@@miguelbayona157 Hey Miguel, check out Pranab Gosh document, it's in the notes hopefully it can answer your questions. This is just my interpretation and its been a while so I don't recall the details. Best!
www.viralml.com
@Miguel Bayona : I just went through the code and the matrices are actually 26x26 (or 24x24 with the data that Manuel had back when he posted this video). I think your confusion comes from the fact that, in the code, Manuel just prints the .head() of the matrices that, by default only shows the first 5 rows of the dataframe...
@Manuel Amunategui : Thanks for this and many other ineteresting videos you post!!
11:08 where is the date from 2010 gone. It started for 2015
Can you optimize this model and increase the accuracy above 70 or 80?
I have tried this method using DJI and SPY stocks. The accuracy didn't cross 70.
Can you post another video on the same topic but using better datasets and optimized code if possible?
Hi, glad you tried, that's the whole point is to teach you how to do it. And if you want to dig more into these topics - check out:
www.viralml.com/learn
Thanks!
@@viralml can you make a video on option pricing using different models like monte Carlo, binomial, black scholes
@@whatwonderfulworld Thanks for the tips, Kaushik - I'll add it to the list.
can any provide with python code link in github
Can you beat a provider spread with 4% positive probability?
It would probably beat 50% of day traders :-)
He can apply martingale
Rest. Does it work ? ;)
What is the time frame for this model?
I thought it was just daily data, but then it does not get along with the following combinations:
th-cam.com/video/sdp49vTanSk/w-d-xo.html
For trading,would you reccomend me the hmm or markov chains?Sorry i am totally new to this...
You need to backtest thoroughly any strategy yourself - recommendations aren't worth anything.
@@viralml Yes,i am aware that i need to backtest my strategy.But rn i am trying to understand the differences between chains and hidden models.As a person who doesn't have a strong mathematical or statistical background it is very challenging to understand those terms when applied to real life situations.It is a bit easier to understand the theory when someone uses analogies with weather or food,but for trading i am still not sure which path is more suited...
I do this thing in Excel :D
Nice !
I love this. HMM?
54.43% is not good. Flipping a coin is 50%. Why don't we simply flip a coin to decide whether to buy a stock?
Most do
Manuel Amunategui really? :o
Ever heard of the compound interest?