Forecasting strength - MSE metric

[andrewczgithub]

Hi @compops !
I hope you're well.
I am looking at using the matlab code for a university project, located at -

https://github.com/compops/pmh-tutorial

I am just wondering, my understanding of state space models is that their accuracy can be determined by the mean squared error. However when I run your code, Im not sure how the forecast accuracy is measured. Is it split into test and train sets, then a metric is determined?
What metric is used to measure the forecast accuracy of the state space model?

Many thanks for your help,
Best,
Andrew

[compops]

Hi,

Mean square error (MSE) can be used as a measure of how good the model is, yes. However, I am not really sure what code you are referering to. If it is example 1, then the MSE if computed by the use of the Kalman filter/smoother which gives the optimal performance on the LGSS model. If you refer to examples 3-5 using the stochastic volatility model, the error bars (in the plots) are the quantiles generated using the (P)MCMC run. These indicate how the state estimate (the volatility) would vary over the range of likely values of theta (the parameters).

If you would like to find the MSE of the state estimate, then one approach would be to simulate a dataset and divide it into estimation and validation set. Fit the model using the estimation set and then compute the MSE (as you know the true state) on the validation set. However, we typically do not do it like this. You basically never recover the true state sequence in state space models. So, we typically compare with a ground truth given by a particle smoother of some sort.

I hope this clarify your questions. Otherwise, get back to me with some specifics and I will try my best to help out.

Johan

[andrewczgithub]

Hey @compops!
I was looking at comparing forecast strategies on return data.
I thought the stochastic volatility model is used to forecast return data of stock prices.
I was going to compare mse, between a few models and write a report on my findings.

The below model was another model, I was going to try -

https://github.com/OxfordML/GPz

Does this make sense or am I choosing the wrong tool for the job.
Many thanks for your help :).
Best,
Andrew

[compops]

Hi again @andrewczgithub,

You could do that but SV models are not good for prediction if you don't use other factors to explain the changes in e.g., the stock price. See https://arxiv.org/pdf/1002.2017.pdf. If you want to compare how well a model fits the data, the marginal likelihood or AIC/BIC is the way to go. A paper doing a similar thing is: https://www.stats.ox.ac.uk/~doucet/pitt_malik_doucet_continuousparticlefilterSVmodel.pdf

I hope this helps!
Best Regards,
Johan

[madpower2000]

First of all thank you for detailed tutorial, I looked for a while for good intro into practical particle filtering, but your work is the best!

As I understand the problem, if you made good stochastic volatility model you can accurately forecast future volatility and then make profit of buying or selling some option combo.

But for me is unclear, how to extract prediction (in general) from particle filtering?
If we are talking about simple kalman filter there exact prediction step, but it’s still vague for me, how to do the same with particle filter?

And one more thing. Some time ago I stumble upon two papers: Trend without hiccups: a Kalman filter approach and Kalman filter demystified: from intuition to probabilistic graphical model to real case in financial markets - as I understand the author trying to predict future price movement (stochastic process) with kalman filter, linear model and a lot greek letters… I’m stupid or author of these papers is genius?

[compops]

Hi @madpower2000

Thank you! I'm happy that the tutorial is useful for you :)

Forecasting is always difficult and the standard stochastic volatility model in the paper is not suited for forecasting future volatility. The model is mainly used in risk management and option pricing to get the current volatility. It's commonly accepted that the model is "wrong" but good enough for the finding the latent stochastic volatility. There are other models (such as factor models) as I discussed above that might be better at forecasting volatility. However, volatility is usually also described in terms of jumps in prices (this is known as leverage) as people tend to sell the assets if the prices drops which increases the volatility. These jumps are rare (think: the crash in 2007-2008) but they have a huge impact on the volatility and they cannot be predicted.

If you want to predict the volatility you just run the prediction/propagation step again and again and again in the particle filter without the resampling or weighting. Use the best guess on your parameters from e.g., PMH and initialise the particle filter in the latest estimate of the stochastic volatility extracted from PMH or the particle filter. You can try it as a teaching experience and look how the predictions will turn out. You can assume that all particles have equal weight when you compute the state estimate. It will essentially just be a random walk and the uncertainty (the spread of the particles) will grow very quickly.

I'm very skeptical when it comes to predicting the market. A lot of people do try to do this but there are a lot of academic support for the efficient market hypothesis which basically means that there are no systematic ways of consistently beating the market. Basically, as more advanced trading schemes are introduced the market adapts so the arbitrage effect disappears. (There is probably loads to read about this on Wikipedia). Also the Quants at the big bank use a lot more advanced tools than this stochastic volatility model in their work and they can't beat the market all the time. It's a gamble.

[madpower2000]

Many thanks for yours detailed answer. I don’t trying to catch a fish in the in muddy water, that is, to predict future market price movement in the long run. And I totally agree with you, this type of venture mostly totally gamble.

I just have practical issue with intraday volatility estimation, I would say HFT-like, tick-by-tick estimation. The tick data is extremely noisy and if used sliding window for estimation of standard deviation aka volatility, next dilemma is arising:

• The window should be long so as to incorporate as much training data as possible.
• The window should be short so as to incorporate only the most recent data, as old data might be obsolete.

And I suppose particle filtering is perfect fit for this purpose. Isn't it?

For my point of view in very short run price movement somewhat predictable - huge wealth of Renaissance Tec. is confirmation for this. But my interest mostly academical, I know to compete with smartest guys in the room - it’s not an easy target.

[compops]

Sounds reasonable and wise!

A stochastic volatility model with the particle filter is a good fit for the intra-day volatility. Any parameter inference algorithm will fit the model to data and automatically find a good balance between the two dilemmas that you raise. A Bayesian inference method such as the PMH will also give you the uncertainty in the volatility estimate which could be useful to see if the estimate is just crap at the moment :P

In terms of prediction, you can probably assume the the volatility is the same as the day before. We call this zero-hold in the engineering literature. But otherwise you can just run the particle filter for one extra step as we discussed earlier.

Good luck with your endeavor and let me know if you need any help with the particle filter / PMH algorithm.