Timeframes with different market situations?

[mczero80]

I have an idea that could potentially improve live results.
Would it make sense to feed the genetic algorithm 90% of the time with the timeframe
as it is now, and 5%-10% with "difficult market timeframes" ? Special timeframes where traders
had really problems applying their strategies. Genetic evolved strategy parameters could
then in this case be rated as a positive result, when it not necessarily makes profit in that time, but is good in keeping loss low.

Does that make sense to implement? Of course manual backtesting afterwards can address different market situations, but genetic backtesting could bring interesting parameters up that would otherwise be discarded.

The reason not to genetic backtest huge timeframes from the beginning (with good and bad market situations right from the start) is that I believe that it is better to change live trading parameters every few days to reflect the current market better.

[Gab0]

This seems to be a very good idea. Do you have such a database of difficult candles, or sources to them, or even ways to label and extract them?
With those candles at hand, one could feed them among the evaluation candles (defined in [dataset_sourceX] at settings/dataset.toml), as a mixture of "regular" candles and those "difficult" candles.

I do agree that its better to have multiple short (1~3 months) candle sequences and evaluate them separately and in parallel (as we do it now, with different locales) than evaluate a single long (1 year+) candle for every individual everytime.

It will make sense to implement a dedicated system to manage those candles once the method proves to be promising...

[mczero80]

No, unfortunately I don't have such datasets. Mind you, I am not an expert in crypto trading.

But I will investigate a bit. In the meantime, I could imagine to extract difficult market situations with the following method:
Using gekko on 1 minute candles (?), and adapting one or more known good strategies (perhaps this time overfitted parameters are wanted?), to write the candle data to csv (or something else) when the strategy is failing for a few days here and there.
This could be refined when extracted data from different strategies is combined (overlayed) with each other, and then matching times when both are failing are again written to a result dataset.

Or perhaps a semiautomatic method works with TradingView, using a script that does a similar thing like what I said about using gekko. I don't know if those scripts are able to write to downloadable files though.

When I think about it, this could address the whole overfitting a bit. By how much, hmm that has to be looked at.

[Gab0]

Maybe we could have a program that breaks a big candle data series into 1 month blocks (or some other random size).
Then it will test a series of strategies over each block, detecting which blocks are easy or complicated thru the mean profits of the strategies at given block. A csv file would be created with one row per block, containing their time boundaries and scores (not candle data... candle data needs to be accessed by gekko which has its own method of storing candles).

One could argue that the most complicated candles are simply the ones where market is bearish, so no need of this complicated machine? I guess that for a good strategy it is easy to know when the market is obviously going bearish, so the AI will outperform it by not exposing at all. The complicated candle cases we are talking about are when the market seems profitable but then some sudden (maybe predictable?) reversal or wathever happens and then we are at loss, right?

TL;DR while there will be a lot of work involved in implementing this idea, it seems very valid to me.

Currently I'm working in making japonicus a general purpose genetic algorithm framework, not specifically related to gekko.... Maybe in a month or something we can discuss this better and implement.

[mczero80]

Yes that sounds very plausible. I am learning in the meantime how this and that works, all related to gekko and trading in general. When I look into the code of japonicus, it doesn't seem to be very complicated to query backtest data from gekko.

I am also waiting for my PC to get ready with a japonicus session... it is running all the time since a week, still not ready, but at epoch 2340 from 3000. It does still make negative profit, not changing very much. I don't know if I should have stopped it, if it was a waste of time with that strategy and timeframe! Do you leave it running no matter what?

Back on topic, I will invest more time and I think I can help programming when you are at a point where you are more happy with japonicus. I will report when I come up with ideas and results!

[Gab0]

Hey @mczero80, epoch 3000 is too much. It should find some viable parameterset for the strategy at epoch 50 or 100. However, very few strategies should achieve this and if it didn't, it should not ever go profitable.

Maybe you should compare your strat's evolution timeline with RSI_BULL_BEAR_ADX or some other, they're all available here.

I don't think you got lucky in your run (even 11 days later XD), but your evolutionary logs could be valuable for further development of the GA, if you can share them (send me an e-mail if that's the case).

cheers

[mczero80]

Hey,

unfortunately I hadn't had much time, I will see if I can upload them tomorrow evening,
thank you for your response! All I can say at the moment is that it was more kind of a generic test, didn't expect much but was wondering why it performed that negative. And...I was testing what would happen if I leave it running that long.

Greetings from Germany!