Hi,

I am backtesting my strategy with 21 Instruments on 7 years of history with a data Interval of 1min
in order to optimize the backtest speed.

It appears that data cache takes more than 100GB on the hard drive due to a very large amount of intraperiod data.

The issue is that since JForex and API ITesterClient both flush intraperiod files after every restart, it often takes
more than 30 minutes to 1 hour to start JForex or a standalone backtest with ITesterClient ! (it takes a long time to flush more than 30GB and thousands of files !)

Can't we just chose to let the data in place and re-use it ? (intraperiod included)

In tried to put the cache on a SSD disk, it's faster, but I it's too small to contain all the cache and with all these write/flush tasks
the disk will be dead in a short time...

This is really a big issue and makes multiyear/multicurrency backtests almost impossible to optimize.

King regards

We will check this

Thanks for your response.

Please check this both with the JForex platform AND the ITesterClient standalone API (a lot of us use the API for complicated optimization processes).

Let us know what kind of solution you have for us.

Best regards.

Available on DEMO. Version 2.32.2.

Folders of intraperiod are deleted automatically now if the test is finished and the tester chart is closed.

Thanks for the news, I tried and it works.

But I wonder why all this intraperiod data has to be written when no chart is opened ?
Why is the Tester creating all these intraperiod candles when the Optimizer does just load
the start candles ?

The tester with optimization checked is 4 times faster than any other backtest method
for exactly the same result !

Will the standalone API be updated to ? Because I have the exact same issues and I
have no choice but to use the API to automate the backtest process.

Please see the following screenshots where I compare both methods of testing.

Intraperiod data might be used in the strategy.

This might be not correct. We will have a look into this.

Yes, it will

Thanks very much for these information.




Wouldn't it be possible to let the user choose if all this intraperiod data should be written or not ?
Loosing 4 times the speed of the optimizer for unused data would be just impossible for a lot of us !

If we have no choice but to write intraperiod data on the disk, then at least let us the ability to generate
that data at the first run, then the tester will re-use it like it is doing with the downloaded cache.

This issue of duplicated intraperiod data has been first discussed in another topic concerning the API, let me take the example from it :

Starting multiple strategies with ITesterClient in a loop like the following code, is creating as much duplicated intraperiod
folders as they are strategies started.

for (int j = 0; j < 80; j++) {
  
            MA_Play strategy1 = new MA_Play();
 
            //start the strategy
            LOGGER.info("Starting strategy");
            client.startStrategy(strategy1)
}

This creates for each currency pair something like :
intraperiod
intraperiod1
intraperiod2
intraperiod3
...
...
intraperiod80

Each intraperiod is dupplicated and has the same data for hundred's of MB on the hard drive !
I don't see the need to duplicate all this data since ITesterClient has only one instance, so every
started strategy uses the same ITesterClient parameter set.


Last thing that bothers me. As you can see in the first screenshoot of my last post, intraperiod
data witting is random and is very much lagging the testers progression, for instance :

2007_12_03_00_ticks.bi5 has been written at 16:34

2007_12_03_02_ticks.bi5 which are the ticks 2 hours later has been written at 16:31

So more recent data has been written about 3 minutes before older data. In 3 minutes
the strategy progression is well ahead of that data in time. If it needed intraperiod it
should have pause a few minutes to wait for that data to be in place.
I have a fast computer and the data is on a SSD drive, I imagine on a regular machine
the gap should be a lot bigger.

Please tell me what you think about all this, the steps you are willing to take and the
schedule so I can organize my work in consequence.

Best regards

Hi support,

Any news on these issues ?

Kind regards.

Hello is anybody still here ?

These are some important issues, last message from support is from February 27th.
I have some projects on hold because of this.

Could I have at least some news from the Dukascopy team, telling what is planned
and when it is scheduled. Every day waiting and not knowing what steps you are willing
to take is costing me money.

Thanks for your understanding.

Best regards.

Will let you know next week. Probably this will be a custom setting.

Thank you Support for your response.

A custom setting would just be fine !
Please note that this setting should be manageable in JForex Gui AND Standalone SDK.

I am looking forward to here from you.

Best regards.

Hi Support,

I expected some news last week. Did you take any decision ?
Thanks.

Best regards.

Will implement the possibility to disable the intraperiod saving under Custom settings of Historical tester.

This will be possible only if visual mode is off, or else scrolling or switching the period on chart with interpolated data will load real data and there will be a confusion.

Hello,

Thank you for this good news.


OK sounds good to me.

When will this option be available on JForex Gui and Standalone API ?
I hope that this option will be available in the optimizer to, will this be the case ?

Best regards.

If optimizer is used, intraperiod data is not saved anyway as there is no Visual mode. However, it can cause problems in case of indicators or some history specific logic in strategy.

And about the schedule for implementation on the Gui and the standalone API ?

As soon as possible

OK, please let me know when its ready.

Best reagards.

Available in JForex version 2.35 (DEMO only at this moment)

You can find it under Custom settings in Historical tester. You can disable it, but it will be saved anyway if Visual mode is enabled and test is run on Interpolated data. This is done in order to avoid a mix between real historical data and interplated data when swiching the visual mode chart from one period to another.

Hi support, this is great news !

I just tested. It's perfect, now that the hard drive bottleneck isn't an issue
anymore, on my computer the speed of a backtest is 3 to 4 times faster !

Did you update the standalone API too ?
Can you please give me the syntax in order to use this option ?

Best regards