Dukascopy Support Board http://www.dukascopy.com/swiss/english/forex/jforex/forum/ |
|
ITesterClient, optimize the code for parallel strategy backtesting http://www.dukascopy.com/swiss/english/forex/jforex/forum/viewtopic.php?f=83&t=50797 |
Page 1 of 1 |
Author: | nicofr0707 [ Wed 12 Feb, 2014, 17:55 ] |
Post subject: | ITesterClient, optimize the code for parallel strategy backtesting |
Hi, I just figured out how to backtest strategies in parallel with ITesterClient and the MA_Play.java example provided with the JFOREX SDK. How would you optimize things in order to use close to 100% of the CPU ? I never managed to stay above 50-60% (I7 4770k 4 cores / 4 threads). Please see the following code, I'm starting the same strategy with a different parameter through a loop. The higher the number of strategies loaded, the higher the number of threads is in the Windows task manager (+6000 threads for 200 strategies) and the higher is the Ram usage. But CPU load still remains at 50-60% Why ? Thanks for your help and knowledge. /* |
Author: | API Support [ Fri 14 Feb, 2014, 09:51 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
To reach 100% the of the CPU usage the ITesterClient should keep all of your 4 processors busy at all times, which it is obviously not doing and it should not be doing this. You should be able reach 100% CPU if you ran 4 instances of the following program: public class WhileTrue { Also mind that even though you are starting the strategies in a cycle, in ITesterClient only one strategy at a time is being back-tested (in IClient in comparison they run in parallel). If you wish to back-test multiple strategies in parallel, you need to run them from separate ITesterClient instances each of which get ran from a separate process. |
Author: | nicofr0707 [ Mon 17 Feb, 2014, 17:24 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
Hi support, I've figured out a lot of things. One thing bothers me, backtesting in JForex is faster than in the standalone API with ITesterClient : For my test I tried to backtest only one strategy. - JForex with "Optimization" selected took about 20 minutes with a CPU load of 10-15% - Standalone API with ITesterClient with no GUI (and no console print) took about 30 minutes with a CPU load of 15-30% How is that possible ? It's a huge difference with even more CPU usage. I tried both of these startStrategy calls, it doesn't make that much difference : public long startStrategy(IStrategy strategy) Could you please give me some advice to solve this ? Thanks |
Author: | API Support [ Thu 20 Feb, 2014, 15:13 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
Could you please provide an example ITesterClient program, an example strategy and Historical Tester setting and optimization setting printscreen? |
Author: | nicofr0707 [ Mon 24 Feb, 2014, 19:03 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
Hello, I did further testing and I may know why the JForex optimizer is faster than API ITesterClient's public long startStrategy(IStrategy strategy) AND even faster than the JForex tester itself (without Visual mode nor Optimization checked). Let's assume we test with 1 minute candles, BID and 4 ticks at OHLC. The tester and ITesterClient are writing very large amounts of 1 minute intraperiod cache files. This is causing the computer to slow down because of heavy disk utilization. Moreover, this intraperiod cache has to be flushed after each restart, it's very painful after a multiyear, multicurrency backtest of 100GB ! In opposition, JForex Tester with "Optimization" checked doesn't write any intraperiod 1 minute cache file. This is for sure the reason why it is faster and does use less cpu than all other backtest method. In the case exposed in the first post, starting multiple strategies with ITesterClient in a loop like the following code, is creating as much duplicated intraperiod folders as they are strategies started. for (int j = 0; j < 80; j++) { This creates for each currency pair : intraperiod intraperiod1 intraperiod2 intraperiod3 ... ... intraperiod80 Each intraperiod is dupplicated and has the same data for hundred's of MB on the hard drive ! I don't see the need to duplicate all this data since ITesterClient has only one instance. What can we do to have the same behavior than the tester with Optimization checked ? Thanks for your support. Nicolas |
Author: | nicofr0707 [ Tue 25 Feb, 2014, 08:59 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
If I can't desactivate Intraperiod data to be written, is there a way to modify the API in order to write Intraperiod data on an other disk than the cache data disk ? The idea would be to use RAMDisk software to load Intraperiods in RAM with a program periodically flushing the unused data. |
Author: | nicofr0707 [ Wed 26 Feb, 2014, 17:35 ] |
Post subject: | Re: ITesterClient, optimize the code for parallel strategy backtesting |
Hi support, I did start a new topic concerning intraperiod data issues and had a response from the platform support. Since my last messages here are related to the same topic, please see the following link with tests and screenshots. I am looking forward for some solution on this because right now it's impossible for me to go further with my strategy because of this intraperiod issue. Thanks viewtopic.php?f=85&t=50828&p=78057#p78057 |
Page 1 of 1 |