Dukascopy
 
 
Wiki JStore Search Login

Historical tester works 5 to 6 times slower since a couple of days
 Post subject: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Sat 28 May, 2016, 13:56 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
Since wednesday/thursday i noticed that while using the same parameters the historical tester got 5 to 6 times slower. This is for different currency pairs i am testing with.

I cleaned up my laptop and even tried testing with another laptop but still the historical tester remains slow. So i am guessing their must be something with the Dukascopy server.

Are there any other people experiencing the same thing?
Or does anybody know of a solution?

Every help is appreciated!! because i am eager to continue testing my strategies at full speed.

Greetings, David


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Mon 30 May, 2016, 08:28 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Hi,

What was the runtime parameters used for your test please ?
Are you testing your strategy on tick basis ? did you tried interpolation in order to speed up the test ?

Thanks


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Mon 30 May, 2016, 11:14 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Hi,

What was the runtime parameters used for your test please ?
Are you testing your strategy on tick basis ? did you tried interpolation in order to speed up the test ?

Thanks


Hi Amine,

Thank you for your quick response.

Yes, tests are running on tick basis.

The runtime parameters was set to 2048 mb and on average max 1000 mb of memory was used. I used optimization and simultaniously tested with 3 historical testers with 30 tests running on each of them and tested a period of about 2,5 months. Depending on which currency pairs this used to take between 1,5 and 2,5 hours. Now it takes between 6 to 15 hours.

Things i already tried:
- Uninstalled and than installed a fresh version of JForex
- Registered for a new demo account and tested with that
- Tested the speed of my internet connection
- cleaned up my computer
- tested on another computer

Untill now with no change in test speed.

I don't know what interpolation is. How does it work?

Greetings


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Mon 30 May, 2016, 16:32 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Thanks a lot for these details.

I would need to see the strategy and test it in order to have a clear picture but I suspect something else:
It is probably related to the number of indicators called by the strategy, I use to remove the built-in indicators "Balance" "Equity" and "P&L" and call them only when I need that. We're trying to optimize the memory load of such situation.
If you did not notice any improvement, I suggest to export the strategy to Jforex platform and test it there.
As said, thorough tests should be conducted before confirming the above.
Interpolation settings are created to speed up historical tester: In short, if your strategy is built on tick basis then there is no need to optimise the data. More information available in our wiki.

Thanks


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Tue 31 May, 2016, 13:02 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Thanks a lot for these details.

I would need to see the strategy and test it in order to have a clear picture but I suspect something else:
It is probably related to the number of indicators called by the strategy, I use to remove the built-in indicators "Balance" "Equity" and "P&L" and call them only when I need that. We're trying to optimize the memory load of such situation.
If you did not notice any improvement, I suggest to export the strategy to Jforex platform and test it there.
As said, thorough tests should be conducted before confirming the above.
Interpolation settings are created to speed up historical tester: In short, if your strategy is built on tick basis then there is no need to optimise the data. More information available in our wiki.

Thanks


Thank you for your response.

Well the sudden difference in speed can not be related to the number of indicators called by the strategy. Because i also tested it with multiple strategies that i have been testing with for the last couple of months. And without making any changes to the strategies the completion time was still 5 to 6 times slower.

Are there any changes that Dukascopy made to the historical tester server since last week wednesday/thursday?

How can i export the strategies to the JForex platform for testing and am i than still able to use the optimization function?

thanks


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Tue 31 May, 2016, 14:40 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Hi,

- Strategy testing doesn't depend on Dukascopy servers but your computer resources: The historical data is downloaded in your hard drice and processed form it. You can see it in
C:\Users\your_user_name\AppData\Local\JForex\.cache (AppData folder might be hidden)
- You can delete the content of the above cache folder thus you force the system to download fresh data from our servers.
- Monitor your processor / Memory load when the strategy is running
- To export your strategy is Java to Jforex: Compiler > Build then, Go to your stratgies folder in the right section (purple icon) "Strategies" > My strategies then locate your recent ly built strategy and right click > View source and then copy paste the code in Jforex new strategy/ from Jforex you should compile it test it afterwards
Image

- Optimization is indeed available in Jforex but slightly different methods. it is available in the historical tester tab in the right side as a drop down menu
Image

Cheers


Attachments:
VJFtoJava.JPG [149.64 KiB]
Downloaded 325 times
HT_Jforex.JPG [52.99 KiB]
Downloaded 326 times
DISCLAIMER: Dukascopy Bank SA's waiver of responsability - Documents, data or information available on this webpage may be posted by third parties without Dukascopy Bank SA being obliged to make any control on their content. Anyone accessing this webpage and downloading or otherwise making use of any document, data or information found on this webpage shall do it on his/her own risks without any recourse against Dukascopy Bank SA in relation thereto or for any consequences arising to him/her or any third party from the use and/or reliance on any document, data or information found on this webpage.
 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Tue 31 May, 2016, 17:32 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Hi,

- Strategy testing doesn't depend on Dukascopy servers but your computer resources: The historical data is downloaded in your hard drice and processed form it. You can see it in
C:\Users\your_user_name\AppData\Local\JForex\.cache (AppData folder might be hidden)
- You can delete the content of the above cache folder thus you force the system to download fresh data from our servers.
- Monitor your processor / Memory load when the strategy is running

Cheers


Hi Amine,

I have been testing with JForex since december 2015 and since may 2016 trading live. Untill now I was completely satisfied with your system, quality of your data and the speed of the historical tester. Just since last week wednesday/thursday the speed of the historical tester slowed down 5 to 6 times. Since i am in the process of expanding my live account with more automated strategies it is really important to me that i get this sorted out as soon as possible.

I have been testing the speed of the historical tester today through using Google's Cloud Computing. I used two different Virtual Computer Setups to test the difference in speed.

Setup 1= 2,75 GHz + 3,75 GB Memory
Setup 2= 5,50 GHz + 7,50 GB Memory

The strange thing is that with the same strategies the testing speed is still the same dispite the fact that setup 2 has double the capacity as setup 1. And compared with the speed that i was used to test with the last couple of months it is about 5 to 6 times slower.

As it seems the speed of the historical tester is not being limited by the capacity of the computer but probably by something else. Can you please check if there have been any changes last week with the installed version of JForex or changes in the speed or capacity of the data transfer with the historical tester?

Greetings,
David


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Wed 01 Jun, 2016, 10:39 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Hi David,

I see, thanks for these important details.
I'll check my team for more information in this regards.
Meanwhile, did you tried to play with the optimization settings (interpolation) ?
Basically, it speeds up the test significantly: Say that you're using 15mn candles in your strategy, when launching the HTest, you can set the period to 15mn interpolated with OHLC for example, and the system will only grab the open high low and close prices to build a given candle: This will not affect your strategy and the speed test will increase ( unless you're using some additional tick prices within the candle itself that could impact the strategy logic/results)
I recommend you to try that and check the results till I get back to you.

Cheers


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Wed 01 Jun, 2016, 12:16 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Hi David,

I see, thanks for these important details.
I'll check my team for more information in this regards.
Meanwhile, did you tried to play with the optimization settings (interpolation) ?
Basically, it speeds up the test significantly: Say that you're using 15mn candles in your strategy, when launching the HTest, you can set the period to 15mn interpolated with OHLC for example, and the system will only grab the open high low and close prices to build a given candle: This will not affect your strategy and the speed test will increase ( unless you're using some additional tick prices within the candle itself that could impact the strategy logic/results)
I recommend you to try that and check the results till I get back to you.

Cheers


Hi Amine,

Thank you very much for diving more deeper in to it. Well i didn't use interpolation because i only test with tick data and tick data is necessary for testing my strategies. And i learned from you that you can't use interpolation while using tick data. Did i interpret that correct?
I will further see if there are other optimization settings that i can speed up the Htester with. Thank you for the advice.

Greetings,
David


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Wed 01 Jun, 2016, 13:24 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Hi,

That's correct, Tick data cannot be optimized as it is mandatory to call/load every tick to process the historical tester. Besides, did you managed to export the strategy in Java code to Jforex platform ?
There is another method to load VJF strategies from your Demo account in Jforex: Once the strategy is built in VJF, do the following:



Image


Cheers


Attachments:
OpenVJF_JF.jpg [805.3 KiB]
Downloaded 330 times
DISCLAIMER: Dukascopy Bank SA's waiver of responsability - Documents, data or information available on this webpage may be posted by third parties without Dukascopy Bank SA being obliged to make any control on their content. Anyone accessing this webpage and downloading or otherwise making use of any document, data or information found on this webpage shall do it on his/her own risks without any recourse against Dukascopy Bank SA in relation thereto or for any consequences arising to him/her or any third party from the use and/or reliance on any document, data or information found on this webpage.
 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Wed 01 Jun, 2016, 16:31 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Just got an important feedback:

Historical Tester relies on Java program allocated resource : Java heap memory that you can manage in Jforex from the login window at the bottom "Advanced settings". In visual Jforex you should edit the downloaded jnlp file called "visual_jforex-demo.jnlp" and change the "...max-heap-size="512m"/" to 6GB when running with the setup 2 for instance. That will make a difference I believe.

Cheers


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Thu 02 Jun, 2016, 09:37 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Hi,

That's correct, Tick data cannot be optimized as it is mandatory to call/load every tick to process the historical tester. Besides, did you managed to export the strategy in Java code to Jforex platform ?
There is another method to load VJF strategies from your Demo account in Jforex: Once the strategy is built in VJF, do the following:

Cheers


Thank you for the extra info Amine.

I prefer to keep working with the VJF Htester because i like the simplicity and practicality and in this way i don't have to mess around with java code. Now i just need to get the testing speed of the Htester back to normal.

Greetings,


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Thu 02 Jun, 2016, 09:43 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Just got an important feedback:

Historical Tester relies on Java program allocated resource : Java heap memory that you can manage in Jforex from the login window at the bottom "Advanced settings". In visual Jforex you should edit the downloaded jnlp file called "visual_jforex-demo.jnlp" and change the "...max-heap-size="512m"/" to 6GB when running with the setup 2 for instance. That will make a difference I believe.

Cheers


Hi Amine,

I already work with a bigger Java heap memory than the standard.

Setup 1 = 2048 mb
Setup 2 = 4096 mb

And when i monitor it while testing with the Htester it doesn't even use half of the available heap memory and the testing speed of both Htesters stay's about the same.

What else can it be?

Greetings,
David


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Thu 02 Jun, 2016, 12:08 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
Hi David,

I see, although you already told me about this but I missed that :)
Can we arrange a troubleshooting session for that ?
Can you please let me know you availabilities by email to [email protected] I'll share with you the needed details.

Cheers


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Fri 03 Jun, 2016, 07:46 
User avatar

User rating: 0
Joined: Fri 01 Jan, 2016, 14:06
Posts: 9
Location: Netherlands,
amine_chourou wrote:
Hi David,

I see, although you already told me about this but I missed that :)
Can we arrange a troubleshooting session for that ?
Can you please let me know you availabilities by email to [email protected] I'll share with you the needed details.

Cheers


Hi Amine,

Thank you very much and i just e-mailed you @support.

Greetings,
David


 
 Post subject: Re: Historical tester works 5 to 6 times slower since a couple of days Post rating: 0   New post Posted: Fri 03 Jun, 2016, 10:30 
Visual JForex expert at Dukascopy
User avatar

User rating: 287
Joined: Thu 18 Apr, 2013, 09:26
Posts: 1496
Location: Switzerland, Geneva
You're welcome !

Please check my reply and let me know

Cheers


 

Jump to:  

  © 1998-2011 Dukascopy® Bank SA
On-line Currency forex trading with Swiss Forex Broker - ECN Forex Brokerage,
Managed Forex Accounts, introducing forex brokers, Currency Forex Data Feed and News
Currency Forex Trading Platform provided on-line by Dukascopy.com