Dukascopy Support Board http://www.dukascopy.com/swiss/english/forex/jforex/forum/ |
|
Strategy Testing Speed http://www.dukascopy.com/swiss/english/forex/jforex/forum/viewtopic.php?f=65&t=57144 |
Page 1 of 1 |
Author: | JP7 [ Fri 23 Aug, 2019, 18:10 ] |
Post subject: | Strategy Testing Speed |
Once a strategy test passed sluggish for the set interval (15 years in max interval for EUR/USD) then becomes faster next time for the same currency pair and same interval. It is faster then for other strategies using the same currency pair and the same interval. The Proof: I did this test on two computers with the same configuration and the one used before to run strategies is faster than the other one who just started running strategies. Then I did a HDD clone from the comp who run faster to the other one and booth computers run faster (same fast speed) for the same tests. Conclusions: 1. trading history is written somewhere in the computer - not in windows temporal files folder OK - thank you ! "On windows cached tick data is stored here C:\Users\<UserName>\AppData\Local\JForex\.cache." 2. those hidden files improve next test on same or other strategies for the same period and currency pair used Yes - they contain history tick data 3. probably if we can locate this hidden files we might copy them on other computers to make the test faster .... Yes confirmed - works faster - files located OK - "On windows cached tick data is stored here C:\Users\<UserName>\AppData\Local\JForex\.cache." can be copied on other computers to make them work faster - by eliminating tick data downloads time from servers. 4. after those hidden files are in your computer, then the computer performance has its say - the faster computers will perform faster Yes - statement confirmed. ....more to come if discovered.... |
Author: | JP7 [ Fri 23 Aug, 2019, 18:12 ] |
Post subject: | Re: Strategy Testing Speed |
mtnfx wrote: Unfortunately, "performance of back testing" and "reporting" are not of high-priority for Dukas's developers, for years ... I'm pretty much sure this highly wanted feature by many users, but instead effort is spent onto adding new fancy (not sure about useful) things ... and I also pretty much sure back-testing may run at least 10 times quicker if Jforex developers would address that thing appropriately.
Ok, back to question (actually - it is better to move onto other thread for this): 1. Xeon CPUs are better used in highly-threaded applications. JForex runs user strategy in the single thread - so ... you'll get better performance with CPU having more "Instructions Per Second" what nowadays same as "having higher clock rate - Ghz". If you would want to solve that issue from hardware perspective - refer to this link (sorted down by SuperPI 32M, which is a "simple single thread test") (... next things depend on how much you are away from programming ... ) 2. I recommend running empty strategy on 1 year interval (1 month / 1 week) - this will give you baseline and let you known how much time on top of JForex is added by you strategy. Then - you may enabled/disable some lines in you strategy to understand what is consuming most of the time. 3. I'm pretty much sure processing all the ticks is not really needed. I would recommend in your strategy skipping ticks coming more frequently than "once per N seconds" and having no significant price change - i think this may speed up testing by 2-3-5 times. Note: at some hot periods tick-rate may rich N*10/sec. 4. Reporting ... the one provided by JForex is !@$!@@#. I would recommend listening to notification messages provided by JForex and writing results to a CSV files - this way you'll be able to open that file in Excel while strategy is still in progress and build graphs, data filtering or whatever else you may need. |
Author: | JP7 [ Fri 23 Aug, 2019, 18:13 ] |
Post subject: | Re: Strategy Testing Speed |
It is clear to me now that the speed is not given by a high end computer but by the speed feed form dealer servers. |
Author: | JP7 [ Fri 23 Aug, 2019, 18:14 ] |
Post subject: | Re: Strategy Testing Speed |
NightOwl wrote: Hello!
Yeah the speed is dependant on the networks speed to the servers. There are for example some VPS that have 1ms latency to dukascopy server, but this does not mean that the data will spaw between the servers at that speed as the protocols and scripts that orchestrate the whole datafeed transfer affects the speed that data can be downloaded. Nice thread! This is cool concept, when you have enough of these with 10 year proofs, you can start e hedge fund where you allocate money on these different strategies. WOW! It is like not only investing in one strategy but, we say 30, and if one 5 fails, the others are safe. This is really nice idea man, lol. EDIT: depentant on other factors also. EDIT2: I would point out that dukascopy demo has some odd anomalities, sometimes the price jumps really really odd to your take profit, like dirty odd. I have seen it twice. |
Author: | JP7 [ Fri 23 Aug, 2019, 18:29 ] | ||||
Post subject: | Re: Strategy Testing Speed | ||||
BE PATIENT Use a computer just for this purpose with minimal software installation Set up the computer to never sleep - HDD and processor Continuous uninterrupted fast internet connection _ LAN in better Test on Demo accounts - looks faster - and no worries First run Empty Strategy just once, for each trading instrument and time period, to load the cached tick data stored in C:\Users\<UserName>\AppData\Local\JForex\.cache. Reduce continuous tick feed if not necessary from Processing All Ticks to lower options - be careful here - my tests gave me different results from reducing from continuous data feed to other intervals - and I have no idea how to adapt this reduced feed intervals in real time strategy trading. For this reason continuous data feed - process all ticks - remains the standard reference for testing in my views. After platform crushing or freezing restart the platform - if still sluggish restart computer. Do not delete the platform cache files C:\Users\<UserName>\AppData\Local\JForex\.cache - they contain history data Restart testing ! P.S. From my point of view I think the strategies are crushing when they are restarting the servers at rollover time each day and on Fridays at close and Sundays on open. When you notice that the strategy barely moves, and it used to work faster before, that means the server is overloaded and there is no point to continue in this conditions. Stop and Restart. Run strategy when Europe People are sleeping start past midnight and when you walk up in the morning look for results! Good Luck !
|
Author: | mtnfx [ Fri 23 Aug, 2019, 21:35 ] |
Post subject: | Re: Strategy Testing Speed |
JP7 wrote: Conclusions: 1. trading history is written somewhere in the computer - not in windows temporal files folder It took quite a while for you to know that On windows cached tick data is stored here C:\Users\<UserName>\AppData\Local\JForex\.cache. Once strategy has fetched some portion - it stays there forever (as far as i see it). Yes - first run depends on speed of internet (plus throughput of Dukas's) servers. Next runs - solely depends on performance of your computer and complexity of the strategy. |
Author: | mtnfx [ Fri 23 Aug, 2019, 21:59 ] |
Post subject: | Re: Strategy Testing Speed |
BTW - it seems like Dukascopy developers enhanced something in terms of backtesting performance. I vaguely remember 1 year ago it took around 5 minutes to play 1 year of "Empty" strategy (EUR/USD) - on pre-cached data. Now it takes around 1:20 minutes (i5-9600K). Roughly - 4 times quicker. Good improvement ... though I still expect it should take less than 20-30 seconds for that dumb thing - so there is still plenty space for improvement . |
Author: | JP7 [ Sat 24 Aug, 2019, 03:26 ] | ||
Post subject: | Re: Strategy Testing Speed | ||
OK - thank you ! "On windows cached tick data is stored here C:\Users\<UserName>\AppData\Local\JForex\.cache." Now the next test is to copy cash files from the old computer to a cleaned one to see if works and if it is faster. My Cache is 80 GB Test confirmed - coping the cache from the old computer to the new one works and computer run faster and uses tick data I have copied from the old computer
|
Author: | goose_ [ Sat 31 Aug, 2019, 16:57 ] |
Post subject: | Re: Strategy Testing Speed |
JP7 wrote: Once a strategy test passed sluggish for the set interval (15 years in max interval for EUR/USD) then becomes faster next time for the same currency pair and same interval. I did this test on two computers with the same configuration and the one used before to run strategies is faster than the other one who just started running strategies. Then I did a HDD clone from the comp who run faster to the other one and booth computers run faster (same fast speed) for the same tests. You tracked how many process was running on this machine. Maybe operating system (which means CPU and scheduler / context of tasks) skip to time frame in other moment because on this 2 computers probably works different task / different programs at the same time. And frame for execute java is different for each one. Maybe check stuff like this and track performance. You slow down program creating very long loop 10 x 1.000.000.000 Of course try this one in historical tester... package jforex; I have Intel i3, 6GB RAM and my average time is something like 852 for this loop settings loop 10 x 1.000.000.000 for loop 1 x 1.000.000.000 something between 66-90 Without visual mode* Linux Ubuntu 14.04 LTS. And then open web browser like firefox, opera sth like that and few tabs on Youtube ;p and check this out... what performance is now. Probably cpu work harder and this means this strategy slow down. And do few tests with this.. You will see different value of "average_time" . btw. this is a deep topic for analysis... there are many variables, many ways, many things that affect performance, but my knowledge about this is very... poor |
Author: | JP7 [ Thu 05 Sep, 2019, 19:02 ] | |||
Post subject: | Re: Strategy Testing Speed | |||
My cache is now 120 GB and there is no room to the laptop HDD so I am moving it to an external HDD and change the cache path in the platform: Settings - Preference - Advanced - Paths I intend to keep the cache on an external HDD and move it from a computer to another ! As external HDD will accumulate cache data and as "growing larger" will not interfere with the internal operation system hard drive. Data cache for each instrument for the max period available and processing all ticks is huge: examples EURGBP 5.39 GB, EURJPY = 7.94 GB, EURCAD = 6.33 GB, EURAUD 6.79 GB and so on.
|
Author: | JP7 [ Fri 06 Sep, 2019, 18:14 ] | ||
Post subject: | Re: Strategy Testing Speed | ||
Test Confirmed - platform uses now data cache from the external HDD
|
Page 1 of 1 |