The timers for the tester engine will be available in a next DEMO platform version.

I'm having similar issue, please check it out: https://www.dukascopy.com/swiss/english/forex/jforex/forum/viewtopic.php?f=5&t=3409

Dear support team,

may I ask you for an updated status on this issue?

a) Have you meanwhile been able to reproduce the issue for example through oktar's sample code?
b) Is a logging now included in the new version JF2.2.6?

Thanks a lot,
RR.

Please take a look at Processing stat. parameter in Historical Tester.

I think I may know the cause of your slow tests issue, though I [saddly] do not have a workaround.

Are you running tests over a very long timespan, eg, more than 6 months; something on the order of 1 year or more?

We've been getting things setup to do large-scale strategy testing using Amazon's EC2 service, but we do not wish to overburden Dukascopy's servers with 2,000 historical tester instances all trying to download historical data to populate their disk cache. Instead, I setup a simple strategy that doesn't really do anything and set it to cache 1 years worth of data for about 7 or 8 pairs, and I populated our amazon AMI image with that cache.

When a given pair has very little data in its disk cache, I could download at a rate of around 1mb/sec. However, the download speed seemed to taper off quite dramatically. I didn't take any statistics to verify this, but it appeared to taper off by a rational power law, logarithmically, or something similar. By the time I had 600mb of data for EUR/USD, it was taking around 1 min / mb of data. Around 800mb it was taking around 5 minutes.

Assuming I'm right about your test covering a large timespan, I haven't delved into it to figure out where the lost time is. It could be something as simple as switching to a filesystem that can do random seeks for many small files very quickly (eg, ReiserFS) -- assuming you're in a *nix environment, or it may be a limitation in either Java or the algorithm Dukas is using to search through the cache to see what needs to be downloaded.

-Brian

We don't search through the cache, we always know where file should be. If we need EUR/USD 2010-07-09 15:00 ticks, then we already know the file name with full path - EUR/USD\2010\07\09\15h_ticks.bi5. If we need some specific tick at 15:25, then it involves some searching inside file to find the correct time, but for candles we also know the exact place in file where any candle resides.
Maybe some filesystems don't work good with many small files, try to run disk defragmenter on it.

Hi Brian,

did you check the memory heap?
We've seen that it's running full over time and this can cause huge performance issues with some of the strategies in longer runs.

You might want to try increasing the heap size.
In addition we hold the DK-cache in a RAM-disk. This helps gaining performance, though performance gains are not that huge (compared to having the cache on a SSD).

And not adding any indicators to the chart while performing a tester run helps too.
Some of the (built-in) indicators can really drag the performance of a run down.

Have a good day,
RR.

Support:

Well, defragmenting isn't likely to help in this instance. Fragmentation only occurs when there are many writes and deletes over time in a given region of a disk. I haven't been deleting anything from my cache, just running my strategy tester to populate the disk cache.

RR, Support:

I setup a script that splits up everything from jan1-2000 through the present day into 6 month time slices -- that is, there's a separate cache directory for each currency pair, for each 6 month slice. Split up like this, I was able to maintain a very rapid download speed, then recombine them after all the downloads complete.

Doing this, the download speeds were improved tremendously. I've also noticed that having a full disk cache (2005 - present) doesn't seem to affect performance of the strategy itself, provided it does not need to download anymore data.

Our main problem now is logistics of how to get what we need on our EC2 instances, but that's not an issue with the API fortunately.

RR:

With our code, I've found a min/max heap size of 512mb seems to work rather well. We kept running into issues with memory (EC2 doesn't give you swap drives, so swap space is all in ram), but we found a way around that and it drastically improved load avg on the instances we were spawning.

-Brian

RR:

As another followup to what I said earlier, the ramdisk thing isn't really an option. We could use more expensive EC2 instances with more ram, but just creating a swap file is a simpler solution that works well.

I'm fairly sure it's the filesystem, but my partner, who is quite a *nix guru, says the FS we're using should be one of the best for this purpose.

Our strategy testing is being performed en-masse from a CLI tool we've created, so charts & indicators on the charts aren't really a concern; yet, we encountered the same speed issues you described, and it would appear it had something to do with the time it takes to populate the disk cache.

-Brian

Hi Brian,

thanks for your feedback.
I've never used Amazon's EC2, but it sounds appealing for bulk backtestings.
Would you be willing to share your experiences with EC2? I would be interesting to "get a feeling" for the service.
a) How many backtesting instances do you normally run parallel?
b) What subscription and services do you use (and what's the range of costs for a month)?
c) What are your experiences regarding performance of the EC2 services?
d) Did you see the same performance issues when running the backtesting on a local desktop PC?
e) How long does an EC2 run of your strategy now last for the first 12 months, for the first 24 months of backtesting data?
f) How satisfied are you with the EC2 service overall?

Any info would be very much appreciated.
Thanks & have a good day,
RR.

I'd be happy to. Sorry for the long delay, I haven't been checking these boards regularly latley.



Per machine, this depends on available resources. Amazon's servers have a variety of machines available. Some have as few as 2 cores per machine with a small amount of disk space & ram, whereas others have as much as 16-32gb (or possibly more) of RAM, and others have 8 cores. They may have recently added 16 core machines as well, but the price per hour on the new machines is not in line with their predecessors.

As a sidenote, disk space is extremely cheap to make use of, as is bandwidth -- as long as the data you transmit/receive goes between machines in the same region. If some are on the east coast, some on the west coast, the bandwidth costs more.

The short answer is: 1 instance per core, unless I use the new modifications to the API that allows multiple strategies to run in parallel. On a 2 core machine I was able to run as many as 16 threads simultaneously and I still saw a performance improvement over 12 threads, which saw an improvement over 2 threads. This improvement is far less dramatic the more threads you add. As an example, going from 12 to 16 threads decreased the average runtime from 5.5 seconds to about 5.25 seconds, whereas going from 2 to 12 dropped it from around 15-20s to around 5.5s.

The drawback is, the parallel strategies thing only allows you to run parallel strategies over the same time period. Each of our tests runs a single days worth of testing, across many permutations of parameters; but it does so across many days. Fortunately our setup still allows us to take advantage of it, but it is a little limiting.



I'd recommend just looking it up. It's a fairly complex pricing structure. You can bulk purchase access to hardware, or order it on a per-hour basis. If you do the latter, it's not at all unlike the commodities market. You offer a bid price for an instance, and if there's unused hardware you'll get an instance rather quickly. If you outbid someone else that's currently using hardware, their instance will shutdown and shortly thereafter given to you.



I'm not at all disappointed with the performance of their hardware. The main performance issues I've seen were either caused by my hand; the dukas API's approach to storing cache; and by some weird performance issue that crops up when a margin call occurs.

On average my tests take less than 10-20s to complete, but some would run for 60-90 minutes stuck on one test, and the only clue I have so far is it occurs at the same time as a margin call, and it appears my code is waiting for something to happen in the API.

I am a little annoyed Amazon doesn't allow you to do broadcast UDP packets, that would help significantly with some AI work I've been working on, but I can see why they don't make it available. Thankfully you get 10 gig ethernet connections between instances, so that helps mitigate the throughput issues tremendously.



Yes.



In the last test I ran, there were over 6 million permutations to run through across 3.5 years of data. On average, a single test took less than 20 seconds to complete (most less than 7, I'm guessing, but I don't have numbers in front of me). It took about a week, partly because we kept getting outbid. Occasionally someone would place an abnormally large bid and take most if not all our instances away and we'd have to wait for the price to return to normal.



Very. Well worth the time investment.

-Brian

Hi Brian,

thanks a lot for your detailled information.
I'll definitely give the Amazon service a closer look!

All the best,
RR.