Could someone give me a tip as to how the incoming "data rate" of incoming IStrategy.onTick events should be "paced" or timed to achieve a near realtime data rate?

I can think of several approaches to this, such as controlled delays within the onTick callbacks, to correct for any variations in the data rate, as the historical data is "driven" into the Strategy.

I'm new to using ITesterClient, so maybe there is an API element which is able to control this pacing fairly close to a realtime data rate?

Any tips are appreciated !

HyperScalper

For now, I'll build a buffer, switch incoming data on and off to control the flow, and use my own timebase to pump data at a controlled rate.

I'll capture the data in onTick, and then pump it into my own onTickImpl processing routine at a controllable rate.

This may result in some small inconsistencies with charting, but these should be minor.

As a Feature Request, it would be nice to have access to an *accurate*, high thread priority, rate control which would make it unnecessary for me to create my own controlled throttling. I notice that Historical Tester in the JForex platform has a slider control, but it's not that "smoothly" controllable.

I realize that for most backtesting, only an approximate rate control is needed.

HyperScalper

CriticalSection, thank you for your contribution to a solution.

I am not sure but maybe a simple Thread.sleep(...) approach as you've outlined it, would
be one solution. Mine is a bit more complex.

I have a tick buffer processor, and I was just going to do "flow control" by switching the incoming data stream ON or OFF as needed to maintain my buffer.

Then I would simply clock the buffered <Instrument,ITick> data (each would have its given timestamp, I'm not interested in the onBar callbacks)
and then just consume them from the buffer at my desired "playback rate".

This technique simply captures ALL of the data incoming on each IStrategy.onTick event
and queues it all up, while managing the "flow control".

Then the queue is drained into some "onTickImpl" private processing routine which
is driven by some Callable<Void> execution packet, invoked by the Queue Consumer
process on the Virtual time line, the desired "playback rate".

HyperScalper

P.S. My snag was that I have to delete JForex charts, as they are not currently supported
in the ITesterClient context. But I think my "flow control" idea will work fairly well. The
objective here is to "trade against" the incoming historical data, etc.

By the way, jlongo in this forum has provided MyOfflineTrades (search for it), but this
executes in the JForex platform in the Historical Tester. So its data rate is controlled
by the Historical Tester's "slider" control.

In my case, because it is standalone code, I can't use the JForex charting, and I don't have
any "fine control" over the incoming historical data. HENCE, my idea of simply buffering it
and doing "flow control" while driving the buffered data into my onTickImpl routine
at my own Controlled Rate.

Hey HyperScalper, when I read this line above I homed in on the 'realtime' requirement presuming you had a desire to have the Tester relay it's historical data to your onTick() with a timing profile similar to that which it would receive had the incoming data been generated in realtime by the quotes server.

Compounding this assumption was another that said your onTick() completes all of it's calculations within the same (stretegy) thread and within the call made to onTick().

My thinking then was that with these assumptions holding, the only thing preventing the realtime consumption of the historical data is the fact that the Tester is spitting the data out as fast as it can, paying no attention to the timestamps contained on each tick. As the ticks are produced in order and are timestamped in milliseconds, all that remains to produce a realtime effect are delays in between the onTicks() that honour the deltas between the timestamps.

The Tester presumes activities related to onTick() are concluded before onTick() returns.

Where onTick() related activies occur in another thread, unlike the situation in a DEMO/LIVE run, when we're using the Tester
there's an opportunity to go ahead and pause the thread onTick() is running in without any adverse impact as until onTick() returns
the Tester is literally stuck in time. Nothing else can happen.

So consider this simplified approach to tick processing in another thread:

MyTickThread tickThread = new MyTickThread();

onTick(ITick tick)
{
   //defer processing of this tick to another thread
   if(tickThread.busy() == false)
   {
      tickThread.update(tick);      //let the thread know about the current tick
      tickThread.setBusy(true);   //the thread now has work
      tickThread.notify();  //tell it to wake up by notifying the monitor object it is waiting on
   }
   else System.out.println("TICK SKIPPED AS TICK THREAD IS BUSY");   <-----
 
   //return here back to Tester which is free to send in another tick unopposed

   //or Thread.sleep() to control rate
}

Class MyTickThread
{
   boolean busy;
   ITick worktick;
   
   void update(Itick t)
   {
      worktick = t;
   }

   void setBusy(boolean b)
   {
      busy = b;
   }

  //separate thread
   void run()
   {
     while(true)
     {
       while(busy == false)
         this.wait();

      //new work is in - work with the worktick that came in and do all the work necessary

      //we've finished, when we loop around we will enter wait state
      busy = false;
      }
   }
}

The overall problem here is that we are open to the problem of our thread being busy for a long time and causing ticks entering onTick() to be missed.

During DEMO/LIVE that's a situation the strategy writer must address depending on the strategy but at least they have the (real) time to.

During Testing what can happen as the tester is spitting it's data out without reference to (real) time, is that the tick thread is notified but until it gets to run
subsequent onTick() ticks go missing as the tick thread is busy. Or the tick thread gets its time quite alright but it's CPU timesliced with the strategy thread which proceeds to call onTick() which again meets a busy tick thread.

Either way the Tester is spitting out data that isn't in realtime so backtesting with threads can be an issue depending on how important the missed ticks are.
In realtime you miss maybe 5 ticks, but during backtesting potentially a lot more as they're are flying out of the tester.

One way to address this is to pause the Tester while the tickthread is running, this allows the tick thread to process every tick coming in:

Class MyWaitMonitor {};
MyWaitMinotor mwm = new MyWaitMinotor();

onTick(ITick tick)
{
   //defer processing of this tick to another thread
   if(tickThread.busy() == false)
   {
      tickThread.update(tick);      //let the thread know about the current tick
      tickThread.setBusy(true);   //the thread now has work
      tickThread.notify();  //tell it to wake up by notifying the monitor object it is waiting on
   }
 
   //Dont return here back to Tester which is free to send in another tick unopposed
   //instead wait for the tick thread to finish and notify us using the MyWaitMonitor object
   mwm.wait();

   //and still use Thread.sleep() to control rate of onTick()

   //overall effect is no ticks are missed by our tick thread and the rate of the ticks are still controlled
}

  //separate thread
   void run()
   {
      while(true)
      {
         while(busy == false)
           this.wait();

      //new work is in
      //work with the tick that came in and do all the work necessary

      //we've finished, when we loop around we will enter wait state
      busy = false;

      //notify onTick() we have finished
      mwm.notify();
      }
   }

Now in this version, during Testing we have another thread but it has the ability to notify ontick() when it has finished so onTick() can return to give us
the next tick. NOTE: that the MyWaitMonitor/mwm should only be used during Testing to support the testing of onTick() calls that rely on other threads in a way that doesn't allow the Tester to run away from us. and in fact, everything written here is from the perspective of the Tester only.

SO WHAT'S THE POINT???

Well, you mention 'Buffer'. Which implies you aim to deal with some kind of inherent decoupling such as that which occurs when you attempt to use another thread inside onTick() while testing as above. Since the tick thread can miss subsequent threads to onTick(), sure a need could arise to buffer those subsequent ticks.

But during Testing, using a wait()/notify() allows those misses to disappear and therefore no more decoupling between onTick() and the tick thread to exist.

You want to buffer the ticks entering onTick(), of course there are a lot of them, potentially as many ticks as appear between ur TEST start time/date and TEST end time/date in the case that they are not consumed from your buffer. So either the Tester is paused by blocking/sleeping inside onTick() (what u call "flow control") or the buffer is pre-calculated for the worst case.

Ok so you have the buffered ticks, you want to "pace" them.

Any pacing not concerning the actual time of the ticks only requires a simple sleep() of any variable amount that achieves the desired effect.

But without reference to the tick times no effect will appear 'realtime' as the tester will send T1 and T2 which are 1 minute apart to onTick() with two calls that are only milliseconds apart.

So if you have say 100 ticks in a buffer. You want to relay them to your strategy at some pace. Each of these 100 ticks has their own time and with respect to the tick before it a delta - difference between tickTime(i) and tickTime(i-1). So perhaps you want the ticks to be replayed every 1 SECOND.

On one hand this is impossible as the ticks are already timestamped and T1 may be 01:23:00.000 and T2 01:23:00.031 and T3 01:23:00.055 so that's 3 ticks occuring in under 1 second. Replaying these ticks back every 1 second wouldn't be realtime but might/would be at a pace one can better cope with.
Realtime playback would require a delay of 31ms after T1, 24ms after T2 and so on.

On the otherhand with the above three ticks in a buffer, one has the option to change their times (directly or by cloning them) and forcing them to be 1 SECOND apart each but by then we're not dealing with accurate market data anymore as altering timebases alters everything relying on them from charts to indicators.

So with the option to pause the Tester in it's tracks with no adverse side affects while using another thread or not - simply by holding up onTick(), I didn't begin to consider the problem with the need for a buffer.

And with the only thing preventing realtime playback of ticks being the deltas between the ticks, I figured those could be slept away before allowing the next tick to come in.

So realtime would be Thread.sleep (difference).

Twice the speed of realtime would be Thread.sleep (difference / 2) - sleep for half the time

Twice as slow the realtime speed would be Thread.sleep (difference * 2) = sleep twice as long

the multiplier we can generalise as a variable 'rate' in Thread.sleep (difference * rate) where the twice as fast rate=0.5 and twice as slow rate=2

Using a virtual timeline is fine, you can even rebase the tick times as part of this virtual timeline but what matters above all is the tick time deltas not the actual times. As long as we know T2 is 31ms ahead of T1 and that T3 is 24ms ahead of T2, we can produce realtme playback without any adverse effects (as long as the charts/indicators work with the rebased times or as long as the original tick times are not touched which the standard price chart presumes - it's busy plotting the ticks that are being sent to onTick() using their actual tick.getTime() values)

If the simple aim is to say, delay onTick() by 1 second, or 5 seconds, 10 seconds or 1 minute, then a sleep() in onTick() achieves that.

So it may be I misunderstood 'realtime' at first but even if so Thread.sleep(difference) would become Thread.sleep(delay) any arbitrary delay to slow things down.

Beyond this, I can see the elegance in your approach but I still wonder if it's just another layer of abstraction just for the sake of it requiring the headache of flow control and a virtual timeline.

Again, I really might not fully understand the effect you are aiming for and further, I'm not privy to your codebase and the complexity in its structure.

oh and re. jlongo's app, it too can be easily modified as above with the Tester slider set to the maximum for best result.

The real question I guess though is, what is that slider doing in terms of method calls to affect the tester and are those methods available via API.

But I wouldn't be surprised if the slider is simply calling Thread.sleep() before making a call to the running strategy's onTick().

Cheers.

Sorry for the Length

Hi CriticalSection,

Yes, that was a long series of mostly good suggestions. I'll have to study them.

Specifically, you do NOT want to "use another thread" inside of the onTick dispatch callback, not at all, nor would I use any "wait loops". I'm sure you know all of that. You just want to capture and queue up the data for some other Consumer type thread to work on later.

If anybody wants to know how to buffer onTick incoming data, free up the calling thread, and then re-dispatch later, I could lend you some code.

I do it ALL the time so that I never miss a single onTick's data, including the full Depth of Market which is the main thing I use for Scalping Trend Following (yes, I know it's unfortunately not available from historical data). It's just a simple Producer/Consumer problem in Java, but it's important always to keep in mind that any API operations may require implementation in a Callable<?> and be dispatched and serialized of course, via IContext.executeTask(Callable), **especially** if they are Order Entry operations.

To me, threading is just second nature, but it can get you in trouble, unless you are strictly aware of the required thread contexts for API work, for Swing GUI work, and just threading in general.

As for replaying data on a timebase, you just need some Quantum of time in a high priority TimeBase type of thread as your clock, and then play data back by timestamp as closely as possible to what would have been the original data rate, given the JVM's ability to do so. Any "jitter" (faster or slower) should also be corrected for, so that the playback throughput is as even as can be achieved in a "soft realtime" system such as Windoze.

I will study your various suggestions. I may be able to hack something together this weekend and do some offline scalping work, "a la jlongo" and his very nice MyOfflineTrades facility. I'll report what I come up with...

HyperScalper.