This is a problem with rather broad scope. I hope you can give me some advice.

I have a number of strategies that, when started, need some time in the market to achieve "steady state", before they can start trading. To speed up the process of starting them, I am streaming a preamble of recent ticks and bars from IHistory in the onStart method.

If I just need to stream ticks from a single currency or bars of a given period, IHistory provides convenient methods for doing this. However, I need to stream all ticks and bars (of all periods) of all subscribed currencies, in the order of their time stamps. (i.e. in the way they would arrive during a live session)

Can you recommend / outline a possible solution?

My currently implemented solution works, but it is quite slow. It obtains each bar (one at a time) for all periods and instruments from IHistory using a recursive algorithm. When it reaches down to the the TEN_SECS bars, it reads the list of ticks (for the 10 seconds) for each instrument, joins them into a single PriorityQueue before streaming it. As you might imagine, this all takes a lot of time. Even if only subscribed to one instrument, it is much slower than the Historical tester.

I'm considering two other solutions:
1. Reading all the required ticks and bars in bulk and then sorting them.
2. Streaming only ticks, and generating the bars from the ticks. But the tick streams from different instruments would still need to be merged.

Any advice or ideas would be greatly appreciated.

Consider subscribing to instruments in onStart method in the following way:

   @Override
   public void onStart(IContext context) throws JFException {
      this.context = context;
      this.console = context.getConsole();
      this.engine = context.getEngine();

      Set<Instrument> instruments = new HashSet<Instrument>();
      instruments.add(Instrument.EURUSD);
      instruments.add(Instrument.AUDUSD);
      context.setSubscribedInstruments(instruments);

      printSubscribed("instruments before subscription");
      
      // wait max 5 seconds for the instruments to get subscribed
      int i = 5;
      while (!context.getSubscribedInstruments().containsAll(instruments)) {
         console.getOut().println(" not subscribed yet");
         try {
            Thread.sleep(1000);
         } catch (InterruptedException e) {
            console.getOut().println(e.getMessage());
         }
         i--;
      }

      printSubscribed("instruments after subscription");
   }
   
   private void printSubscribed(String prefix){
      console.getOut().print(prefix +": ");
      for (Instrument in : context.getSubscribedInstruments()) {
         console.getOut().print( in + ", ");
      }
      console.getOut().println();
   }

Mind that if you are using Historical Tester, you will also need to select those instruments by clicking "+" in Historical Data panel.

Thank you for your reply, but unfortunately my problem was misunderstood.

The problem I am trying to solve is not how to subscribe to multiple instruments, but the following:
How to efficiently merge/stream historical bars and ticks from multiple instruments, chronologically.

I know I can get chronologically sorted bars for each instrument using:
IHistory.getBars / IHistory.readBars

And I can get chronologically sorted ticks for each instrument using:
IHistory.getTicks / IHistory.readTicks

What I need is (a chronologically correct) implementation of
readBarsAndTicksForMultipleInstruments()
if you know what I mean.

I have tried a couple of solutions (see original post), which work, but are very slow. With the provided IHistory API, it seems unavoidable that any solution will have low performance because:
1. If the data is read in small chunks, there will be lots of read operations.
2. If the data is read in large chunks, there will be lots of sorting to do.

I am hoping you may have some advice, since the same problem has been solved (efficiently) by whoever implemented the Historical Tester.

I have a new idea to use ITesterClient to do the streaming more automatically. An outline of the solution is below. I am trying to run this (i.e. the main code fragment) inside onStart() method of a strategy running inside the JForex client.

1. Is the general approach advisable from a framework/threading perspective?
2. The strategy tries to make an "external"/additional connection to the Dukascopy servers (because of the call to connect()). Could that be problematic?

Main code fragment

TesterFactory.getDefaultInstance().connect(jnlp, userName, password);

HistoryStreamer.stream(instruments, time - preambleDuration, time, new PreambleListener() {
    @Override
    public void onPreambleTick(Instrument i, ITick itick) throws JFException {
        // tick handler goes here
    }
    @Override
    public void onPreambleBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException {
        // bar handler goes here
     }
});

PreambleListener

interface PreambleListener {
    void onPreambleTick(Instrument i, ITick itick) throws JFException;
    void onPreambleBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException;
}

HistoryStreamer

class HistoryStreamer {

    public static void stream(Set<Instrument> instruments, long start, long end, final PreambleListener listener) throws JForexException {
        ITesterClient client;
        try {
            client = TesterFactory.getDefaultInstance();
        } catch (Exception ex) {
            throw new JForexException("Failed to get tester client instance: " + ex, ex);
        }

        //wait for it to connect
        int i = 10; //wait max ten seconds
        while (i > 0 && !client.isConnected()) {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException ex) {
                Debug.trace("wu", "Sleep interrupted.");
            }
            i--;
        }
        if (!client.isConnected()) {
            throw new JForexException("Tester client failed to connect Dukascopy servers.");
        }
        client.setSubscribedInstruments(instruments);
        client.setDataInterval(Period.TICK, OfferSide.BID, ITesterClient.InterpolationMethod.OPEN_TICK, start, end);
        //load data
        Debug.trace("Downloading historical data.");
        Future<?> future = client.downloadData(null);
        try {
            //wait for downloading to complete
            future.get();
        } catch (InterruptedException ex) {
            throw new JForexException("Failed to get historical data: " + ex, ex);
        } catch (ExecutionException ex) {
            throw new JForexException("Failed to get historical data: " + ex, ex);
        }
        Debug.trace("Streaming historical data.");
        final AtomicBoolean completed = new AtomicBoolean(false);
        client.startStrategy(new IStrategy(){
            @Override
            public void onTick(Instrument i, ITick itick) throws JFException {
                listener.onPreambleTick(i, itick);
            }
            @Override
            public void onBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException {
                listener.onPreambleBar(i, period, ibar, ibar1);
            }
            @Override
            public void onMessage(IMessage im) throws JFException {
            }
            @Override
            public void onAccount(IAccount ia) throws JFException {
            }
            @Override
            public void onStart(IContext ic) throws JFException {
            }
            @Override
            public void onStop() throws JFException {
                if (!completed.compareAndSet(false, true)) {
                    Debug.assertFail();
                }
                synchronized (completed) {
                    completed.notifyAll();
                }
            }
        });

        synchronized (completed) {
            while (!completed.get()) {
                try {
                    completed.wait();
                } catch (InterruptedException ex) {
                }
            }
        }
    }
}

Note: For brevity, some irrelevent dependencies such as (Debug and JForexException, which extends JFException) have not been included.

I've tried to run the solution in my last post, but the client platform doesn't seem to allow making the additional connection with TesterFactory.getDefaultInstance().connect(). I hope you can give me some advice.

See the last post in
viewtopic.php?f=61&t=35436&p=45925&hilit=process#p45925
If this is still an active issue - Historical tester does not intertwine bars and ticks in the same list, it downloads data to local machine and furtheron works with that as cache (You can see the cache location when you run the ITesterClient.downloadData method). Obviously, from IStrategy anything similar to this is not possible, since you cannot arbitrary create ITick and IBar objects. So there is nothing close to IHistory.getReadBarsAndTicksForMultipleInstruments implemented yet.

Thank you for your response.
Ok, so I understand that each process can have exactly one ITesterClient object, given by TesterFactory.getDefaultInstance(). I suppose this is also true of the JForex client platform process. So my strategy (running in the JForex client platform) is trying to make use of that instance.

But:
1. TesterFactory.getDefaultInstance().isConnected() returns false.
2. TesterFactory.getDefaultInstance().connect(...) throws an exception.

Then is it even possible to use ITesterClient in this situation (from a strategy running in the client platform)? Should I perhaps be calling reconnect() rather than connect(), since the platform has already authenticated?

Essentially all the time you work with the same IClient instance - it shouldn't be a problem to call connect on a client who has already connected, maybe you had missed some nuance. We managed to successfully implement a strategy from snippets that you had provided (for us it worked both from API (called from IClient.startStrategy) and from client):

package jforex.history;

import java.text.SimpleDateFormat;
import java.util.HashSet;
import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicBoolean;

import com.dukascopy.api.*;
import com.dukascopy.api.system.ITesterClient;
import com.dukascopy.api.system.TesterFactory;

@RequiresFullAccess
public class HistoryUsage implements IStrategy {
    
    private static String jnlp = "https://www.dukascopy.com/client/demo/jclient/jforex.jnlp";
    private static String userName = "username";
    private static String password = "password";
    
    private IConsole console;
    private SimpleDateFormat sdf;

    @Override
    public void onStart(IContext context) throws JFException {
        
        sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss") {{setTimeZone(TimeZone.getTimeZone("GMT"));}};
        
        console = context.getConsole();
        
        print("start " + this.getClass().getSimpleName());
        
        try {
            TesterFactory.getDefaultInstance().connect(jnlp, userName, password);
        } catch (Exception e1) {
            printErr(e1);
        }
         
        Set<Instrument> instruments = new HashSet<Instrument>();
        instruments.add(Instrument.EURUSD);
        
        int i = 5;
        while (!context.getSubscribedInstruments().containsAll(instruments)) {
            print(" not subscribed yet");
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                print(e.getMessage());
            }
            i--;
        }
        
        long time = System.currentTimeMillis(); 
        long preambleDuration = Period.DAILY.getInterval();
        
        HistoryStreamer historyStreamer = new HistoryStreamer();
        try {
            historyStreamer.stream(instruments, time - preambleDuration, time, new PreambleListener() {
                @Override
                public void onPreambleTick(Instrument i, ITick itick) throws JFException {
                    print("TICK ARRIVED " + sdf.format(itick.getTime()));
                }
                @Override
                public void onPreambleBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException {
                    print("BAR ARRIVED " + sdf.format(ibar.getTime()) + " " + period);
                 }
            });
        } catch (Exception e) {
            printErr(e);
        }

    }

    private void print(Object o){
        //to easily distinct from logger statements in case if ran from IDE
        console.getOut().println("_________________" + o);
    }
    
    private void printErr(Object o){
        console.getErr().println(o);
    }
    
    public void onAccount(IAccount account) throws JFException {    }
    public void onMessage(IMessage message) throws JFException {    }
    public void onStop() throws JFException {    }
    public void onTick(Instrument instrument, ITick tick) throws JFException {    
        print("onTick: " + sdf.format(tick.getTime()) );
    }
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {    }
    
    interface PreambleListener {
        void onPreambleTick(Instrument i, ITick itick) throws JFException;
        void onPreambleBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException;
    }
    
    class HistoryStreamer {
         
        public void stream(Set<Instrument> instruments, long start, long end, final PreambleListener listener) throws Exception {
            ITesterClient client;
            try {
                client = TesterFactory.getDefaultInstance();
            } catch (Exception ex) {
                throw new Exception("Failed to get tester client instance: " + ex, ex);
            }
     
            //wait for it to connect
            int i = 10; //wait max ten seconds
            while (i > 0 && !client.isConnected()) {
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException ex) {
                    //Debug.trace("wu", "Sleep interrupted.");
                }
                i--;
            }
            if (!client.isConnected()) {
                throw new Exception("Tester client failed to connect Dukascopy servers.");
            }
            client.setSubscribedInstruments(instruments);
            client.setDataInterval(Period.TICK, OfferSide.BID, ITesterClient.InterpolationMethod.OPEN_TICK, start, end);
            //load data
            //Debug.trace("Downloading historical data.");
            Future<?> future = client.downloadData(null);
            try {
                //wait for downloading to complete
                future.get();
            } catch (InterruptedException ex) {
                throw new Exception("Failed to get historical data: " + ex, ex);
            } catch (ExecutionException ex) {
                throw new Exception("Failed to get historical data: " + ex, ex);
            }
            //Debug.trace("Streaming historical data.");
            final AtomicBoolean completed = new AtomicBoolean(false);
            client.startStrategy(new IStrategy(){
                @Override
                public void onTick(Instrument i, ITick itick) throws JFException {
                    listener.onPreambleTick(i, itick);
                }
                @Override
                public void onBar(Instrument i, Period period, IBar ibar, IBar ibar1) throws JFException {
                    listener.onPreambleBar(i, period, ibar, ibar1);
                }
                @Override
                public void onMessage(IMessage im) throws JFException {
                }
                @Override
                public void onAccount(IAccount ia) throws JFException {
                }
                @Override
                public void onStart(IContext ic) throws JFException {
                }
                @Override
                public void onStop() throws JFException {
                    if (!completed.compareAndSet(false, true)) {
                        //Debug.assertFail();
                    }
                    synchronized (completed) {
                        completed.notifyAll();
                    }
                }
            });
     
            synchronized (completed) {
                while (!completed.get()) {
                    try {
                        completed.wait();
                    } catch (InterruptedException ex) {
                    }
                }
            }
        }
    }

}

Thank you!
After looking at your code, I added @RequiresFullAccess and now mine works. connect() no longer throws java.security.AccessControlException: access denied (java.util.PropertyPermission auth.ssl.ignore.errors read)

In all my strategies so far, I have taken pains to avoid @RequiresFullAccess. I guess it is unavoidable here?

As of JForex wiki:
Method IContext.getFilesDir() returns path to the directory where read/write operations are allowed. Use @RequiresFullAccess annotation whenever you need to write/read files outside this directory.
So presumably java keeps its properties in different directory from the one that is returned by IContext.getFilesDir().

Admittedly this is not a train smash, but it would be nice if ITesterClient.connect() would not need @RequiresFullAccess.
Surely other API calls (I imagine for example IHistory.readTicks()) also access files (ouside context.getFilesDir()) but do not need @RequiresFullAccess.

Is it
"Use @RequiresFullAccess annotation whenever you need to write/read files outside this directory."
or is it
"Use @RequiresFullAccess annotation whenever you or JForex might need to write/read files outside this directory."
?

By the way, your support has been splendid.

If java itself (like reading java security settings this time) implies JForex then it is the second. Also note that ITesterClient was not originally designed for being run from JForex client (if you run your strategy from API then it should work without @RequiresFullAccess).
If you look for ways to avoid using @RequiresFullAccess you might consider browsing for you exception here:
https://java.sun.com/developer/onlineTra ... urity.html

Yes, it worked in the client for a few days until the new platform version was released (build 402 I think), which broke your code and mine. It now throws the exception regardless of @RequiresFullAccess:
java.security.AccessControlException: access denied (java.util.PropertyPermission auth.ssl.ignore.errors read)
What can I do?

NOTE: My username is slightly different because for some reason I can no longer log in to the forum as of Friday 17 June evening.

We are currently working on it and the fix will be available with the next relaease.

Great
While you are at it, please consider doing away altogether with @RequiresFullAccess for ITesterClient.connect(). As far as the user is concerned, the "Full Access" is happening on your side of the JF API.

We took another look at it and the answer stays the same - ITesterClient is not designed for being run from JForex client. Meaning that it is rather an unintended feature for ITesterClient to be available from JForex client, not that it is bug that it can't be used in JForex client without @RequiresFullAccess.

Thank you for looking into it. Will this unintended feature continue working in future versions?

It is working well at the moment with the rolled back client version, but before it was rolled back...
I am relying on this

Yes.