Dear Support,
Or anyone else with some experience on the subject.

I am in the planning phase of a strategy that is operating mainly in the onTick() function. If the order conditions are met, the strategy will submit orders in the onTick() function. After order submitting an order merge can occur, if there is another open position (with the same direction: short/long).
As order merging could halt execution for some time (seconds?) (because of IOrder.waitForUpdate), I have to consider that some ticks will be skipped, as it is nicely described in this wiki page.

Would that be a safer implementation (to avoid ticks to be skipped) if the strategy would only create orders in the onTick() function, and in a separate thread (based on scheduled asynchronous execution) it would do the merging?
With this there would be no ticks skipped, and by scheduling the merge-task to be executed for every 5 seconds, I have a quite reasonable follow-up with merging.

Or another idea is to implement merging on onMessage(). Whenever an order is filled, and onMessage() is called with the proper message, I would check in the onMessage() function if there is another order that could be merged.
But I am not sure what will exactly happen in the strategy that halts onMessage() execution. If onMessage() is being halted for let's say a second (due to IOrder.waitForUpdate), will onTick() be called whenever a new tick arrives? Or no, the complete strategy will be halting and no other function (ontick(), onBar()) will be called till onMessage() finishes.


Here are the above mentioned ideas, as a kind of flow-chart:
1)
new tick arrives, onTick() is called.
implement in onTick():
- if order conditions are met, submit an order (with TP and SL levels)
- check if we can merge. if yes, merge (and set TP and SL levels)

2)
new tick arrives, onTick() is called.
implement in onTick():
- if order conditions are met, check if we can merge. if yes, a). if not, b)
- a) submit order without TP and SL. merge (and set TP and SL levels)
- b) submit order (with TP and SL levels)

3)
new tick arrives, onTick() is called.
implement in onTick():
- if order conditions are met, submit an order (with TP and SL levels)
implement somewhere else (async. scheduler?, onMessage()?):
- check if we can merge. if yes, merge (and set TP and SL levels)




I would appreciate any tips on this manner. I am still in the planning phase, so it would be better to tackle to issue properly, instead of trying to fix a not working implementation.


Thanks in advance.

Do you really need to wait for update after the merge request? Do you need to change the merged position right after its merge success? If skipping ticks is a concern as an alternative you can also consider handling the IMessage.Type.ORDERS_MERGE_OK message in onMessage.
Using onMessage makes more sense, since you will execute your logic only when an actual fill has happened.

You can avoid using the waitForUpdate by using onMessage and java.util.Map:

1. new tick arrives, onTick is called, if order conditions are met, submit an order.
2. in onMessage handle the ORDER_FILL_OK - if can merge - remove the order SL an TP and merge it to the corresponding position. If you need to save the removed SL/TP consider storing them in two java.util.Map<IOrder, Double>'s.
3. in onMessage handle the ORDER_MERGE_OK - set the SL/TP, if necessary fetch the values from the Map's of the previous point and afterwards remove them.

There are multiple examples in wiki that use the Map class, for instance:
https://www.dukascopy.com/wiki/#Manage_Order_State/Resubmit_order
https://www.dukascopy.com/wiki/#Merge_Positions/Limit_merge_count_per_position

Thanks for the super-fast replies,

And sorry for submiting-editing-re-submitting .


No, not at all. I only call waitForUpdate if I remove TP and/or SL of an order, before merge. Isn't that necessary?


Could you explain what will happen if the onMessage() execution is halted (due to waitForUpdate)? Is onMessage() is being called/executed in a separate thread, so onTick() execution doesnt/wont rely on finishing onMessage() first?


I had the impression that the waitForUpdate call is necessary during merges, as we have to make sure that the order is in the proper state. For example, if we want to merge positions where TP and/or SL is filled, we have to remove it and call waitForUpdate to make sure the remove was 'registered'.
How is this avoidable with java.util.Map? Is that not only a container/array of my orders? What does it have to do with calling/not calling waitForUpdate? Could you explain a bit more detailed?

I've created a little test strategy to check if a halt in onMessage() is stopping/halting onTick execution.

package jforex;

import java.util.*;

import com.dukascopy.api.*;
import com.dukascopy.api.IEngine.OrderCommand;

public class onMessageHalt implements IStrategy {
    private IEngine engine;
    private IConsole console;
    private IHistory history;
    private IContext context;
    private IIndicators indicators;
    private IUserInterface userInterface;

    @Configurable("Instrument")
    public Instrument selectedInstrument = Instrument.HKDJPY;
    @Configurable("Period1")
    public Period selectedPeriod1 = Period.TEN_SECS;

    public void onStart(IContext context) throws JFException {
        this.engine = context.getEngine();
        this.console = context.getConsole();
        this.history = context.getHistory();
        this.context = context;
        this.indicators = context.getIndicators();
        this.userInterface = context.getUserInterface();
    }

    public void onAccount(IAccount account) throws JFException {
    }

    public void onMessage(IMessage message) throws JFException {
        switch (message.getType()) {
            case ORDER_FILL_OK:
                console.getOut().println("onMessage called. Sleep starting");
                IOrder anOrder = message.getOrder();
                anOrder.waitForUpdate(4000);
                console.getOut().println("Sleep finished.");
                break;
        }
    }

    public void onStop() throws JFException {
    }

    public void onTick(Instrument instrument, ITick tick) throws JFException {
        console.getOut().println("onTick called");
    }
    
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
        if ( (instrument.equals(selectedInstrument)) && (period.equals(selectedPeriod1))) {
            engine.submitOrder("_onMessageTest" + (System.currentTimeMillis()/1000), instrument, OrderCommand.BUY, 0.001, 0.0, 0.0);
        }
    }
}

And running this strategy shows that onTick() execution is halted while we are in a wait/sleep state in onMessage(). Probably it was obvious for you (Support), but I wanted to test it.
So the question is still open:
How to avoid a tick not being processed because of waitForUpdate command (or any other operation that takes more time).


Edit: Just found in the javadoc:

So it was obvious .

Sorry to continue replying myself, but I have time now...

Let me conclude what is my problem:
During merge, in the preparation phase I have to clear existing TP and SL prices of orders. Based on some wiki examples, this pre-merge phase is executing waitForUpdate(2000);
This 2 second halt is too much. How can I workaround this problem?

- Placing merging in the onMessage() saves nothing, as waitForUpdate(2000) in onMessage() also halts onTick() execution.
- How about placing the merging in a different thread, executed from time to time (every 2-4 seconds?)? Is it safe to place merging out from strategy context (if a different thread means different context at all)? Or would that be a better approach to only start this other thread whenever an order is submitted? As there is no point in running it from time to time if there are no new orders created...

Latest idea:
If we could catch the exact moment when the TP/SL prices were removed, we don't have to wait always for 2 seconds. Maybe the TP/SL change is done faster.
Based on the Order State diagram I don't see a state change upon SL/TP modification. So calling waitForUpdate(State... states) with the proper (non-existing) state is not an option.
However there is a message (ORDER_CHANGE_OK) generated whenever TP and SL is modified. So theoretically we could catch that message in onMessage(), and execute merging of the orders there (in onMessage()).
This could possible speed up the merging, as we are not for sure waiting for 2 seconds for the TP/SL modification, but whenever it is ready, we can start merging.
Now the question is, what is the average time of a SL/TP modification (set both to 0 for merging)? If that is more-or-less close to 2 seconds, then there is no point in trying to catch the ORDER_CHANGE_OK message and execute merge afterwards. In this case the only possible way to prevent 2 second halts in the strategy is to implement merging in a different thread.

Hi tcsabina:

Well, i've done just a fast read on your posts...

If you are using mainly ticks, why you do not use elapsed time between ticks to control the flow instead of sleeping the thread. I don't see the need for that. Other better aproach maybe using another thread. Also i do not understand why in your message example you are using sleep... at that point it's doing nothing. Other thing i can't understand is why your urgency to get the orders merged... merging now or after some time (suposing the orders still open depeding on the objectives) we get exactly the same result.

Trade well

JL

Hi jlongo,

Thanks for the thoughts. Let me reply.

I am not completely understand, but if you say that I could use the time between 2 ticks (between 2 onTick() calls), than it is indeed my problem. As the process of the current tick (check order conditions, submit order, check merge condition, submit merge) is not finished before the new tick arrives. That is my whole problem.


Ok, but how? With asynchronous execution, as described in the wiki, for example?

I used anOrder.waitForUpdate(4000), just to check if there is an order that is waiting for state change is going to halt the whole strategy thread. It is, as described in the javadoc...


True. But there must be a place (and time) when I do merge.
The issue is that during merging of orders, when I clear TP and SL there can be a 2 second (more? less?) delay/halt when we wait for the new TP/SL prices to be filled/validated/etc. During this waiting time the strategy halts, avoiding further ticks to be processed, new orders to be submitted, etc.
If I move merging out from onTick() and place in a less frequently executed location, this strategy halt will be less frequent of course, but it will be still there.
I am looking for a way to avoid this halt, or if that is not possible at least try to minimize the halt, or the effect of this halt on the strategy.

Hi tc_sabina:

Not sure if this helps you...

Get the initial time of the event in milisseconds (bar.getTime(), tick.getTime(), etc), on each tick, get the time of the tick and get the diference between them, if greater than some value do the action you pretend. You can also use the onMessage to get the events of the order... for example if you receive a ORDER_CHANGE_OK after a stop loss/take profit change, right after you do the next step.

Trade well

JL

Your assumption is wrong. IOrder.waitForuUpdate means that it could take up to 2 seconds, normally the order updates happen in a couple of millis. Run the examples to see yourself how fast actually the IOrder.waitForuUpdate does respond.

Thanks for the replies, guys!
I am busy nowdays. Hopefully can get back to this in the weekend...

I had some time finally, and this is what I came up with.

package singlejartest;
/**************************************************************************************************************************/

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.PrintStream;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.Callable;

import com.dukascopy.api.*;
import com.dukascopy.api.IEngine.OrderCommand;
import com.dukascopy.api.IIndicators.AppliedPrice;
/**************************************************************************************************************************/

@RequiresFullAccess
public class SubmitAndMergeOnTick implements IStrategy {
    @Configurable("Instrument")
    public Instrument selectedInstrument = Instrument.EURUSD;
    @Configurable("Period")
    public Period selectedPeriod = Period.ONE_MIN;
    @Configurable("Order Amount")
    public double orderAmount = 0.01;
    @Configurable("Stop Loss (in pips)")
    public int stopLoss = 0;
    @Configurable("Take Profit (in pips)")
    public int takeProfit = 0;
    @Configurable("Indicator Applied Price")
    public AppliedPrice appliedPrice = AppliedPrice.CLOSE;
    @Configurable("Debug logging")
    public boolean debugLog = true;

    private class MergeTask implements Callable<Object> {
        private IOrder currentOrder;
        private IOrder prevOrder;

        public MergeTask(IOrder anOrder){
            this.currentOrder = anOrder;
        }

        public Object call() throws Exception {
            printWTime("MergeTask::call() called (%s FILLED)", currentOrder.getComment());
            if (currentOrder.isLong())
                prevOrder = searchOrder(LONG, selectedPeriod, currentOrder);
            else
                prevOrder = searchOrder(SHORT, selectedPeriod, currentOrder);

            if (prevOrder != null) {
                printWTime("start merging [%s;%s]", prevOrder.getComment(), currentOrder.getComment());
                try {
                    mergeWithSlAndTp(selectedPeriod, prevOrder, currentOrder);
                }
                catch (JFException e) {
                    printWTime("merge failed: %s", e.getMessage());
                }
            }
            else
                printWTime("no mergable order found in MergeTask::call()");

            return null;
        }
    }

    private IEngine          engine;
    private IConsole         console;
    private IContext         context;
    private double           tickSize;
    private int              longOrderCounter;
    private int              shortOrderCounter;
    private int              longSerieCounter;
    private int              shortSerieCounter;
    private int              tickCounter;
    private SimpleDateFormat sdf;
    private String           strategyOwnName;
    private static final int LONG              = -1;
    private static final int SHORT             = 1;
    private MergeTask        mergeTask;

    /**************************************************************************************************************************/

    public void onStart(final IContext context) throws JFException {
        this.engine       = context.getEngine();
        this.console      = context.getConsole();
        this.context      = context;
        sdf               = new SimpleDateFormat("yyyy-MM-dd_HHmmss");
        sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
        tickSize          = selectedInstrument.getPipValue();
        longOrderCounter  = 1;
        shortOrderCounter = 1;
        longSerieCounter  = 1;
        shortSerieCounter = 1;
        tickCounter       = 0;
        strategyOwnName   = this.getClass().getSimpleName();
        
        try {
            console = new IConsole() {
                String logFileName = "E:\\work\\JForex\\Messages\\" + strategyOwnName + "_" + sdf.format(new Date()) + ".log";
                PrintStream printStream = new PrintStream(new FileOutputStream(logFileName, true)) {
                    @Override
                    public void println(String x) {
                        super.println(x);
                        context.getConsole().getOut().println(x);
                    }
                };

                @Override
                public PrintStream getOut() {
                    return printStream;
                }

                @Override
                public PrintStream getErr() {
                    return printStream;
                }

                @Override
                public PrintStream getWarn() {
                    return printStream;
                }

                @Override
                public PrintStream getInfo() {
                    return printStream;
                }

                @Override
                public PrintStream getNotif() {
                    return printStream;
                }
            };
        }catch (FileNotFoundException e) {
            e.printStackTrace();
        }
    }
    /**************************************************************************************************************************/

    public void onAccount(IAccount account) throws JFException {
    }
    /**************************************************************************************************************************/

    public void onMessage(IMessage message) throws JFException {
        IOrder anOrder;

        switch (message.getType()) {
            case ORDER_SUBMIT_REJECTED:
                anOrder = message.getOrder();
                printWOTime("%s - %s CANCELED: amount=%.2f; price=%.5f; SL=%.5f; TP=%.5f", sdf.format(message.getCreationTime())
                          , anOrder.getLabel(), anOrder.getAmount(), anOrder.getOpenPrice()
                          , anOrder.getStopLossPrice(), anOrder.getTakeProfitPrice());
                break;

            case ORDER_FILL_OK:
                anOrder = message.getOrder();
                printWOTime("%s - %s FILLED: amount=%.2f; price=%.5f; SL=%.5f; TP=%.5f", sdf.format(message.getCreationTime())
                          , anOrder.getLabel(), anOrder.getAmount(), anOrder.getOpenPrice(), anOrder.getStopLossPrice(), anOrder.getTakeProfitPrice());
                this.mergeTask = new MergeTask(anOrder);
                final IContext finalContext = context;
                Thread thread = new Thread(new Runnable() {
                    @Override
                    public void run() {
                        try {
                            finalContext.executeTask(mergeTask);
                        } catch (Exception e) {
                            console.getErr().println(Thread.currentThread().getName() + " " + e);
                        }
                    }
                });

                thread.start();
                break;

            case ORDER_CLOSE_OK:
                anOrder = message.getOrder();
                if (message.getReasons().contains(IMessage.Reason.ORDER_CLOSED_BY_SL))
                    printWOTime("%s - %s closed by SL. PL=%f", sdf.format(message.getCreationTime()), anOrder.getLabel(), anOrder.getProfitLossInPips());
                else if (message.getReasons().contains(IMessage.Reason.ORDER_CLOSED_BY_TP))
                    printWOTime("%s - %s closed by TP. PL=%f", sdf.format(message.getCreationTime()), anOrder.getLabel(), anOrder.getProfitLossInPips());
                else if (message.getReasons().contains(IMessage.Reason.ORDER_CLOSED_BY_MERGE))
                    printWOTime("%s - %s closed by MERGE. PL=%f", sdf.format(message.getCreationTime()), anOrder.getLabel(), anOrder.getProfitLossInPips());
                break;

            default:
                break;
        }
    }
    /**************************************************************************************************************************/

    public void onTick(Instrument instrument, ITick tick) throws JFException {
        String  newLabel;

        ++tickCounter;
        if ( (instrument.equals(selectedInstrument)) && (tickCounter > 20)) {
            tickCounter = 0;
                newLabel = createLabel(instrument, selectedPeriod, LONG);
                printWTime("%s prepared for submit", newLabel);
                engine.submitOrder(newLabel + "_" + (System.currentTimeMillis()/1000), instrument, OrderCommand.BUY, orderAmount, 0.0, 0.0
                                , (stopLoss!=0?tick.getBid()-stopLoss*tickSize:0.0), (takeProfit!=0?tick.getBid()+takeProfit*tickSize:0.0), 0, newLabel);
        }
    }
    /**************************************************************************************************************************/

    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
    }
    /**************************************************************************************************************************/

    public void onStop() throws JFException {
        console.getOut().close();
    }
    /**************************************************************************************************************************/

    public void printPlain(String message) {
        console.getOut().println(message);
    }
    /**************************************************************************************************************************/

    public void printWTime(boolean debug, String format, Object...args) {
        if (debug)
            printWTime(format, args);
    }
    /**************************************************************************************************************************/

    public void printWTime(String format, Object...args) {
        printPlain(sdf.format(new Date()) + " - " + String.format(format, args));
    }
    /**************************************************************************************************************************/

    public void printWOTime(String format, Object...args) {
        printPlain(String.format(format, args));
    }
    /**************************************************************************************************************************/

    /*
     * This function is called when the order creation conditions are fulfilled.
     * The functions returns a suitable order that can be merged with the order that is just about to be created.
     * And order is suitable if it is the same type (LONG or SHORT), and if the order was created with the same strategy period.
     */
    private IOrder searchOrder(int orderType, Period period, IOrder currentOrder) throws JFException {
        int orderCount;

        try {
            for (IOrder order : engine.getOrders()) {
                orderCount = engine.getOrders().size();
                if (orderCount < 1)
                    return null;

                // if order is not valid anymore (due to merge, for eg.), continue to next order (if there is any)
                if (!engine.getOrders().contains(order))
                    continue;

                // if order is the same as the current order, continue to next order (if there is any)
                if (engine.getOrder(order.getLabel()).equals(currentOrder))
                    continue;

                // search for other LONG orders to merge
                if (orderType == LONG) {
                    if (engine.getOrder(order.getLabel()).isLong()) {
                        if (engine.getOrder(order.getLabel()).getLabel().contains(periodShortName(period)))
                            return order;
                    }
                }
                // search for other SHORT orders to merge
                else {
                    if (!engine.getOrder(order.getLabel()).isLong()) {
                        if (engine.getOrder(order.getLabel()).getLabel().contains(periodShortName(period)))
                            return order;
                    }
                }
            }
        }
        catch (JFException e) {
            printPlain("  JF Exception in searchOrder(): " + e.getMessage());
        }
        catch (Exception ex) {
            printPlain("  General Exception in searchOrder(): " + ex.getMessage());
        }

        return null;
    }
    /**************************************************************************************************************************/

    // Creates a unique label for an order.
    // For example: eurusd_m1_s_1_1
    private String createLabel(Instrument instrument, Period period, int orderType) {
        String label;

        label = instrument.name();
        label = label.toLowerCase();
        label = label + periodShortName(period);
        switch (orderType) {
            case LONG:
                label = label + "L_" + longSerieCounter + "_" + longOrderCounter;
                ++longOrderCounter;
                break;

            case SHORT:
                label = label + "S_" + shortSerieCounter + "_" + shortOrderCounter;
                ++shortOrderCounter;
                break;
        }

        return label;
    }
    /**************************************************************************************************************************/

    // Converting parameter period to short format.
    private String periodShortName(Period period) {
        String label;

        if (period == Period.TICK)              label = "_tk_";
        else if (period == Period.ONE_SEC)      label = "_s1_";
        else if (period == Period.TWO_SECS)     label = "_s2_";
        else if (period == Period.TEN_SECS)     label = "_s10_";
        else if (period == Period.TWENTY_SECS)  label = "_s20_";
        else if (period == Period.THIRTY_SECS)  label = "_s30_";
        else if (period == Period.ONE_MIN)      label = "_m1_";
        else if (period == Period.FIVE_MINS)    label = "_m5_";
        else if (period == Period.TEN_MINS)     label = "_m10_";
        else if (period == Period.FIFTEEN_MINS) label = "_m15_";
        else if (period == Period.TWENTY_MINS)  label = "_m20_";
        else if (period == Period.THIRTY_MINS)  label = "_m30_";
        else if (period == Period.ONE_HOUR)     label = "_h1_";
        else if (period == Period.FOUR_HOURS)   label = "_h4_";
        else if (period == Period.DAILY)        label = "_d_";
        else if (period == Period.WEEKLY)       label = "_w_";
        else                                    label = "_-_";

        return label;
    }
    /**************************************************************************************************************************/

    // Merge orders with weighted average SL and TP.
    private void mergeWithSlAndTp(Period period, IOrder... orders) throws JFException {
        IOrder  mergedOrder;
        double  price 
              , weight
              , priceWeighted
              , priceWeightedTotal;
//              , weightTotal
//              , priceWeightedAverage
//              , slPrice = 0
//              , tpPrice = 0;
        String  label = "";

        priceWeightedTotal = 0;
//        weightTotal = 0;
        //remove SL and TP attached orders if any
        for (IOrder o: orders) {            
            price = o.getOpenPrice();
            weight = o.getAmount();
            priceWeighted = price * weight;
            priceWeightedTotal += priceWeighted;
//            weightTotal += weight;
            if (Double.compare(o.getStopLossPrice(),0) != 0) {
                printWTime("remove SL for %s: price=%.5f; amount=%.5f; SL=%.5f; TP=%.5f; priceWeighted=%.5f; priceWeightedTotal=%.5f"
                         , o.getLabel(), price, weight, o.getStopLossPrice(), o.getTakeProfitPrice(), priceWeighted, priceWeightedTotal);
                o.setStopLossPrice(0);
                o.waitForUpdate(2000);
            }

            if (Double.compare(o.getTakeProfitPrice(),0) != 0) {
                printWTime("remove TP for %s: price=%.5f; amount=%.5f; SL=%.5f; TP=%.5f; priceWeighted=%.5f; priceWeightedTotal=%.5f"
                        , o.getLabel(), price, weight, o.getStopLossPrice(), o.getTakeProfitPrice(), priceWeighted, priceWeightedTotal);
                o.setTakeProfitPrice(0);                
                o.waitForUpdate(2000);        
            }

            label = o.getComment(); // order comment is the order label without timestamp (for eg.: eurusd_1m_s_23_2)
        }
        
//        priceWeightedAverage = priceWeightedTotal / weightTotal;       
        label = label + "m";
        mergedOrder = engine.mergeOrders(label + "_" + (System.currentTimeMillis()/1000), label, orders);
        mergedOrder.waitForUpdate(2000);
        if (mergedOrder.getState() != IOrder.State.FILLED)
        {
            printWTime(String.format("  merge failed. Order state: %s", mergedOrder.getState()));

            return;
        }

//        slPrice = mergedOrder.isLong() 
//                ? priceWeightedAverage - (stopLoss*tickSize)
//                : priceWeightedAverage + (stopLoss*tickSize);
//        tpPrice = mergedOrder.isLong() 
//                ? priceWeightedAverage + (takeProfit*tickSize)
//                : priceWeightedAverage - (takeProfit*tickSize);
//        mergedOrder.setStopLossPrice(slPrice);
//        mergedOrder.setTakeProfitPrice(tpPrice);        
//        mergedOrder.waitForUpdate(2000); 
        printWTime("%s submitted: mergedAmount=%.2f; price=%.5f; SL=%.5f; TP=%.5f;", mergedOrder.getLabel()
                , mergedOrder.getAmount(), mergedOrder.getOpenPrice(), mergedOrder.getStopLossPrice(), mergedOrder.getTakeProfitPrice());

    }
    /**************************************************************************************************************************/
}

In a nutshell:
- there is a tick counter, for easy test. Whenever the counter reaches 20, I submit a LONG order.
- when an order is FILLED, I am going to check if there is a suitable order to be merged with. If yes, I merge the 2 orders.

I am testing this strategy on the Historical Tester. The concept, and the above strategy, is working, however I have an issue:
If the Historical Tester is running at Max speed, the strategy cannot handle properly the submit-merge-submit-merge rotation. Let me explain it with the output of a test run from 2008.01.04:

2013-10-07_193629 - eurusd_m1_L_1_1 prepared for submit
2008-01-04_000004 - eurusd_m1_L_1_1_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - eurusd_m1_L_1_2 prepared for submit
2008-01-04_000008 - eurusd_m1_L_1_2_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_2 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_1;eurusd_m1_L_1_2]
2008-01-04_000008 - eurusd_m1_L_1_1_1381174589 closed by MERGE. PL=0.000000
2008-01-04_000008 - eurusd_m1_L_1_2_1381174589 closed by MERGE. PL=0.000000
2013-10-07_193629 - eurusd_m1_L_1_2m_1381174589 submitted: mergedAmount=0.02; price=1.47450; SL=0.00000; TP=0.00000;

This already shows some issues, as after an order is FILLED the mergeTask should be called from onMessage(). This call is not executed for the 1st order (eurusd_m1_L_1_1). MergeTask() is only called for the 2nd order (eurusd_m1_L_1_2). Why is that happening? Why do I miss a call to MergeTask()?
Of course the MergeTask() would give no mergable orders after 1 order, but still, it must be called.

If we continue:

2013-10-07_193629 - eurusd_m1_L_1_3 prepared for submit
2008-01-04_000013 - eurusd_m1_L_1_3_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - eurusd_m1_L_1_4 prepared for submit
2008-01-04_000017 - eurusd_m1_L_1_4_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_4 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_2m;eurusd_m1_L_1_4]
2008-01-04_000017 - eurusd_m1_L_1_2m_1381174589 closed by MERGE. PL=0.000000
2008-01-04_000017 - eurusd_m1_L_1_4_1381174589 closed by MERGE. PL=0.000000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_4 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_3;eurusd_m1_L_1_4]
2013-10-07_193629 - merge failed: Cannot merge orders in state other than FILLED
2013-10-07_193629 - eurusd_m1_L_1_4m_1381174589 submitted: mergedAmount=0.03; price=1.47450; SL=0.00000; TP=0.00000;
2013-10-07_193629 - eurusd_m1_L_1_5 prepared for submit
2008-01-04_000021 - eurusd_m1_L_1_5_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_5 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_3;eurusd_m1_L_1_5]
2008-01-04_000021 - eurusd_m1_L_1_3_1381174589 closed by MERGE. PL=0.000000
2008-01-04_000021 - eurusd_m1_L_1_5_1381174589 closed by MERGE. PL=0.000000
2013-10-07_193629 - eurusd_m1_L_1_5m_1381174589 submitted: mergedAmount=0.02; price=1.47450; SL=0.00000; TP=0.00000;
2013-10-07_193629 - eurusd_m1_L_1_6 prepared for submit
2008-01-04_000025 - eurusd_m1_L_1_6_1381174589 FILLED: amount=0.01; price=1.47450; SL=0.00000; TP=0.00000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_6 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_4m;eurusd_m1_L_1_6]
2008-01-04_000026 - eurusd_m1_L_1_4m_1381174589 closed by MERGE. PL=0.000000
2008-01-04_000026 - eurusd_m1_L_1_6_1381174589 closed by MERGE. PL=0.000000
2013-10-07_193629 - eurusd_m1_L_1_6m_1381174589 submitted: mergedAmount=0.04; price=1.47450; SL=0.00000; TP=0.00000;
2013-10-07_193629 - eurusd_m1_L_1_7 prepared for submit
2008-01-04_000030 - eurusd_m1_L_1_7_1381174589 FILLED: amount=0.01; price=1.47460; SL=0.00000; TP=0.00000
2013-10-07_193629 - MergeTask::call() called (eurusd_m1_L_1_7 FILLED)
2013-10-07_193629 - start merging [eurusd_m1_L_1_5m;eurusd_m1_L_1_7]

We see again that the MergeTask call is not working as it should. It is never being called for order eurusd_m1_L_1_3, and it is being called 2 times with the same order (eurusd_m1_L_1_4).

Why is this happening?
Is implementing order merge in a separate thread that is being triggered from the onMessage() function whenever an order is filled is not a good idea?
I am also not sure about how I instantiate the MergeTask and the thread. These are in onMessage(), in the ORDER_FILL_OK case. Is this the proper way to do such thing?
Is it overkill at all to separate order merge to a different thread? Shall I just call my MergeTask from onTick(), whenever a tick is suitable for submitting an order? I would say that maybe this is a better approach. As if the merge is truly implemented in the onTick() function, there wouldn't be multiple order submits while the MergeTask is being processed, as it (the MergeTask) would 'halt' onTick() and prevent any new orders to be submitted.

Sorry for the wall of text...

If you will run the MergeTask in the strategy thread you will see that the strategy works as you expected, i.e.:

this.mergeTask = new MergeTask(anOrder);
try {
   mergeTask.call();
} catch (Exception e) {
   e.printStackTrace();
}

There is no real necessity to run the task in a separate thread, especially when back-testing, where the order requests get processed instantly (there is no server involved). If you find it necessary to run the task in a separate thread then consider checking if IEngine.Type != TEST.
In general if you can attain your goal without multi-threading then better do so.
According the strategy logic as we see it - it is more logical to call merge from the onMessage as you don't need to "scan" all the active orders on every tick, rather only a particular order and when an actual order change happens.

When I backtest I have conditional checks that introduce synchronization between the back tester and the multi-threaded strategy it is trying to test.
My onTick() and onBar() methods return straight after enqueuing/message passing their parameters.
This means the back tester is free to speed ahead with its testing regime while it maintains the CPU's time and is the current thread of execution, generating many calls into onBar() and onTick() before the threads comprising the bulk of the strategy's work even get a chance to run.

If you can avoid it, for your own sanity, look for logical and straight forward ways to attack your problems.
The thread creation above didn't add anything, it simply delayed the request for the strategy thread to execute the Callable.
That request could have been made in the same thread or indeed as Support indicated, the task itself could have been executed right there inline.
If it can be done in onAccount(), onBar(), onMessage(), onTick() it's best to do it there in the long run.

Also generally, you could do with a few more key console.getOut().println()'s in order to debug your execution paths and gain more insight into what is actually being assigned, called, referenced, etc.

I'll have to read the posts above again to properly understand the problem that has been identified with merging and the timing of its occurrence and what you are trying to achieve.

I'll be back

Thanks for the constructive replies!

I am still not sure how should I implement what I am up to. These are my ideas, with pros and cons:

In general, the strategy should do the following:
- Work on ticks. Whenever a tick arrives (to the defined instrument), we should decide if we submit an order or not.
- An order must have a TP, but SL level is only necessary when certain conditions are met.
- An order must be merged with an other (older) order if they are from the same tpye (LONG with LONG; SHORT with SHORT).
- The merged order must have a TP price, and must have SL price also, if certain conditions are met (same conditions as for a single order).

This is plain simple. The only issue that makes this a bit more complicated is the fact that you have the clear SL/TP prices for orders before merging. And I think that this clearing part could slow down the whole strategy (and its processing of ticks properly), as there could be situations when this clear take some time. Support says it is done in milliseconds, but the actual code that is used is this:

order.setStopLossPrice(0);
order.waitForUpdate(2000);

I haven't been able to thoroughly test these 2000 millisecond conditions, and if I am not mistaken, the above code means that the order update must be finished within 2 seconds. But I don't know if this 2 seconds should be taken into account. As stated above, Support mentioned that the actual update is done quite fast (within milliseconds), I still want to make sure, and create a strategy that deals properly with situations when the above operation really take the maximum allowed time, which is 2 seconds.

Now, keeping in mind this 2 second halt (again saying to avoid misunderstanding, that I know that in general this is not the case, and order update (due to clear SL/TP levels) are done within milliseconds) I came up with the following strategy plans:

A) version
1st. in onTick(): If order conditions are met, we submit on order without TP and SL levels.
2nd. in onMessage(): When the submitted order gets filled, we check if there is a suitable mergeble order.
3rd-a. We can merge. We clear the older order's TP and SL levels (if there is one), and we merge these 2 orders. We check if we have to set SL level. If yes, we set SL and TP level of the merged order. If not (no need to set SL), we only set TP level of the merged order.
3rd-b. We cannot merge. We check if we have to set SL level. If yes, we set SL and TP levels of the submitted order. If not, we only set TP level of the submitted order.

B) version
1st. in onTick(): If order conditions are met, we check if we have to set SL level. If yes, we submit and order with SL and TP levels. If not (no need to set SL level), we submit an order with TP level.
2nd. in onMessage(): When the submitted order gets filled, we check if there is a suitable order to be merged.
3rd-a. There is a suitable order. We clear SL and TP levels of these orders. We merge them. We check if we have to set SL level for the merged order. If yes, we set SL and TP levels for the merged order. If not, we only set TP level.
3rd-b. There is no order to be merged. We do nothing.

C) version
Same as B) version, but the merging in onMessage() is done in a separate thread.


A) version is probably faster in execution, as we only have to deal with clearing of 1 order's TP (and maybe) SL levels. This order is the older order we submitted in the past, and we will merge it with the currently submitted order.
This version has an open question about safety: is it safe to submit and order without TP and SL levels? And 'trust' that onMessage() will be properly called, where we set finally the TP (and maybe) the SL levels or the order. What happens if the strategy stops or connection is lost during order submission? And order will be 'hanging' out there without TP and SL levels. Is this something that should warn me?

B) version is probably safer, as we submit every order with at least a TP level. But, as stated above in the post, this version has to deal with more SL and TP level clear operation, which gives me a headache. The strategy should operate on ticks, so slowing down the strategy is something I want to avoid.

C) version seems to be the best option: it is safe as there are no orders without TP/SL levels (keeping in mind that SL is not always set). And the delay that is (possibly) coming from merge won't stop further ticks to be processed. But I am not sure that my attempts to put merging in a separate thread are reached at all (in the sample strategy I posted several posts above), and if it is necessary at all.


Now, after this Wall Of Text, my question(s):
Is it necessary to 'design' a strategy that can process ticks even if there is a command (order.waitForUpdate in this case) that might halt the strategy for a second (or two)? Or I should live with the fact that sometimes a tick is skipped? There is no way to completely prevent it.
If it can be prevented, how? Is version C) a good try?
If it cannot be prevented completely, then I should not care about submitting an order with a TP level in onTick(), and in onMessage() when the order gets FILLED, maybe remove this TP?

Anything else than using waitForUpdate in onMessage while performing order operations in the strategy thread would immensely sophisticate your strategy logic. Theoretically it is possible to do this without using any waitForUpdate call, but this would possibly involve introducing special order wrapper classes with self-defined states or extensive usage of java.util.Map to hold the relevant order change data. For the latter approach consider doing something like this:

package jforex.orders.merge;

import java.math.BigDecimal;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;

import com.dukascopy.api.*;
import com.dukascopy.api.IEngine.OrderCommand;
import com.dukascopy.api.util.DateUtils;

/**
 * The strategy demonstrates how one can go about automatically merging positions
 * with SL/TP values without needing to use the IOrder.waitForUpdate.
 * 
 * 
 * The strategy on its start creates 4 orders which have SL or TP prices.
 * Since orders with  SL or TP can't be merged, both price conditions have to be removed and after
 * successful merge put back to the merged order.
 * 
 */
public class MergeRemoveSlTpOnMessage implements IStrategy {

    private IConsole console;
    private IEngine engine;
    private IHistory history;

    private Map<Instrument, Map<IOrder, Double>> slMap = new HashMap<Instrument, Map<IOrder, Double>>();
    private Map<Instrument, Map<IOrder, Double>> tpMap = new HashMap<Instrument, Map<IOrder, Double>>();
    private Map<String, Set<IOrder>> mergedFrom = new HashMap<String, Set<IOrder>>();
    private Set<IOrder> tpAddInProgress = new HashSet<IOrder>();
    private Set<IOrder> slAddInProgress = new HashSet<IOrder>();
    private Set<IOrder> mergedWithCompletedSlTpAdd = new HashSet<IOrder>();    
    private Map<IOrder, Long> slChangeTimes = new HashMap<IOrder, Long>();
    private Map<IOrder, Long> tpChangeTimes = new HashMap<IOrder, Long>();
    private int mergeCount;

    @Override
    public void onStart(IContext context) throws JFException {
        engine = context.getEngine();
        console = context.getConsole();
        history = context.getHistory();
        console.getOut().println("Start");

        Instrument instrument = Instrument.EURUSD;

        double price = history.getLastTick(instrument).getBid();
        // 0.01 BUY market order with SL and TP of 10 pips
        engine.submitOrder("order1", instrument, OrderCommand.BUY, 0.01, 0, 20, price - 0.0010, price + 0.0010);
        // 0.02 SELL market order without SL and TP
        engine.submitOrder("order2", instrument, OrderCommand.SELL, 0.02, 0, 20, 0, 0);
        // 0.02 BUY market order with SL of 10 pips
        engine.submitOrder("order3", instrument, OrderCommand.BUY, 0.02, 0, 20, price - 0.0010, 0);
        engine.submitOrder("order4", instrument, OrderCommand.BUY, 0.02, 0, 20, price - 0.0010, 0);

    }

    @Override
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {}

    @Override
    public void onMessage(IMessage message) throws JFException {
        IOrder order = message.getOrder();
        if (order == null || message.getType() == IMessage.Type.NOTIFICATION) {
            return;
        }

        console.getOut().format("<html><font color=\"gray\">%s - %s %s SL=%.5f TP=%.5f</font>", 
                DateUtils.format(message.getCreationTime()), message, message.getReasons(), order.getStopLossPrice(), order.getTakeProfitPrice()).println();

        Instrument instrument = order.getInstrument();
        
        int filledPositionCount = 0;
        for(IOrder o : engine.getOrders(instrument)){
           if(o.getState() == IOrder.State.FILLED){
              filledPositionCount++;
           }
        }


        try {
            // check if the SL/TP has been put back
            mergeSlTpPutBackChecks(message);

            // remove SL/TP for "eligible" positions - just filled ones and the merged ones with just having been put back their SL/TP's
            if (filledPositionCount > 1 && (isJustFilledWithSlTp(message) || isMergedWithJustCompletedSlTpAdd(message))) {
                removeSlTpPrices(order);
                if (isMergedWithJustCompletedSlTpAdd(message)) {
                    mergedWithCompletedSlTpAdd.remove(order);
                }
                return;
            // put back SL/TP to just merged positions
            } else if (message.getType() == IMessage.Type.ORDERS_MERGE_OK && mergedFrom.containsKey(order.getLabel())) {
                putBackSlTpPrices(order);
            //we can merge this position - if the other order has no SL/TP - we will merge them;
            //otherwise - we will remove the other position's SL/TP and merge in the next iteration
            } else if (filledWithoutSlTp(order) && filledPositionCount > 1) {
                Set<IOrder> inMergeProgress = new HashSet<IOrder>();
                for (Set<IOrder> from : mergedFrom.values()) {
                    inMergeProgress.addAll(from);
                }
                for (IOrder o2 : engine.getOrders(instrument)) {
                    if (order != o2 && !inMergeProgress.contains(o2)) {
                        // merge filled positions with no SL/TP
                        if (filledWithoutSlTp(o2)) {
                            IOrder merged = engine.mergeOrders("merged" + ++mergeCount, o2, order);
                            mergedFrom.put(merged.getLabel(), new HashSet<IOrder>(Arrays.asList(order, o2)));
                            break;
                        //this would normally be a position with SL/TP which for a while has been the only position
                        } else if (isFilledWithSlTp(o2)) {
                            removeSlTpPrices(o2);
                            break;
                        }
                    }
                }
            }
        } catch (Exception e) {
            console.getErr().println(order + " " + e);
            e.printStackTrace();
        }
    }  
    
    private boolean isMergedWithJustCompletedSlTpAdd(IMessage message){
        return mergedWithCompletedSlTpAdd.contains(message.getOrder())
                && message.getType() == IMessage.Type.ORDER_CHANGED_OK
                && isFilledWithSlTp(message.getOrder());
    }
    
    private boolean isJustFilledWithSlTp(IMessage message){
        return message.getType() == IMessage.Type.ORDER_FILL_OK 
                && ( message.getOrder().getStopLossPrice() > 0 
                || message.getOrder().getTakeProfitPrice() > 0);
    }
    
    private boolean isFilledWithSlTp(IOrder order){
        return order.getState() == IOrder.State.FILLED 
                && (order.getStopLossPrice() > 0 
                || order.getTakeProfitPrice() > 0);
    }
    
    private boolean filledWithoutSlTp(IOrder order){
        return order.getState() == IOrder.State.FILLED 
                && order.getStopLossPrice() == 0 
                && order.getTakeProfitPrice() == 0;
    }
    
    private void mergeSlTpPutBackChecks(IMessage message){
        mergeSlTpUpdateCheck(message, true);
        mergeSlTpUpdateCheck(message, false);
    }
    
    private void mergeSlTpUpdateCheck(IMessage message, boolean isSl){
        IOrder order = message.getOrder();
        Set<IOrder> priceAddInProgress = isSl ? slAddInProgress : tpAddInProgress;
        Set<IOrder> oppositePriceAddInProgress = isSl ? tpAddInProgress : slAddInProgress;
        IMessage.Reason reason = isSl ? IMessage.Reason.ORDER_CHANGED_SL : IMessage.Reason.ORDER_CHANGED_TP;
        if (message.getReasons().contains(reason) && priceAddInProgress.contains(order)) {
            priceAddInProgress.remove(order);
            if (!oppositePriceAddInProgress.contains(order)) {
                mergedWithCompletedSlTpAdd.add(order);
            }
        }
    }
    
    private void putBackSlTpPrices(IOrder order) throws JFException{
        putBackPrice(order, true);
        putBackPrice(order, false);
        mergedFrom.remove(order.getLabel());
    }    
    
    // simply take the average of pip distances of the two merged-from orders
    private void putBackPrice(IOrder order, boolean isSl) throws JFException{
        Instrument instrument = order.getInstrument();
        Iterator<IOrder> it = mergedFrom.get(order.getLabel()).iterator();
        IOrder o1 = it.next();
        IOrder o2 = it.next();
        Map<Instrument, Map<IOrder, Double>> priceMap = isSl ? slMap : tpMap;
        Set<IOrder> priceAddInProgress = isSl ? slAddInProgress : tpAddInProgress;
        Double tp1 = priceMap.get(instrument).remove(o1);
        Double tp2 = priceMap.get(instrument).remove(o2);
        double tpNew = 0;
        if (anyGtZero(tp1, tp2)) {
            double tp1Pips = nullZero(tp1) ? 0 : Math.abs(o1.getOpenPrice() - tp1) / instrument.getPipValue();
            double tp2Pips = nullZero(tp2) ? 0 : Math.abs(o2.getOpenPrice() - tp2) / instrument.getPipValue();
            tpNew = order.getOpenPrice() + (tp1Pips + tp2Pips) / 2 * instrument.getPipValue() * (order.isLong() ^ isSl ? +1 : -1);
            setPrice(order, roundToPippette(tpNew, instrument), isSl);
            priceAddInProgress.add(order);
        }
        console.getNotif()
                .format("%6$s[%1$s]=%3$s %6$s[%2$s]=%4$s => %6$s[%7$s]=%5$.5f",
                        o1.getLabel(), o2.getLabel(), tp1, tp2, tpNew, isSl? "SL" : "TP", order.getLabel()).println();
    }
    
    
    private void removeSlTpPrices(IOrder order) throws JFException{
        setPrice(order, 0, true);
        setPrice(order, 0, false);
    }

    private void setPrice(IOrder order, double price, boolean isSl) throws JFException{
        double prevPrice = isSl ? order.getStopLossPrice() : order.getTakeProfitPrice();
        if( Double.compare(price, prevPrice)==0){
            return;
        }
        long time = engine.getType() == IEngine.Type.TEST ? history.getLastTick(order.getInstrument()).getTime() : System.currentTimeMillis();
        Instrument instrument = order.getInstrument();
        Map<Instrument, Map<IOrder, Double>> priceMap = isSl ? slMap : tpMap;
        if (priceMap.get(instrument) == null) {
            priceMap.put(instrument, new HashMap<IOrder, Double>());
        }
        priceMap.get(instrument).put(order, prevPrice);
        Map<IOrder, Long> changeTimes = isSl ? slChangeTimes : tpChangeTimes;
        if (changeTimes.get(order) != null && time - changeTimes.get(order) < 1000) {
            // make sure there is at least 1 second between the price change times
            // if the position meanwhile got closed - we don't need to wait anymore
            order.waitForUpdate(1001 - time + changeTimes.get(order), IOrder.State.CLOSED);
        }
        if(order.getState() != IOrder.State.FILLED){
            return;
        }
        if(isSl){
            order.setStopLossPrice(price);
        } else {
            order.setTakeProfitPrice(price);
        }
        changeTimes.put(order, System.currentTimeMillis());
    }
    
    private boolean nullZero(Double d) {
        if (d == null || Double.compare(d, 0) == 0) {
            return true;
        }
        return false;
    }
    
    private boolean anyGtZero(Double...values){
        for(Double d : values){
            if(d != null && d>0){
                return true;
            }
        }
        return false;
    }
    
    
    private static double roundToPippette(double amount, Instrument instrument) {
        return round(amount, instrument.getPipScale() + 1);
    }

    private static double round(double amount, int decimalPlaces) {
        return (new BigDecimal(amount)).setScale(decimalPlaces, BigDecimal.ROUND_HALF_UP).doubleValue();
    }

    @Override
    public void onAccount(IAccount account) throws JFException {
    }

    @Override
    public void onStop() throws JFException {
        for (IOrder o : engine.getOrders()){
            if(o.getState() == IOrder.State.FILLED || o.getState() == IOrder.State.OPENED){
                o.close();
            }
        }
    }

    @Override
    public void onTick(Instrument instrument, ITick tick) throws JFException {
    }

}

Dear Support,

Thank you for the time and effort for putting this example together!