Dukascopy Support Board • View topic - JFOREX-1466 IBar returns wrong values

RoadRunner

Dear support team!

The following lines of code get called with the same arguments, but return different values, depending on whether called earlier or later into the run:

Code:

      IBar iBar = history.getBars(Instrument.GBPUSD, Period.THIRTY_MINS, OfferSide.ASK, Filter.ALL_FLATS, 0, _lTickStart, 1).get(0);

      console.getOut().println("Bar.time=" + fDT1.format(iBar.getTime()) + " Open=" + iBar.getOpen() + " High=" + iBar.getHigh() + " Low=" + iBar.getLow() + " Close=" + iBar.getClose());

Loggings:

Quote:

beginning of run:
Bar.time=Mon 2008.01.07 16:00:00.000 Open=1.9754 High=1.97565 Low=1.96875 Close=1.9688
Bar.time=Mon 2008.01.07 16:30:00.000 Open=1.9688 High=1.97105 Low=1.96805 Close=1.97025
Bar.time=Mon 2008.01.07 17:00:00.000 Open=1.97025 High=1.97195 Low=1.9699 Close=1.9719
...
later into the run:
Bar.time=Mon 2008.01.07 16:00:00.000 Open=1.9754 High=1.9754 Low=1.9754 Close=1.9754
Bar.time=Mon 2008.01.07 16:30:00.000 Open=1.9688 High=1.9688 Low=1.9688 Close=1.9688
Bar.time=Mon 2008.01.07 17:00:00.000 Open=1.97025 High=1.97025 Low=1.97025 Close=1.97025

The IBar object in later calls always returns the open price for all price calls, regardless if getOpen(), getHigh(), getLow(), getClose() are called.

This is a most critical bug as it can lead to wrong trading signals based on the interpretation of the false values.

As far as I can see, our strategy code that is calling and using the values retrieved, is performing correctly. We don't see any other field values to be distorted.

Any ideas of what might go on here?

Best, RR.

API Support

Is it DEMO or LIVE system? What version of the client are you using? Can you provide reproducing example (this would allow us to provide fix faster)

RoadRunner

I've observed this behaviour on the current DEMO version. It is reproducable in our strategy system.
I'll try to extract a test code that reproduces this behaviour, but it might take some time for me to provide this.

Meanwhile we have replaced all calls to history.getBars(..) in our code by a subroutine that checks the returned bars for this error before passing the bar-list to the strategy code.

Best, RR.

RoadRunner

the routine mentioned in my last post that now checks all bars loaded from IHistory for flats does log a lot of flat candlesticks, that are returned even when the Filter.ALL_FLATS is set.

Examples (run with JFroex 2.1.14):

Quote:

[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2008.01.07 00:00:00.000 OHLC=1.9717] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 92 bars before and 0 behind Mon 2008.01.07 00:00:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
...
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2008.01.07 00:00:00.000 OHLC=1.9717] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 123 bars before and 0 behind Mon 2008.01.07 00:00:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
...
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2007.12.31 17:30:00.000 OHLC=1.98345] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 1 bars before and 0 behind Tue 2008.01.01 08:30:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2007.12.31 18:00:00.000 OHLC=1.98345] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 1 bars before and 0 behind Tue 2008.01.01 09:00:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2007.12.31 18:30:00.000 OHLC=1.98345] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 1 bars before and 0 behind Tue 2008.01.01 09:30:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2007.12.31 19:00:00.000 OHLC=1.98345] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 1 bars before and 0 behind Tue 2008.01.01 10:00:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Mon 2007.12.31 19:30:00.000 OHLC=1.98345] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 1 bars before and 0 behind Tue 2008.01.01 10:30:00.000[GBP/USD, 30 Mins, Bid, ALL_FLATS]
...
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Thu 2007.12.27 21:34:00.000 OHLC=1.9956] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 3601 bars before and 0 behind Mon 2007.12.31 02:29:00.000[GBP/USD, 1 Min, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Thu 2007.12.27 21:36:00.000 OHLC=1.99555] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 3601 bars before and 0 behind Mon 2007.12.31 02:29:00.000[GBP/USD, 1 Min, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Thu 2007.12.27 22:12:00.000 OHLC=1.9955] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 3601 bars before and 0 behind Mon 2007.12.31 02:29:00.000[GBP/USD, 1 Min, Bid, ALL_FLATS]
[getIBarsFromDKHistory] ...debug: CheckError: Flat bar [Thu 2007.12.27 22:20:00.000 OHLC=1.99565] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving 3601 bars before and 0 behind Mon 2007.12.31 02:29:00.000[GBP/USD, 1 Min, Bid, ALL_FLATS]
...

Best, RR.

API Support

We don't filter the bars that have open price different than the close price of the previous bar, even if it has all the prices at one level

RoadRunner

Hi,

yes, I'm very aware of the fact that flats by definition have to hold the same OHLC prices. The point here is a different one.

As I mentioned earlier, getOpen/Close/High/Low() did return identical values, even though when checking with the charts, etc. it is clear that the bars at that time should hold different values!

It is the getOpen/Close/High/Low()-functions that return wrong values! Furthermore, calling getOpen... on the same bars earlier in the run, did return the correct values! From one point onwards, the getOpen...-functions did return wrong values. That is what makes this issue *most critical*.

Best, RR.

API Support

I wrote an example. I didn't run it for days, only for a few hours, but it always returns correct results. Please check it, try it... maybe it needs some modifications. For now we can't reproduce the problem

package jforex;

import java.util.*;
import java.text.*;

import com.dukascopy.api.*;

public class JFOREX1466 implements IStrategy {
   private IEngine engine;
   private IConsole console;
   private IHistory history;
   private IContext context;
   private IIndicators indicators;
   private IUserInterface userInterface;
    private DateFormat fDT1;
    private long time;
   
   public void onStart(IContext context) throws JFException {
      this.engine = context.getEngine();
      this.console = context.getConsole();
      this.history = context.getHistory();
      this.context = context;
      this.indicators = context.getIndicators();
      this.userInterface = context.getUserInterface();
        
        fDT1 = new SimpleDateFormat("yyyy.MM.dd HH:mm:ss");
        fDT1.setTimeZone(TimeZone.getTimeZone("GMT"));
        try {
            time = fDT1.parse("2008.01.07 16:00:00").getTime();
        } catch (Exception e) {
            console.getOut().println(e.getMessage());
            context.stop();
        }
   }

   public void onAccount(IAccount account) throws JFException {
   }

   public void onMessage(IMessage message) throws JFException {
   }

   public void onStop() throws JFException {
   }

   public void onTick(Instrument instrument, ITick tick) throws JFException {
   }
   
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
        if (instrument == Instrument.EURUSD && period == Period.TEN_SECS) {
            IBar iBar = history.getBars(Instrument.GBPUSD, Period.THIRTY_MINS, OfferSide.ASK, Filter.ALL_FLATS, 0, time, 1).get(0);
            console.getOut().println("reqest.time=" + fDT1.format(time) + " Bar.time=" + fDT1.format(iBar.getTime()) + " Open=" + iBar.getOpen() + " High=" + iBar.getHigh() + " Low=" + iBar.getLow() + " Close=" + iBar.getClose());
        }
    }
}

RoadRunner

Hi,

thank you very much for looking into this issue.
The above code however is not likely to uncover the issue.
As I've mentioned in my first post and you can see from the logging snippet in that post, the first calls of the bars through history.getBars(...) did return bars with the correct values. If calling the very same bar again later in the same run, it did return a flat bar with the open price being correct and the HLC prices showing the same value as the open price (which is wrong).

From what I've seen, it happens from a certain point in the strategy execution onwards.
The log-lines shown in my post as of Wed Mar 17, 2010 10:48 were logged directly after the history.getBars(..) call, so no code in between.

Here is the code, that we meanwhile use to retrieve bars from IHistory throughout our strategy code instead of calling history.getBars directly in order to check for wrong bars and take action whenever we discover a false bar (ignore/log/throw error):

        public List<IBar> getIBarsFromDKHistory(Instrument _iInstrument, Period _pPeriod, OfferSide _oOfferSide, 
            Filter _fFilter, int _nBarsBeforeAndIncludingTickTime,
            long _lTickTime, int _nBarsAfterTickTime) throws JFLException
      {
           List<IBar> aIBars = null;
            try
            {
               _lTickTime = history.getBarStart(_pPeriod, _lTickTime);
               aIBars = history.getBars(_iInstrument, _pPeriod, _oOfferSide, _fFilter, _nBarsBeforeAndIncludingTickTime, _lTickTime, _nBarsAfterTickTime);      
               if (gnDebugCheckDKGetBarsResults > 0)
               {
                  if (aIBars != null && _fFilter == Filter.ALL_FLATS)
                  {
                     for (IBar iBar: aIBars)
                     {
                        if (iBar.getOpen() == iBar.getClose() && iBar.getOpen() == iBar.getHigh() && iBar.getOpen() == iBar.getLow())
                        {
                           if (gnDebugCheckDKGetBarsResults == 1)   //warning/logmessage
                           {
                                 this.debug.log("getIBarsFromDKHistory", "CheckError: Flat bar [" + cfgMain.fDateTimeFormat.format(iBar.getTime()) + " OHLC=" + iBar.getOpen() + "] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving " + _nBarsBeforeAndIncludingTickTime + " bars before and " + _nBarsAfterTickTime + " behind " + cfgMain.fDateTimeFormat.format(_lTickTime) + "[" + _iInstrument + ", " + _pPeriod + ", " + _oOfferSide + ", " + _fFilter + "]");      
                           }
                           else
                           {
                                 throw new JFLException("getIBarsFromDKHistory", "LBSDH#91", "CheckError: Flat bar [" + cfgMain.fDateTimeFormat.format(iBar.getTime()) + " OHLC=" + iBar.getOpen() + "] returned by DukasCopy history.getBars() despite filter.ALL_FLATS while retrieving " + _nBarsBeforeAndIncludingTickTime + " bars before and " + _nBarsAfterTickTime + " behind " + cfgMain.fDT.format(_lTickTime) + "[" + _iInstrument + ", " + _pPeriod + ", " + _oOfferSide + ", " + _fFilter + "]");      
                           }
                        }
                     }
                  }
               }
            }
            catch (JFException e)
            {
               throw new JFLException("getIBarsFromDKHistory", "LBSDH#92", "Error loading bars from DukasCopy history. Couldn't retrieve " + _nBarsBeforeAndIncludingTickTime + " bars before and " + _nBarsAfterTickTime + " behind " + cfgMain.fDT.format(_lTickTime) + "[" + _iInstrument + ", " + _pPeriod + ", " + _oOfferSide + ", " + _fFilter + "]", e);      
            }
            return aIBars;
      }

Best, RR.

API Support

Please look at the following example. It checks flats and outputs error if finds one. Running it on real time demo and in tester doesn't return any flats. The problem is not reproducible and we don't see why it could return normal candles at the start and flats after some time

package jforex; 

import java.util.*; 
import java.text.*; 

import com.dukascopy.api.*; 

public class JFOREX1466 implements IStrategy { 
   private IEngine engine; 
   private IConsole console; 
   private IHistory history; 
   private IContext context; 
   private IIndicators indicators; 
   private IUserInterface userInterface; 
    private DateFormat fDT1; 
    
   public void onStart(IContext context) throws JFException { 
      this.engine = context.getEngine(); 
      this.console = context.getConsole(); 
      this.history = context.getHistory(); 
      this.context = context; 
      this.indicators = context.getIndicators(); 
      this.userInterface = context.getUserInterface(); 
         
        fDT1 = new SimpleDateFormat("yyyy.MM.dd HH:mm:ss"); 
        fDT1.setTimeZone(TimeZone.getTimeZone("GMT")); 
   } 

   public void onAccount(IAccount account) throws JFException { 
   } 

   public void onMessage(IMessage message) throws JFException { 
   } 

   public void onStop() throws JFException { 
   } 

   public void onTick(Instrument instrument, ITick tick) throws JFException { 
   } 
    
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException { 
        if (instrument == Instrument.GBPUSD && period == Period.TEN_SECS) { 
            //take next bar start time to get current bar time, and then take last formed bar start time for 30mins 
            long time = history.getPreviousBarStart(Period.THIRTY_MINS, history.getNextBarStart(period, askBar.getTime())); 
            List<IBar> bars = history.getBars(Instrument.GBPUSD, Period.THIRTY_MINS, OfferSide.BID, Filter.ALL_FLATS, 1000, time, 0); 
            IBar prevBar = null; 
            for (IBar bar : bars) { 
                //check flats also comparing it's open price with previous bar close time 
                if (bar.getOpen() == bar.getClose() && bar.getClose() == bar.getHigh() && bar.getHigh() == bar.getLow() && prevBar != null && bar.getOpen() == prevBar.getClose()) { 
                    console.getErr().println("reqest.time=" + fDT1.format(time) + " Bar.time=" + fDT1.format(bar.getTime()) + " Open=" + bar.getOpen() + " High=" + bar.getHigh() + " Low=" + bar.getLow() + " Close=" + bar.getClose()); 
                } 
                prevBar = bar; 
            } 
            //console.getOut().println("."); 
        } 
    } 
}

RoadRunner

Hi!

Could the bug be related to a maximum time you are looking backwards to get past a period of flat bars? In the below case there was an extended flat period because of New Year and weekend.

I'm still getting the following error. Please note, that the log-message was compiled by the application. It shows the exact arguments that were passed to IHistory.getBars(...) (no later editing...)

Quote:

10:27:18 [getIBarsFromDKHistory] ...debug: CheckError@tick=Mon 2010.01.04 05:35:01.636 1 flat bar(s) in 92 bars returned by DukasCopy history.getBars() despite filter.ALL_FLATS [arguments passed: EUR/USD, 5 Mins, Bid, ALL_FLATS, 92, Mon 2010.01.04 05:30:00.000, 0]
==> 1st flat bar returned idx=0 of 91, getTime=Fri 2010.01.01 13:15:00.000 OHLC=1.43215

Best, RR.

RoadRunner

here is another instance where I can reproduce a return of a flat bar:

Quote:

[getIBarsFromDKHistory] ...debug: CheckError@tick=So 2009.01.25 22:00:00.000 1 flat bar(s) in 92 bars returned by DukasCopy history.getBars() despite filter.ALL_FLATS [arguments passed: EUR/USD, 5 Mins, Bid, ALL_FLATS, 92, So 2009.01.25 21:55:00.000, 0] ==> 1st flat bar returned idx=91 of 91, getTime=So 2009.01.25 21:45:00.000 OHLC=1.29645

The flat bar returned in the list by IHistory.getBars with the Filter.ALL_FLAT on is definitely in the weekend flat zone.
The bar before the flat bar was the last before the weekend period with a starttime at Fr 2009.01.23 21:55:00.000 with O=1.29715 H=1.29725 L=1.2959 C=1.29725.

Best, RR.

RoadRunner

JFYI - the bug only seems to happen with Filter.ALL_FLATS and not with Filter.WEEKENDS