Dukascopy Support Board • View topic - Can be Historical price data wrong/corrupted that we retrieved via IHistory?

NonikNone

I am about to go crazy. My strategy fires orders wrongly because the close price of previous bar is WRONG?

 public void onTick(Instrument instrument, ITick tick) throws JFException {
        if (!instrument.equals(Instrument.EURUSD)) return;               

        DateFormat formatter = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss");
        formatter.setTimeZone(TimeZone.getTimeZone("Etc/GMT-3"));
        Calendar iCalendar = Calendar.getInstance();
        iCalendar.setTimeInMillis(tick.getTime());
        int MinuteOfTime= iCalendar.get(Calendar.MINUTE);
        
        if(MinuteOfTime==0){ // If the new ONE_HOUR bar created
            IBar previousBar = history.getBar(Instrument.EURUSD, Period.ONE_HOUR, OfferSide.BID, 1); 
            IBar newBar = history.getBar(Instrument.EURUSD, Period.ONE_HOUR, OfferSide.BID, 0); 
            writeToLogFile(previousBar.getClose() + " " + newBar.getOpen(), formatter.format(iCalendar.getTime()));
        }
}

Log Output:

09/05/2018 01:00:00 (GMT)  previousBar.getClose: 1.1866 newBar.getOpen: 1.18542

"previousBar.getClose: 1.1866" is wrong, it is 1.18540 actually. And in the following ticks, previousBar.getClose still coming as 1.1866, until restart the strategy. So, I understand that history.getBar does not request data every time, must be a cache policy, right?

In my log, newBar.getOpen always correct but previousBar.getClose sometimes incorrect, I tried the same code in onBar event but same thing happened. At least one time in a day it comes wrong.

I got two questions?
Can be Historical price data wrong/corrupted that we retrieved via IHistory?
And can we delete the cache of IHistory via JForex API?

Dear support team, please help.

mtnfx

I'm curious why you are not using 'onBar' method and emulating the same in onTick method.

I'm not sure what happens internally in JForex when trying to get bar via reverse index. It may happen in this 'edge case', when current tick is 'between 2 bars', that
-> '0' - points to current 'just started' bar,
-> whereas '1' actually points 2 bars back.

    @Override
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
        if (instrument != Instrument.EURUSD || period != Period.ONE_HOUR )
            return;
        
        // do your staff
    }

NonikNone

mtnfx wrote:

I'm curious why you are not using 'onBar' method and emulating the same in onTick method.

I'm not sure what happens internally in JForex when trying to get bar via reverse index. It may happen in this 'edge case', when current tick is 'between 2 bars', that
-> '0' - points to current 'just started' bar,
-> whereas '1' actually points 2 bars back.

    @Override
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
        if (instrument != Instrument.EURUSD || period != Period.ONE_HOUR )
            return;
        
        // do your staff
    }

You're right, it's just for testing purpose, I use onBar method normally and the same thing happens in the onBar event, the incorrect price is not the closing price of any previous bars by the way, there is no one_hour bar closing with that price. So it is not that edge point you mentioned.

I used onTick method to detect what happens in the following ticks. And keeps coming wrong price. But if I restart the strategy, onTick method starts logging the correct price. So IHistory must be reading from cache in some cases, I guess.

Please check the price on EURUSD One_Hour chart for following dates the result is different with the historical data:

09/05/2018 01:00:00 (GMT) previousBar.getClose: 1.1866 newBar.getOpen: 1.18542
01/05/2018 09:00:00 (GMT) previousBar.getClose: 1.2059 newBar.getOpen: 1.20343

mtnfx

All I can say about that is

- testing via 'onTick' is not the way which 'can show something' - it is wrong test, use 'onBar'
- it may happen that Jforex has some bug inside it - support team may help you if you have reproducible strategy (though ... they are not very quick in that)
- you can check what's in history using export to CSV - see 'Main menu -> View -> Historic Data Manager' - there you can export bars or ticks.

Note: as far as I see - 'previous close' is not always the same as 'next open'. As far as i understand 'close' is the price of last tick within some bar, 'open' is price of the first tick withing some bar.
Gaps of different size may occurs between two consecutive ticks, especially during weekends - last tick on Friday -> First tick on Monday. Therefore some price difference between close/next-open is normal.

I've looked into CSV export for your period on my machine - 'previousBar.getClose: 1.1866' at '09/05/2018 01:00:00' does not match - 1.1854

Time (UTC)   Open   High   Low   Close   Volume 
2018.05.08 23:00:00   1.18594   1.18682   1.18594   1.18671   2428.64
2018.05.09 00:00:00   1.18673   1.18697   1.18534   1.1854   7803.1
2018.05.09 01:00:00   1.18542   1.18549   1.18427   1.1853   9928.69
2018.05.09 02:00:00   1.1853   1.18597   1.18505   1.18537   5764.71
2018.05.09 03:00:00   1.18536   1.18592   1.1851   1.18519   6084.78
2018.05.09 04:00:00   1.1852   1.18559   1.18505   1.1855   4058.99

mtnfx

All I can say about that is

- testing via 'onTick' is not the way which 'can show something' - it is wrong test, use 'onBar'
- it may happen that Jforex has some bug inside it - support team may help you if you have reproducible strategy (though ... they are not very quick in that)
- you can check what's in history using export to CSV - see 'Main menu -> View -> Historic Data Manager' - there you can export bars or ticks.

Note: as far as I see - 'previous close' is not always the same as 'next open'. As far as i understand 'close' is the price of last tick within some bar, 'open' is price of the first tick withing some bar.
Gaps of different size may occurs between two consecutive ticks, especially during weekends - last tick on Friday -> First tick on Monday. Therefore some price difference between close/next-open is normal.

I've looked into CSV export for your period on my machine - 'previousBar.getClose: 1.1866' at '09/05/2018 01:00:00' does not match - 1.1854

Time (UTC)   Open   High   Low   Close   Volume 
2018.05.08 23:00:00   1.18594   1.18682   1.18594   1.18671   2428.64
2018.05.09 00:00:00   1.18673   1.18697   1.18534   1.1854   7803.1
2018.05.09 01:00:00   1.18542   1.18549   1.18427   1.1853   9928.69
2018.05.09 02:00:00   1.1853   1.18597   1.18505   1.18537   5764.71
2018.05.09 03:00:00   1.18536   1.18592   1.1851   1.18519   6084.78
2018.05.09 04:00:00   1.1852   1.18559   1.18505   1.1855   4058.99

NonikNone

mtnfx wrote:

- it may happen that Jforex has some bug inside it - support team may help you if you have reproducible strategy (though ... they are not very quick in that)

Thank you for your reply,
Yes, I think there is a bug, because there is another post in the forum from 2011 saying same thing.
I am trying to find gaps within that strategy, but sometimes it behaves like there is a gap when there is not. It is very easy to reproduce the case here is the code, by logging close and open prices for 2 days and checking gaps if they are real or not.

    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
        if (!instrument.equals(Instrument.EURUSD)) return;         
        if(period.equals(Period.ONE_HOUR)){          
            newBar = history.getBar(Instrument.EURUSD, Period.ONE_HOUR, OfferSide.BID, 0); //I also tried getBars method with bidBar.getTime parameter but same thing happens.
            previousBar = history.getBar(Instrument.EURUSD, Period.ONE_HOUR, OfferSide.BID, 1);          
            writeToLog(previousBar.getClose() + " " + newBar.getOpen() + "");
    }

    public void writeToLog(String text){
        try{
            FileWriter fileOpenWidthAppendMode = new FileWriter("C:/Users/Ugur/Documents/Visual Studio 2015/Projects/Unicorn/Unicorn/log_test.txt", true);
            BufferedWriter contentToWrite = new BufferedWriter(fileOpenWidthAppendMode);
            contentToWrite.append(text + "\r\n");
            contentToWrite.close();
        }catch(Exception e){
                
        }       
    }

This is very dangerous, it means that strategies will use wrong data without knowing that.

Platform Support

We have found the possible reason why not finished candle could be stored in historical data cache. The chages will be available in next version. We plan to release it next week on DEMO.

NonikNone

Platform Support wrote:

We have found the possible reason why not finished candle could be stored in historical data cache. The chages will be available in next version. We plan to release it next week on DEMO.

Great news, thank you.