Dukascopy
 
 
Wiki JStore Search Login

Attention! Read the forum rules carefully before posting a topic.

    Try to find an answer in Wiki before asking a question.
    Submit programming questions in this forum only.
    Off topics are strictly forbidden.

Any topics which do not satisfy these rules will be deleted.

Wait For Data
 Post subject: Wait For Data Post rating: 0   New post Posted: Mon 29 Jul, 2013, 21:13 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
I am working with a strategy which collects a lot of sentiment data. Upon my first execution, it seems as though the data isn't fully received before the next part of the strategy exectutes. This produces an error (NullPointerException) later on in the strategy.

Is there a way for me to confirm the data is received before the rest of the strategy executes?


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Mon 29 Jul, 2013, 22:26 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
An example of the error I am encountering is attached.

Using the example from the wiki, if you request 1000 bars at once, the following errors are received:
Quote:
21:20:27 Strategy "SentimentIndexInterval" is stopped at 2013-07-29 21:20:27.194 GMT on the local computer with parameters ""=[4 Hours], ""=[EUR/USD], ""=[1000]. Reason: Stopped by Engine
21:20:27 Stopping "SentimentIndexInterval" strategy at 2013-07-29 21:20:27.169 GMT on the local computer
21:20:27 java.lang.NullPointerException @ jforex.strategies.indicators.test.SentimentIndexInterval.printIndices(SentimentIndexInterval.java:39)
21:20:24 Strategy thread queue overloaded with tasks. Ticks in queue - 3, bars - 468, other tasks - 105
21:20:19 Strategy thread queue overloaded with tasks. Ticks in queue - 3, bars - 416, other tasks - 93
21:19:26 Strategy "SentimentIndexInterval" Strategy ID: ADF060E465AE84AF3AB03BA162E9757E is started at 2013-07-29 21:19:26.759 GMT on the local computer with parameters ""=[4 Hours], ""=[EUR/USD], ""=[1000]
21:19:26 Starting "SentimentIndexInterval" strategy at 2013-07-29 21:19:26.622 GMT on the local computer


I look forward to a solution to this error.

Thank you,

Brad


Attachments:
SentimentIndexInterval.java [2.04 KiB]
Downloaded 373 times
DISCLAIMER: Dukascopy Bank SA's waiver of responsability - Documents, data or information available on this webpage may be posted by third parties without Dukascopy Bank SA being obliged to make any control on their content. Anyone accessing this webpage and downloading or otherwise making use of any document, data or information found on this webpage shall do it on his/her own risks without any recourse against Dukascopy Bank SA in relation thereto or for any consequences arising to him/her or any third party from the use and/or reliance on any document, data or information found on this webpage.
 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Wed 31 Jul, 2013, 12:37 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
API Support, are you able to see this error? It generally only happens on the first run (as I believe the data is cached afterwards which eliminates the error). To recreate the error, you simply need to change your input request amount (number still needs to be fairly large). Any ideas on how to work around/fix this?

Thanks,

Brad


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Wed 31 Jul, 2013, 14:09 
User avatar

User rating:
Joined: Fri 31 Aug, 2007, 09:17
Posts: 6139
And what is the start time of the strategy? How long does it take for the exception to occur? Does the problem persist when you read data in smaller chunks?


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Thu 01 Aug, 2013, 02:28 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
According to the attached error log, the exception occurs on the 61st second of execution. I have ran this again to confirm that this always stops executing on the 61st second.

A more recent log demonstrates this:
01:01:16 Strategy "SentimentIndexInterval" is stopped at 2013-08-01 01:01:16.284 GMT on the local computer with parameters ""=[4 Hours], ""=[EUR/USD], ""=[1000]. Reason: Stopped by Engine
01:01:16 Stopping "SentimentIndexInterval" strategy at 2013-08-01 01:01:16.272 GMT on the local computer
01:01:16 java.lang.NullPointerException @ jforex.strategies.indicators.test.SentimentIndexInterval.printIndices(SentimentIndexInterval.java:39)
01:00:15 Strategy "SentimentIndexInterval" Strategy ID: ADF060E465AE84AF3AB03BA162E9757E is started at 2013-08-01 01:00:15.325 GMT on the local computer with parameters ""=[4 Hours], ""=[EUR/USD], ""=[1000]
01:00:15 Starting "SentimentIndexInterval" strategy at 2013-08-01 01:00:15.277 GMT on the local computer


I am presuming that Dukascopy has some sort of limit in place of approximately 60 seconds of start time.

By modifying the code to ensure that all bars requested are broken down into 5 pieces, more data can be requested. The code is as follows:

package jforex.strategies.indicators.test;

import java.util.List;
import com.dukascopy.api.*;

/**
 * The strategy retrieves sentiment index aggregation bars
 * for the instrument and its both currencies
 *
 */
public class SentimentIndexInterval implements IStrategy {

    private IConsole console;
    private IHistory history;
    private IDataService dataService;

    @Configurable("")
    public Period period = Period.FOUR_HOURS;
    @Configurable("")
    public Instrument instr = Instrument.EURUSD;
    @Configurable("")
    public int barCount = 1000;

    @Override
    public void onStart(IContext context) throws JFException {
        console = context.getConsole();
        history = context.getHistory();
        dataService = context.getDataService();
       
        for (int i = 5; i > 0; i--) {
            long lastTickTime = history.getTimeOfLastTick(instr);
            long from = history.getTimeForNBarsBack(period, lastTickTime, i/5*barCount);
            long to = history.getTimeForNBarsBack(period, lastTickTime, i/5*barCount-(barCount/5+1));
            printIndices(dataService.getFXSentimentIndex(instr, period, from, to), instr);
            printIndices(dataService.getFXSentimentIndex(instr.getPrimaryCurrency(), period, from, to), instr.getPrimaryCurrency());
            printIndices(dataService.getFXSentimentIndex(instr.getSecondaryCurrency(), period, from, to), instr.getSecondaryCurrency());
        }
    }
   
    private void printIndices(List<IFXSentimentIndexBar> sentimentBars, Object key){
        for (IFXSentimentIndexBar sentimentBar : sentimentBars) {
            console.getOut().println(key.toString() + " - " + sentimentBar + ", close price: " + sentimentBar.getClose());
        }
    }

    @Override
    public void onTick(Instrument instrument, ITick tick) throws JFException {}
    @Override
    public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {}
    @Override
    public void onMessage(IMessage message) throws JFException {}
    @Override
    public void onAccount(IAccount account) throws JFException {}
    @Override
    public void onStop() throws JFException {}

}


This work-around is not perfect and is not robust. Even by breaking down the requests, there is still the potential to receive this error. Is there any possibility this bug can be fixed?


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Thu 01 Aug, 2013, 07:19 
User avatar

User rating:
Joined: Fri 31 Aug, 2007, 09:17
Posts: 6139
Brad wrote:
I am presuming that Dukascopy has some sort of limit in place of approximately 60 seconds of start time.
Yes, there is a 1 minute timeout, that is why we asked for the strategy execution time to make sure.
Brad wrote:
This work-around is not perfect and is not robust. Even by breaking down the requests, there is still the potential to receive this error. Is there any possibility this bug can be fixed?
Consider working with smaller chunks and handling the case when the method call returns null.


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Thu 01 Aug, 2013, 12:47 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
Would another approach be to simply place the code outside of OnStart()?

I have tried handling the null but with no success. Can you please provide an example of how one would do this?

Thank you,

Brad


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Fri 02 Aug, 2013, 10:23 
User avatar

User rating:
Joined: Fri 31 Aug, 2007, 09:17
Posts: 6139
Please tell what are you trying to achieve?


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Wed 07 Aug, 2013, 03:19 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
My goal is for my strategy to be able to receive sufficient amounts of data without producing errors.

Upon further testing, I note that this error is received even if the data request does not originate from onStart() (however, it does then exceed the 60 second timeout).

Any guidance on how to receive large amounts of data within a strategy without receiving this error would be most appreciated.


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Thu 08 Aug, 2013, 15:10 
User avatar

User rating:
Joined: Fri 31 Aug, 2007, 09:17
Posts: 6139
Brad wrote:
My goal is for my strategy to be able to receive sufficient amounts of data without producing errors.
Your request simply times out and returning null is the API behavior for such case. The error occurs when you don't handle this case properly - when you don't you decrease the readable data chunk size when your selected amount of data can't be read from the server within a minute.
Brad wrote:
Any guidance on how to receive large amounts of data within a strategy without receiving this error would be most appreciated.
Simply decrease the chunk sizes that you read, e.g., first try to read sentiment indices with chunk size of 100, if the request returns null, then downgrade to, for example, 10.
If you face difficulties in implementing this, please provide your example program such that we can assist you.


 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Fri 09 Aug, 2013, 02:03 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
I have produced the following code to break down a 10,000 bar request into smaller chunks of 100 (as an example).

Now, I receive warnings and errors regarding the strategy thread being overloaded:
Quote:
2013-08-09 00:58:00 Strategy thread queue overloaded with tasks. Ticks in queue - 2, bars - 1715, other tasks - 465
2013-08-09 00:57:55 Strategy thread queue overloaded with tasks. Ticks in queue - 3, bars - 1715, other tasks - 465
2013-08-09 00:57:49 Strategy thread queue overloaded with tasks. Ticks in queue - 2, bars - 1710, other tasks - 465


Can you please provide an example of how I might correct this based on my code?

Thanks,

Brad


Attachments:
ErrorFixAttempt.java [3.87 KiB]
Downloaded 372 times
DISCLAIMER: Dukascopy Bank SA's waiver of responsability - Documents, data or information available on this webpage may be posted by third parties without Dukascopy Bank SA being obliged to make any control on their content. Anyone accessing this webpage and downloading or otherwise making use of any document, data or information found on this webpage shall do it on his/her own risks without any recourse against Dukascopy Bank SA in relation thereto or for any consequences arising to him/her or any third party from the use and/or reliance on any document, data or information found on this webpage.
 
 Post subject: Re: Wait For Data Post rating: 0   New post Posted: Sun 11 Aug, 2013, 23:00 
User avatar

User rating: 0
Joined: Fri 19 Apr, 2013, 11:35
Posts: 11
Location: Canada, Ottawa
I have fixed this error by sending the data requests from their own thread. In case anyone encounters this error, I have posted my fix for this.


Attachments:
ErrorFixAttempt.java [4.32 KiB]
Downloaded 383 times
DISCLAIMER: Dukascopy Bank SA's waiver of responsability - Documents, data or information available on this webpage may be posted by third parties without Dukascopy Bank SA being obliged to make any control on their content. Anyone accessing this webpage and downloading or otherwise making use of any document, data or information found on this webpage shall do it on his/her own risks without any recourse against Dukascopy Bank SA in relation thereto or for any consequences arising to him/her or any third party from the use and/or reliance on any document, data or information found on this webpage.
 

Jump to:  

  © 1998-2025 Dukascopy® Bank SA
On-line Currency forex trading with Swiss Forex Broker - ECN Forex Brokerage,
Managed Forex Accounts, introducing forex brokers, Currency Forex Data Feed and News
Currency Forex Trading Platform provided on-line by Dukascopy.com