Dukascopy Support Board
http://www.dukascopy.com/swiss/english/forex/jforex/forum/

NullPointerException in TesterClientImpl.connect()
http://www.dukascopy.com/swiss/english/forex/jforex/forum/viewtopic.php?f=16&t=57160
Page 1 of 1

Author:  Edwin [ Thu 26 Sep, 2019, 11:20 ]
Post subject:  NullPointerException in TesterClientImpl.connect()

Caused by - java.lang.NullPointerException: null
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1690)
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.getOrCreateListenersSet(FeedDataProvider.java:3190) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.addCacheDataUpdatedListener(FeedDataProvider.java:2477) *
      * at com.dukascopy.charts.data.datacache.IntraperiodCandlesGenerator.<init>(IntraperiodCandlesGenerator.java:140) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.<init>(FeedDataProvider.java:286) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.<init>(FeedDataProvider.java:234) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.createFeedDataProvider(FeedDataProvider.java:367) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.createFeedDataProvider(FeedDataProvider.java:341) *
      * at com.dukascopy.charts.data.datacache.FeedDataProvider.createFeedDataProvider(FeedDataProvider.java:331) *
      * at com.dukascopy.api.impl.connect.TesterClientImpl.connect(TesterClientImpl.java:255) *
      * at com.dukascopy.api.impl.connect.TesterClientImpl.connect(TesterClientImpl.java:221) *
      * at de.invesdwin.trading.jforex.runtime.JForexRuntimeProperties.getAvailableInstrumentsFromTesterClient(JForexRuntimeProperties.java:57) *


(I guess Instrument.values() somehow contains null values due to some initialization race condition on first connect?)

The code that is doing the connect is:

    private static Set<Instrument> getAvailableInstrumentsFromTesterClient()
            throws JFAuthenticationException, JFVersionException, Exception {
        final ITesterClient client = TesterFactory.getDefaultInstance();
        try {
            if (!client.isConnected()) {
                client.connect(JFOREX_JNLP_URL.toString(), JFOREX_JNLP_USERNAME, JFOREX_JNLP_PASSWORD);
            }
            final Set<Instrument> availableInstruments = client.getAvailableInstruments();
            return availableInstruments;
    }


This happens only recently after I have upgraded to the current version of jforex:
<version.jforex-3-sdk>[3.6.5,)</version.jforex-3-sdk>
<version.jforex-api>[2.13.76,)</version.jforex-api>
<version.greed-common>[318.4.87,)</version.greed-common>
<version.dds2-charts>[6.24.89,)</version.dds2-charts>

Author:  EdwinStang [ Fri 04 Oct, 2019, 09:31 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

The nullpointer did not occur anymore for a few days now. Though I experience tick messages stopping to arrive after a few minutes of running. All threads are then idle, no deadlocks. I am connected from multiple machines to my permanent demo account. All of them stop after some individual time between a few minutes and a few hours (tested with a simple strategy that logs the tick time in the handler method). Though Client.isConnected still reports being connected.

Author:  EdwinStang [ Fri 04 Oct, 2019, 09:43 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

And I am getting lots of these errors then:
Caused by - com.dukascopy.api.JFException: Error while loading ticks
      * at com.dukascopy.api.impl.HistoryData.getTicksSecured(HistoryData.java:103) *
      * at com.dukascopy.api.impl.History.lambda$getTicks$4(History.java:370) *
      * at com.dukascopy.api.impl.History.lambda$doPrivileged$116(History.java:3498) *
        at java.base/java.security.AccessController.doPrivileged(Native Method)
      * at com.dukascopy.api.impl.History.doPrivileged(History.java:3498) *
      * at com.dukascopy.api.impl.History.getTicks(History.java:368) *
      * at de.invesdwin.trading.jforex.broker.instrument.internal.data.tick.JForexTickCache.getTicks(JForexTickCache.java:406) *

Author:  EdwinStang [ Fri 04 Oct, 2019, 12:07 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

Nope, the initially reported error is still happening as of today. I guess I was just lucky the last few days.
Maybe there is a race condition with multiple java processes using the same demo account login?

Author:  EdwinStang [ Thu 10 Oct, 2019, 08:19 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

I verified that the connection drops occur even when only one client is connected to the account.

Now it got worse that even the exception is not thrown, instead the client just does not receive any tick events

Author:  Edwin [ Mon 18 Nov, 2019, 14:53 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

The connection drops happened because I had multiple processes share the same JForex folder. I fixed that by only having one process/connection per folder now.

The "Error loading ticks" still happens sometimes, though not as often as before and I just made the process reconnect in that case to work around the issue.

The initialization error (NullPointerException) still cannot be recovered from on first connect (any subsequent connect attemps also throw this exception). If that happens I now restart my process to then try to connect again. Since I cannot have my strategy processes restart all the time, I have separated the processes now with my own middleware and IPC mechanisms. The strategies processes will simply wait for the JForex connection process to restart itself if it has gone away.

It is not ideal, but the best I can do on my side to live with these problems.

Author:  API Support [ Fri 20 Dec, 2019, 16:20 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

Hello.

NPE during connect could occur if several threads are connecting concurrently.
Could your method de.invesdwin.trading.jforex.runtime.JForexRuntimeProperties.getAvailableInstrumentsFromTesterClient be called from several threads simultaneously?

Kind regards, Support Team.

Author:  Edwin [ Fri 03 Jan, 2020, 22:31 ]
Post subject:  Re: NullPointerException in TesterClientImpl.connect()

You were right. While the ITesterClient from JForexRuntimeProperties.getAvailableInstrumentsFromTesterClient was connecting, an DCClientImpl connection for a live account was established at the same time. Synchronizing both connects fixed the NPE error. Though the stale connections were still happening. I traced it down to two things:

1. When ITesterClient is connected before DCClientImpl data feeds are not feeding new data (despite instrument subscriptions because I guess Backtest mode is enabled for the live connection)
2. When disconnecting ITesterClient while DCClientImpl is connected, DCClientImpl's data feed disconnects too while the DCClientImpl still thinks it is connected itself.

I guess the same problem would occur when trying to connect multiple servers (demo vs live) and accounts in the same java process. The connections would all break each other since they seem to be sharing the data feed and so on. Thus there are likely some singletons they share and close individually despite other connections still using those singletons.

I have added additional checks to throw exceptions when one of these problems are occuring. I feel this should fix my stability problems mostly. It is a bit sad that JForex API does not support multiple account connections. But I guess I could work around that by using classloader separation (might be more light weight than separating each connection via processes).

Though I guess in the future I will have to request FIX API access in order to support multiple accounts trading via Dukascopy. This should be easier to handle in my trading platform backend/middleware.

Thanks for your help!

  Page 1 of 1