Dukascopy
 
 
Wiki JStore Search Login

Attention! Read the forum rules carefully before posting a topic.

    Try to find an answer in Wiki before asking a question.
    Submit programming questions in this forum only.
    Off topics are strictly forbidden.

Any topics which do not satisfy these rules will be deleted.

Can't get ticks data
 Post subject: Can't get ticks data Post rating: 0   New post Posted: Thu 17 Dec, 2015, 14:56 
User avatar

User rating: 0
Joined: Sun 08 Mar, 2015, 23:36
Posts: 7
Location: Russian FederationRussian Federation
Hello, I'm trying to get ticks from 54 weeks before to current time, so I get an exception and can't get moar than 20 weeks.
Why I need this? Because I need tick data to learn my neural net.
Please, what I need to do?
CODE:
public class MA_Play implements IStrategy {
...
    private final int WEEKS=54;
...
    @Override
    public void onStart(IContext context) throws JFException {
    ...
        /**
         * GETTING TICKS.
         */
        ITick tick = history.getLastTick(instrument);
        long time = tick.getTime();
        List<ITick> ticks = new ArrayList<>();
        long TC = System.currentTimeMillis();
        for(int i=WEEKS; i>=0; i--) {
            //dwatchdogTimer.refresh();
            DateTime beginWeek = new DateTime(time).minusWeeks(i).withDayOfWeek(DateTimeConstants.MONDAY).withTimeAtStartOfDay();
            DateTime endWeek = i>0?new DateTime(time).minusWeeks(i).withDayOfWeek(DateTimeConstants.SUNDAY).withTimeAtStartOfDay():new DateTime(time);
            System.out.print("Week:"+i+". Getting ticks from "+beginWeek.toString("dd.MM.yy")+" to "+endWeek.toString("dd.MM.yy"));
           
            List<ITick> t = history.getTicks(instrument, beginWeek.getMillis(), endWeek.getMillis());
            System.out.print(", size = "+t.size());
            ticks.addAll(t);
            System.out.println(", overall size = "+ticks.size());
            if (i%20==0 && i>0) {
                try {
                    System.out.println("Please, wait 6 min.");
                    Thread.sleep(1000*60*6);
                } catch (InterruptedException ex) {
                    Logger.getLogger(MA_Play.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        }
        System.out.println("Recieved "+ticks.size()+" ticks in "+(System.currentTimeMillis()-TC)+" ms.");
    ...
    }
...
}


LOG:
2015-12-17 16:27:43.789 INFO  Reflections - Reflections took 173 ms to scan 6 urls, producing 116 keys and 460 values 
Week:54. Getting ticks from 01.12.14 to 07.12.14, size = 415296, overall size = 415296
Week:53. Getting ticks from 08.12.14 to 14.12.14, size = 461303, overall size = 876599
Week:52. Getting ticks from 15.12.14 to 21.12.14, size = 483296, overall size = 1359895
Week:51. Getting ticks from 22.12.14 to 28.12.14, size = 179969, overall size = 1539864
Week:50. Getting ticks from 29.12.14 to 04.01.15, size = 217229, overall size = 1757093
Week:49. Getting ticks from 05.01.15 to 11.01.15, size = 417475, overall size = 2174568
Week:48. Getting ticks from 12.01.15 to 18.01.15, size = 560573, overall size = 2735141
Week:47. Getting ticks from 19.01.15 to 25.01.15, size = 495910, overall size = 3231051
Week:46. Getting ticks from 26.01.15 to 01.02.15, size = 491280, overall size = 3722331
Week:45. Getting ticks from 02.02.15 to 08.02.15, size = 444021, overall size = 4166352
Week:44. Getting ticks from 09.02.15 to 15.02.15, size = 409768, overall size = 4576120
Week:43. Getting ticks from 16.02.15 to 22.02.15, size = 376471, overall size = 4952591
Week:42. Getting ticks from 23.02.15 to 01.03.15, size = 385276, overall size = 5337867
Week:41. Getting ticks from 02.03.15 to 08.03.15, size = 372401, overall size = 5710268
Week:40. Getting ticks from 09.03.15 to 15.03.15, size = 526213, overall size = 6236481
Please, wait 6 min.
Week:39. Getting ticks from 16.03.15 to 22.03.15, size = 508339, overall size = 6744820
Week:38. Getting ticks from 23.03.15 to 29.03.15, size = 600441, overall size = 7345261
Week:37. Getting ticks from 30.03.15 to 05.04.15, size = 474770, overall size = 7820031
Week:36. Getting ticks from 06.04.15 to 12.04.15, size = 470034, overall size = 8290065
Week:35. Getting ticks from 13.04.15 to 19.04.15, size = 551013, overall size = 8841078
Week:34. Getting ticks from 20.04.15 to 26.04.152015-12-17 16:37:17.292 INFO  ClientConnector - Child session opened for [d-ja-msk-10.dukascopy.com/195.208.187.197:10443]
2015-12-17 16:37:37.561 INFO  ClientProtocolHandler - Disconnect task in queue, reason [CONNECTION_PROBLEM], server address [d-ja-msk-10.dukascopy.com/195.208.187.197:10443], transport name [null]
Exception in thread "FeedDataProvider_ActionsThread_4" Exception in thread "LocalCacheTimer" java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
   at java.lang.StringBuffer.<init>(StringBuffer.java:128)
   at java.io.UnixFileSystem.normalize(UnixFileSystem.java:68)
   at java.io.UnixFileSystem.normalize(UnixFileSystem.java:92)
   at java.io.File.<init>(File.java:279)
   at com.dukascopy.charts.data.datacache.LocalCacheManager.clearIntraperiodCacheFolderAndKeepIt(LocalCacheManager.java:337)
   at com.dukascopy.charts.data.datacache.time.TimeManager$5.run(TimeManager.java:312)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
Exception in thread "FeedDataProvider_ActionsThread_2" java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2035)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
Exception in thread "FeedDataProvider_ActionsThread_5" java.lang.OutOfMemoryError: GC overhead limit exceeded
2015-12-17 16:41:52.689 ERROR LoadDataAction - GC overhead limit exceeded
2015-12-17 16:42:57.469 WARN  TimeManager - Local time was changed, was<2015-12-17 13:39:54:978>, now<2015-12-17 13:40:41:516>
2015-12-17 16:43:24.471 ERROR StrategyRunner - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Strategy tester: java.lang.OutOfMemoryError: GC overhead limit exceeded
GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded
2015-12-17 16:43:24.650 WARN  TimeManager - Local time was changed, was<2015-12-17 13:39:54:978>, now<2015-12-17 13:40:41:516>
2015-12-17 16:43:24.658 ERROR TesterClientImpl$DefaultStrategyExceptionHandler - Exception thrown while running onStart method: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2015-12-17 16:43:24.789 INFO  TesterMainGUIMode - Strategy stopped: 1


 
 Post subject: Re: Can't get ticks data Post rating: 0   New post Posted: Fri 18 Dec, 2015, 12:12 
User avatar

User rating:
Joined: Fri 31 Aug, 2007, 09:17
Posts: 6139
This happens because all of the ticks are being loaded into the memory List<ITick> ticks.
We would suggest using Historical Data Manager to download tick data.
If you really need to run this from a strategy, here is an example of receiving historical data without blocking whole platform:
https://www.dukascopy.com/wiki/#Feed_history


 

Jump to:  

cron
  © 1998-2025 Dukascopy® Bank SA
On-line Currency forex trading with Swiss Forex Broker - ECN Forex Brokerage,
Managed Forex Accounts, introducing forex brokers, Currency Forex Data Feed and News
Currency Forex Trading Platform provided on-line by Dukascopy.com