hi,

When using array based Indicator Calculation it is required to feed into the indicator an input array (in some cases a multi dimensional array) of numbers.

So how would you handle array manipulation in order to reduce CPU and Garbage Collection to a minimum?

The naive approach is to create a new array on each iteration. It is very costly due to the intensive GC.

Another approach would be to let the indicator save a state and feed into it only the new available numbers. From inspecting the Indicators (at least those that come with the API), I don't think that there is a State Mechanism built into them (meaning they are stateless) so this approach is not practical. (Is this the case or did I miss something?)

The third approach to deal with this kind of inefficiency is saving the array in our Strategy's state management (thus avoiding the GC issue). Still we will need to shift the number in the array (CPU intensive) on each iteration (throwing old items out, shifting numbers and finally adding the new items in == FIFO).

My question - what is your recommendation? is there any way around this number shifting operation? for example - can we play with the calculation indexes on each iteration?

any other suggestion that I'm not aware of?

thx,
cb888trader.

Consider:
1) Keeping your values inside an array with a buffer and just keep track on the last value index.
2) Calculate the indicator by passing start and end indexes.
Consider the following strategy:

package jforex.indicators.test;

import java.text.DecimalFormat;

import com.dukascopy.api.*;
import com.dukascopy.api.IIndicators.MaType;
import com.dukascopy.api.indicators.IIndicator;

@RequiresFullAccess
public class InputsManagerStrategy2 implements IStrategy {

   @Configurable("Instrument")
   public Instrument instrument = Instrument.EURUSD;
   @Configurable("Offer Side")
   public OfferSide offerSide = OfferSide.BID;
   @Configurable("MA Type")
   public MaType maType = MaType.SMA;
   @Configurable("MA TimePeriod")
   public int maTimePeriod = 3;
   @Configurable("MA Shift")
   public int maShift = 0;
   @Configurable("Log values")
   public boolean logValues = true;

   private IConsole console;
   private IIndicators indicators;
   private Period period = Period.TEN_SECS;

   //initial price array
   double[][] inputs = { { 1.4000, 1.4001, 1.4002, 1.4003, 1.4004, 1.4005, 1.4006, 1.4007, 1.4008, 1.4009 },
         { 1.5000, 1.5001, 1.5002, 1.5003, 1.5004, 1.5005, 1.5006, 1.5007, 1.5008, 1.5009 },
         { 1.6000, 1.6001, 1.6002, 1.6003, 1.6004, 1.6005, 1.6006, 1.6007, 1.6008, 1.6009 },
         { 1.7000, 1.7001, 1.7002, 1.7003, 1.7004, 1.7005, 1.7006, 1.7007, 1.7008, 1.7009 },
         { 25.000, 25.001, 25.002, 25.003, 25.004, 25.005, 25.006, 25.007, 25.008, 25.009 } };

   public static DecimalFormat df = new DecimalFormat("0.00000");
   private InputManager inputsManager;

   @Override
   public void onStart(IContext context) throws JFException {

      console = context.getConsole();
      indicators = context.getIndicators();

      //inputs manager will keep an array of size 5 * (10+100)
      inputsManager = new InputManager(inputs, 100);
      
      log("5 close prices with shift 3: " + arrayToString(inputsManager.getPrices(3, 5, InputManager.CLOSE)));
   }

   @Override
   public void onTick(Instrument instrument, ITick tick) throws JFException {
      if (instrument != this.instrument) {
         return;
      }
      inputsManager.updateLatestInput(tick);
      // log last 3 close prices
      log("Last 3 close prices: " + arrayToString(inputsManager.getPrices(0, 3, InputManager.CLOSE)));
   }

   @Override
   public void onBar(Instrument instrument, Period period, IBar askBar, IBar bidBar) throws JFException {
      if(instrument != this.instrument || period != this.period){
         return;
      }
      double price = bidBar.getClose();
      //on new bar all prices are the same - they match the previous close price and volume is 0
      inputsManager.addInput(price, price, price, price, 0);
      
      //log last 3 close prices
      log("last 3 close prices: " + arrayToString(inputsManager.getPrices(0, 3, InputManager.CLOSE)));
      
      double ma = getMa(this.maTimePeriod, this.maShift);
      print("ma: " + ma + " on bar: " + bidBar);
   }

   /**
    * Calculates MA over an array, see more:
    * https://www.dukascopy.com/wiki/index.php?title=Strategy_API:_Indicators#Calculate_indicator_on_array
    * 
    * @param timePeriod
    * @param shift
    * @return
    */
   private double getMa(int timePeriod, int shift) {
      IIndicator maIndicator = indicators.getIndicator("MA");

      double result = 0;
      int outputCount = shift + timePeriod;

      // set optional inputs
      maIndicator.setOptInputParameter(0, timePeriod);
      maIndicator.setOptInputParameter(1, maType.ordinal()); 

      // set inputs
      // Note that we pass the whole 100+10 array
      double [] priceArr = inputsManager.getAllPrices(InputManager.CLOSE);
      maIndicator.setInputParameter(0, priceArr);

      // set outputs
      double[] resultArr = new double[outputCount];
      maIndicator.setOutputParameter(0, resultArr);
      
      // calculate
      // Note that we specify the range of values that we will calculate on
      maIndicator.calculate(inputsManager.getLastPriceIndex() - outputCount + 1, inputsManager.getLastPriceIndex());
      
      log("ma result array: " + arrayToString(resultArr));
      log("Moving Indicator Lookback = " + maIndicator.getLookback());
      log("resultArr.length = " + resultArr.length);
      int index = resultArr.length - shift - 1 - maIndicator.getLookback();
      log("Retreiving MA from resultArr[" + index + "]");
      result = resultArr[index];

      return result;
   }

   @Override
   public void onMessage(IMessage message) throws JFException {
   }

   @Override
   public void onAccount(IAccount account) throws JFException {
   }

   @Override
   public void onStop() throws JFException {
   }

   private void print(Object o) {
      console.getOut().println(o);
   }
   
   private void log(Object o) {
      if(logValues) print(o);
   }

   public static String arrayToString(double[] arr) {
      String str = "";
      for (int r = 0; r < arr.length; r++) {
         str += "[" + r + "] " + df.format(arr[r]) + "; ";
      }
      return str;
   }
}

class InputManager {

   private double[][] inputs;
   private int curPos;
   private final int buffer;// = 100;

   public static final int PRICE_INPUT_SIZE = 5;

   // Indices in price input array (according to the order in input type PRICE)
   public static final int OPEN = 0;
   public static final int CLOSE = 1;
   public static final int HIGH = 2;
   public static final int LOW = 3;
   public static final int VOLUME = 4;

   public InputManager(double[][] priceInputs, int bufferSize) {

      buffer = bufferSize;
      inputs = new double[PRICE_INPUT_SIZE][priceInputs[0].length + buffer];
      for (int i = 0; i < PRICE_INPUT_SIZE; i++) {
         System.arraycopy(priceInputs[i], 0, inputs[i], 0, priceInputs[i].length);
      }
      curPos = priceInputs[0].length - 1;
   }

   // add input at the end of the array
   public void addInput(double open, double close, double high, double low, double volume) {
      curPos++;

      if (curPos >= inputs[0].length) {
         resetBuffers();
      }
      
      inputs[OPEN][curPos] = open;
      inputs[CLOSE][curPos] = close;
      inputs[HIGH][curPos] = high;
      inputs[LOW][curPos] = low;
      inputs[VOLUME][curPos] = volume;
      
   }

   public void updateLatestInput(ITick tick) {
      inputs[CLOSE][curPos] = tick.getBid();
      inputs[HIGH][curPos] = inputs[HIGH][curPos] < tick.getBid() ? tick.getBid() : inputs[HIGH][curPos];
      inputs[LOW][curPos] = inputs[LOW][curPos] > tick.getBid() ? tick.getBid() : inputs[LOW][curPos];
      inputs[VOLUME][curPos] += tick.getBidVolume();
   }

   private void resetBuffers() {
      double[][] emptyBufferArray = new double[PRICE_INPUT_SIZE][buffer];

      for (int i = 0; i < PRICE_INPUT_SIZE; i++) {
         System.arraycopy(inputs[i], buffer, inputs[i], 0, inputs[i].length - buffer);
         System.arraycopy(emptyBufferArray[i], 0, inputs[i], inputs[i].length - buffer, buffer);
      }

      curPos = inputs[0].length - buffer;
   }

   public double[] getPrices(int shift, int count, int priceIndex) {
      double[] resultArray = new double[count];
      System.arraycopy(inputs[priceIndex], curPos - count - shift + 1, resultArray, 0, count);
      return resultArray;
   }

   public double getPrice(int priceIndex) {
      return getPrice(0, priceIndex);
   }

   public double getPrice(int shift, int priceIndex) {
      return inputs[priceIndex][curPos - shift];
   }

   // for indicator calculations
   public double[] getAllPrices(int priceIndex) {
      return inputs[priceIndex];
   }

   public int getLastPriceIndex() {
      return curPos;
   }

   public int getFirstPriceIndex() {
      return 0;
   }

}

Note that the strategy only copies the arrays once the buffer got full and on logging (which is optional).

hi Support,

This is very interesting. I have a few follow up questions -

1. When resetting the buffer - why do you need to fill empty values? isn't it just enough to update the curPos? looks to me like it is redundant code (please correct me if I'm wrong). It will save both CPU cycles and Heap allocations/GC. In any case there is no need to declare a new empty array since it is not getting filled with anything and it is GC when the method returns.

2. Wouldn't it be more efficient to do a manual shift of numbers instead of using the System.arraycopy? this will reduce another heap allocation/ GC . From the Java API documentation:

"If the src and dest arguments refer to the same array object, then the copying is performed as if the components at positions srcPos through srcPos+length-1 were first copied to a temporary array with length components and then the contents of the temporary array were copied into positions destPos through destPos+length-1 of the destination array. "

3. Why do you update the input array on each tick by calling updateLatestInput? Since we calculate the indicator only on new bar arrival isn't that redundant?

thx,
cb888trader.

It is sufficient. However, if you want to log the full array then zeroes make it explicit where the buffer is and which is the last value.
System.arraycopy only gets applied only once (if we don't count the optional logging), it is up to you to opt for this method or simple iteration over an array. Besides, in the following performance test - arraycopy significantly outperforms simple iteration:

package utils;

public class CopyTest {
   static byte[] in;
   static byte[] out;

   public static void main(String[] args) {
      arrayTest(100000, 200);
   }

   public static void arrayTest(int size, int iterations) {
      try {

         in = new byte[size];
         out = new byte[size];

         long arraycopyTime = 0;
         long loopcopyTime = 0;

         long before = System.currentTimeMillis();
         for (int i = 0; i < iterations; i++) {
            System.arraycopy(in, 0, out, 0, size);
         }
         arraycopyTime = System.currentTimeMillis() - before;
         before = System.currentTimeMillis();
         for (int i = 0; i < iterations; i++) {
            for (int j = 0; j < in.length; j++) {
               out[j] = in[j];
            }
         }
         loopcopyTime = System.currentTimeMillis() - before;
         System.out.println("Array size: " + size + ", Repetitions: " + iterations);
         System.out.println("Loop: " + loopcopyTime + "ms, System.arrayCopy:" + arraycopyTime + "ms");

      } catch (Exception e) {
         e.printStackTrace();
      }
   }
}

This is redundant for the logic of the particular case. It is there to show how you can operate with the buffered array.

hi Support,

In regards to System.arraycopy - my bad. I should have tested more carefully before criticizing . Since it is a native implementation it actually runs an order of magnitude faster than any java bytecode counterpart. See more details here (#3)

Still - there is the issue of memory re-allocation on every 100 bars (buffer reset). Which makes me wonder why indicators are not working directly with IBar objects instead of raw arrays. This way we could have just used Object Queues and pass an iterator of that queue on each calculation.

In regards to the example provided there are 2 issues -

First issue - there is a bug in the following line:

inputs[VOLUME][curPos] = tick.getBidVolume();

volume should accumulate:

inputs[VOLUME][curPos] += tick.getBidVolume();

Second issue -
It is worth pointing out that the OCHL price calculations assume time based bars. Using the example as is for Range Bars will result in invalid calculations.

Consider doing a bit research on java primitives vs objects. Also you are free do implement and performance-test any implementation of InputManager you please (i.e. add a constructor which takes as parameter List<IBar>). Also you can compare the performance against API provided calculation of MA (i.e. IIndicators.ma).
This is fixed in the upper post
It would take a few adjustments to use it with range bars:
1) Remove code from onTick
2) Move IStrategy.onBar content to IRangeBarFeedListener.onBar.

hi Support,

In another thread we discussed the issue of unstable period. I came to the conclusion that in order to stay in line with the UI indicator value I should use 100 samples for each calculation.

In your InputManager utility - which is a great piece of code you use the following for index calculation:

int outputCount = shift + timePeriod;

and

// calculate
// Note that we specify the range of values that we will calculate on
maIndicator.calculate(inputsManager.getLastPriceIndex() - outputCount + 1, inputsManager.getLastPriceIndex());

My question is - how is this taking into account the unstable issue. Shouldn't we calculate 100 items on each iteration?

For the sake of simplicity it does not consider this case, the example is not so general.
In general case - yes.