I was wondering if its possible to load a different matrix in each iteration of a loop. I can't load the full data together as GAUSS runs into "error G0030 : Insufficient memory" after a few iterations. As I will need matrix1 only in iteration1 and so on, I wanted to reuse the space allotted to the matrix in each iteration by loading a different matrix, but storing it under the same name (all the matrices have the same dimension). However, I am not sure if there is a way to dynamically change the load statement so that it reads different files in each iteration.
Any suggestion on how to handle this issue would be greatly appreciated.
Do all these matrices have the same number of columns and same column types?
Yes, they do.
All the matrices have the same number of rows and columns and same column types.
I would greatly appreciate any suggestion in this issue.
If they all have the same number of columns, you can the
readr function to load sequential chunks of rows from a dataset. For example:
new; /* ** Create dataset for example */ rndseed 432143; X = seqa(1, 1, 10) ~ rndn(10,3); fname = "test_data.dat"; call saved(X, fname, 0); // Open file handle to access dataset fh = dataOpen(fname, "read"); // Load 2 rows per iteration for i(1, 5, 1); X_tmp = readr(fh, 2); print "Chunk "$+ntos(i)$+" of "$+ntos(5); print X_tmp; endfor; call close(fh);
If all the data won't fit in memory at once, you can use
writer to create the dataset.