"Out of memory" error during sequence acquisition

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

"Out of memory" error during sequence acquisition

Philipp Rinklin
Dear all,

in our lab, we are trying to realize a sequence acquisition of ~7500 frames (should correspond to 3.7 GB) with a Leica DFC9000. However, after recording approx. 4000 frames (i.e. ~1.9 GB), the acquisition stops and the systems throws the following error message: "Out of memory to store images: Direct buffer memory". The system is set up as follows.

We are using a Win 10 Enterprise PC (64bit) with 16 GB of DDR3 RAM (4 x 4 GB), an Intel i7-3770K processor , and a 2 GB Nvidia GeForce Graphics card. The MicroManager installation is v1.4.22 (64bit) and the acquisition is launched via Matlab R2016b.

After searching this list we tried the following approaches to mitigate the problem:
- Image J memory setting set to 2048 MB
- Sequence Buffer Memory Setting set to 5120 MB (we also tried 50/100 MB which would correspond to the recommended 20-100 times size of an individual image as per the MM wiki)
- We have disabled the "Save positions in separate Image Stack Files" option as was successful here.

Unfortunately, none of these helped with the error or even changed the behavior. Does anybody here have experience with this particular error or camera and can suggest a possible approach to solve this?

Many thanks in advance and best,
Philipp

Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" error during sequence acquisition

JonD
Administrator
Hi Phillip, some musings below, hopefully others will chime in.

Philipp Rinklin wrote
in our lab, we are trying to realize a sequence acquisition of ~7500 frames (should correspond to 3.7 GB) with a Leica DFC9000. However, after recording approx. 4000 frames (i.e. ~1.9 GB), the acquisition stops and the systems throws the following error message: "Out of memory to store images: Direct buffer memory". The system is set up as follows.

We are using a Win 10 Enterprise PC (64bit) with 16 GB of DDR3 RAM (4 x 4 GB), an Intel i7-3770K processor , and a 2 GB Nvidia GeForce Graphics card. The MicroManager installation is v1.4.22 (64bit) and the acquisition is launched via Matlab R2016b.
It is worth testing a similar acquisition set up within Micro-Manager, maybe there is something funny happening at the Matlab/MM interface.

Some tidbits you didn't mention are the rate at which you are taking images in MB/s and the speed of your hard drive (benchmark with CrystalDiskMark).  The sequence buffer holds images until they can be written to disk so maybe you are overflowing the buffer.  But it sounds like you changed the sequence buffer size by an order of magnitude without noticeable effect, suggesting the problem is elsewhere.   The sequence buffer monitor plugin might help confirm that.

A quick glance at the source code suggests you are hitting an out of memory error when trying to insert an image into the cache at this line of code.


Philipp Rinklin wrote
After searching this list we tried the following approaches to mitigate the problem:
- Image J memory setting set to 2048 MB
That seems low for ImageJ memory setting but I'm no expert there.

Jon

-------------------------------------------
Jon Daniels
Applied Scientific Instrumentation
29391 West Enid Rd, Eugene, OR 97402
Phone: (541) 461-8181 x118
-------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" error during sequence acquisition

Nico Stuurman-2
In reply to this post by Philipp Rinklin
Hi Philipp,


> in our lab, we are trying to realize a sequence acquisition of ~7500 frames
> (should correspond to 3.7 GB) with a Leica DFC9000.

Interesting.  I was not aware of a Leica brander sCMOS.  What drivers
and Micro-Manager adapter do you use to run this?

> However, after recording approx. 4000 frames (i.e. ~1.9 GB), the acquisition stops and the systems throws the following error message: "Out of memory to store images: Direct buffer memory".

Are you sure of those numbers?  If you are using full frame images, then
each frame is ~ 8.4MB (2048x2048x2).  So, 4000 frames corresponds to
33.6GB.  Are you using an ROI or binning?  If so, what ROI size and what
binning?

> The system is set up as follows.
>
> We are using a Win 10 Enterprise PC (64bit) with 16 GB of DDR3 RAM (4 x 4
> GB), an Intel i7-3770K processor , and a 2 GB Nvidia GeForce Graphics card.
> The MicroManager installation is v1.4.22 (64bit) and the acquisition is
> launched via Matlab R2016b.
>
> After searching this list we tried the following approaches to mitigate the
> problem:
> - Image J memory setting set to 2048 MB
> - Sequence Buffer Memory Setting set to 5120 MB (we also tried 50/100 MB
> which would correspond to the recommended 20-100 times size of an individual
> image as per the MM wiki)
> - We have disabled the "Save positions in separate Image Stack Files" option
> as was successful  here
> <http://micro-manager.3463995.n2.nabble.com/SOLVED-quot-Out-of-memory-to-store-images-Direct-buffer-memory-quot-td7587404.html#a7587407>

16GB - 2GB (ImageJ) - 5 GB (Sequence Buffer) = 9 GB for the system and
your images.  You can use the task manager to get an idea of the amount
of memory used by other parts of the system.  If your acquisition is
really only 3.7Gb, then there should be no problem, if it is more, then
your disk will need to be fast enough to keep up, or you will run into a
problem sooner or later. Of course, it is always possible that Matlab
does something strange (for instance, how much memory is assigned to
Matlab?  It will certainly be worth testing if Micro-Manager by itself
does this correctly.  If it does, you know where to look).

Best,

Nico


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" error during sequence acquisition

Philipp Rinklin
In reply to this post by JonD
Hi Jon, hi Nico,

many thanks for the swift reply. Please find answers to your
questions/results of your recommendations below:


JonD wrote
> It is worth testing a similar acquisition set up within Micro-Manager,
> maybe there is something funny happening at the Matlab/MM interface.

I tested starting MicroManager via MatLab or on its own and ran a
multi-dimensional acquisition with settings identical to those in previous
tests. For some reason, the instance created by Matlab crashes at the same
point as before, while the "standalone" instance records the full sequence.


JonD wrote
> Some tidbits you didn't mention are the rate at which you are taking
> images in MB/s and the speed of your hard drive (benchmark with
> CrystalDiskMark).

In the experiment, we would like to record at around 5 fps with a binning of
4x4. When we tried our first experiment, we got the error. To save time
during debugging, we increased the frame rate to 50 fps. This should thus
correspond to 512 x 512 x 16 bit x 50 fps = 26.2 MB/s. In order to prevent
any issues in this regards, we save the sequence to our SSD (write speeds
~100-500 MB/s according to CrystalDiskMark).


JonD wrote
> The sequence buffer holds images until they can be written to disk so
> maybe you are overflowing the buffer.  But it sounds like you changed the
> sequence buffer size by an order of magnitude without noticeable effect,
> suggesting the problem is elsewhere. The sequence buffer monitor plugin
> might help confirm that.

Somehow, I cannot open the plugins menu when MicroManager is launched via
Matlab. After the above described issues I started paying more attention to
differences between the instance launched via Matlab and the regular MM and
found that when starting MM via Matlab no (visible) ImageJ instance starts.


JonD wrote
>
> Philipp Rinklin wrote
>> After searching this list we tried the following approaches to mitigate
>> the problem:
>> - Image J memory setting set to 2048 MB
> That seems low for ImageJ memory setting but I'm no expert there.

I tried setting the ImageJ memory setting to 8192 MB and launched another
acquisition from Matlab. The result was the same error after approx. 1.9 GB
(~4000 frames).


Nico Stuurman-2 wrote
> Interesting.  I was not aware of a Leica brander sCMOS.  What drivers and
> Micro-Manager adapter do you use to run this?

If I am not mistaken the DFC9000 is a re-branded Andor Zyla 4.2P. As such,
we used the Andor sCMOS camera adapter in MM in combination with (I believe)
the SDK3 drivers from Andor. It was not exactly straight-forward to get it
to run, but after trying a few things found here it worked.


Nico Stuurman-2 wrote
>
> Philipp Rinklin wrote
>> However, after recording approx. 4000 frames (i.e. ~1.9 GB), the
>> acquisition stops and the systems throws the following error message:
>> "Out of memory to store images: Direct buffer memory".
> Are you sure of those numbers?  If you are using full frame images, then
> each frame is ~ 8.4MB (2048x2048x2).  So, 4000 frames corresponds to
> 33.6GB.  Are you using an ROI or binning?  If so, what ROI size and what
> binning?

Correct. We use full frame acquisition but 4x4 binning.


Nico Stuurman-2 wrote
> [...] how much memory is assigned to Matlab?

The Java heap memory for Matlab is cranked up to the maximum value available
on this system (~4 GB). I also tried setting it to ~2 GB to see if this
changed symptoms, but it didn't.


"Nico Stuurman-2 wrote
> It will certainly be worth testing if Micro-Manager by itself does this
> correctly. ( If it does, you know where to look).

As mentioned above, launching a multi-dimensional acquisition with similar
parameters in MM works fine. I am not sure if this is identical to the
"sequence" acquisition launched via Matlab. If not, how can I start a
"sequence" from within MM to test this?

Best,
Philipp



--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" error during sequence acquisition

Philipp Rinklin
Nico Stuurman wrote

> Hi Philipp,
>
>> [...] how much memory is assigned to Matlab?
>> The Java heap memory for Matlab is cranked up to the maximum value
>> available on this system (~4 GB). I also tried setting it to ~2 GB to
>> see if this changed symptoms, but it didn't.
>>
>>
>> "Nico Stuurman-2 wrote
>>> It will certainly be worth testing if Micro-Manager by itself does
>>> this correctly. ( If it does, you know where to look).
>> As mentioned above, launching a multi-dimensional acquisition with
>> similar parameters in MM works fine. I am not sure if this is
>> identical to the "sequence" acquisition launched via Matlab. If not,
>> how can I start a "sequence" from within MM to test this?
> If you use a 0 msec interval between frames, then the MDA in MM will use
> sequence acquisition.
>
> I have the feeling that for one reason or another memory is not
> relinquished after saving data when you use Matlab to run the acquisition
> and that for one reason or another you have only 2.0 GB available to
> address directly (as the data are stored in direct memory buffers that are
> kept outside of the JVM).  Is it possible that Matlab runs as a 32-bit
> application?
>
> Best,
>
> Nico

Hi Nico,

I checked the Matlab installation on the PC we used and it actually is
64bit. I couldn't find any info in the MM instance launched by Matlab, but
we only have the 64bit version installed and refer to its installation
directory when calling StartMMStudio. To get an idea if the Java heap memory
set in Matlab, I also tried to lower the memory to ~500 MB. However, the
recording still stopped at ~1.9 GB (technically, it's always at frame 4065).

Don't know if this provides any additional cues, but I also found out how to
launch the sequence buffer monitor when MM is launched via Matlab (the menu
was simply hidden, resizing the window revealed it). Activating the "live"
view leads to the buffer haapily cycling through 0-500 (i.e. 0-100%). The
original sequence settings (i.e. 50 fps, 4x4 binning) don't show anything on
the buffer. Using Multi-D acquisition with 0 ms interval leads to a
fluctuation between 0 and 5... I assume this means/confirms that the
sequence buffer or the write speed to the hard drive is not the problem,
right?

Best,
Philipp





--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general