Circular buffer overflow with low resource utilization

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Circular buffer overflow with low resource utilization

Noah Young
I'm running an Orca Flash 4.0 v2 (2048x2048px 16 bit) at 50fps using micromanager 2.0.0-beta3. ImageJ is set to use 24 threads (24 logical processors available) and has a 8GB buffer. Data is being saved to an NVMe SSD.

After a few frames I notice the circular buffer starting to fill up in the buffer monitor, limiting my acquisitions to only a few seconds. My disk write utilization is only about 30% and CPU usage is also around 25%. If I drop the frame rate to 20fps I don't see buffering initially, but then it fills up rapidly after 3-5 minutes.

With the hardware I have there should be no problem keeping up with the images being acquired, so it seems like there's a software bottleneck in micromanger. Any idea what's going on?

Noah Young
PhD candidate, Bioengineering
Stanford University

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Circular buffer overflow with low resource utilization

JonD
Administrator
Hi Noah,

You have probably tried allocating more memory to the circular buffer.

You have probably tried separately characterizing the write speed of your
SSD to make sure it is performing to expectations.  You need 200 MB/s to
keep up with your camera.

Besides watching the circular buffer, also watch the disk utilization (e.g.
Performance Monitor in Windows)

For fastest speeds, I suggest using a disk that is dedicated to data, i.e.
it should not be the OS disk nor should it contain the page/swap file.
Otherwise other processes can be competing for the same disk, which can
cause hiccups that spiral out of control.

Finally, for debugging purposes it would be helpful to know if the same
thing happens in a recent 1.4.x nightly build and/or a recent "gamma"
nightly build (http://valelab4.ucsf.edu/~nstuurman/fom/)... maybe the
problem is specific to this beta.

-------------------------------------------
Jon Daniels
Applied Scientific Instrumentation
29391 West Enid Rd, Eugene, OR 97402
Phone: (541) 461-8181 x118
-------------------------------------------



--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Circular buffer overflow with low resource utilization

Noah Young
Thanks Jon,

Crystal disk mark measures my sequential write speed at 1300MB/s and the disk is only used for data, no programs or OS is loaded from it.

Disk utilization is only 30% during acquisition. Since I'm using 16 bit depth I calculate that at 50fps I need 400MB/s of write speed, so actually 30% utilization is about right. But even though I have all that headroom, I still start buffering frames. The limit seems to be about 20fps. Any faster and the buffer starts building up. 

The buffer size is set for about 1000 images. I should be able to set it at only 20 or so images - just enough to account for jitter. I will need to do experiments that will last for hundreds of thousands of frames so just having a gigantic buffer and winding it down at the end of an experiment won't work... plus, I have more than sufficient hardware to stream to disk so I'd like to do so.

I tried installing the latest "gamma" version and I'm seeing the same behavior unfortunately.

On Fri, Jun 22, 2018 at 11:10 AM JonD <[hidden email]> wrote:
Hi Noah,

You have probably tried allocating more memory to the circular buffer.

You have probably tried separately characterizing the write speed of your
SSD to make sure it is performing to expectations.  You need 200 MB/s to
keep up with your camera.

Besides watching the circular buffer, also watch the disk utilization (e.g.
Performance Monitor in Windows)

For fastest speeds, I suggest using a disk that is dedicated to data, i.e.
it should not be the OS disk nor should it contain the page/swap file.
Otherwise other processes can be competing for the same disk, which can
cause hiccups that spiral out of control.

Finally, for debugging purposes it would be helpful to know if the same
thing happens in a recent 1.4.x nightly build and/or a recent "gamma"
nightly build (http://valelab4.ucsf.edu/~nstuurman/fom/)... maybe the
problem is specific to this beta.

-------------------------------------------
Jon Daniels
Applied Scientific Instrumentation
29391 West Enid Rd, Eugene, OR 97402
Phone: <a href="tel:(541)%20461-8181" value="+15414618181" target="_blank">(541) 461-8181 x118
-------------------------------------------



--
Sent from: http://micro-manager.3463995.n2.nabble.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general