Saving images at high speed

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Saving images at high speed

Nico Stuurman-2
Lukas Hille is looking into the issues saving images fast enough to keep
up with current sCMOS cameras and wrote me the following:

> Until now it is not clear to me where to start debugging/ boosting the system.
> On our pco.edge setup the system struggles sometimes with 1 camera at full resolution and speed (2048x2048, 100fps) without storing the data into ancreaseny Datastore.
> The return times of the steps from the burstAcquisition are:
> mmc.popNextTaggedImage(): 2-4ms
> mm.data().convertTaggedImage(): 4-16ms
> pipeline.insertImage(): 0ms (into RAMDatastore), 4-8ms into MultipageTIFFDatastore():
> I tried with fixed metadata (created by builder) and reduced devices,
> without any change.
> The timings seem to scale with different chip sizes (512², 1024², 2048², 4096²). Some things are confusing,  like the timings scale also with the exposure time (with shorter exposure time, the functions return quicker). This effect is small and there is a limit.
>
> time for popNextTaggedImage()+convertTaggedImage to return:
>
> chip size : min. exposure time (into RAM)
> 512² : 1ms
> 1024²: 4ms
> 2048²: 11ms
> 4096²: 38ms
>
> I tried 8bt, 16bit and 32bit, the bit depth doesn't influence the return time. I would expect, that the metadata is equal for all of this trials.

Great work!  This suggests that the function call:

rawPixels_ = DirectBuffers.bufferFromArray(tagged.pix);

in org.micromanager.data.internal.DefaultImage is the main culprit (but
it would be very good to confirm this).  This call allocates the
ByteBuffer memory and transfers the data into it.  It is unclear to me
why the 8, 16, and 32 bit images act about the same.  In any case, it
will help tremendously to figure out the real culprit here.  If
allocation is an issue, MM could pre-allocate multiple buffers (painful,
but doable).

I played a bit with the following (Beanshell/Java) script:

import java.nio.ByteBuffer;

ByteBuffer bb = ByteBuffer.allocateDirect(2048 * 2048);
byte[] bytes = new byte[bb.capacity()];

int runs = 1000;
long start = System.nanoTime();
for (int i = 0; i < runs; i++) {
     bb.clear();
     bb.put(bytes);
}
long time = System.nanoTime() - start;
mm.scripter().message("Average time to copy 4 MB was " + (time / runs /
1000) + "us");

On my system, copying the data takes less than 1 ms, however, when I add
allocation into the loop, the total average time goes up to about 4ms.

I hope that you will be able to pinpoint the time-consuming operations
even better, and hopefully we will find ways to work around them.

Best,


Nico


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general