Andor iXon+, iXon3 bug?, need help

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Andor iXon+, iXon3 bug?, need help

Lukas Hille
Hi,

I need help with two Andor cameras.

Two Andor cameras are used in dual view (Multi Camera adapter with
Andor: iXon+ and Andor: iXon3).
Acquisition is done by a burstAcquisition macro (equal to the script),
but the described behavior applies also to MDA.
With Frame transfer enabled and at higher Acquisition speeds (binning 2,
0.9us vertical clock, 20ms exposure, full ROI) i get an buffer overflow.

It appears that the device adapter before 14.April2019 "old" returns
very quick from mmc.popNextTaggedImage() and therefore produces no
buffer overflow, while the adapter after 14.April2019 "new" takes a very
long time to return from this function.
This also occurs with only one Camera in use, but with two cameras the
return time is to long to handle the data flow.

A second issue is, that with Multi Camera adapter, the camera referring
to channel 0 (independent which one) stops acquiring after some time
(~60.000 images in Trigger: software).
This applies to the "old" and the "new" Andor device adapter.

With Multi Camera adapter and external trigger (one source for both
cameras, Trigger set to External on both cameras), the camera referring
to channel 0 stops acquiring within 0 to some thousand images)

I don't get any debugging error message with the new adapter.
With the old adapter i get for example: "GetNumberNewImages error:
first: 3473 last: 3473" for the camera on channel 0 before it stops working.

The goal would be to acquire about 200,000 time points with external
trigger at 20ms exposure time.

I tried to understand the differences in the code from commits on
14.April, but I didn't figure out why this makes a difference for the
popNextTaggedImage() function.
Also: the error message "GetNumberNewImages error:" doesn't make any
sense to me, referring to the situation that no new images are acquired.

I would be very happy to get assistance on this one.

Software:
AndorDriverPack2 - 2.103.30031.0
Micro-Manager 2.0.0-gamma1 20200222
mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019 (tried many
many more)

best
Lukas

--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

P.Almada
Hi Lukas,

Just as a sanity check, have you set the circular memory buffer to an appropriate size?

Kind regards,
Pedro Almada 

On Wed, 11 Mar 2020 at 18:04, Lukas Hille <[hidden email]> wrote:
Hi,

I need help with two Andor cameras.

Two Andor cameras are used in dual view (Multi Camera adapter with
Andor: iXon+ and Andor: iXon3).
Acquisition is done by a burstAcquisition macro (equal to the script),
but the described behavior applies also to MDA.
With Frame transfer enabled and at higher Acquisition speeds (binning 2,
0.9us vertical clock, 20ms exposure, full ROI) i get an buffer overflow.

It appears that the device adapter before 14.April2019 "old" returns
very quick from mmc.popNextTaggedImage() and therefore produces no
buffer overflow, while the adapter after 14.April2019 "new" takes a very
long time to return from this function.
This also occurs with only one Camera in use, but with two cameras the
return time is to long to handle the data flow.

A second issue is, that with Multi Camera adapter, the camera referring
to channel 0 (independent which one) stops acquiring after some time
(~60.000 images in Trigger: software).
This applies to the "old" and the "new" Andor device adapter.

With Multi Camera adapter and external trigger (one source for both
cameras, Trigger set to External on both cameras), the camera referring
to channel 0 stops acquiring within 0 to some thousand images)

I don't get any debugging error message with the new adapter.
With the old adapter i get for example: "GetNumberNewImages error:
first: 3473 last: 3473" for the camera on channel 0 before it stops working.

The goal would be to acquire about 200,000 time points with external
trigger at 20ms exposure time.

I tried to understand the differences in the code from commits on
14.April, but I didn't figure out why this makes a difference for the
popNextTaggedImage() function.
Also: the error message "GetNumberNewImages error:" doesn't make any
sense to me, referring to the situation that no new images are acquired.

I would be very happy to get assistance on this one.

Software:
AndorDriverPack2 - 2.103.30031.0
Micro-Manager 2.0.0-gamma1 20200222
mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019 (tried many
many more)

best
Lukas

--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

Lukas Hille

Hi Pedro,

I am using the 64bit version.
ImageJ.cfg is set with -Xmx6000m
The Sequence Buffer Size is 512MB which can hold 2048 images.
The computer has 32Gb of RAM.

Single Andor camera:
With the old adapter the average return time of mmc.popNextTaggedImage(); is 1ms (2x binning --> 256x256 pixel, 16bit)
With the new adapter (past 14.April19) the average return time of mmc.popNextTaggedImage(); is 18ms (exactly the same config file, just changed mmgr_dal_Andor.dll)

With the Multi Camera adapter:
old adapter: 2ms
new adapter: 36ms --> not sufficient for 100fps (2x 50fps)

I am also curious why the Multi Camera adapter doubles the time to return a single image.

With the "new" adapter the Sequence Buffer fills up quite fast (observed with Sequence Buffer Monitor).
With the "old" adapter the Sequence Buffer Monitor shows rarely 1-2 images.
I can't just increase the buffer size, it would exceed due to the long recording time.
Also: the System is tracking on the camera stream, therefore a large amount of images in a huge buffer causes a fatal delay for the tracker.

Further Interesting:
With "Trigger: Software", on the "new" adapter the cameras are slowing down a lot and therefore the buffer is kept below 100 images..
But with external Trigger the buffer overflow occurs quite quick (but the cameras stay at pace).
The readout time of the cameras is indicated with about 16ms.

The script i used to investigate in the timings:

mm.scripter().resetInterpreter(); //Reset Interpreter to get rid of old stuff
mm.scripter().clearMessageWindow(); //Clear Script output window
/**
 * A burst aquisition test script
 */
import org.micromanager.interal.utils.JavaUtils;
//import org.micromanager.data.Datastore;

// User defined variables:
int nrFrames = 1000;
int loopSleepTime = 3;

// defining variables
long taggedTime = 0;
long endSequenceTime = 0;
int loopCount = 0;

// stop running acquisitions
if (mmc.isSequenceRunning()) {
    mmc.stopSequenceAcquisition();   
}

// RW ram store
storeRW = mm.data().createRewritableRAMDatastore();
// Create a display to show images as they are acquired.
mm.displays().createDisplay(storeRW);

// actual acqusition in a try block to catch errors
try{
// start acqusition and run until it is stopped by command
// the sequence acquistion with preset number of Images did cause problems
mmc.startContinuousSequenceAcquisition(0);
// Set up a Coords.CoordsBuilder for applying coordinates to each image.
builder = mm.data().getCoordsBuilder().z(0).channel(0).stagePosition(0);

while (mmc.getRemainingImageCount() > 0 || mmc.isSequenceRunning(mmc.getCameraDevice())) {
   if (mmc.getRemainingImageCount() > 0) {
      timeStart = System.currentTimeMillis();
      tagged = mmc.popNextTaggedImage();
      timeEnd = System.currentTimeMillis();
      taggedTime = taggedTime + (timeEnd - timeStart);
      frameNumber = tagged.tags.getInt("ImageNumber");
      // stop sequence after nrFrames images
      if (frameNumber >= nrFrames) {
          timeStart = System.currentTimeMillis();
          print("stop sequence");
          mmc.stopSequenceAcquisition();
          print("sequence stopped");
          timeEnd = System.currentTimeMillis();
          endSequenceTime = taggedTime + (timeEnd - timeStart);
      }
      // for visualisation
      image = mm.data().convertTaggedImage(tagged,
            builder.time(0).build(), null);
      storeRW.putImage(image);
      print("image Nr.: " + frameNumber);
      loopCount++;
      }
   else {
      java.lang.Thread.sleep(loopSleepTime);
   }
}
}
catch (Exception exception) {
    print(exception + " - something went wrong");
}
if (mmc.isSequenceRunning()) {
    mmc.stopSequenceAcquisition();   
}
storeRW.close();
print ("acqusition done");
print ("sum tagged time: " + taggedTime);
print ("end sequence time: " + endSequenceTime);


Thanks for helping me.

kind regards Lukas


On 2020-03-12 10:07, Pedro Almada wrote:
Hi Lukas,

Just as a sanity check, have you set the circular memory buffer to an appropriate size?

Kind regards,
Pedro Almada 

On Wed, 11 Mar 2020 at 18:04, Lukas Hille <[hidden email]> wrote:
Hi,

I need help with two Andor cameras.

Two Andor cameras are used in dual view (Multi Camera adapter with
Andor: iXon+ and Andor: iXon3).
Acquisition is done by a burstAcquisition macro (equal to the script),
but the described behavior applies also to MDA.
With Frame transfer enabled and at higher Acquisition speeds (binning 2,
0.9us vertical clock, 20ms exposure, full ROI) i get an buffer overflow.

It appears that the device adapter before 14.April2019 "old" returns
very quick from mmc.popNextTaggedImage() and therefore produces no
buffer overflow, while the adapter after 14.April2019 "new" takes a very
long time to return from this function.
This also occurs with only one Camera in use, but with two cameras the
return time is to long to handle the data flow.

A second issue is, that with Multi Camera adapter, the camera referring
to channel 0 (independent which one) stops acquiring after some time
(~60.000 images in Trigger: software).
This applies to the "old" and the "new" Andor device adapter.

With Multi Camera adapter and external trigger (one source for both
cameras, Trigger set to External on both cameras), the camera referring
to channel 0 stops acquiring within 0 to some thousand images)

I don't get any debugging error message with the new adapter.
With the old adapter i get for example: "GetNumberNewImages error:
first: 3473 last: 3473" for the camera on channel 0 before it stops working.

The goal would be to acquire about 200,000 time points with external
trigger at 20ms exposure time.

I tried to understand the differences in the code from commits on
14.April, but I didn't figure out why this makes a difference for the
popNextTaggedImage() function.
Also: the error message "GetNumberNewImages error:" doesn't make any
sense to me, referring to the situation that no new images are acquired.

I would be very happy to get assistance on this one.

Software:
AndorDriverPack2 - 2.103.30031.0
Micro-Manager 2.0.0-gamma1 20200222
mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019 (tried many
many more)

best
Lukas

--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
-- 
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

Nico Stuurman-2
Hi Lukas,

I am afraid that I can not help much at this point in time, but we ran
into similar issues with our dual camera setup (2 iXons), and we are
still using the "old" Andor adaptor code on our dual camera microscope. 
The Andor engineer at the time told me that he could reproduce the
issues I was seeing (will have to go back to see if those were the same
you were seeing) with VS2010, but that they went away with the newer VS
that he was using.  Although that did not make a lot of sense to me at
the time, I very much do want to see MM to migrate soon to new
compilers, to at least make sure whether or not that is an issue.

It would be great if you could contact Andor technical support about
these issues, and see if they can raise it to a point that their
software developers get involved.  At that point, please do get me in
the loop as well.  Andor maintains the device adaptor code, and the
April 2019 update was quite drastic, but certainly left a number of
issues behind.  I am afraid that they are in the best position to fix
this (but I'll be happy to advice).

Best,


Nico


> Hi Pedro,
>
> I am using the 64bit version.
> ImageJ.cfg is set with -Xmx6000m
> The Sequence Buffer Size is 512MB which can hold 2048 images.
> The computer has 32Gb of RAM.
>
> Single Andor camera:
> With the old adapter the average return time of
> mmc.popNextTaggedImage(); is 1ms (2x binning --> 256x256 pixel, 16bit)
> With the new adapter (past 14.April19) the average return time of
> mmc.popNextTaggedImage(); is 18ms (exactly the same config file, just
> changed mmgr_dal_Andor.dll)
>
> With the Multi Camera adapter:
> old adapter: 2ms
> new adapter: 36ms --> not sufficient for 100fps (2x 50fps)
>
> I am also curious why the Multi Camera adapter doubles the time to
> return a single image.
>
> With the "new" adapter the Sequence Buffer fills up quite fast
> (observed with Sequence Buffer Monitor).
> With the "old" adapter the Sequence Buffer Monitor shows rarely 1-2
> images.
> I can't just increase the buffer size, it would exceed due to the long
> recording time.
> Also: the System is tracking on the camera stream, therefore a large
> amount of images in a huge buffer causes a fatal delay for the tracker.
>
> Further Interesting:
> With "Trigger: Software", on the "new" adapter the cameras are slowing
> down a lot and therefore the buffer is kept below 100 images..
> But with external Trigger the buffer overflow occurs quite quick (but
> the cameras stay at pace).
> The readout time of the cameras is indicated with about 16ms.
>
> The script i used to investigate in the timings:
>
> mm.scripter().resetInterpreter(); //Reset Interpreter to get rid of
> old stuff
> mm.scripter().clearMessageWindow(); //Clear Script output window
> /**
>  * A burst aquisition test script
>  */
> import org.micromanager.interal.utils.JavaUtils;
> //import org.micromanager.data.Datastore;
>
> // User defined variables:
> int nrFrames = 1000;
> int loopSleepTime = 3;
>
> // defining variables
> long taggedTime = 0;
> long endSequenceTime = 0;
> int loopCount = 0;
>
> // stop running acquisitions
> if (mmc.isSequenceRunning()) {
>     mmc.stopSequenceAcquisition();
> }
>
> // RW ram store
> storeRW = mm.data().createRewritableRAMDatastore();
> // Create a display to show images as they are acquired.
> mm.displays().createDisplay(storeRW);
>
> // actual acqusition in a try block to catch errors
> try{
> // start acqusition and run until it is stopped by command
> // the sequence acquistion with preset number of Images did cause problems
> mmc.startContinuousSequenceAcquisition(0);
> // Set up a Coords.CoordsBuilder for applying coordinates to each image.
> builder = mm.data().getCoordsBuilder().z(0).channel(0).stagePosition(0);
>
> while (mmc.getRemainingImageCount() > 0 ||
> mmc.isSequenceRunning(mmc.getCameraDevice())) {
>    if (mmc.getRemainingImageCount() > 0) {
>       timeStart = System.currentTimeMillis();
>       tagged = mmc.popNextTaggedImage();
>       timeEnd = System.currentTimeMillis();
>       taggedTime = taggedTime + (timeEnd - timeStart);
>       frameNumber = tagged.tags.getInt("ImageNumber");
>       // stop sequence after nrFrames images
>       if (frameNumber >= nrFrames) {
>           timeStart = System.currentTimeMillis();
>           print("stop sequence");
>           mmc.stopSequenceAcquisition();
>           print("sequence stopped");
>           timeEnd = System.currentTimeMillis();
>           endSequenceTime = taggedTime + (timeEnd - timeStart);
>       }
>       // for visualisation
>       image = mm.data().convertTaggedImage(tagged,
>             builder.time(0).build(), null);
>       storeRW.putImage(image);
>       print("image Nr.: " + frameNumber);
>       loopCount++;
>       }
>    else {
>       java.lang.Thread.sleep(loopSleepTime);
>    }
> }
> }
> catch (Exception exception) {
>     print(exception + " - something went wrong");
> }
> if (mmc.isSequenceRunning()) {
>     mmc.stopSequenceAcquisition();
> }
> storeRW.close();
> print ("acqusition done");
> print ("sum tagged time: " + taggedTime);
> print ("end sequence time: " + endSequenceTime);
>
>
> Thanks for helping me.
>
> kind regards Lukas
>
>
> On 2020-03-12 10:07, Pedro Almada wrote:
>> Hi Lukas,
>>
>> Just as a sanity check, have you set the circular memory buffer to an
>> appropriate size?
>> https://micro-manager.org/wiki/Micro-Manager_Configuration_Guide#Memory_Settings 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__micro-2Dmanager.org_wiki_Micro-2DManager-5FConfiguration-5FGuide-23Memory-5FSettings&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=PG8_MP-bp5ljDiggl6RgaGpdZCvv5zr-LEynV0WaowQ&e=>
>>
>>
>> Kind regards,
>> Pedro Almada
>>
>> On Wed, 11 Mar 2020 at 18:04, Lukas Hille <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     Hi,
>>
>>     I need help with two Andor cameras.
>>
>>     Two Andor cameras are used in dual view (Multi Camera adapter with
>>     Andor: iXon+ and Andor: iXon3).
>>     Acquisition is done by a burstAcquisition macro (equal to the
>>     script),
>>     but the described behavior applies also to MDA.
>>     With Frame transfer enabled and at higher Acquisition speeds
>>     (binning 2,
>>     0.9us vertical clock, 20ms exposure, full ROI) i get an buffer
>>     overflow.
>>
>>     It appears that the device adapter before 14.April2019 "old" returns
>>     very quick from mmc.popNextTaggedImage() and therefore produces no
>>     buffer overflow, while the adapter after 14.April2019 "new" takes
>>     a very
>>     long time to return from this function.
>>     This also occurs with only one Camera in use, but with two
>>     cameras the
>>     return time is to long to handle the data flow.
>>
>>     A second issue is, that with Multi Camera adapter, the camera
>>     referring
>>     to channel 0 (independent which one) stops acquiring after some time
>>     (~60.000 images in Trigger: software).
>>     This applies to the "old" and the "new" Andor device adapter.
>>
>>     With Multi Camera adapter and external trigger (one source for both
>>     cameras, Trigger set to External on both cameras), the camera
>>     referring
>>     to channel 0 stops acquiring within 0 to some thousand images)
>>
>>     I don't get any debugging error message with the new adapter.
>>     With the old adapter i get for example: "GetNumberNewImages error:
>>     first: 3473 last: 3473" for the camera on channel 0 before it
>>     stops working.
>>
>>     The goal would be to acquire about 200,000 time points with external
>>     trigger at 20ms exposure time.
>>
>>     I tried to understand the differences in the code from commits on
>>     14.April, but I didn't figure out why this makes a difference for
>>     the
>>     popNextTaggedImage() function.
>>     Also: the error message "GetNumberNewImages error:" doesn't make any
>>     sense to me, referring to the situation that no new images are
>>     acquired.
>>
>>     I would be very happy to get assistance on this one.
>>
>>     Software:
>>     AndorDriverPack2 - 2.103.30031.0
>>     Micro-Manager 2.0.0-gamma1 20200222
>>     mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019
>>     (tried many
>>     many more)
>>
>>     best
>>     Lukas
>>
>>     --
>>     ____________________________________________
>>     Lukas Hille, M.Sc.
>>     microscopy engineer
>>     [hidden email] <mailto:[hidden email]>
>>     M +43 660 349 169 3
>>
>>     Department of Neurobiology, University of Vienna
>>     Campus-Vienna-Biocenter 1
>>     1030 Vienna
>>     AUSTRIA
>>
>>     https://neuro.univie.ac.at/
>>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__neuro.univie.ac.at_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=vOEXQZsKGgi3nnbjzvgQn42ESKwizdX8-w-cF9pJs3k&e=>
>>     https://www.imp.ac.at/groups/manuel-zimmer/
>>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.imp.ac.at_groups_manuel-2Dzimmer_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=asG_6hmBgrmDz6Rbwm0hll60QnyPPogr1kqoz9eEq6Y&e=>
>>     Part of Vienna BioCenter
>>     www.viennabiocenter.org
>>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.viennabiocenter.org&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=JQvGS7Y-Uc6IqcKD3lPntn5yVlZO0i7rpltQBfj9trE&e=>
>>     ____________________________________________
>>
>>
>>
>>     _______________________________________________
>>     micro-manager-general mailing list
>>     [hidden email]
>>     <mailto:[hidden email]>
>>     https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_micro-2Dmanager-2Dgeneral&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=9i6pdTjg_qT2toVJkbzjoQnylKRxs8CVRcuuhVjnU9Q&e=>
>>
>>
>>
>> _______________________________________________
>> micro-manager-general mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
> --
> ____________________________________________
> Lukas Hille, M.Sc.
> microscopy engineer
> [hidden email]
> M +43 660 349 169 3
>
> Department of Neurobiology, University of Vienna
> Campus-Vienna-Biocenter 1
> 1030 Vienna
> AUSTRIA
>
> https://neuro.univie.ac.at/
> https://www.imp.ac.at/groups/manuel-zimmer/
> Part of Vienna BioCenter
> www.viennabiocenter.org
> ____________________________________________
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_micro-2Dmanager-2Dgeneral&d=DwICAg&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=9i6pdTjg_qT2toVJkbzjoQnylKRxs8CVRcuuhVjnU9Q&e=



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

Lukas Hille
Hi Nico,

good to know, thanks!

I will contact Andor.

Apart from the "speed / buffer overflow" issue, can you also verify that
one of the cameras (camera 0) stops acquiring after some images?
(~60,000 with Trigger:software and varying with Trigger:external)
This applies to all adapter versions.
I am fine with the old driver, but dealing with breaks every some
thousand images is a pain.

We have some "professional" software too, but they do even worse (i am
super happy with MM).
I have to check if the "dual camera stop" on channel 0 is also a Andor
issue, i didn't try long lasting acquisitions with other cameras.

best
Lukas

On 2020-03-12 18:55, Nico Stuurman wrote:

> Hi Lukas,
>
> I am afraid that I can not help much at this point in time, but we ran
> into similar issues with our dual camera setup (2 iXons), and we are
> still using the "old" Andor adaptor code on our dual camera
> microscope.  The Andor engineer at the time told me that he could
> reproduce the issues I was seeing (will have to go back to see if
> those were the same you were seeing) with VS2010, but that they went
> away with the newer VS that he was using.  Although that did not make
> a lot of sense to me at the time, I very much do want to see MM to
> migrate soon to new compilers, to at least make sure whether or not
> that is an issue.
>
> It would be great if you could contact Andor technical support about
> these issues, and see if they can raise it to a point that their
> software developers get involved.  At that point, please do get me in
> the loop as well.  Andor maintains the device adaptor code, and the
> April 2019 update was quite drastic, but certainly left a number of
> issues behind.  I am afraid that they are in the best position to fix
> this (but I'll be happy to advice).
>
> Best,
>
>
> Nico
>
>
>> Hi Pedro,
>>
>> I am using the 64bit version.
>> ImageJ.cfg is set with -Xmx6000m
>> The Sequence Buffer Size is 512MB which can hold 2048 images.
>> The computer has 32Gb of RAM.
>>
>> Single Andor camera:
>> With the old adapter the average return time of
>> mmc.popNextTaggedImage(); is 1ms (2x binning --> 256x256 pixel, 16bit)
>> With the new adapter (past 14.April19) the average return time of
>> mmc.popNextTaggedImage(); is 18ms (exactly the same config file, just
>> changed mmgr_dal_Andor.dll)
>>
>> With the Multi Camera adapter:
>> old adapter: 2ms
>> new adapter: 36ms --> not sufficient for 100fps (2x 50fps)
>>
>> I am also curious why the Multi Camera adapter doubles the time to
>> return a single image.
>>
>> With the "new" adapter the Sequence Buffer fills up quite fast
>> (observed with Sequence Buffer Monitor).
>> With the "old" adapter the Sequence Buffer Monitor shows rarely 1-2
>> images.
>> I can't just increase the buffer size, it would exceed due to the
>> long recording time.
>> Also: the System is tracking on the camera stream, therefore a large
>> amount of images in a huge buffer causes a fatal delay for the tracker.
>>
>> Further Interesting:
>> With "Trigger: Software", on the "new" adapter the cameras are
>> slowing down a lot and therefore the buffer is kept below 100 images..
>> But with external Trigger the buffer overflow occurs quite quick (but
>> the cameras stay at pace).
>> The readout time of the cameras is indicated with about 16ms.
>>
>> The script i used to investigate in the timings:
>>
>> mm.scripter().resetInterpreter(); //Reset Interpreter to get rid of
>> old stuff
>> mm.scripter().clearMessageWindow(); //Clear Script output window
>> /**
>>  * A burst aquisition test script
>>  */
>> import org.micromanager.interal.utils.JavaUtils;
>> //import org.micromanager.data.Datastore;
>>
>> // User defined variables:
>> int nrFrames = 1000;
>> int loopSleepTime = 3;
>>
>> // defining variables
>> long taggedTime = 0;
>> long endSequenceTime = 0;
>> int loopCount = 0;
>>
>> // stop running acquisitions
>> if (mmc.isSequenceRunning()) {
>>     mmc.stopSequenceAcquisition();
>> }
>>
>> // RW ram store
>> storeRW = mm.data().createRewritableRAMDatastore();
>> // Create a display to show images as they are acquired.
>> mm.displays().createDisplay(storeRW);
>>
>> // actual acqusition in a try block to catch errors
>> try{
>> // start acqusition and run until it is stopped by command
>> // the sequence acquistion with preset number of Images did cause
>> problems
>> mmc.startContinuousSequenceAcquisition(0);
>> // Set up a Coords.CoordsBuilder for applying coordinates to each image.
>> builder = mm.data().getCoordsBuilder().z(0).channel(0).stagePosition(0);
>>
>> while (mmc.getRemainingImageCount() > 0 ||
>> mmc.isSequenceRunning(mmc.getCameraDevice())) {
>>    if (mmc.getRemainingImageCount() > 0) {
>>       timeStart = System.currentTimeMillis();
>>       tagged = mmc.popNextTaggedImage();
>>       timeEnd = System.currentTimeMillis();
>>       taggedTime = taggedTime + (timeEnd - timeStart);
>>       frameNumber = tagged.tags.getInt("ImageNumber");
>>       // stop sequence after nrFrames images
>>       if (frameNumber >= nrFrames) {
>>           timeStart = System.currentTimeMillis();
>>           print("stop sequence");
>>           mmc.stopSequenceAcquisition();
>>           print("sequence stopped");
>>           timeEnd = System.currentTimeMillis();
>>           endSequenceTime = taggedTime + (timeEnd - timeStart);
>>       }
>>       // for visualisation
>>       image = mm.data().convertTaggedImage(tagged,
>>             builder.time(0).build(), null);
>>       storeRW.putImage(image);
>>       print("image Nr.: " + frameNumber);
>>       loopCount++;
>>       }
>>    else {
>>       java.lang.Thread.sleep(loopSleepTime);
>>    }
>> }
>> }
>> catch (Exception exception) {
>>     print(exception + " - something went wrong");
>> }
>> if (mmc.isSequenceRunning()) {
>>     mmc.stopSequenceAcquisition();
>> }
>> storeRW.close();
>> print ("acqusition done");
>> print ("sum tagged time: " + taggedTime);
>> print ("end sequence time: " + endSequenceTime);
>>
>>
>> Thanks for helping me.
>>
>> kind regards Lukas
>>
>>
>> On 2020-03-12 10:07, Pedro Almada wrote:
>>> Hi Lukas,
>>>
>>> Just as a sanity check, have you set the circular memory buffer to
>>> an appropriate size?
>>> https://micro-manager.org/wiki/Micro-Manager_Configuration_Guide#Memory_Settings 
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__micro-2Dmanager.org_wiki_Micro-2DManager-5FConfiguration-5FGuide-23Memory-5FSettings&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=PG8_MP-bp5ljDiggl6RgaGpdZCvv5zr-LEynV0WaowQ&e=>
>>>
>>>
>>> Kind regards,
>>> Pedro Almada
>>>
>>> On Wed, 11 Mar 2020 at 18:04, Lukas Hille <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>     Hi,
>>>
>>>     I need help with two Andor cameras.
>>>
>>>     Two Andor cameras are used in dual view (Multi Camera adapter with
>>>     Andor: iXon+ and Andor: iXon3).
>>>     Acquisition is done by a burstAcquisition macro (equal to the
>>>     script),
>>>     but the described behavior applies also to MDA.
>>>     With Frame transfer enabled and at higher Acquisition speeds
>>>     (binning 2,
>>>     0.9us vertical clock, 20ms exposure, full ROI) i get an buffer
>>>     overflow.
>>>
>>>     It appears that the device adapter before 14.April2019 "old"
>>> returns
>>>     very quick from mmc.popNextTaggedImage() and therefore produces no
>>>     buffer overflow, while the adapter after 14.April2019 "new" takes
>>>     a very
>>>     long time to return from this function.
>>>     This also occurs with only one Camera in use, but with two
>>>     cameras the
>>>     return time is to long to handle the data flow.
>>>
>>>     A second issue is, that with Multi Camera adapter, the camera
>>>     referring
>>>     to channel 0 (independent which one) stops acquiring after some
>>> time
>>>     (~60.000 images in Trigger: software).
>>>     This applies to the "old" and the "new" Andor device adapter.
>>>
>>>     With Multi Camera adapter and external trigger (one source for both
>>>     cameras, Trigger set to External on both cameras), the camera
>>>     referring
>>>     to channel 0 stops acquiring within 0 to some thousand images)
>>>
>>>     I don't get any debugging error message with the new adapter.
>>>     With the old adapter i get for example: "GetNumberNewImages error:
>>>     first: 3473 last: 3473" for the camera on channel 0 before it
>>>     stops working.
>>>
>>>     The goal would be to acquire about 200,000 time points with
>>> external
>>>     trigger at 20ms exposure time.
>>>
>>>     I tried to understand the differences in the code from commits on
>>>     14.April, but I didn't figure out why this makes a difference for
>>>     the
>>>     popNextTaggedImage() function.
>>>     Also: the error message "GetNumberNewImages error:" doesn't make
>>> any
>>>     sense to me, referring to the situation that no new images are
>>>     acquired.
>>>
>>>     I would be very happy to get assistance on this one.
>>>
>>>     Software:
>>>     AndorDriverPack2 - 2.103.30031.0
>>>     Micro-Manager 2.0.0-gamma1 20200222
>>>     mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019
>>>     (tried many
>>>     many more)
>>>
>>>     best
>>>     Lukas
>>>
>>>     --     ____________________________________________
>>>     Lukas Hille, M.Sc.
>>>     microscopy engineer
>>>     [hidden email] <mailto:[hidden email]>
>>>     M +43 660 349 169 3
>>>
>>>     Department of Neurobiology, University of Vienna
>>>     Campus-Vienna-Biocenter 1
>>>     1030 Vienna
>>>     AUSTRIA
>>>
>>>     https://neuro.univie.ac.at/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__neuro.univie.ac.at_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=vOEXQZsKGgi3nnbjzvgQn42ESKwizdX8-w-cF9pJs3k&e=>
>>>     https://www.imp.ac.at/groups/manuel-zimmer/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.imp.ac.at_groups_manuel-2Dzimmer_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=asG_6hmBgrmDz6Rbwm0hll60QnyPPogr1kqoz9eEq6Y&e=>
>>>     Part of Vienna BioCenter
>>>     www.viennabiocenter.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.viennabiocenter.org&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=JQvGS7Y-Uc6IqcKD3lPntn5yVlZO0i7rpltQBfj9trE&e=>
>>>     ____________________________________________
>>>
>>>
>>>
>>>     _______________________________________________
>>>     micro-manager-general mailing list
>>>     [hidden email]
>>>     <mailto:[hidden email]>
>>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_micro-2Dmanager-2Dgeneral&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=9i6pdTjg_qT2toVJkbzjoQnylKRxs8CVRcuuhVjnU9Q&e=>
>>>
>>>
>>>
>>> _______________________________________________
>>> micro-manager-general mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>> --
>> ____________________________________________
>> Lukas Hille, M.Sc.
>> microscopy engineer
>> [hidden email]
>> M +43 660 349 169 3
>>
>> Department of Neurobiology, University of Vienna
>> Campus-Vienna-Biocenter 1
>> 1030 Vienna
>> AUSTRIA
>>
>> https://neuro.univie.ac.at/
>> https://www.imp.ac.at/groups/manuel-zimmer/
>> Part of Vienna BioCenter
>> www.viennabiocenter.org
>> ____________________________________________
>>
>>
>> _______________________________________________
>> micro-manager-general mailing list
>> [hidden email]
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_micro-2Dmanager-2Dgeneral&d=DwICAg&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=9i6pdTjg_qT2toVJkbzjoQnylKRxs8CVRcuuhVjnU9Q&e= 
>>
>
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general

--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

Nico Stuurman-2
Hi Lukas,
>
> Apart from the "speed / buffer overflow" issue, can you also verify
> that one of the cameras (camera 0) stops acquiring after some images?
> (~60,000 with Trigger:software and varying with Trigger:external)

May be a while before I can get behind a microscope again;(  That number
sounds a bit like an overflow (awfully close to the maximum of an
unsigned short: 65535), and checking with the democamera (possibly using
the fastimage=1 option to avoid generating new images all the time)
would be very helpful to localize the problem.

> This applies to all adapter versions.
> I am fine with the old driver, but dealing with breaks every some
> thousand images is a pain.
>
> We have some "professional" software too, but they do even worse (i am
> super happy with MM).

Great to hear!  Spending so much time supporting MM that now and then
hearing from people that it is actually useful helps me stay motivated
to continue doing so;)

> I have to check if the "dual camera stop" on channel 0 is also a Andor
> issue, i didn't try long lasting acquisitions with other cameras.

If possible, check with the democamera, although  external triggers are
not an option there, and I do not know how well that code has been
exercised with multiple cameras.


Best,



Nico




_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Andor iXon+, iXon3 bug?, need help

Lukas Hille
Hi Nico,

with the demo cameras everything works fine (which is already a good thing).

I will investigate more on that on Monday.

Have a nice weekend!

best
Lukas

On 2020-03-12 23:00, Nico Stuurman wrote:

> Hi Lukas,
>>
>> Apart from the "speed / buffer overflow" issue, can you also verify
>> that one of the cameras (camera 0) stops acquiring after some images?
>> (~60,000 with Trigger:software and varying with Trigger:external)
>
> May be a while before I can get behind a microscope again;(  That
> number sounds a bit like an overflow (awfully close to the maximum of
> an unsigned short: 65535), and checking with the democamera (possibly
> using the fastimage=1 option to avoid generating new images all the
> time) would be very helpful to localize the problem.
>
>> This applies to all adapter versions.
>> I am fine with the old driver, but dealing with breaks every some
>> thousand images is a pain.
>>
>> We have some "professional" software too, but they do even worse (i
>> am super happy with MM).
>
> Great to hear!  Spending so much time supporting MM that now and then
> hearing from people that it is actually useful helps me stay motivated
> to continue doing so;)
>
>> I have to check if the "dual camera stop" on channel 0 is also a
>> Andor issue, i didn't try long lasting acquisitions with other cameras.
>
> If possible, check with the democamera, although  external triggers
> are not an option there, and I do not know how well that code has been
> exercised with multiple cameras.
>
>
> Best,
>
>
>
> Nico
>
>
>
>
> _______________________________________________
> micro-manager-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general

--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
[hidden email]
M +43 660 349 169 3

Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA

https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general