Using Opencv Lib with MicroManager plugin

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Opencv Lib with MicroManager plugin

Vikram Kopuri
Hi Everyone ,

I've been working on a MicroManager plugin that uses Camera images for XY tracking. I found OpenCV has extensive libraries for tracking (like the phaseCorrelate and Optical flow ones like Lucas Kanade).
I'm also finding opencv functions(like Canny Edge Detector for finding edges ) seem to run faster. than imagej or micro manager equivalents.

In this email I'm sharing what I had to do to get OpencV libraries working in a MicroManager plugin , incase others find it useful. Also to solicit feedback and advice on a better and safer methods.

I'm working with Opencv 2.4.11 as it seems to work with Java Compiler 1.6 , the newer Opencv 3.0 need complier 1.7 or newer I think.

I started out with open the Opencv Java eclipse tutorial , http://docs.opencv.org/2.4/doc/tutorials/introduction/java_eclipse/java_eclipse.html#java-eclipse

First you get opencv 2.4 and install it , then in eclipse you refer to opencv-2411.jar and the dll opencv-2411.dll either the x64 or x86. Then in the code before using any Opencv function , use the syntax "System.loadLibrary( Core.NATIVE_LIBRARY_NAME );" To load the libraries.

Outside the Eclipse debug environment , this didn't work for me . Kept getting the "java.lang.UnsatisfiedLinkError" error.

To work around this , what I needed to export my project as a "Runnable Jar" or "fat Jar" with "Extract required lib into generated jar" selected. And copy the opencv-2411.dll in "C:\Windows\System32" (because the dll has to be in a java library path) or instead of "System.loadLibrary(lib_name)" use the "java.lang.Runtime.getRuntime().load(path of the dll)" instead because it accepts the path of the dll directly , and doesn't care if the path part of the library or not.

Couple of problems with this workaround is that MicroManager plugins are not fat jars and a fat jar won't work. Next is when I deploy the plugin I want it to load the correct dll (x64 or x86 correctly).

To work around this . What I needed to do was , copy the opencv/build/java folder which just has the opencv-2411.jar and the opencv-2411.dll to micromanager plugin folder like here , "C:\Program Files\Micro-Manager-1.4\plugins\opencv_2411"

Then to load the correct dll , I found some code on stack overflow http://stackoverflow.com/a/18780559 , and I modified it a little to work for my case. Below is my modified code to load the correct opencv dll based on architecture.


String osName = System.getProperty("os.name");
if(osName.startsWith("Windows")){
int bitness = Integer.parseInt(System.getProperty("sun.arch.data.model"));
ReportingUtils.logMessage("OS is windows");
if(bitness == 32){
ReportingUtils.logMessage("32 bit detected");
//the run time method is more thread safe, and it seems to ignore repeated reloads
java.lang.Runtime.getRuntime().load("C:/Program Files/Micro-Manager-1.4/plugins/opencv_2411/x86/opencv_java2411.dll");


}
else if (bitness == 64){
ReportingUtils.logMessage("64 bit detected");
java.lang.Runtime.getRuntime().load("C:/Program Files/Micro-Manager-1.4/plugins/opencv_2411/x64/opencv_java2411.dll");

}
else{
ReportingUtils.logMessage("Cannot figure out os bit , so loading 32 bit");
java.lang.Runtime.getRuntime().load("C:/Program Files/Micro-Manager-1.4/plugins/opencv_2411/x86/opencv_java2411.dll");
}
}
else{ //else if(osName.equals("Mac OS X")){
ReportingUtils.logMessage("!!!!! Non Windows OS not supported !!!!!");
}


} catch (Exception e) {
ReportingUtils.logMessage("!!!!!opencv code library failed to load.!!!!!" + e);
}


Question for the pros , is there a way from inside the MicroManager plugin to figure out MicroManager install path "C:/Program Files/Micro-Manager-1.4" . If I can deduce that automatically , then deploying it becomes very easy. All I have to tell the user is , put the plugin jar in \Micro-Manager-1.4\mmplugins and copy the opencv_2.4.11 jar and dll to Micro-Manager-1.4\plugins and it works.

I tested it on a few computers , just need the opencv-2411.jar and the right opencv-2411.dll . Users don't need to download the entire opencv library or add any additional folder to system path. And I export my MicroManager plugin just as a regular jar.

Let me know if i'm doing anything unsounds. If there is a way to figure out MicroManager install path that would be very useful to me. Like to also hear your thoughts on using opencv and micromanager together, has anyone done this before , I would like to learn more.

Thanks.



With Regards

Vikram Kopuri
Electronic Engineer | Applied Scientific Instrumentation , Eugene OR, USA | +1 541-461-8181 x119


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Stuurman, Nico
Hi Vik,


> In this email I'm sharing what I had to do to get OpencV libraries working in a MicroManager plugin , incase others find it useful. Also to solicit feedback and advice on a better and safer methods.
Thanks for sharing!  It is a pitty that the data have to go from the C++
layer in the core, to the Java layer, than back to OpenCV (C++).  There
used to be ImageProcessors in the C++ layer, but those were never
developed further, and I don't think that there would be a clean way to
feed-back results of the processing, so your approach is definitely the
most straight forward.

> Question for the pros , is there a way from inside the MicroManager plugin to figure out MicroManager install path "C:/Program Files/Micro-Manager-1.4" .

Does:

System.getProperty("user.dir");

work?  It returns the correct path for me.

Best,

Nico



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Vikram Kopuri
Hi Nico ,

"System.getProperty("user.dir");"  worked perfectly :) Thanks a lot.

Vikram
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Micro-Manager mailing list
In reply to this post by Vikram Kopuri
Hi Vikram,

I tried to follow your solution in MM2.0, the library loads using
java.lang.Runtime.getRuntime().load, however, I still get
"java.lang.UnsatisfiedLinkError" error when calling OpenCV library in my
plugin, do you have any suggestions? I've put opencv.jar in plugins folder
and using java.lang.Runtime.getRuntime().load(absolute path to dll) and my
plugin can be detected and shown in MM2, but reports this error when using
the functions calling opencv.

Here is the error message that I got:
java.lang.UnsatisfiedLinkError:
org.opencv.imgcodecs.Imgcodecs.imread_1(Ljava/lang/String;)J

Thanks,
Bin Li



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Vikram Kopuri
Hi Bin Li, 

Yeah it was tricky getting OpenCv working with java and micro-manager. I'm not sure I understand fully how I got it working :). The things that helped were

* Picking the version of opencv that ran on the same version of java as micro-manager. I found opencv 2.4.11 ran on java 1.6 as well. I believe MM2.0 runs on java 8 , so you might be able to use a later version of opencv.  
* Adding the opencv jar to the project's built path in eclipse
* Next I used the java.lang.Runtime.getRuntime().load(dll path) to load the dll , make sure to call the right version of the dll , 64bit or 32 bit. 

Have a look at mycode as well 

Hope that helps. 
~~~~~~~~~~~~~~~~~~~~~~
Vikram G Kopuri



On Wed, Oct 3, 2018 at 3:48 PM Bin via micro-manager-general <[hidden email]> wrote:
Hi Vikram,

I tried to follow your solution in MM2.0, the library loads using
java.lang.Runtime.getRuntime().load, however, I still get
"java.lang.UnsatisfiedLinkError" error when calling OpenCV library in my
plugin, do you have any suggestions? I've put opencv.jar in plugins folder
and using java.lang.Runtime.getRuntime().load(absolute path to dll) and my
plugin can be detected and shown in MM2, but reports this error when using
the functions calling opencv.

Here is the error message that I got:
java.lang.UnsatisfiedLinkError:
org.opencv.imgcodecs.Imgcodecs.imread_1(Ljava/lang/String;)J

Thanks,
Bin Li



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Micro-Manager mailing list
Hi Vikram,

It seems like that the plugin loading and initialization mechanism in MM2.0
is different from 1.4. I moved my plugin jar that loads opencv library to
the "plugins" folder (instead of "mmplugins" folder) and the library can be
successfully called. However, menu plugins should normally live in the
"mmplugins" folder. I checked that in both folders opencv_java.dll can be
loaded, but only in the "plugins" folder that it can be correctly called.

Thanks,
Bin Li



I
Vikram Kopuri wrote

> Hi Bin Li,
>
> Yeah it was tricky getting OpenCv working with java and micro-manager. I'm
> not sure I understand fully how I got it working :). The things that
> helped
> were
>
> * Picking the version of opencv that ran on the same version of java as
> micro-manager. I found opencv 2.4.11 ran on java 1.6 as well. I believe
> MM2.0 runs on java 8 , so you might be able to use a later version of
> opencv.
> * Adding the opencv jar to the project's built path in eclipse
> * Next I used the java.lang.Runtime.getRuntime().load(dll path) to load
> the
> dll , make sure to call the right version of the dll , 64bit or 32 bit.
>
> Have a look at mycode as well
> https://www.dropbox.com/s/qam337vtwbpaeh4/LoadOpenCV.java?dl=0
>
> Hope that helps.
> ~~~~~~~~~~~~~~~~~~~~~~
> Vikram G Kopuri
>
>
>
> On Wed, Oct 3, 2018 at 3:48 PM Bin via micro-manager-general <

> micro-manager-general@.sourceforge

>> wrote:
>
>> Hi Vikram,
>>
>> I tried to follow your solution in MM2.0, the library loads using
>> java.lang.Runtime.getRuntime().load, however, I still get
>> "java.lang.UnsatisfiedLinkError" error when calling OpenCV library in my
>> plugin, do you have any suggestions? I've put opencv.jar in plugins
>> folder
>> and using java.lang.Runtime.getRuntime().load(absolute path to dll) and
>> my
>> plugin can be detected and shown in MM2, but reports this error when
>> using
>> the functions calling opencv.
>>
>> Here is the error message that I got:
>> java.lang.UnsatisfiedLinkError:
>> org.opencv.imgcodecs.Imgcodecs.imread_1(Ljava/lang/String;)J
>>
>> Thanks,
>> Bin Li
>>
>>
>>
>> --
>> Sent from: http://micro-manager.3463995.n2.nabble.com/
>>
>>
>> _______________________________________________
>> micro-manager-general mailing list
>>

> micro-manager-general@.sourceforge

>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>>
>
>
> _______________________________________________
> micro-manager-general mailing list

> micro-manager-general@.sourceforge

> https://lists.sourceforge.net/lists/listinfo/micro-manager-general





--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Stuurman, Nico
Hi Bin and Vikram,
> It seems like that the plugin loading and initialization mechanism in MM2.0
> is different from 1.4. I moved my plugin jar that loads opencv library to
> the "plugins" folder (instead of "mmplugins" folder) and the library can be
> successfully called. However, menu plugins should normally live in the
> "mmplugins" folder. I checked that in both folders opencv_java.dll can be
> loaded, but only in the "plugins" folder that it can be correctly called.

That sounds familiar.  Chris and Mark decided that plugins should live
in their own class-loaders to avoid interference with other stuff.  I
can see the reasoning, but in practice it is a bit of a pain.

To make plugin code available to other plugins, Chris and Mark added the
"libraries" directory in the Micro-Manager source code repository. 
Plugins in that directory end up in the plugins folder in the final
installation.  The Localization Microscopy plugin has a very thin layer
in the mmplugins directory that calls code living in the "libraries"
directory.  It would be nice to simplify this organization.

Best,

Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Vikram Kopuri
Interesting , I'll have to remember this when I'm ready to port my plugin over to MM2.0. 
~~~~~~~~~~~~~~~~~~~~~~
Vikram G Kopuri



On Fri, Oct 5, 2018 at 1:58 PM Stuurman, Nico <[hidden email]> wrote:
Hi Bin and Vikram,
> It seems like that the plugin loading and initialization mechanism in MM2.0
> is different from 1.4. I moved my plugin jar that loads opencv library to
> the "plugins" folder (instead of "mmplugins" folder) and the library can be
> successfully called. However, menu plugins should normally live in the
> "mmplugins" folder. I checked that in both folders opencv_java.dll can be
> loaded, but only in the "plugins" folder that it can be correctly called.

That sounds familiar.  Chris and Mark decided that plugins should live
in their own class-loaders to avoid interference with other stuff.  I
can see the reasoning, but in practice it is a bit of a pain.

To make plugin code available to other plugins, Chris and Mark added the
"libraries" directory in the Micro-Manager source code repository. 
Plugins in that directory end up in the plugins folder in the final
installation.  The Localization Microscopy plugin has a very thin layer
in the mmplugins directory that calls code living in the "libraries"
directory.  It would be nice to simplify this organization.

Best,

Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Vikram Kopuri
Hi Bin, 

I had some luck :), I found a way to incorporate the opencv dlls , opencv java source code and the mm plugins code into just one jar file. 

I've been playing around with JXinput library(https://github.com/StrikerX3/JXInput) for another mm plugin I've been working on. The author was able to include all the dlls he needed into his release jar. Its tricky but he figured out how to do it.  

So I tried it out , I included the opencv dlls into my project and was able to load them at runtime. Then I unziped the opencv jar and copied all the java files into my project. So now when I build it, everything is in a nice single jar file. Since everything is self contained, I can even submit it to MM SVN in the future. 


My test plugin code is in src/org/vik/mm_opencv_test , a simple plugin only tried to load opencv and run test code.
Opencv java interface code is in src/org/opencv
Then the dll needed are in src/lib/native

I copied NativeLibraryHelper class from JXinput and placed it in src/org/opencv/ . This is where the magic happens , its able to load dlls from a relative path and inside a jar as well, which is the key. 


mm_opencv_test.jar is the final product. Ready to be placed in  mmplugin folder of your MM installation. 
~~~~~~~~~~~~~~~~~~~~~~
Vikram G Kopuri



On Fri, Oct 5, 2018 at 4:25 PM Vikram G Kopuri <[hidden email]> wrote:
Interesting , I'll have to remember this when I'm ready to port my plugin over to MM2.0. 
~~~~~~~~~~~~~~~~~~~~~~
Vikram G Kopuri



On Fri, Oct 5, 2018 at 1:58 PM Stuurman, Nico <[hidden email]> wrote:
Hi Bin and Vikram,
> It seems like that the plugin loading and initialization mechanism in MM2.0
> is different from 1.4. I moved my plugin jar that loads opencv library to
> the "plugins" folder (instead of "mmplugins" folder) and the library can be
> successfully called. However, menu plugins should normally live in the
> "mmplugins" folder. I checked that in both folders opencv_java.dll can be
> loaded, but only in the "plugins" folder that it can be correctly called.

That sounds familiar.  Chris and Mark decided that plugins should live
in their own class-loaders to avoid interference with other stuff.  I
can see the reasoning, but in practice it is a bit of a pain.

To make plugin code available to other plugins, Chris and Mark added the
"libraries" directory in the Micro-Manager source code repository. 
Plugins in that directory end up in the plugins folder in the final
installation.  The Localization Microscopy plugin has a very thin layer
in the mmplugins directory that calls code living in the "libraries"
directory.  It would be nice to simplify this organization.

Best,

Nico



_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Reply | Threaded
Open this post in threaded view
|

Re: Using Opencv Lib with MicroManager plugin

Micro-Manager mailing list
Hi Vikram,

It sounds fascinating, I will give it a try later to see whether it works
with 2.0, thanks!

Bin



--
Sent from: http://micro-manager.3463995.n2.nabble.com/


_______________________________________________
micro-manager-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/micro-manager-general