Fiji Bio-Formats is not opening large czi?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Fiji Bio-Formats is not opening large czi?

Mario Emmenlauer-3
Hi,

I have a 85 GB CZI file (one out of a series of 40 files in the
same folder) that I wanted to open with latest Fiji (updates from
today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
   Plugins > Bio-Formats > Bio-Formats Importer

with options:
 - View with Hyperstack
 - Open files individually
 - Use virtual stack
 - all other options unchecked

Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
I've waited 20 minutes now, then gave up. About Bio-Formats says
"5.0.7 built 3.2.2015 revision 274796b".

The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
the --debug option, but the command prompt console does not show any
messages (to track down whats happening).

Thanks for your help, and all the best,

    Mario




--
Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
D-81669 München                          http://www.biodataanalysis.de/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

ctrueden
Hi Mario,

> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> I've waited 20 minutes now, then gave up.

Did you try reading it with the Bio-Formats showinf command line tool?
http://openmicroscopy.org/info/bio-formats/users/comlinetools/

Did you check out the "Debugging JVM hangs" section of the Debugging page?
http://imagej.net/Debugging#Debugging_JVM_hangs

Getting a stack trace will tell you more about what is going on -- i.e.,
what Bio-Formats is doing internally, whether it has crashed, etc.

Did you try with the Bio-Formats update site enabled? This enables the
5.1.x development builds, which contain additional bug-fixes.
http://fiji.sc/Bio-Formats#Daily_builds

If this is a bug in Bio-Formats (if showinf also does not work, etc.), then
please report this bug to the OME team.
http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html

Regards,
Curtis

On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
wrote:

> Hi,
>
> I have a 85 GB CZI file (one out of a series of 40 files in the
> same folder) that I wanted to open with latest Fiji (updates from
> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>    Plugins > Bio-Formats > Bio-Formats Importer
>
> with options:
>  - View with Hyperstack
>  - Open files individually
>  - Use virtual stack
>  - all other options unchecked
>
> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> I've waited 20 minutes now, then gave up. About Bio-Formats says
> "5.0.7 built 3.2.2015 revision 274796b".
>
> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
> the --debug option, but the command prompt console does not show any
> messages (to track down whats happening).
>
> Thanks for your help, and all the best,
>
>     Mario
>
>
>
>
> --
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> D-81669 München                          http://www.biodataanalysis.de/
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Douglas Richardson
In reply to this post by Mario Emmenlauer-3
Hi Mario,

Is this a Lightsheet file that's been separated by time (or color, view,
etc.) so that you have a series of files in the format: filename.czi;
filename(1).czi; filename(2).czi?  If so, open the complete file
(filename.czi) with Bioformats. You will then see a number of thumbnail
images representing filename(1).czi, filename(2).czi, etc.  Check the one
you want to open.  If you do not have enough RAM to open the file, make
sure you check "Open in virtual Stack."

This will take Fiji a long time to open (possibly minutes) as it has to
read all of the header data.  Even longer than ZEN.

-Doug


On Tue, Mar 3, 2015 at 9:45 AM, Mario Emmenlauer <[hidden email]>
wrote:

> Hi,
>
> I have a 85 GB CZI file (one out of a series of 40 files in the
> same folder) that I wanted to open with latest Fiji (updates from
> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>    Plugins > Bio-Formats > Bio-Formats Importer
>
> with options:
>  - View with Hyperstack
>  - Open files individually
>  - Use virtual stack
>  - all other options unchecked
>
> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> I've waited 20 minutes now, then gave up. About Bio-Formats says
> "5.0.7 built 3.2.2015 revision 274796b".
>
> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
> the --debug option, but the command prompt console does not show any
> messages (to track down whats happening).
>
> Thanks for your help, and all the best,
>
>     Mario
>
>
>
>
> --
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> D-81669 München                          http://www.biodataanalysis.de/
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Stephan Preibisch
Hi Mario,

I did not experience this problem using Bioformats so far, even for very long timelapses acquired with the Lightsheet Z.1.

You can try my Multiview Reconstruction pipeline which is part of Fiji (http://fiji.sc/Multiview-Reconstruction <http://fiji.sc/Multiview-Reconstruction>) to test it. Just

1) Define a new XML dataset (Plugins > Multiview Reconstruction > Define Multi-View Dataset). Press OK, then choose the first filename.CZI (the one without (1), (2), etc.)
2) It will show all the metadata and you should just have to click OK.
3) Then, for example display one of the views (Plugins > Multiview Reconstruction > Tools > Display View)

You can also resave the entire or parts of the dataset as TIFF or HDF5 (Plugins > Multiview Reconstruction > Resave > As TIFF). It also supports registration, fusion, deconvolution of the dataset!

Hope this helps,
Stephan
---

Dr. Stephan Preibisch
HFSP Fellow
Robert H. Singer / Eugene Myers lab

Albert Einstein College of Medicine / HHMI Janelia Farm / MPI-CBG

email: [hidden email] <mailto:[hidden email]> / [hidden email] <mailto:[hidden email]> / [hidden email] <mailto:[hidden email]>
web: http://www.preibisch.net/ <http://fly.mpi-cbg.de/preibisch>

> On Mar 3, 2015, at 11:49 , Douglas Richardson <[hidden email]> wrote:
>
> Hi Mario,
>
> Is this a Lightsheet file that's been separated by time (or color, view,
> etc.) so that you have a series of files in the format: filename.czi;
> filename(1).czi; filename(2).czi?  If so, open the complete file
> (filename.czi) with Bioformats. You will then see a number of thumbnail
> images representing filename(1).czi, filename(2).czi, etc.  Check the one
> you want to open.  If you do not have enough RAM to open the file, make
> sure you check "Open in virtual Stack."
>
> This will take Fiji a long time to open (possibly minutes) as it has to
> read all of the header data.  Even longer than ZEN.
>
> -Doug
>
>
> On Tue, Mar 3, 2015 at 9:45 AM, Mario Emmenlauer <[hidden email]>
> wrote:
>
>> Hi,
>>
>> I have a 85 GB CZI file (one out of a series of 40 files in the
>> same folder) that I wanted to open with latest Fiji (updates from
>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>   Plugins > Bio-Formats > Bio-Formats Importer
>>
>> with options:
>> - View with Hyperstack
>> - Open files individually
>> - Use virtual stack
>> - all other options unchecked
>>
>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>> "5.0.7 built 3.2.2015 revision 274796b".
>>
>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>> the --debug option, but the command prompt console does not show any
>> messages (to track down whats happening).
>>
>> Thanks for your help, and all the best,
>>
>>    Mario
>>
>>
>>
>>
>> --
>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>> D-81669 München                          http://www.biodataanalysis.de/
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Mario Emmenlauer-3
In reply to this post by Douglas Richardson
Hi,

On 03.03.2015 17:49, Douglas Richardson wrote:
> Is this a Lightsheet file that's been separated by time (or color, view,
> etc.) so that you have a series of files in the format: filename.czi;
> filename(1).czi; filename(2).czi?  If so, open the complete file
> (filename.czi) with Bioformats. You will then see a number of thumbnail
> images representing filename(1).czi, filename(2).czi, etc.  Check the one
> you want to open.  If you do not have enough RAM to open the file, make
> sure you check "Open in virtual Stack."

Yes, its a Lightsheet file that has been separated by time. I tried
to open the complete file drlGFPlmo2Red7ss.czi with Bio-Formats, and
nothing shows up, even after many minutes (no thumnails, no popup,
no status message, nothing). Its as if I never tried "open" at all.
Fiji just stands there, idle...

Could it be that opening files from a (very fast fiber-connected)
network share (mounted as drive letter L:) is not supported?

All the best,

    Mario



> This will take Fiji a long time to open (possibly minutes) as it has to
> read all of the header data.  Even longer than ZEN.
>
> -Doug
>
>
> On Tue, Mar 3, 2015 at 9:45 AM, Mario Emmenlauer <[hidden email]>
> wrote:
>
>> Hi,
>>
>> I have a 85 GB CZI file (one out of a series of 40 files in the
>> same folder) that I wanted to open with latest Fiji (updates from
>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>    Plugins > Bio-Formats > Bio-Formats Importer
>>
>> with options:
>>  - View with Hyperstack
>>  - Open files individually
>>  - Use virtual stack
>>  - all other options unchecked
>>
>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>> "5.0.7 built 3.2.2015 revision 274796b".
>>
>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>> the --debug option, but the command prompt console does not show any
>> messages (to track down whats happening).
>>
>> Thanks for your help, and all the best,
>>
>>     Mario
>>
>>
>>
>>
>> --
>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>> D-81669 München                          http://www.biodataanalysis.de/
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?

Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
D-81669 München                          http://www.biodataanalysis.de/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Mario Emmenlauer-3
In reply to this post by Stephan Preibisch
Hi,

On 03.03.2015 18:45, Stephan Preibisch wrote:
> I did not experience this problem using Bioformats so far, even for very long timelapses acquired with the Lightsheet Z.1.
>
> You can try my Multiview Reconstruction pipeline which is part of Fiji (http://fiji.sc/Multiview-Reconstruction <http://fiji.sc/Multiview-Reconstruction>) to test it. Just
>
> 1) Define a new XML dataset (Plugins > Multiview Reconstruction > Define Multi-View Dataset). Press OK, then choose the first filename.CZI (the one without (1), (2), etc.)
> 2) It will show all the metadata and you should just have to click OK.
> 3) Then, for example display one of the views (Plugins > Multiview Reconstruction > Tools > Display View)

I tried this, also this seems not to work. Here is what I see:

 1) Define a new XML dataset (Plugins > Multiview Reconstruction > Define Multi-View Dataset). Press OK, then choose drlGFPlmo2Red7ss.czi
 2) It shows in the log window:

    2
    Investigating file 'L:\MarioEmmenlauer\Lightsheet_RAW\drlGFPlmo2Red7ss.czi'.

No output comes after that in a very long time.

All the best,

    Mario



> You can also resave the entire or parts of the dataset as TIFF or HDF5 (Plugins > Multiview Reconstruction > Resave > As TIFF). It also supports registration, fusion, deconvolution of the dataset!
>
> Hope this helps,
> Stephan
> ---
>
> Dr. Stephan Preibisch
> HFSP Fellow
> Robert H. Singer / Eugene Myers lab
>
> Albert Einstein College of Medicine / HHMI Janelia Farm / MPI-CBG
>
> email: [hidden email] <mailto:[hidden email]> / [hidden email] <mailto:[hidden email]> / [hidden email] <mailto:[hidden email]>
> web: http://www.preibisch.net/ <http://fly.mpi-cbg.de/preibisch>
>> On Mar 3, 2015, at 11:49 , Douglas Richardson <[hidden email]> wrote:
>>
>> Hi Mario,
>>
>> Is this a Lightsheet file that's been separated by time (or color, view,
>> etc.) so that you have a series of files in the format: filename.czi;
>> filename(1).czi; filename(2).czi?  If so, open the complete file
>> (filename.czi) with Bioformats. You will then see a number of thumbnail
>> images representing filename(1).czi, filename(2).czi, etc.  Check the one
>> you want to open.  If you do not have enough RAM to open the file, make
>> sure you check "Open in virtual Stack."
>>
>> This will take Fiji a long time to open (possibly minutes) as it has to
>> read all of the header data.  Even longer than ZEN.
>>
>> -Doug
>>
>>
>> On Tue, Mar 3, 2015 at 9:45 AM, Mario Emmenlauer <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a 85 GB CZI file (one out of a series of 40 files in the
>>> same folder) that I wanted to open with latest Fiji (updates from
>>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>>   Plugins > Bio-Formats > Bio-Formats Importer
>>>
>>> with options:
>>> - View with Hyperstack
>>> - Open files individually
>>> - Use virtual stack
>>> - all other options unchecked
>>>
>>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>>> "5.0.7 built 3.2.2015 revision 274796b".
>>>
>>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>>> the --debug option, but the command prompt console does not show any
>>> messages (to track down whats happening).
>>>
>>> Thanks for your help, and all the best,
>>>
>>>    Mario
>>>
>>>
>>>
>>>
>>> --
>>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>>> D-81669 München                          http://www.biodataanalysis.de/
>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?

Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
D-81669 München                          http://www.biodataanalysis.de/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Stephan Preibisch
In reply to this post by Mario Emmenlauer-3
Yes, this might be the issue … over the network it might be much slower from my experience.

Cheers,
Stephan

> On Mar 4, 2015, at 7:28 , Mario Emmenlauer <[hidden email]> wrote:
>
> Hi,
>
> On 03.03.2015 17:49, Douglas Richardson wrote:
>> Is this a Lightsheet file that's been separated by time (or color, view,
>> etc.) so that you have a series of files in the format: filename.czi;
>> filename(1).czi; filename(2).czi?  If so, open the complete file
>> (filename.czi) with Bioformats. You will then see a number of thumbnail
>> images representing filename(1).czi, filename(2).czi, etc.  Check the one
>> you want to open.  If you do not have enough RAM to open the file, make
>> sure you check "Open in virtual Stack."
>
> Yes, its a Lightsheet file that has been separated by time. I tried
> to open the complete file drlGFPlmo2Red7ss.czi with Bio-Formats, and
> nothing shows up, even after many minutes (no thumnails, no popup,
> no status message, nothing). Its as if I never tried "open" at all.
> Fiji just stands there, idle...
>
> Could it be that opening files from a (very fast fiber-connected)
> network share (mounted as drive letter L:) is not supported?
>
> All the best,
>
>    Mario
>
>
>
>> This will take Fiji a long time to open (possibly minutes) as it has to
>> read all of the header data.  Even longer than ZEN.
>>
>> -Doug
>>
>>
>> On Tue, Mar 3, 2015 at 9:45 AM, Mario Emmenlauer <[hidden email] <mailto:[hidden email]>>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a 85 GB CZI file (one out of a series of 40 files in the
>>> same folder) that I wanted to open with latest Fiji (updates from
>>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>>   Plugins > Bio-Formats > Bio-Formats Importer
>>>
>>> with options:
>>> - View with Hyperstack
>>> - Open files individually
>>> - Use virtual stack
>>> - all other options unchecked
>>>
>>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>>> "5.0.7 built 3.2.2015 revision 274796b".
>>>
>>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>>> the --debug option, but the command prompt console does not show any
>>> messages (to track down whats happening).
>>>
>>> Thanks for your help, and all the best,
>>>
>>>    Mario
>>>
>>>
>>>
>>>
>>> --
>>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>>> D-81669 München                          http://www.biodataanalysis.de/
>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> A: Yes.
>> Q: Are you sure?
>>> A: Because it reverses the logical flow of conversation.
>>>> Q: Why is top posting annoying in email?
>
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch <http://unibas.ch/>
> D-81669 München                          http://www.biodataanalysis.de/ <http://www.biodataanalysis.de/>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Mario Emmenlauer-3
In reply to this post by ctrueden
Hi Curtis,

On 03.03.2015 17:11, Curtis Rueden wrote:
>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>> I've waited 20 minutes now, then gave up.
>
> Did you try reading it with the Bio-Formats showinf command line tool?
> http://openmicroscopy.org/info/bio-formats/users/comlinetools/

I tried that now, and it seems Stephan Preibisch might be right that
opening the files just takes forever. But why? The network mount is
reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
file in ~30sec, and I can scroll through the timepoints and stacks at
reasonable speed. I opened the czi with "Open files individually" and
"Use virtual stack" in Fiji, and I hoped this might speed things up.

So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
25% of the time it would take to read *all* the files over the 100MB/s
network. I can not think Bio-Formats would not read 25% of the 4TB
data, just to show thumbnails?


When using showinf, it prints first:
  Checking file format [Zeiss CZI]
  Initializing reader
  ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP lmo2Red 7ss .czi

... and then after around 1 hour, it fails with out-of-memory. Can
it really take >2 hrs before I can see anything? Should the option
"Open files individually" not limit it to the first timepoint?

Cheers,

    Mario



> Did you check out the "Debugging JVM hangs" section of the Debugging page?
> http://imagej.net/Debugging#Debugging_JVM_hangs
>
> Getting a stack trace will tell you more about what is going on -- i.e.,
> what Bio-Formats is doing internally, whether it has crashed, etc.
>
> Did you try with the Bio-Formats update site enabled? This enables the
> 5.1.x development builds, which contain additional bug-fixes.
> http://fiji.sc/Bio-Formats#Daily_builds
>
> If this is a bug in Bio-Formats (if showinf also does not work, etc.), then
> please report this bug to the OME team.
> http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html
>
> Regards,
> Curtis
>
> On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
> wrote:
>
>> Hi,
>>
>> I have a 85 GB CZI file (one out of a series of 40 files in the
>> same folder) that I wanted to open with latest Fiji (updates from
>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>    Plugins > Bio-Formats > Bio-Formats Importer
>>
>> with options:
>>  - View with Hyperstack
>>  - Open files individually
>>  - Use virtual stack
>>  - all other options unchecked
>>
>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>> "5.0.7 built 3.2.2015 revision 274796b".
>>
>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>> the --debug option, but the command prompt console does not show any
>> messages (to track down whats happening).
>>
>> Thanks for your help, and all the best,
>>
>>     Mario
>>
>>
>>
>>
>> --
>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>> D-81669 München                          http://www.biodataanalysis.de/
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
A: Yes.
> Q: Are you sure?
>> A: Because it reverses the logical flow of conversation.
>>> Q: Why is top posting annoying in email?

Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
D-81669 München                          http://www.biodataanalysis.de/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

ctrueden
Hi Mario,

> Stephan Preibisch might be right that opening the files just takes
> forever. But why? The network mount is reasonably fast, it achieves
> easily 100MB/s.

There are multiple sorts of "fast" at work here: burst throughput,
sustained throughput, latency, etc.

If you really want to know why it is slow, then take a few stack traces
while you are waiting for the files to be read. Just press Ctrl+\ while
showinf is thinking. Cut and paste them from your console, saving them to
files. Look for the string "loci.formats." and find out which Bio-Formats
classes/method/lines are being hit. That will be 90% likely to answer your
"why" question in this instance.

Regards,
Curtis

On Wed, Mar 4, 2015 at 9:26 AM, Mario Emmenlauer <[hidden email]>
wrote:

> Hi Curtis,
>
> On 03.03.2015 17:11, Curtis Rueden wrote:
> >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> >> I've waited 20 minutes now, then gave up.
> >
> > Did you try reading it with the Bio-Formats showinf command line tool?
> > http://openmicroscopy.org/info/bio-formats/users/comlinetools/
>
> I tried that now, and it seems Stephan Preibisch might be right that
> opening the files just takes forever. But why? The network mount is
> reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
> file in ~30sec, and I can scroll through the timepoints and stacks at
> reasonable speed. I opened the czi with "Open files individually" and
> "Use virtual stack" in Fiji, and I hoped this might speed things up.
>
> So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
> 25% of the time it would take to read *all* the files over the 100MB/s
> network. I can not think Bio-Formats would not read 25% of the 4TB
> data, just to show thumbnails?
>
>
> When using showinf, it prints first:
>   Checking file format [Zeiss CZI]
>   Initializing reader
>   ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP
> lmo2Red 7ss .czi
>
> ... and then after around 1 hour, it fails with out-of-memory. Can
> it really take >2 hrs before I can see anything? Should the option
> "Open files individually" not limit it to the first timepoint?
>
> Cheers,
>
>     Mario
>
>
>
> > Did you check out the "Debugging JVM hangs" section of the Debugging
> page?
> > http://imagej.net/Debugging#Debugging_JVM_hangs
> >
> > Getting a stack trace will tell you more about what is going on -- i.e.,
> > what Bio-Formats is doing internally, whether it has crashed, etc.
> >
> > Did you try with the Bio-Formats update site enabled? This enables the
> > 5.1.x development builds, which contain additional bug-fixes.
> > http://fiji.sc/Bio-Formats#Daily_builds
> >
> > If this is a bug in Bio-Formats (if showinf also does not work, etc.),
> then
> > please report this bug to the OME team.
> > http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html
> >
> > Regards,
> > Curtis
> >
> > On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
> > wrote:
> >
> >> Hi,
> >>
> >> I have a 85 GB CZI file (one out of a series of 40 files in the
> >> same folder) that I wanted to open with latest Fiji (updates from
> >> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
> >>    Plugins > Bio-Formats > Bio-Formats Importer
> >>
> >> with options:
> >>  - View with Hyperstack
> >>  - Open files individually
> >>  - Use virtual stack
> >>  - all other options unchecked
> >>
> >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> >> I've waited 20 minutes now, then gave up. About Bio-Formats says
> >> "5.0.7 built 3.2.2015 revision 274796b".
> >>
> >> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
> >> the --debug option, but the command prompt console does not show any
> >> messages (to track down whats happening).
> >>
> >> Thanks for your help, and all the best,
> >>
> >>     Mario
> >>
> >>
> >>
> >>
> >> --
> >> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> >> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> >> D-81669 München                          http://www.biodataanalysis.de/
> >>
> >> --
> >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >>
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
>
> --
> A: Yes.
> > Q: Are you sure?
> >> A: Because it reverses the logical flow of conversation.
> >>> Q: Why is top posting annoying in email?
>
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> D-81669 München                          http://www.biodataanalysis.de/
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Douglas Richardson
Slow file opening from a network drive via Bioformats is a know issue:

http://lists.openmicroscopy.org.uk/pipermail/ome-users/2014-December/004952.html

Performance can be improved slightly by reducing the number of
sub-directories your file in placed in.

-Doug

On Wed, Mar 4, 2015 at 12:50 PM, Curtis Rueden <[hidden email]> wrote:

> Hi Mario,
>
> > Stephan Preibisch might be right that opening the files just takes
> > forever. But why? The network mount is reasonably fast, it achieves
> > easily 100MB/s.
>
> There are multiple sorts of "fast" at work here: burst throughput,
> sustained throughput, latency, etc.
>
> If you really want to know why it is slow, then take a few stack traces
> while you are waiting for the files to be read. Just press Ctrl+\ while
> showinf is thinking. Cut and paste them from your console, saving them to
> files. Look for the string "loci.formats." and find out which Bio-Formats
> classes/method/lines are being hit. That will be 90% likely to answer your
> "why" question in this instance.
>
> Regards,
> Curtis
>
> On Wed, Mar 4, 2015 at 9:26 AM, Mario Emmenlauer <[hidden email]>
> wrote:
>
> > Hi Curtis,
> >
> > On 03.03.2015 17:11, Curtis Rueden wrote:
> > >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> > >> I've waited 20 minutes now, then gave up.
> > >
> > > Did you try reading it with the Bio-Formats showinf command line tool?
> > > http://openmicroscopy.org/info/bio-formats/users/comlinetools/
> >
> > I tried that now, and it seems Stephan Preibisch might be right that
> > opening the files just takes forever. But why? The network mount is
> > reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
> > file in ~30sec, and I can scroll through the timepoints and stacks at
> > reasonable speed. I opened the czi with "Open files individually" and
> > "Use virtual stack" in Fiji, and I hoped this might speed things up.
> >
> > So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
> > 25% of the time it would take to read *all* the files over the 100MB/s
> > network. I can not think Bio-Formats would not read 25% of the 4TB
> > data, just to show thumbnails?
> >
> >
> > When using showinf, it prints first:
> >   Checking file format [Zeiss CZI]
> >   Initializing reader
> >   ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP
> > lmo2Red 7ss .czi
> >
> > ... and then after around 1 hour, it fails with out-of-memory. Can
> > it really take >2 hrs before I can see anything? Should the option
> > "Open files individually" not limit it to the first timepoint?
> >
> > Cheers,
> >
> >     Mario
> >
> >
> >
> > > Did you check out the "Debugging JVM hangs" section of the Debugging
> > page?
> > > http://imagej.net/Debugging#Debugging_JVM_hangs
> > >
> > > Getting a stack trace will tell you more about what is going on --
> i.e.,
> > > what Bio-Formats is doing internally, whether it has crashed, etc.
> > >
> > > Did you try with the Bio-Formats update site enabled? This enables the
> > > 5.1.x development builds, which contain additional bug-fixes.
> > > http://fiji.sc/Bio-Formats#Daily_builds
> > >
> > > If this is a bug in Bio-Formats (if showinf also does not work, etc.),
> > then
> > > please report this bug to the OME team.
> > > http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html
> > >
> > > Regards,
> > > Curtis
> > >
> > > On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I have a 85 GB CZI file (one out of a series of 40 files in the
> > >> same folder) that I wanted to open with latest Fiji (updates from
> > >> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
> > >>    Plugins > Bio-Formats > Bio-Formats Importer
> > >>
> > >> with options:
> > >>  - View with Hyperstack
> > >>  - Open files individually
> > >>  - Use virtual stack
> > >>  - all other options unchecked
> > >>
> > >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> > >> I've waited 20 minutes now, then gave up. About Bio-Formats says
> > >> "5.0.7 built 3.2.2015 revision 274796b".
> > >>
> > >> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
> > >> the --debug option, but the command prompt console does not show any
> > >> messages (to track down whats happening).
> > >>
> > >> Thanks for your help, and all the best,
> > >>
> > >>     Mario
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Mario Emmenlauer BioDataAnalysis             Mobil:
> +49-(0)151-68108489
> > >> Balanstrasse 43                    mailto: mario.emmenlauer *
> unibas.ch
> > >> D-81669 München
> http://www.biodataanalysis.de/
> > >>
> > >> --
> > >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >>
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> >
> > --
> > A: Yes.
> > > Q: Are you sure?
> > >> A: Because it reverses the logical flow of conversation.
> > >>> Q: Why is top posting annoying in email?
> >
> > Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> > Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> > D-81669 München                          http://www.biodataanalysis.de/
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Pavel Tomancak
In reply to this post by Mario Emmenlauer-3
Hi Mario,

Try moving one of those files to either your local file system or to another folder on the remote machine. The idea is to remove it from the place where the first .czi file lives. I think what takes a lot of time is the analysis of master .czi.

Why is another question, probably best asked at the Bio-formats mailing list.

All the best

PAvel


Why this happens is another question, probably

> Hi Curtis,
>
> On 03.03.2015 17:11, Curtis Rueden wrote:
>>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>>> I've waited 20 minutes now, then gave up.
>>
>> Did you try reading it with the Bio-Formats showinf command line tool?
>> http://openmicroscopy.org/info/bio-formats/users/comlinetools/
>
> I tried that now, and it seems Stephan Preibisch might be right that
> opening the files just takes forever. But why? The network mount is
> reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
> file in ~30sec, and I can scroll through the timepoints and stacks at
> reasonable speed. I opened the czi with "Open files individually" and
> "Use virtual stack" in Fiji, and I hoped this might speed things up.
>
> So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
> 25% of the time it would take to read *all* the files over the 100MB/s
> network. I can not think Bio-Formats would not read 25% of the 4TB
> data, just to show thumbnails?
>
>
> When using showinf, it prints first:
>  Checking file format [Zeiss CZI]
>  Initializing reader
>  ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP lmo2Red 7ss .czi
>
> ... and then after around 1 hour, it fails with out-of-memory. Can
> it really take >2 hrs before I can see anything? Should the option
> "Open files individually" not limit it to the first timepoint?
>
> Cheers,
>
>    Mario
>
>
>
>> Did you check out the "Debugging JVM hangs" section of the Debugging page?
>> http://imagej.net/Debugging#Debugging_JVM_hangs
>>
>> Getting a stack trace will tell you more about what is going on -- i.e.,
>> what Bio-Formats is doing internally, whether it has crashed, etc.
>>
>> Did you try with the Bio-Formats update site enabled? This enables the
>> 5.1.x development builds, which contain additional bug-fixes.
>> http://fiji.sc/Bio-Formats#Daily_builds
>>
>> If this is a bug in Bio-Formats (if showinf also does not work, etc.), then
>> please report this bug to the OME team.
>> http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html
>>
>> Regards,
>> Curtis
>>
>> On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a 85 GB CZI file (one out of a series of 40 files in the
>>> same folder) that I wanted to open with latest Fiji (updates from
>>> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
>>>   Plugins > Bio-Formats > Bio-Formats Importer
>>>
>>> with options:
>>> - View with Hyperstack
>>> - Open files individually
>>> - Use virtual stack
>>> - all other options unchecked
>>>
>>> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
>>> I've waited 20 minutes now, then gave up. About Bio-Formats says
>>> "5.0.7 built 3.2.2015 revision 274796b".
>>>
>>> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
>>> the --debug option, but the command prompt console does not show any
>>> messages (to track down whats happening).
>>>
>>> Thanks for your help, and all the best,
>>>
>>>    Mario
>>>
>>>
>>>
>>>
>>> --
>>> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
>>> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
>>> D-81669 München                          http://www.biodataanalysis.de/
>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> A: Yes.
>> Q: Are you sure?
>>> A: Because it reverses the logical flow of conversation.
>>>> Q: Why is top posting annoying in email?
>
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> D-81669 München                          http://www.biodataanalysis.de/
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
Reply | Threaded
Open this post in threaded view
|

Re: Fiji Bio-Formats is not opening large czi?

Melissa Linkert-2
In reply to this post by Mario Emmenlauer-3
Hi Mario,

> I tried that now, and it seems Stephan Preibisch might be right that
> opening the files just takes forever. But why? The network mount is
> reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
> file in ~30sec, and I can scroll through the timepoints and stacks at
> reasonable speed. I opened the czi with "Open files individually" and
> "Use virtual stack" in Fiji, and I hoped this might speed things up.
>
> So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
> 25% of the time it would take to read *all* the files over the 100MB/s
> network. I can not think Bio-Formats would not read 25% of the 4TB
> data, just to show thumbnails?

Reading large files over NFS on Windows is known to be slow in general.
We are currently reviewing improvements to this for the 5.1.0 release of
Bio-Formats later this month.  Note though that these changes are not
yet available through the Bio-Formats update site in Fiji.

Please keep in mind that the initialization step isn't just showing thumbnails,
it's also reading all of the metadata from the entire dataset and assembling
offsets to all of the planes (to improve performance of reading images on demand).

> When using showinf, it prints first:
>   Checking file format [Zeiss CZI]
>   Initializing reader
>   ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP lmo2Red 7ss .czi
>
> ... and then after around 1 hour, it fails with out-of-memory. Can
> it really take >2 hrs before I can see anything? Should the option
> "Open files individually" not limit it to the first timepoint?

If you use the "Open files individually", then it's important to pick
one of the files that ends with a number in parentheses.  The "master"
file (the one without that doesn't end with a number) typically doesn't
contain pixel data, and is meant to indicate which files constitute the
full dataset - so if you pick the master file, it's going to group all
of the files together no matter what.

Regards,
-Melissa

On Wed, Mar 04, 2015 at 04:26:01PM +0100, Mario Emmenlauer wrote:

> Hi Curtis,
>
> On 03.03.2015 17:11, Curtis Rueden wrote:
> >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> >> I've waited 20 minutes now, then gave up.
> >
> > Did you try reading it with the Bio-Formats showinf command line tool?
> > http://openmicroscopy.org/info/bio-formats/users/comlinetools/
>
> I tried that now, and it seems Stephan Preibisch might be right that
> opening the files just takes forever. But why? The network mount is
> reasonably fast, it achieves easily 100MB/s. Zeiss Zen 2012 opens the
> file in ~30sec, and I can scroll through the timepoints and stacks at
> reasonable speed. I opened the czi with "Open files individually" and
> "Use virtual stack" in Fiji, and I hoped this might speed things up.
>
> So in Fiji, I can select the thumbnails finally after > 2hrs. Thats
> 25% of the time it would take to read *all* the files over the 100MB/s
> network. I can not think Bio-Formats would not read 25% of the 4TB
> data, just to show thumbnails?
>
>
> When using showinf, it prints first:
>   Checking file format [Zeiss CZI]
>   Initializing reader
>   ZeissCZIReader initializing L:\MarioEmmenlauer\Lightsheet_RAW\drlGFP lmo2Red 7ss .czi
>
> ... and then after around 1 hour, it fails with out-of-memory. Can
> it really take >2 hrs before I can see anything? Should the option
> "Open files individually" not limit it to the first timepoint?
>
> Cheers,
>
>     Mario
>
>
>
> > Did you check out the "Debugging JVM hangs" section of the Debugging page?
> > http://imagej.net/Debugging#Debugging_JVM_hangs
> >
> > Getting a stack trace will tell you more about what is going on -- i.e.,
> > what Bio-Formats is doing internally, whether it has crashed, etc.
> >
> > Did you try with the Bio-Formats update site enabled? This enables the
> > 5.1.x development builds, which contain additional bug-fixes.
> > http://fiji.sc/Bio-Formats#Daily_builds
> >
> > If this is a bug in Bio-Formats (if showinf also does not work, etc.), then
> > please report this bug to the OME team.
> > http://openmicroscopy.org/info/bio-formats/about/bug-reporting.html
> >
> > Regards,
> > Curtis
> >
> > On Tue, Mar 3, 2015 at 8:45 AM, Mario Emmenlauer <[hidden email]>
> > wrote:
> >
> >> Hi,
> >>
> >> I have a 85 GB CZI file (one out of a series of 40 files in the
> >> same folder) that I wanted to open with latest Fiji (updates from
> >> today, on Win7 x64 with 42GB RAM, Sun Java 7, -Xmx 12GB. I tried:
> >>    Plugins > Bio-Formats > Bio-Formats Importer
> >>
> >> with options:
> >>  - View with Hyperstack
> >>  - Open files individually
> >>  - Use virtual stack
> >>  - all other options unchecked
> >>
> >> Fiji shows "Analyzing XXX" in the status bar, then nothing happens.
> >> I've waited 20 minutes now, then gave up. About Bio-Formats says
> >> "5.0.7 built 3.2.2015 revision 274796b".
> >>
> >> The file opens fine in Zeiss ZEN 2012 in ~30sec. I tried Fiji with
> >> the --debug option, but the command prompt console does not show any
> >> messages (to track down whats happening).
> >>
> >> Thanks for your help, and all the best,
> >>
> >>     Mario
> >>
> >>
> >>
> >>
> >> --
> >> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> >> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> >> D-81669 München                          http://www.biodataanalysis.de/
> >>
> >> --
> >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >>
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
>
> --
> A: Yes.
> > Q: Are you sure?
> >> A: Because it reverses the logical flow of conversation.
> >>> Q: Why is top posting annoying in email?
>
> Mario Emmenlauer BioDataAnalysis             Mobil: +49-(0)151-68108489
> Balanstrasse 43                    mailto: mario.emmenlauer * unibas.ch
> D-81669 München                          http://www.biodataanalysis.de/
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html