bugGNU Octave - Bugs: bug #47707, hist takes up a lot of memory

 
 

bug #47707: hist takes up a lot of memory

Submitter:  Carnë Draug <carandraug>
Submitted:  Fri 15 Apr 2016 06:05:52 PM UTC
   
 
Category:  Performance Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 16 Apr 2016 06:22:33 PM UTC, comment #3: 

After investigation, there is no issue with saturation (or at least not anymore -- maybe there was in 2007).  There was an issue when adding the bin edges (cutoff variable) to the array which caused the edges to be rounded.  Fix was to floor the cut off points ourselves.

Fixed with  http://hg.savannah.gnu.org/hgweb/octave/rev/225d88a97104 which adds several tests for class integer in the second argument.

Of special note that Matlab does not even support input of class integers.

Carnë Draug <carandraug>
Group Member
Sat 16 Apr 2016 02:07:16 PM UTC, comment #2: 


> The script doesn't say what issues it is trying to avoid. Do you know what they are? I'd have thought that working with integers would be the way to go.


hg blame points to e8d953d03f6a .  This precedes meaningful commit messages, but adds the following note on the ChangeLog file.

  • plot/hist.m: Avoid saturation when the x values are in an inetger type.


There are no tests in hist for integer input so I will write them and see if this conversion to double is really necessary or if it can be avoided in some cases.

> Alternatively, do you need a histogram of your whole 4-D image, or is a histogram over one or two dimensions? A compromise may be to convert only one row/column/page/... at a time to double.


No, I want a single histogram for the whole image (this is to compute global threshold value).  I have worked around it by using histc like so:


  ## We use histc() instead of hist() to save memory (see bug #47707), but
  ## then we need to adjust the weird output of histc.
  h = histc (im(:), [linspace(min (im(:)), max(im(:)), nbins+1)(1:end-1) Inf])(1:end-1);


> However, I thought that "hist" was being phased out by Matlab in favour of "histogram". (I haven't checked the differences.) Perhaps we should just write an efficient implementation of that and leave hist.


Doesn't mean that the function hist will be removed. I think it's been around for far too long and widely used to just be dropped like that.  If I can also make it better, good.

---

Another issue that I didn't mention at the start is that somewhere in the code, this actually crashed Octave.  Sometimes it would error about not having enough memory (fair enough), but roughly half of the times the kernel would just kill the process due to out-of-memory.

Carnë Draug <carandraug>
Group Member
Sat 16 Apr 2016 05:32:41 AM UTC, comment #1: 

The script doesn't say what issues it is trying to avoid.  Do you know what they are?  I'd have thought that working with integers would be the way to go.

Alternatively, do you need a histogram of your whole 4-D image, or is a histogram over one or two dimensions?  A compromise may be to convert only one row/column/page/... at a time to double.

However, I thought that "hist" was being phased out by Matlab in favour of "histogram".  (I haven't checked the differences.)  Perhaps we should just write an efficient implementation of that and leave hist.

Lachlan Andrew <lachlan>
Fri 15 Apr 2016 06:05:52 PM UTC, original submission:  

The function hist converts the input vector or matrix to class double to avoid any issues with integers

http://hg.savannah.gnu.org/hgweb/octave/file/1dda942a2514/scripts/plot/draw/hist.m


  ## Avoid issues with integer types for x and y
  x = double (x);
  y = double (y);


If the input is uint8 (very common in images), that's an 8x increase in memory usage.  In my current case, I have a 480x480x15x28 image (uint16) which is roughly 190Mb but Octave needs 4GB of memory to make an histogram of the image.

Carnë Draug <carandraug>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by carandraug (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-16 carandraug StatusNone Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code