Mozilla MemShrink Set To Fix Firefox Memory

darthcamaro writes "If you're like a lot of Firefox 4 users out there, you've probably noticed that Firefox has a serious memory problem — it uses more than it really should. At long last, Mozilla developers are finally set to take this issue seriously with a dedicated team called MemShrink that are focused on the problem. 'It's pretty clear by now that this is a much bigger problem than any one person can likely tackle,' Mozilla Developer Johnny Stenback said."

Pinpoint

[globalist]

This is easily reproduced in FF4 when you load a page with lots of images. The mem tends to grow proportionally to the size of the images on the page. But this is the only scenariou where the mem usage is different from other browsers and needs looking into.

Re:Problem of perception?

[adri]

Ah, the sounds of someone spewing the 80's virtual memory rhetoric.

There's more to it then that. I could go into it, but I'm supposed to be studying for a psychology exam, so I'll be brief.

You assume that the OS will make sensible paging decisions. You assume you can hint to the OS that you're going to make sensible paging decisions. You hope the application, which is likely big, multithreaded and such, is doing the sensible thing of not wrapping large accesses to "memory things" (eg big trees of data, as an example, or image caches, or whatever takes up more than a small bit of RAM) in mutexes. You assume that your application is using memory in a sensible fashion, and not simply using a few bytes here and there in each allocated chunk.

The trouble is, application writers have been taught from an "early" age that hey, memory is cheap, the OS will handle paging out unused bits, so please go right ahead and use it without caring about how it's used. This is how you end up with application behaviours which include, but aren't limited to:

* walking a tree requires a page in (ie, a random disk read) for each tree node touched. Because each node is malloc()'ed, and although on modern implementations small objects are packed into pages, your 10,000 tree node is going to likely be spread across multiple pages based on when and how often you allocated them ('temporal location');
* this also means your memory use versus memory allocation isn't terribly efficient ('fragmentation');
* your mutex protected data structures are suddenly now mutex'ing _page disk access_, so whilst the OS is busy paging in your data, all other threads currently trying to do stuff that requires that mutex (which may even not require paging in) suddenly has to stop and wait for your page-in to complete.

It's a real shame that memory management has really stopped progressing since virtual memory systems were made. They're convenient, but they hide the worst case behaviours from unknowing coders. Then those worse case behaviours become _very_ worse case behaviours which can't be changed without a fundamental rearchitecture of your software. Likely what people are realising here.

Enjoy!

Re:Let's do a test

[Dwedit]

Looks like the Url Classifier (worthless anti-phishing feature) is eating up a fair amount of resources for nothing.  You look in your user profile, and you see the urlclassifier3.sqlite file keep getting bigger and bigger for no real reason, and you see that Firefox needs 24MB of RAM just to have it loaded and working.

Here's my list (sorted top 25, but may be misleading since some of the numbers look like sums of others)

win32/workingset                             432,160,768
win32/privatebytes                           420,995,072
malloc/committed                             376,307,712
DescriptionValuemalloc/allocated             354,080,976
storage/sqlite/pagecache                     79,423,848
js/gc-heap                                   66,060,288
storage/places.sqlite/Cache_Used             52,400,928
storage/urlclassifier3.sqlite/Cache_Used     24,250,888
js/string-data                               7,282,684
layout/all                                   6,261,185
images/content/used/raw                      4,043,747
images/content/used/uncompressed             3,444,586
gfx/surface/win32                            3,398,208
storage/sqlite/other                         1,589,512
malloc/dirty                                 1,097,728
js/mjit-code                                 913,507
storage/cookies.sqlite/Cache_Used            528,376
storage/formhistory.sqlite/Cache_Used        508,664
storage/extensions.sqlite/Cache_Used         494,144
storage/addons.sqlite/Cache_Used             362,512
content/canvas/2d_pixel_bytes                360,000
images/chrome/used/uncompressed              353,440
shmem/allocated                              344,064
shmem/mapped                                 344,064