in progress…

March 10, 2012

Oracle Coherence for C++ Heap Analysis

Filed under: Coherence, Software — Mark Falco @ 1:43 pm

Recently I was asked what we do detect and prevent memory leaks in Oracle Coherence for C++. I produced a rather long-winded answer, but the recipient liked it enough to ask that I provide it to a bit wider of an audience so here goes:

  • Our C++ managed object model has automatic memory management, ensuring that objects which aren’t referenced are automatically freed (yes reference counting). The reference counting is thread-safe and includes features such as weak-references, and referencing-chaining. So we for the most part have around the same likelihood as leaking as java does, i.e. you can leak if you put things in global collections for instance. Unlike Java we can’t handle object cycles automatically, so we have to manually defend against them. On a side note it is interesting to note that we had very few data structures which included cyclical references.
  • We routinely test, specifically we’ve ensured that for a variety of use cases we can run them in a tight loop and not have memory grow unbounded.
  • Our C++ managed object model includes its own heap analysis tooling, allowing our unit and functional tests to validate that things do not leak.

Our object model’s heap analysis is included in the product and using it you may get a better view into what is being allocated and for how long it is retained. As an example here is now to use it through the coherence c++ console, which is included in the distribution:

  • enable the analyzer by setting the TangosolCoherenceHeapAnalyzer environment variable to one of (object, class, alloc), the following example uses “alloc”
  • run the examples console (in the examples directory) against a local cache
coherence-cpp/examples> ./run console local-foo
  • type “memory delta” and you will see a delta of how much memory is held since the last time “memory delta” was run, in this case it will be a lot, since it has never been run, so ignore this output
  • in fact type “memory delta” again to zero things out, as we don’t want to see the lazy initialization costs from the first time we collect a delta either
  • type “put a b”, to insert a key value pair of two strings into the local cache
  • type “memory delta” to see the memory cost of this operation, and you get:
Space     Count    Allocs    Class
0 B       0        1         coherence::lang::ClassBasedHeapAnalyzer::ClassStats
0 B       0        1         coherence::util::(anonymous namespace)::EntryIterator
0 B       0        1         coherence::util::SafeHashMap::EntrySet
0 B       0        1         coherence::util::SafeHashMap::EntrySetIterator
80 B      1        1         coherence::lang::ClassInfo
104 B     1        1         coherence::util::SafeHashMap::Entry
282 B     3        6         coherence::lang::String

The output is sorted by Space, where Space is the retained heap for a particular class type. Count is the number of retained instances of that class, and Allocs is the number of allocations. So here we see that the “put a b” operation allocated some stuff such as EntrySet but didn’t retain it because we have a Count of 0. We do see though that 5 objects were retained. Lets try a second put to ensure that we’re not measuring lazy initialization though. So now we do “put c d”, and “memory delta” again, and we see:

Space    Count    Allocs    Class
0 B      0        1         coherence::lang::ClassBasedHeapAnalyzer::ClassStats
0 B      0        1         coherence::util::(anonymous namespace)::EntryIterator
0 B      0        1         coherence::util::SafeHashMap::EntrySet
0 B      0        1         coherence::util::SafeHashMap::EntrySetIterator
156 B    2        5         coherence::lang::String
172 B    1        1         coherence::net::cache::LocalInvocableCache::Entry

Now we see the true costs of a local cache put. We retain three objects, which is the entry, the key, and the value. So no leak, and further invocations will show the same thing. But now lets try “remove c”, and “memory delta”:

Space    Count    Allocs    Class
-172 B   -1       0         coherence::net::cache::LocalInvocableCache::Entry
-156 B   -2       4         coherence::lang::String
0 B      0        1         coherence::lang::ClassBasedHeapAnalyzer::ClassStats
0 B      0        1         coherence::util::(anonymous namespace)::EntryIterator
0 B      0        1         coherence::util::SafeHashMap::EntrySet
0 B      0        1         coherence::util::SafeHashMap::EntrySetIterator

Here we see that our Count actually went down by three objects, which is of course expected as we removed and automatically freed the entry, key, and value we’d previously inserted. Oh and in case you were wondering what the 4 string allocations are, they are for the following strings: “remove”, “c”, “memory”, “delta”. This is because even the console is written in the object model and thus all it’s allocations are tracked.

Just to be clear this heap analysis is not a feature of the console, I’m just demonstrating how to access it from the console. Once enabled via the environment variable you can get similar reports just by sending SIGQUIT or hitting CTRL+\, which will generate both a java like thread dump, and also a heap analysis histogram.

Users can even choose to access the heap analyzer programmatically see the associated API docs. The simplest way to make use of the analyzer programmatically is via a COH_ENSURE_HEAP code block. Lets say we just wanted to ensure that a particular block of code did not leave any objects allocated. We can replicate our put/remove console example but in code as follows:

COH_ENSURE_HEAP
   {
   hCache->put(hKey, hValue);
   hCache->remove(kKey);
   }

The COH_ENSURE_HEAP macro block will throw an IllegalStateException is the total live object count at the end of the block does not equal what it was at the beginning. If using the feature it is important to account for lazy initialization, which pretty much means never ensure the first call to a tested block of code, or be sure to sufficiently warm things up before testing. The COH_ENSURE_HEAP feature is intended only for use in unit tests, though the general heap analyzer is very low cost and could be enabled in real apps, and accessed via SIGQUIT or CTRL+/ to monitor what is using the heap.

All in all I hope this demonstrates we’ve definitely done our due diligence in the area of memory leaks. Very very few have cropped up in the lifetime of our C++ API, but when they have the analyzer has allowed us to track them down quite quickly.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: