[concurrency-interest] Monitoring Tool

William Louth (JINSPIRED.COM) william.louth at jinspired.com
Sun Aug 1 14:00:21 EDT 2010

  Hi Gregg,

We employ many techniques some of which have enabled us to support Java 
1.4 VM's up to the release of OpenCore 6.0. But as in any endeavor that 
has constraints the key is to start with a good design in terms of 
model, interface and runtime. OpenCore has achieved this via its 
metering concepts of activities (probes and cost group hierarchies) and 
meters (typically thread specific counters, measures, resources). With 
any global state management the goal should always to balance cost and 
value especially when value itself allows one to dynamic reduce cost 
itself. We employ various low level techniques as well as many 
architectural ones which are touched on in this article.


I will be giving a Google tech talk next week on the usage of smart 
metering within software stacks and across cloud service interactions 
next week which hopefully be made available the following week. During 
this talk I will discuss the techniques employed today to reduce such 
overhead and new techniques we are currently investigating which nearly 
eliminate this via a synch(point) probes provider optional add-on to our 

Some other tools employ the use of non-application threads (and pools) 
for collection & update but unfortunately these approaches are generally 
more expensive (globally) than ones that do not but are much smarter and 
efficient in how it is does within the thread context - a benefit of 
which is that scalability is predictable and data is real-time which can 
be important if the metering information is used itself within a 
grid/task scheduler.



On 29/07/2010 21:02, Gregg Wonderly wrote:
> Again, one of the big issues for me, is that typically, such metering 
> requires atomic counting, which, itself, is a form of contention.  
> This contention can get injected into previously uncontended blocks 
> and redistribute the timing through particular parts of the code which 
> then don't reveal the actual behavior in normal execution.
> Where can I see more information about how issues like this are dealt 
> with?
> Gregg Wonderly
> William Louth (JINSPIRED.COM) wrote:
>>  It can do this and more but maybe I should have factor in the 
>> likelihood that you would have only skimmed over the links I email - 
>> time is money. Sorry for not directly pointing this out but as the 
>> architect of both the Probes and Metrics Open API's I thought this 
>> would be rather obvious.
>> OpenCore supports many meters some of which are contention related: 
>> http://opencore.jinspired.com/?page_id=981#p:core A lot of these are 
>> not time related.
>> OpenCore's Probes metering can be automatically mapped to Metrics to 
>> correlate across multiple threads with other measurements not metered 
>> based.
>> http://opencore.jinspired.com/?page_id=377
>> OpenCore supports the reporting of metering at thread and probes 
>> level. You indicate transaction points to the runtime via config and 
>> then see which particular probes (activities) and meters (resources: 
>> monitors) contributed the most. See transaction probes provider under 
>> Guides.
>> http://opencore.jinspired.com/?page_id=772
>> On 26/07/2010 23:56, Gregg Wonderly wrote:
>>> This doesn't really point at concurrency issues, only in execution 
>>> time of compute bound execution, and perhaps some simple linear 
>>> scaling of complexity.
>>> When you have 1000s of instructions and multiple code paths, which 
>>> include synchronization, and random latency injection, I am not sure 
>>> that you can see how threads are "waiting in line" until you see the 
>>> quadratic change in execution time that is typically visible when 
>>> high contention occurs.
>>> Maybe you can point out where this tool provides for the ability to 
>>> monitor lock contention and count stalled threads and otherwise see 
>>> the real contention that develops over time as load increases on a 
>>> highly contended code segment?
>>> If you could "time" the execution interval through all code paths, 
>>> and then look at the "percentage" of threads and time through each 
>>> path, and then see that the "largest" latency block was in a very 
>>> common code path, you could perhaps then say this was the area to 
>>> look at.  All of that analysis might not really be possible though, 
>>> because of the exponential potential complexity of code path coverage.
>>> Gregg Wonderly
>>> William Louth (JINSPIRED.COM) wrote:
>>>>  JINSPIRED'S OpenCore (http://opencore.jinspired.com) metering & 
>>>> metrics runtime has built in meters for thread contention metering 
>>>> (activity based) @see blocking.time and blocking.count.
>>>> You can easily extend it with your own custom counters or resource 
>>>> measures mapped to meters. And you don't always need to measure 
>>>> time: 
>>>> http://williamlouth.wordpress.com/2010/06/11/no-latency-application-performance-analysis-when-wall-clock-time-is-simply-too-slow/ 
>>>> Queuing can also be aggregated at various namespace levels: 
>>>> http://williamlouth.wordpress.com/2010/05/20/metered-software-service-queues/ 
>>>> More related articles: 
>>>> http://williamlouth.wordpress.com/category/profiling/
>>>> On 26/07/2010 21:13, Gregg Wonderly wrote:
>>>>> Per thread latency measurements with 1, 2, 10 and 100 threads will 
>>>>> often tell you a lot about how contention is affecting the 
>>>>> execution time.  When you get to 100, a thread dump will often 
>>>>> reveal where everyone is standing in line...
>>>>> Gregg Wonderly
>>>>> David Holmes wrote:
>>>>>> Kendall,
>>>>>> In my opinion a monitoring tool looking at the lock acquisition 
>>>>>> time, or CAS attempts, won't give you much insight into whether 
>>>>>> to use a blocking or non-blocking approach. You need to measure 
>>>>>> the performance of your application logic as a whole, utilising 
>>>>>> the two different approaches. Afterall how can you compare 
>>>>>> locking times with number of CAS attempts in general?
>>>>>> David Holmes
>>>>>>     -----Original Message-----
>>>>>>     *From:* concurrency-interest-bounces at cs.oswego.edu
>>>>>>     [mailto:concurrency-interest-bounces at cs.oswego.edu]*On Behalf Of
>>>>>>     *Kendall Moore
>>>>>>     *Sent:* Sunday, 25 July 2010 4:14 PM
>>>>>>     *To:* concurrency-interest at cs.oswego.edu
>>>>>>     *Subject:* [concurrency-interest] Monitoring Tool
>>>>>>     Greetings all,
>>>>>>     Is there a common consensus on which monitoring tools are 
>>>>>> best to
>>>>>>     use when writing parallel apps?  To be more specific, I would 
>>>>>> like
>>>>>>     to be able to know how many times a given thread has to try 
>>>>>> to CAS
>>>>>>     before succeeding.   Also, the ability to see how long a thread
>>>>>>     waits to acquire a lock would be useful as well.  The end goal
>>>>>>     would, in my particular case, would be to compare these in 
>>>>>> order to
>>>>>>     determine if a non-blocking approach would be more effective 
>>>>>> in a
>>>>>>     given situation than a lock-based approach.  Any help would 
>>>>>> be much
>>>>>>     appreciated!
>>>>>>     --     Kendall Moore
>>>>>> ------------------------------------------------------------------------ 
>>>>>> _______________________________________________
>>>>>> Concurrency-interest mailing list
>>>>>> Concurrency-interest at cs.oswego.edu
>>>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>>> _______________________________________________
>>>>> Concurrency-interest mailing list
>>>>> Concurrency-interest at cs.oswego.edu
>>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>> _______________________________________________
>>>> Concurrency-interest mailing list
>>>> Concurrency-interest at cs.oswego.edu
>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest

More information about the Concurrency-interest mailing list