[concurrency-interest] Single producer, single consumer: unexpected delays for producer

Daniel Harvey dharvey at tachyoncm.com
Fri Aug 8 22:32:48 EDT 2008


David, many thanks for your response. I'm running on Linux (CentOS)  
with a 2 processor machine, though am planning on moving to an 8  
processor machine in the next few days. I'm currently using jrockit  
(java 1.6) as the JVM. Today I downloaded jrockit-realtime so perhaps  
that will help reduce the amount of dropped messages.

What confuses me is where the delay is showing up on the producer.  
Its not showing up as a slowdown in reading from the socket, just in  
the call to queue.offer(). That queue is really the only shared  
object that is being used by both threads. I was wondering whether  
the thread signaling (when the message is put in the queue) is an OS  
or system level call, and whether that can get held up by the socket  
writing call? That might explain why the socket sending could hold up  
the queue insertions. If so, can you think of any way around my issue?

Thanks,
Dan

On Aug 8, 2008, at 9:49 PM, David Holmes wrote:

> Dan,
>
> The simple answer is that the consumer is using a resource that the  
> producer
> needs. It could be processor cycles, or network access, or  
> contention on
> some shared object at the application, VM or OS level. Assuming you're
> multi-processor I'd suspect contention somewhere in the TCP/IP  
> stack. What
> OS are you on? How many processors do you have?
>
> At 100,000 messages per second you only have 10 microseconds to fully
> process each one. That's a tall order on a non-real-time system.  
> Even if
> your best processing time is within 10us, it isn't a sustainable  
> rate, so
> you should expect to see a lot of variation in the message rate  
> that you can
> handle. When GC kicks in you're going to drop a lot of messages.
>
> Cheers,
> David Holmes
>
>> -----Original Message-----
>> From: concurrency-interest-bounces at cs.oswego.edu
>> [mailto:concurrency-interest-bounces at cs.oswego.edu]On Behalf Of  
>> Daniel
>> Harvey
>> Sent: Saturday, 9 August 2008 1:11 AM
>> To: concurrency-interest at cs.oswego.edu
>> Subject: [concurrency-interest] Single producer,single consumer:
>> unexpected delays for producer
>>
>>
>> Hi,
>> I'm writing a program that receives a very high message rate
>> (~100,000 per second) over a TCP socket, filters out the useful
>> information (about 1/5 of the above), and forwards that information
>> over another TCP socket. I have been having trouble keeping up with
>> the incoming messages, and I think its an oddity of the
>> LinkedBlockingQueue I am using to queue the useful updates for later
>> sending. The code looks something like this:
>>
>> public class MessageForwarder {
>> 	SendThread	sendThread = new SendThread();
>> 	 ...
>>
>> 	public void onMessage(RapidMessage message) {
>> 		long start = System.nanoTime();
>>
>> 		// Do a bunch of checks to decide if useful...
>> 		boolean useful = ...
>>
>> 		if (useful) {
>> 			sendThread.send(throttled.toMessageString())
>> 		}
>> 		processingTime += (System.nanoTime() - start);
>> 	}
>>
>> 	// inner class
>> 	public class SendThread extends Thread {
>>
>> 		private final BlockingQueue<String> queued = new
>> LinkedBlockingQueue<String>(10000);
>> 		private final CharsetEncoder              encoder =
>> Charset.forName
>> ("ISO-8859-1").newEncoder();
>>
>> 		public void send(String message) {
>> 			long start = System.nanoTime();
>> 	 		boolean success = queued.offer(message);
>> 			sendingTime +=  (System.nanoTime() - start);
>>
>> 			if (!success) {
>> 				log.error("Failed to send message:
>> queue is full");
>> 			}
>> 		}
>>
>> 		public void run() {
>> 			while (true) {
>> 				String message = queued.take();
>> 				ByteBuffer buffer =
>> encoder.encode(CharBuffer.wrap(message));
>> 				socketChannel.write(buffer);
>> 			}
>> 		}
>> 	}
>> ...
>> }
>>
>> When I print out periodically the value of the "processingTime" and
>> the "sendingTime", they are similar: the majority of the processing
>> time is taking place in the call to sendThread.send(). The checks to
>> see if the message is useful are fast enough that it could run
>> 100,000 messages per second, but the calls to "send" are much slower,
>> only 20,000 messages per second at best. Note that the queue is not
>> filling up (not seeing the error message anywhere). What I really
>> don't understand is that if I have the consumer (Sending) thread
>> throw away the message after it takes it from the queue (ie just do
>> repeated "take"s), then the call to sendThread.send() speeds up by a
>> factor of around 6. I don't understand how if the queue isn't filling
>> up, why would the actions of the consumer matter to the speed of the
>> producer?
>>
>> Any help or advice you can give would be much appreciated!
>>
>> Thanks,
>> Dan Harvey
>>
>> _______________________________________________
>> Concurrency-interest mailing list
>> Concurrency-interest at cs.oswego.edu
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>



More information about the Concurrency-interest mailing list