CachingClientConnectionFactory gives an UnsupportedOperation exception on incoming data

I often (but not always) get the following exception right after I call getConnection()

java.lang.UnsupportedOperationException: This should never be called
at org.springframework.integration.ip.tcp.connection.CachingClientConnectionFactory$2.onMessage(CachingClientConnectionFactory.java:322)
at org.springframework.integration.ip.tcp.connection.TcpNioConnection.sendToChannel(TcpNioConnection.java:357)
at org.springframework.integration.ip.tcp.connection.TcpNioConnection.run(TcpNioConnection.java:230)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

The server I am connecting to sends data as soon as a client connects to it (the client doesn't send the first message the server does).

What I suspect is happening is that as soon as createForPool() calls targetConnectionFactory.getConnection() the server sends a message. However targetConnectionFactory isn't configured with my listener it is configured with the 'This should never be called listener'. I suspect AbstractClientConnectionFactory.initializeConnection() is too late in the process to be configuring the real listeners.