After some investigation believe I found the issue pointed out by:
INT-2609 - JDBC Message Store locking up with parallel transactional (long-running) consumers
The issue is in all instances where Query.UPDATE_GROUP gets executed. If all messages are coming from the same group, all polled messages from that channel want to update that same group. But because e.g. a Service Activator downstream is "busy", that record is locked and thus, the TX may time out.
When I avoid updating the INT_MESSAGE_GROUP table, the timeout issues are not observed.
As for JdbcMessageStore that back a channel, the group table does not add too much value and can either be bi-passed or a more efficient way of executing updates needs to be found.