[INT-2821] ImapMailReceiver: imapFolder.hasNewMessages() always return false if server do not support RECENT flag Created: 15/Nov/12  Updated: 25/Jan/13  Resolved: 20/Nov/12

Status: Closed
Project: Spring Integration
Component/s: Mail Support
Affects Version/s: 2.2 RC2
Fix Version/s: 2.2 RC3, 2.1.5

Type: Defect Priority: Minor
Reporter: Alexander Kharitonov Assignee: Gary Russell
Resolution: Complete Votes: 0
Labels: PullRequest, specification-compatibility
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Reference URL: https://forums.oracle.com/forums/thread.jspa?threadID=1588617

 Description   

ImapMailReceiver may wait long for some new messages if IMAP server do not support RECENT flag (gmail, CommuniGate ...):

public void waitForNewMessages() throws MessagingException {
...
if (imapFolder.hasNewMessages())

{ return; }

...
try

{ imapFolder.idle(); }

...
}

Because imapFolder.hasNewMessages() can return true only if the server supports RECENT flag and the Folder always remain in the open state (see JavaMail 1.4.5 source).

Perhaps you can replace method hasNewMessages() by searchForNewMessages() here if server do not support RECENT.



 Comments   
Comment by Alexander Kharitonov [ 15/Nov/12 ]

When you first connect to a IMAPFolder containing the new messages, hasNewMessages() method always returns false if the RECENT flag is not supported.
And we have to wait for a IMAP idle timeout 30 minutes to receive this messages.

Comment by Alexander Kharitonov [ 15/Nov/12 ]

After IMAP IDLE timeout (30 minutes) FolderClosedException exception will be thrown but the new message (this new message in the inbox already at startup) has not been received.

Stacktrace:

2012-11-15 18:01:23,155 [task-scheduler-1] DEBUG o.s.i.mail.ImapIdleChannelAdapter - waiting for mail
2012-11-15 18:01:23,169 [catalina-exec-11] INFO o.s.i.mail.ImapIdleChannelAdapter - started customAdapter
2012-11-15 18:01:23,218 [task-scheduler-1] DEBUG o.s.integration.mail.ImapMailReceiver - connecting to store [imap://***:*****@mx.***.ru:143/INBOX]
2012-11-15 18:01:23,487 [task-scheduler-1] DEBUG o.s.integration.mail.ImapMailReceiver - opening folder [imap://***:*****@mx.***.ru:143/INBOX]
2012-11-15 18:31:40,075 [task-scheduler-1] WARN o.s.i.mail.ImapIdleChannelAdapter - error occurred in idle task
javax.mail.FolderClosedException: * BYE JavaMail Exception: java.io.IOException: Connection dropped by server?
at com.sun.mail.imap.IMAPFolder.throwClosedException(IMAPFolder.java:2552) ~[mail-1.4.5.jar:1.4.5]
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:2350) ~[mail-1.4.5.jar:1.4.5]
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:2266) ~[mail-1.4.5.jar:1.4.5]
at org.springframework.integration.mail.ImapMailReceiver.waitForNewMessages(ImapMailReceiver.java:115) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:177) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:155) [spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) [spring-context-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [na:1.6.0_35]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_35]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_35]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) [na:1.6.0_35]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) [na:1.6.0_35]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_35]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_35]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_35]
2012-11-15 18:31:40,076 [task-scheduler-1] WARN o.s.i.mail.ImapIdleChannelAdapter - Failed to execute IDLE task. Will attempt to resubmit in 10000 milliseconds.
java.lang.IllegalStateException: Failure in 'idle' task. Will resubmit.
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:196) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:155) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) [spring-context-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-3.1.3.RELEASE.jar:3.1.3.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [na:1.6.0_35]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_35]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_35]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) [na:1.6.0_35]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) [na:1.6.0_35]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_35]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_35]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_35]
Caused by: javax.mail.FolderClosedException: * BYE JavaMail Exception: java.io.IOException: Connection dropped by server?
at com.sun.mail.imap.IMAPFolder.throwClosedException(IMAPFolder.java:2552) ~[mail-1.4.5.jar:1.4.5]
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:2350) ~[mail-1.4.5.jar:1.4.5]
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:2266) ~[mail-1.4.5.jar:1.4.5]
at org.springframework.integration.mail.ImapMailReceiver.waitForNewMessages(ImapMailReceiver.java:115) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:177) ~[spring-integration-mail-2.2.0.BUILD-20121114.064637-202.jar:na]
... 11 common frames omitted
2012-11-15 18:31:50,078 [task-scheduler-1] DEBUG o.s.i.mail.ImapIdleChannelAdapter - waiting for mail
2012-11-15 18:31:50,087 [task-scheduler-1] DEBUG o.s.integration.mail.ImapMailReceiver - opening folder [imap://***:*****@mx.***.ru:143/INBOX]

Comment by Oleg Zhurakousky [ 15/Nov/12 ]

Alex

Can you also share what brand of server you are using so we can test with it.

Comment by Alexander Kharitonov [ 15/Nov/12 ]

CommuniGate 5.4c1

Comment by Alexander Kharitonov [ 15/Nov/12 ]

IMAP Server monitor always reports about two imap connections if ImapIdleChannelAdapter running. I think that the PingTask runs on second connection. Maybe this will help.

ID IP Address Account Connected Status Running
1751 [*.*.*.1] support 49 min idling 19 min
1752 [*.*.*.1] support 19 min authenticated

Comment by Gary Russell [ 19/Nov/12 ]

I think the simplest solution is to just short-circuit the idle on just the first attempt if the server doesn't support RECENT.

Comment by Gary Russell [ 19/Nov/12 ]

PR Issued: https://github.com/SpringSource/spring-integration/pull/678

Comment by Alexander Kharitonov [ 19/Nov/12 ]

I looked.

But if new message arrive between
AbstractMailReceiver.receive() and next ImapMailReceiver.waitForNewMessages()
then we never get the message. imapFolder.idle() method will never detect this new message.

All subsequent requests will not give us a new message because folder not always stays in open state (this required for idle command by spec).

Comment by Gary Russell [ 20/Nov/12 ]

I see your point, but I don't see how this is any different to your suggestion in the original description because receive() calls searchForNewMessages().

Unless I am missing something, with your suggestion, there is still a race condition between calling search and going into idle mode.

I think this needs some deeper thought and testing, so we may need to push out this issue.

Comment by Alexander Kharitonov [ 20/Nov/12 ]

// <-- this is the second and subsequent launch of IdleTask:
...
// <-- At the moment, the folder is CLOSED and empty.
// <-- A new message has arrived in the folder at server now.
mailReceiver.waitForNewMessages(); // <-- inside: openFolder() ... hasNewMessages()==false, isNewFolder()==false and idle() does not detect the new message (because the folder was opened after the arrival of new message). Throws FolderClosedException because IMAP IDLE timeout 30 sec was reached.

// <-- We have not reached that point, IdleTask has been restarted
// <-- We never get the new message
if (mailReceiver.getFolder().isOpen()) {
Message[] mailMessages = mailReceiver.receive(); //There is folder.close() inside
...
}

Comment by Alexander Kharitonov [ 20/Nov/12 ]

Of course, you should read "timeout 30 minutes" instead of "30 sec" in the previous comments.

Comment by Gary Russell [ 20/Nov/12 ]

Thanks, I see; I was assuming we'd only get notice of new messages once the idle() is called but I just ran a test and confirmed that if a message arrives between the open() and idle() calls, the idle() immediately returns.

I'll go ahead and change it to do the search instead.

Comment by Gary Russell [ 20/Nov/12 ]

Updated pull request: https://github.com/SpringSource/spring-integration/pull/678

Generated at Tue Nov 12 00:23:58 UTC 2019 using Jira 7.13.8#713008-sha1:1606a5c1e7006e1ab135aac81f7a9566b2dbc3a6.