Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Complete
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.2 M3 Sprint 2, 2.2 M3
    • Component/s: Mail Support
    • Labels:
      None

      Description

      We would like to use Spring Integration 2.1.1 to read mail off a POP3 mailbox transactionally. This transactionality is so that we can read an email message and persist it to Oracle and remove from the mailbox. In the event of a transaction rollback we want the email to remain in the mailbox (i.e. not be deleted).

      Our problem is that, despite being in a transaction (more detail on that in a second) the mailbox is opened, the mail is retrieved, marked for deletion and the connection closed irrespective of the rest of the transaction. This means it is too late to not delete the mail if something fails later on (as the connection is closed and the mailbox gone into UPDATE state which permanently removes the email).

      For more details on this see the forum post at: http://forum.springsource.org/showthread.php?127246-Transactional-POP3-Mailbox-access-with-Spring-Integration

        Activity

        Hide
        Andrew Harmel-Law added a comment -

        We've done a little more investigation on this. The problem lies specifically in the use of the org.springframework.integration.mail.AbstractMailReceiver.receive( ) method. This is called by our inbound-channel-adaptor and unfortunately not only does this open the mailbox, and get the mail, but also (in the finally block) calls MailTransportUtils.closeFolder(this.folder, this.shouldDeleteMessages); This means that effectively the POP3 part of the transaction is "committed" before the rest of the channel transaction is started.

        What we think needs to be provided is another Receiver with the current receive() split into separate methods; one for getting the mail and marking got mail as "to be deleted", and a second for closing the mailbox (and making the deletions permanent, or undoing them). The former could then be called at the start of a transaction, and the latter could be called at the end of a transaction, specifying true for "shouldDeleteMessages" is we are committing everything, and false if we are rolling back.

        Consequently, it seems that another inbound channel adaptor will also be required to exert this control based on the outcome of the transaction which it will start.

        Show
        Andrew Harmel-Law added a comment - We've done a little more investigation on this. The problem lies specifically in the use of the org.springframework.integration.mail.AbstractMailReceiver.receive( ) method. This is called by our inbound-channel-adaptor and unfortunately not only does this open the mailbox, and get the mail, but also (in the finally block) calls MailTransportUtils.closeFolder(this.folder, this.shouldDeleteMessages); This means that effectively the POP3 part of the transaction is "committed" before the rest of the channel transaction is started. What we think needs to be provided is another Receiver with the current receive() split into separate methods; one for getting the mail and marking got mail as "to be deleted", and a second for closing the mailbox (and making the deletions permanent, or undoing them). The former could then be called at the start of a transaction, and the latter could be called at the end of a transaction, specifying true for "shouldDeleteMessages" is we are committing everything, and false if we are rolling back. Consequently, it seems that another inbound channel adaptor will also be required to exert this control based on the outcome of the transaction which it will start.
        Hide
        Gary Russell added a comment -

        The "Spring" way of doing this would probably be to bind the session/folder to the thread in a ResourceHolder and have a transaction manager take the appropriate action when the thread returns.

        Bear in mind, though, that this can never be a "real" transaction because POP3 is not transactional; it would be an approximation only. That said, worse case is you'll get duplicates (e.g. if the power goes off after the DB commit and before the mail is deleted). Transaction synchronization between the DB tx and the POP3 pseudo tx would minimize this window, but it still exists.

        If your app can deal with, or reject, duplicates; all the better.

        I am making this a subtask of INT-1849.

        Show
        Gary Russell added a comment - The "Spring" way of doing this would probably be to bind the session/folder to the thread in a ResourceHolder and have a transaction manager take the appropriate action when the thread returns. Bear in mind, though, that this can never be a "real" transaction because POP3 is not transactional; it would be an approximation only. That said, worse case is you'll get duplicates (e.g. if the power goes off after the DB commit and before the mail is deleted). Transaction synchronization between the DB tx and the POP3 pseudo tx would minimize this window, but it still exists. If your app can deal with, or reject, duplicates; all the better. I am making this a subtask of INT-1849 .
        Hide
        Mark Fisher added a comment -

        Thank you Gary. I was going to make the link to INT-1849 here myself. This is a good use-case, parallel to those involving file reading.

        Show
        Mark Fisher added a comment - Thank you Gary. I was going to make the link to INT-1849 here myself. This is a good use-case, parallel to those involving file reading.
        Hide
        Andrew Harmel-Law added a comment -

        Hi Gary,

        We definitely would prefer to go with the grain of spring-integration rather than against it. Consequently, what you suggest sounds like a good solution and I presume that this then means when you add a <transaction> attribute to a <poller> in a mail inbound-channel-adapter it'll work more like you would expect?

        We're more than happy if this results in duplicates. As you say the "transactional" that POP3 thinks it is is not what everyone else means. We're already designing things along the "Best Efforts 1PC" pattern with the POP3 "commit" as the last step so I think this would chime exactly with how this will be implemented. We have, as a result, already built things so we can deal with duplicates.

        All in all, it sounds like something we'd like to get our hands on as soon as possible. Is there an ETA for the next release (which I guess this might be included in?)

        TIA

        Cheers, Andrew

        Show
        Andrew Harmel-Law added a comment - Hi Gary, We definitely would prefer to go with the grain of spring-integration rather than against it. Consequently, what you suggest sounds like a good solution and I presume that this then means when you add a <transaction> attribute to a <poller> in a mail inbound-channel-adapter it'll work more like you would expect? We're more than happy if this results in duplicates. As you say the "transactional" that POP3 thinks it is is not what everyone else means. We're already designing things along the "Best Efforts 1PC" pattern with the POP3 "commit" as the last step so I think this would chime exactly with how this will be implemented. We have, as a result, already built things so we can deal with duplicates. All in all, it sounds like something we'd like to get our hands on as soon as possible. Is there an ETA for the next release (which I guess this might be included in?) TIA Cheers, Andrew
        Hide
        Gary Russell added a comment -

        2.2. is scheduled for late summer.

        We'll see what we can do to get it into the next milestone, due at the end of this month.

        Show
        Gary Russell added a comment - 2.2. is scheduled for late summer. We'll see what we can do to get it into the next milestone, due at the end of this month.
        Hide
        Andrew Harmel-Law added a comment -

        Cheers Gary,

        We'll watch out for the milestone. We'd be happy to try it out in the wild once it's available. Thanks for the update.

        Regs, Andrew

        Show
        Andrew Harmel-Law added a comment - Cheers Gary, We'll watch out for the milestone. We'd be happy to try it out in the wild once it's available. Thanks for the update. Regs, Andrew
        Hide
        Gary Russell added a comment -

        PR: https://github.com/SpringSource/spring-integration/pull/493

        Note: this still has to go through review, but you can give it a spin if you want; my branch is here https://github.com/garyrussell/spring-integration/tree/INT-1849

        Show
        Gary Russell added a comment - PR: https://github.com/SpringSource/spring-integration/pull/493 Note: this still has to go through review, but you can give it a spin if you want; my branch is here https://github.com/garyrussell/spring-integration/tree/INT-1849
        Hide
        Andrew Harmel-Law added a comment -

        Superb. Cheers Gary. We start a new iteration today so should be able to try it out today / tomorrow.

        Show
        Andrew Harmel-Law added a comment - Superb. Cheers Gary. We start a new iteration today so should be able to try it out today / tomorrow.

          People

          • Assignee:
            Gary Russell
            Reporter:
            Andrew Harmel-Law
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1.5d
              1.5d
              Remaining:
              Remaining Estimate - 0d
              0d
              Logged:
              Time Spent - 1.5d
              1.5d