Attached SPR-7856.patch. It fixes this issue by removing appropriate bean definition when singleton beans get destroyed. I guess it was design decision not to do this - maybe there is a reason for this; anyway, all existing tests pass with the fix applied. I wish spring reference docs contained more details on container destruction phase.
Destroying listener bean, with this patch applied, still doesn't remove listener from another registry - applicationListenerBeans of AbstractApplicationEventMulticaster's ListenerRetriever, because multicaster is not available from AbstractBeanFactory, multicaster is in AbstractApplicationContext. As workaround for this, patch adjusts AbstractApplicationEventMulticaster to skip retrieving (creating) listener bean from BeanFactory if BeanFactory doesn't contain listener bean.
Maybe a different fix could be to make a DestructionAwareBeanPostProcessor which would implement ApplicationContextAware and for any context which implements AbstractApplicationContext and for any bean which implements ApplicationListener call removeApplicationListener and removeApplicationListenerBean on AbstractApplicationContext's ApplicationEventMulticaster. Container would have to enable that DestructionAwareBeanPostProcessor always or by default.