Although the first listener in my web.xml is the Log4jConfigListener. the log4j configuration happens before getting there.
It happens because I also have HttpSessionEventPublisher listener that has a static Log, when the static Log is initialized it causes the log4j default configuration.
The problem is that some loggers from the default configuration are still alive even after the Log4jConfigListener is called.
Changing the Log to not being static should solve the issue.