Spring Batch
  1. Spring Batch
  2. BATCH-1704

Allow log file location to depend on Job context (e.g. job name)

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      Clearly this depends on the output log framework, so we will probably not be trying to cover them all. Log4J seems like the obvious choice to support since there is some Spring support for that already.

        Activity

        Hide
        Giovanni Dall'Oglio Risso added a comment -

        We developed a project, in which the job instances "used" a NAS directory ( //nas/jobName/instanceId ).
        The log analysis was a pain (the logs of the different chains was mixed in the same file).

        The same day in which we can easily configure the log in a file like //nas/jobName/instanceId/execution.log we'll hit the bottle (drinking to your health)

        Show
        Giovanni Dall'Oglio Risso added a comment - We developed a project, in which the job instances "used" a NAS directory ( //nas/jobName/instanceId ). The log analysis was a pain (the logs of the different chains was mixed in the same file). The same day in which we can easily configure the log in a file like //nas/jobName/instanceId/execution.log we'll hit the bottle (drinking to your health)
        Hide
        Giovanni Dall'Oglio Risso added a comment -

        Update to the prev post:

        we run in a WAS 7.0, and we have tons of log.

        It is very diffucult to filter the concerning log, so we developed a workaround:

        we put a listener that renames the thread's name with "jobName - instanceId" (all our chains are single-threaded), so we can easily trim the log file, using that approach.

        Do you like it? Do you have better ideas?

        public class ThreadNameListener implements JobExecutionListener
        {

        private Map<Long, String> thNames = new HashMap<Long, String>();

        public void afterJob(JobExecution jobExecution)

        { Long instId = jobExecution.getJobInstance().getId(); Thread.currentThread().setName(thNames.get(instId)); }

        public void beforeJob(JobExecution jobExecution)

        { JobInstance jobInstance = jobExecution.getJobInstance(); Long instId = jobInstance.getId(); Thread currentThread = Thread.currentThread(); thNames.put(instId, currentThread.getName()); currentThread.setName(jobInstance.getJobName() + " - " + instId); }

        }

        Show
        Giovanni Dall'Oglio Risso added a comment - Update to the prev post: we run in a WAS 7.0, and we have tons of log. It is very diffucult to filter the concerning log, so we developed a workaround: we put a listener that renames the thread's name with "jobName - instanceId" (all our chains are single-threaded), so we can easily trim the log file, using that approach. Do you like it? Do you have better ideas? public class ThreadNameListener implements JobExecutionListener { private Map<Long, String> thNames = new HashMap<Long, String>(); public void afterJob(JobExecution jobExecution) { Long instId = jobExecution.getJobInstance().getId(); Thread.currentThread().setName(thNames.get(instId)); } public void beforeJob(JobExecution jobExecution) { JobInstance jobInstance = jobExecution.getJobInstance(); Long instId = jobInstance.getId(); Thread currentThread = Thread.currentThread(); thNames.put(instId, currentThread.getName()); currentThread.setName(jobInstance.getJobName() + " - " + instId); } }
        Hide
        Dave Syer added a comment -

        Unfortunately, thread-bound configuration is probably the best option right now. I did some work that is relevant to this issue (http://github.com/dsyer/log4j-utils), but didn't get as far as integrating it with Spring Batch because of the thread-bound limitation.

        Show
        Dave Syer added a comment - Unfortunately, thread-bound configuration is probably the best option right now. I did some work that is relevant to this issue ( http://github.com/dsyer/log4j-utils ), but didn't get as far as integrating it with Spring Batch because of the thread-bound limitation.
        Hide
        Adrian Shum added a comment -

        I believe MDC for logging frameworks (SLF4J, Log4J etc) aims for this purpose. Things done by Giovanni should better be done by putting Job Name/ Job Instance ID etc in MDC.

        Combination of SLF4J with LogBack allow you to separate log file base on MDC too.

        The only one issue (that is the same as in Giovanni's way) is , whenever new thread is spawned in the job, we need to find some way to set the MDC again. For example, when we are using a Thread Pool Executor in chunk-oriented processing.

        Show
        Adrian Shum added a comment - I believe MDC for logging frameworks (SLF4J, Log4J etc) aims for this purpose. Things done by Giovanni should better be done by putting Job Name/ Job Instance ID etc in MDC. Combination of SLF4J with LogBack allow you to separate log file base on MDC too. The only one issue (that is the same as in Giovanni's way) is , whenever new thread is spawned in the job, we need to find some way to set the MDC again. For example, when we are using a Thread Pool Executor in chunk-oriented processing.
        Hide
        Dave Syer added a comment -

        If you look at my utils project link on github I think you will find it is using MDC, but the problem still remains that thread local context is not preserved automatically. We talked at SpringOne about maybe getting some thread context helpers into Spring Core because it is quite a common concern, but I don't really know what it would look like (and if I did I would have already implemented it).

        Show
        Dave Syer added a comment - If you look at my utils project link on github I think you will find it is using MDC, but the problem still remains that thread local context is not preserved automatically. We talked at SpringOne about maybe getting some thread context helpers into Spring Core because it is quite a common concern, but I don't really know what it would look like (and if I did I would have already implemented it).
        Hide
        Michael Minella added a comment - - edited

        I spent a bit of time on this on this past Friday/over the weekend. Log4j 2 is currently in beta and has some new thread specific features:

        1. They added a ThreadContext that allows a developer to set values at a thread level and key off of them for other processing.
        2. The values set in a ThreadContext are inherited from parent threads. This would theoretically allow you to set an id of a jobname_instanceid_executionid for example when a job is launched and have those values be inherited by any threads spawned by the job. While it doesn't address the issue of remote execution...I think it would be a big step in the right direction.

        I have a unit test that illustrates the functionality of storing thread specific values and including them in the log output currently (it launches a couple instances of a class via TaskExecutor that does some logging. The message in the log file is prepended with the id set in the ThreadContext). I'm going back and forth with the log4j devs on how to configure an apender that will switch based on the id set. More to come...

        Show
        Michael Minella added a comment - - edited I spent a bit of time on this on this past Friday/over the weekend. Log4j 2 is currently in beta and has some new thread specific features: They added a ThreadContext that allows a developer to set values at a thread level and key off of them for other processing. The values set in a ThreadContext are inherited from parent threads. This would theoretically allow you to set an id of a jobname_instanceid_executionid for example when a job is launched and have those values be inherited by any threads spawned by the job. While it doesn't address the issue of remote execution...I think it would be a big step in the right direction. I have a unit test that illustrates the functionality of storing thread specific values and including them in the log output currently (it launches a couple instances of a class via TaskExecutor that does some logging. The message in the log file is prepended with the id set in the ThreadContext). I'm going back and forth with the log4j devs on how to configure an apender that will switch based on the id set. More to come...
        Hide
        Dave Syer added a comment -

        Michael, is log4j compatible with commons-logging? I know it's in the pipeline but I don't really see how it can help - Spring Batch can't even control all the thread contexts in a job in the most general case, so a logging library is even less able to do so. Using an inheritable thread local helps but only if new threads are spawned directly (no thread pooling). I believe the github link I gave before is the best anyone can do right now (happy to be proved wrong but I don't think it's worth spending a lot of time on). We could provide an appender that works with "official" Spring Batch threads and any created directly by them, but I'm not sure it gets any better than that.

        Show
        Dave Syer added a comment - Michael, is log4j compatible with commons-logging? I know it's in the pipeline but I don't really see how it can help - Spring Batch can't even control all the thread contexts in a job in the most general case, so a logging library is even less able to do so. Using an inheritable thread local helps but only if new threads are spawned directly (no thread pooling). I believe the github link I gave before is the best anyone can do right now (happy to be proved wrong but I don't think it's worth spending a lot of time on). We could provide an appender that works with "official" Spring Batch threads and any created directly by them, but I'm not sure it gets any better than that.

          People

          • Assignee:
            Unassigned
            Reporter:
            Dave Syer
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 2d
              2d
              Remaining:
              Remaining Estimate - 2d
              2d
              Logged:
              Time Spent - Not Specified
              Not Specified