There are a number of ways to rotate your log files, and I spent a bit of time looking into all the major ones. Unfortunately, none of them satisfy all my requirements. Here are my requirements:
- put timestamp in archive file names
- rotate weekly (I hate the clutter of daily logfiles)
- work with awstats and other log processing programs
I find it fascinating that even though tens or hundreds of thousands of people use the Apache web server, that a critical mass of developers have not desired what I want. I think it’s because many sysadmins who install Apache find it’s fairly easy to write a small custom script to complete the functionality they’re missing out of the box.
Let me summarize the major tools available to you right now, and then I’ll list their deficiencies.
- logrotate: This is a standard Unix tool to rotate any application logfiles, not just Apache logfiles. It’s nice that the default Debian install sets up a logrotate job for Apache.
- Downside: The default configuration in /etc/logrotate.d/apache2 has the entries “weekly” and “rotate 52″, which means that after 52 weeks, it will start discarding your log files! That really disturbed me, I don’t want to start losing my log files a year from now just because I forgot to move my logfiles elsewhere.
- Downside: The naming scheme sucks: access.log, access.log.1, access.log.2.gz, access.log.3.gz, etc. And all the files get renamed during each rotation, so this isn’t compatible with rsync’ing a local copy of the files.
- rotatelogs: This little program comes with Apache. You use it by configuring Apache to pipe its logfiles through this program. In your configuration you can specify the template file name to use, using the well-known strftime abbreviations (e.g., “/var/log/apache2/access.%Y-%m.log” would make a log file named access.2006-05.log now).
- Downside: Not readily compatible with log processing programs like awstats, which want to read a file named access.log. You have to do extra work to make programs like awstats work.
- Downside: Since it’s configured as a pipe, Apache actually runs it as a separate process, and this increases the amount of resources used. If you have lots of logfiles open simultaneously (say, because you have virtual domains each with their own logfiles), then you better watch out that you don’t hit a soft or hard limit on number of processes or file handles.
- Downside: Does not compress after rotating.
- cronolog: This is a fork of rotatelogs above, with a couple nice enhancements: 1) you can create a symbolic link access.log file which always points to the latest timestamped logfile, and 2) you can create directory structures — “/var/log/apache2/%Y/access-%M.log” would create one subdirectory for each year.
- Downside: You can’t specify the time period for rotations, it infers it by your template file name. In the above example, it would be monthly. I wanted weekly rotations.
- Downside: Same resource issue as rotatelogs above since it is a piped logging program.
- Downside: Does not compress after rotating.
So what did I do? In the end, I stuck with logrotate since it was already working, and then:
- increased number of weeks before it would delete old files to 20 years (20*52=1040). The darn program doesn’t have a setting saying “keep indefinitely”
- wrote a little script to rename the archived logfiles to have timestamps in the file names. I can just run it in a cron job. The script just does:
cd /var/log/apache2
for i in $( find . -name "access.log.*.gz" ); do mv -i $i access.log-`date -r $i +%Y-%m-%d__%H_%M_%S`.gz; done
for i in $( find . -name "error.log.*.gz" ); do mv -i $i error.log-`date -r $i +%Y-%m-%d__%H_%M_%S`.gz; done