Golden Nugget Watch

So I was in the office the other day minding my own business when all of a sudden sprung upon me were over 16,000 log files, totalling in excess of 30GB. I could be a bit more dramatic and say they fell out of the sky, or came flying down the chimney like letters to an orphan boy who lives under the stairs… but alas, this is not Pottermore.

So, amongst the seasoned of you, there are shouts and cries and groans of “gzip *.log”. And yes, indeed that’s where we went. But this little nugget is not about the use of gzip.

The problem: How do I know how much longer it has to go?

I needed to not only see how far IN we were to the operation, but also how far to go. I COULD ‘ls -lah *.gz’ in order to show me the current files and this will give me a long list of files which will absorb the console cache in seconds. But in order to give something more intuitive I decided to be a bit created on the bash command line, utilising “watch”.

watch runs command repeatedly, displaying its output (the first screenfull). This allows you to watch the program output change over time. By default, the program is run every 2 seconds; use -n or –interval to specify a different interval. – https://linux.die.net/man/1/watch

So which command or list of commands do I parse into watch to see what I want.

1. Display a tag of what we’re counting (GZ)

   echo GZ

2. Count GZ

   ls -l *.gz | wc -l

3. Display a tag of what we’re NOT counting (non-GZ)

   echo Uncompressed

4. Count non-GZ

   ls -I “*.gz” -l | wc -l

Most of the above is self explanatory other than item 4, where we use -I to exclude anything that has been given the GZ extension. So to string it all together, like the plumbing in a dodgy hotel, we use pipes and then wrap it in quotes and use it as a parameter for ‘watch’.

watch "echo GZ && ls -l .gz | wc -l && echo Uncompressed && ls -I ".gz" -l | wc -l"

The end result, as below, is a nice, easy to read output that shows you how many GZ’s we have vs how many left.

As a boot note, it’s worth understanding that the folder used here was full of log files ONLY and had no other content. If you had mixed file types you’d obviously have to adjust all of the above to gzip *.log and then ls -l *.log | wc -l for Step 4, instead of excluding everything else.