Replacing new lines with nulls in bash

If you have a list of file names separated by new lines, and you want to turn it into a list of file names separated by null characters, for instance for use as input to xargs, then you can use the tr command, like this:

  $ cat /path/to/new-lines | tr '\n' '\0' > /path/to/null-separated

I found a whole article on various ways to replace text on *nix: rmnl — remove new line characters with tr, awk, perl, sed or c/c++.

Error handling in bash

I’ve been learning about approaches to error handling in bash. I found Error handling in BASH, Exit Shell Script Based on Process Exit Code, Writing Robust Bash Shell Scripts and Bash: Error handling all of which contained a nugget of useful information.

Particularly I learned about $LINENO, set -e (equiv: set -o errexit), set -o pipefail, set -E, set -u (equiv: set -o nounset), set -C (equiv: set -o noclobber), #!/bin/bash -eEu, trap, shopt -s expand_aliases, alias, $PIPESTATUS, subshells, unset, and more.

If you call exit without specifying a return code then $? is used, I didn’t know that.

I found this code snippet that demos a relatively race-condition free method of acquiring a lock file:

if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; 
   trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT

   rm -f "$lockfile"
   trap - INT TERM EXIT
   echo "Failed to acquire lockfile: $lockfile." 
   echo "Held by $(cat $lockfile)"

Auto-extracting archives

I have a directory structure with archived files, many of which are zipped or tarballed up. So if I want to search for a file, I can’t really be sure that the file I’m looking for isn’t in a compressed file. So I wrote some scripts to automatically run over a directory tree and automatically extract any compressed files that it finds there. There are a few extra features, such as if the file is larger than 10MB then it will prompt for whether to extract it or not. I have a few other features for handling errors to add in, but I’m happy to post this version up now.

search() {
	find -iname "*$1" -print0 | xargs -i -0 `dirname "$0"`/ "$1" "$2" "$err" "{}"
search ".tar.gz" "tar xf" 
search ".tgz" "tar xf" 
search ".zip" "unzip -q"

exec 0< /dev/tty
file_name=`basename "$path"`
dir_name=`dirname "$path"`
new_name=`basename "$path" "$1"`
[ -e "$new_path" ] && exit 0
echo $dir_name/$file_name
file_size=`stat -c %s "$path"`
#echo -n "File is $file_size bytes."
printf "File is %'d bytes." $file_size
check=`echo "$file_size < 10485760" | bc`
if [ "$check" = "1" ]; then answer="y"; fi
while [ "$answer" != "y" ]; do
	read -n 1 -s -p " Extract? " answer
	[ "$answer" = "n" ] && echo && echo && exit 0
mkdir "$new_path"
pushd "$new_path" > /dev/null 2>&1
$2 "../$file_name"
if [ "$?" != "0" ]; then
	popd > /dev/null 2>&1
	rm -rf "$new_path"
	while [ "$answer" != "y" ]; do
  		read -n 1 -s -p "Do you want to ignore this file? " answer
		[ "$answer" = "n" ] && exit $3
popd > /dev/null 2>&1

Shell scripting for archive restoration

I’ve been restoring my archives. Basically I have a bit over 1.3TB of data that I’ve tarballed up and stashed on some disconnected SATA disks, and now that I have a computer with the capacity to hold all that data I’m resurrecting the file shares from the archived tarballs. You can see my restore script here:

cd "`dirname $0`"
tar xf $data_path/1999.tar.gz --hard-dereference >  output.txt 2>&1
tar xf $data_path/2001.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2002.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2003.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2004.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2005.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2006.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2007.tar.gz --hard-dereference >> output.txt 2>&1
tar xf $data_path/2008.tar.gz --hard-dereference >> output.txt 2>&1

The script creates an output.txt file that lists any errors from tar during the restore process. I then have a set of scripts that process this output.txt file fixing up two types of common errors.

Fixing dates

The first error is that the date of the file in the archive isn’t a reasonable value. For example, I had files reporting modification time somewhere back in 1911, before computers. To fix the dates with this problem I run the following scripts:


cd "`dirname $0`";
./bad-date | xargs -0 touch --no-create


awk -f bad-date.awk < output.txt | while read line
  # note: both -n and \c achieve the same end.
  echo -n -e "$line\0\c"


  if ( /tar: ([^:]*): implausibly old time stamp/ ) {
    split( $0, array, ":" )
    filepath = array[ 2 ]
    sub( / /, "", filepath )
    printf( "%s\n", filepath )

Fixing hard links

The second class of error that I can receive is that the file that is being extracted from the archive is a hard link to an already existing file, but the hard link cannot be created because the number of links to the target has reached its limit. I think I used ReiserFS as my file system the archives were on originally, and I’m using Ext4 now. Ext4 seems to have limitations that ReiserFS didn’t. Anyway, it’s not big deal, because I can just copy the target to the path that failed to link. This creates a duplicate file, but that’s not a great concern. I’ll try to fix up such duplicates with my pcdedupe project.


cd "`dirname $0`";
./bad-link | xargs -0 ./fix-link


awk -f bad-link.awk < output.txt | while read line
  # note: both -n and \c achieve the same end.
  echo -n -e "$line\0\c"


  if ( /tar: ([^:]*): Cannot hard link to `([^']*)': Too many links/ ) {
    split( $0, array, ":" )
    linkpath = array[ 2 ]
    sub( / /, "", linkpath )
    filepath = array[ 3 ]
    sub( / Cannot hard link to `/, "", filepath )
    filepath = substr( filepath, 0, length( filepath ) )
    printf( "%s:%s\n", filepath, linkpath )


cd "`dirname $0`";
file=`echo "$spec" | sed 's/\([^:]*\):.*/\1/'`
link=`echo "$spec" | sed 's/[^:]*:\(.*\)/\1/'`
#echo "$spec"
#echo Linking "'""$link""'" to "'""$file""'"...
#echo ""
if [ ! -f "$file" ]; then
  echo Missing "'""$file""'"...
  exit 1;
cp "$file" "$link"


I then checked for anything that I’d missed with my scripts with the following:

cd "`dirname $0`";
cat output.txt | grep -v "Cannot hard link" | grep -v "implausibly old time"

Bash aliases

I was reading my default .bashrc file, and found the following:

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases

This suggested that I wanted a .bash_aliases file for my aliases, so I set one up:

jj5@sixsigma:~$ cat .bash_aliases
alias home='cd ~'
alias jj5='cd /var/www/'
alias profile='cd /var/www/'
alias chomsky='vim /var/www/'
alias henney='vim /var/www/'
alias lakoff='vim /var/www/'
alias norvig='vim /var/www/'

This is my basic “CMS” system for