Home

Using awk/sed to detect/remove the byte order mark (BOM)

Tue, 2011.11.08 - 12:13 — müzso
http://stackoverflow.com/questions/1068650/using-awk-to-remove-the-byte-order-ma...
BOM is nothing more than a nuisance. Detecting and removing it is plain simple. There's a utility called bomstrip that pretty much does what is says and there's the one-liner AWK implementation that you find on stackoverflow (I've added whitespaces for better readability):
awk '{ if (NR == 1) sub(/^\xef\xbb\xbf/, ""); print }' INFILE > OUTFILE
Or a sed version (replaces in place):
sed -i -e '1s/^\xEF\xBB\xBF//' FILE
To do this (or just simply detect BOM) recursively in a dir, you'll find tons of ideas in this other stackoverflow article.

For recursively detecting files with a BOM, I prefer this one:
find . -type f -print0 | xargs -0r awk '/^\xEF\xBB\xBF/ {print FILENAME} {nextfile}'

To recursively remove BOM, you could use something like this:
find . -type f -exec sed -i -e '1s/^\xEF\xBB\xBF//' {} \;

Or a slightly safer variant:
find . -type f -exec sed -i.bak -e '1s/^\xEF\xBB\xBF//' {} \; -exec rm '{}.bak' \;