Home

Using sed with curl to grab the filename from the Content-Disposition header

Sun, 2013.02.17 - 10:57 — müzso
http://stackoverflow.com/questions/6881034/curl-to-grab-remote-filename-after-fo...
The answer to the Stackoverflow question sounds like this:
Do two requests: a HEAD to get the file name from response header, then a GET:
url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename="$(curl -sI  "$url" | grep -o -E 'filename=.*$' | sed -e 's/filename=//')"
curl -o "$filename" -L "$url"

It assumes a pretty simple format for the Content-Disposition header, which is most of the time not true. Meanwhile curl has a new option called --remote-header-name (or just -J), which does exactly what we intend ... ie. takes the filename for the --remote-name option from the Content-Disposition header. Actually even this option of curl was flawed at some point. So to be sure you might take the matter into your own hands and use a proper sed command to fetch the filename from the Content-Disposition line:
url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename="$(curl -sIL "$url" | sed -r -e 's/^ *Content-Disposition[ \t]*:[ \t]*[^ \t;]+;[ \t]*filename[ \t]*=[ \t]*("(([^"]|\")*)".*|([^; \t\r"]+)(([^;\r]*[^; \t\r]+)*)[ \t]*(;.*|[\r]?)$)/\2\4\5/' -e 't' -e 'd')"
[ -n "$filename" ] && curl -o "$filename" -L "$url" || curl -OL "$url"

You might have noticed that my regular expression allows for some irregularities in the Content-Disposition header (eg. HTTP headers should not start with a whitespace ... unless it's a folded header which started in a previous line). That's totally intentional.

Using the above technique you can easily download files from Sourceforge too.
Eg. to download the latest sources of LAME (mp3 codec), use this:
url="http://sourceforge.net/projects/lame/files/latest/download"
filename="$(curl -sIL "$url" | sed -r -e 's/^ *Content-Disposition[ \t]*:[ \t]*[^ \t;]+;[ \t]*filename[ \t]*=[ \t]*("(([^"]|\")*)".*|([^; \t\r"]+)(([^;\r]*[^; \t\r]+)*)[ \t]*(;.*|[\r]?)$)/\2\4\5/' -e 't' -e 'd')"
[ -n "$filename" ] && curl -o "$filename" -L "$url" || curl -OL "$url"

P.S.: the same sed expression looks like this for the version of sed in Mac OS X ...
filename="$(curl -sIL "$url" | sed -E -e 's/^[[:space:]]*Content-Disposition[[:space:]]*:[[:space:]]*[^[:space:];]+;[[:space:]]*filename[[:space:]]*=[[:space:]]*("(([^"]|\\")*)".*|([^;[:space:][:cntrl:]"]+)(([^;[:cntrl:]]*[^;[:space:][:cntrl:]]+)*)[[:space:]]*(;.*|[[:cntrl:]]?)$)/\2\4\5/' -e 't' -e 'd')"