Wednesday, January 25, 2012

Text File Processing With Bash

This took me way too long to figure out, and I'll never remember how to do it, so I document it here for all time. I had this list of files I needed to download using wget or curl or whatever. The trick was that each file had a unique URL and required a unique output filename. So I had this long-ish text file which looked like:

127_368_2.pdf http://localhost:52225/Cache/Embedded/127/368/2.pdf
127_368_3.pdf http://localhost:52225/Cache/Embedded/127/368/3.pdf
923_7_1.pdf http://localhost:52225/Cache/Embedded/923/7/1.pdf

where the first string was the output name and the second was it's source URL. After bouncing around scripting forums for way, way too long, I came up with the following:

cat files.txt | while read x y; do curl -b authcookie.txt -o $x $y; done

What took me so long to figure out was the parsing of each line of the file, which "while" takes care of on it's own when multiple arguments are provided. Neat.

No comments:

Post a Comment