There are often times when we want to fetch a variety of URLs that vary in only a small way. For example, you might want to fetch variations on a product page by varying the PRODUCTID parameter. You might need to fetch a variety of URLs that vary in a small part (e.g., news, careers, blog). cURL makes this all easy by not only allowing you to specify variations in a concise format, but also letting you name the corresponding output file according to the variation you specified.
# Fetch all the categories from 00 to 99.curl -o 'category-#1#2.html' 'http://www.example.com/category.php?CATID=[0-9][0-9]'curl -o 'category-#1.html' 'http://www.example.com/category.php?CATID=[0-99]'# Fetch several main pages and store them in files named accordinglycurl -o '#1.html' 'http://www.example.com/{news,blog,careers,contact,sitemap}/'Note the use of single quotes. In a Unix environment, that’s necessary to avoid problems where the shell interprets the #, ?, and brackets. In Windows, that’s less necessary, but it doesn’t hurt. The first example fetches pages where CATID must be exactly 2 digits, i.e., 00, 01, 02, ..., 97, 98, 99. The second example fetches the same sorts of pages, but where CATID is a single digit for values 0–9 and double digits thereafter, i.e., 0, 1, 2, ..., 97, 98, 99.
You can put many varying parameters into a single command. cURL will do all the permutations and combinations. Consider an item page that takes a product ID (0001–9999), a category (0–9), a color (red, yellow, blue, or green) and a size (S, M, L, or XL). The following single invocation of cURL will fetch every possible combination (all 1,599,840 of them!).
curl -o '#1-#2-#3-#4.html' \"http://www.example.com/cgi-bin/item.cgi?prod=[0001-9999] &cat=[0-9]&color={red,yellow,blue,green}&size={s,m,l,xl}"Of course, in security testing, we would test for weird values: alphabetic product IDs, numeric sizes, and interesting boundary cases like 65,537 for color.
