recursively scan all pages under the given path

Suppose http://example.com/ has two sub-pages besides the root: http://example.com/foo and http://example.com/bar/baz. If the URL given is http://example.com, all three pages should be scanned. If the URL given is http://example.com/foo, only that and http://example.com/bar/baz should all be scanned, not http://example.com/.

This should probably be a flag, but not sure if scanning recursively should be the default or not.