Class Crawler

Array of ongoing requests. These are cancelled when calling the reset method. Normally you don't need to modify this array.

The PQueue where the crawl requests are waiting.

The array of visited URL. If you have already visited an URL, crawl will immediately return; else the URL will be added to the array. Normally you don't need to modify this array.