Spider / Download Website with Wget on Windows

Datetime:2017-04-15 05:40:32         Topic: Web Crawlers          Share        Original >>
Here to See The Original Article!!!

Wget has been around forever it seems, but is still get great tool for spidering or downloading content (all content) from a website to your local machine. Below are some simple steps to install Wget and to run it.

To install Wget:

  1. Visit  http://gnuwin32.sourceforge.net/packages/wget.htm and download the setup file , labeled “Complete package, except sources”.
  2. Run the setup .exe and leave everything as defaults.
  3. In a command prompt (cmd.exe) change to the C:\Program Files (x86)\GnuWin32\bin > directory (or C:\Program Files\GnuWin32\bin if your on a 32 bit OS)
  4. To test, if you just run wget .exe it should return wget : missing URL

To download a website:

  1. In a command prompt (cmd.exe) change to the C:\Program Files (x86)\GnuWin32\bin > directory (or C:\Program Files\GnuWin32\bin if your on a 32 bit OS)
  2. Now run:  wget -e robots=off –no-check-certificate –recursive –level=0 -P “c:\\somefolder” http://chrisbitting.com (obviously replace my website with your own)

In that folder, you should soon files from the site Wget copied.

I’ve only had issues with a site that I had forced to use SSL. Wget didn’t like that.








New