The wget command is used to download files from a specified URL. wget is very stable and highly adaptable even under very limited bandwidth or unstable network conditions. If a download fails due to network issues, wget will keep retrying until the entire file has been downloaded. If the server interrupts the download process, it will reconnect to the server and continue downloading from where it stopped. This is very useful for downloading large files from servers that limit connection time.

Syntax

wget(options)(parameters)

Options

-a<log file>: Record the execution process in the specified log file;
-A<suffix>: Specify the suffixes of files to download; separate multiple suffixes with commas;
-b: Run wget in the background;
-B<connection address>: Set the base address for reference links;
-c: Continue a task interrupted in the previous session;
-C<flag>: Set the server data block function flag; on enables it, off disables it, the default is on;
-d: Run the command in debug mode;
-D<domain list>: Set the list of domains to follow, separated by “,”;
-e<command>: Execute the specified command as part of the “.wgetrc” file;
-h: Display command help information;
-i<file>: Get the URLs to download from the specified file;
-l<directory list>: Set the list of directories to follow; separate multiple directories with “,”;
-L: Follow only related links;
-r: Recursive download mode;
-nc: If the file exists, do not overwrite the original file when downloading;
-nv: Show only update and error messages during download, without displaying the detailed execution process;
-q: Do not display the command execution process;
-nh: Do not query host names;
-v: Display the detailed execution process;
-V: Display version information;
--passive-ftp: Use passive mode PASV to connect to the FTP server;
--follow-ftp: Download FTP link files from HTML files.

Parameters

URL: Download the specified URL address.

Examples

Use wget to download a single file

wget http://www.linuxde.net/testfile.zip

The following example downloads a file from the Internet and saves it in the current directory. During the download, a progress bar will be displayed, including (download completion percentage, bytes already downloaded, current download speed, and remaining download time).

Download and save with a different file name

wget -O wordpress.zip http://www.linuxde.net/download.aspx?id=1080

By default, wget uses the characters after the last matching / as the file name. For dynamic link downloads, the file name is often incorrect.

Error: The example below will download a file and save it with the name download.aspx?id=1080:

wget http://www.linuxde.net/download?id=1

Even if the downloaded file is in zip format, it will still use the name download.php?id=1080.

Correct: To solve this problem, we can use the -O parameter to specify a filename:

wget -O wordpress.zip http://www.linuxde.net/download.aspx?id=1080

Limit download speed with wget

wget --limit-rate=300k http://www.linuxde.net/testfile.zip

When you run wget, by default it will use all available bandwidth for downloading. But when you are about to download a large file and still need to download other files, it becomes necessary to limit the speed.

Resume an interrupted download with wget

wget -c http://www.linuxde.net/testfile.zip

Using wget -c to restart an interrupted download is very helpful when a large file download is suddenly interrupted due to network issues or other reasons. We can continue the download instead of downloading the file again. You can use the -c parameter whenever you need to resume an interrupted download.

Download in the background with wget

wget -b http://www.linuxde.net/testfile.zip

Continuing in background, pid 1840.
Output will be written to `wget-log'.

When downloading very large files, we can use the -b parameter to download in the background. You can use the following command to check the download progress:

tail -f wget-log

Spoof the user agent for downloading

wget --user-agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16" http://www.linuxde.net/testfile.zip

Some websites can detect that the user agent is not a browser and reject your download request. However, you can disguise it with the --user-agent parameter.

Test download links

When you plan to schedule downloads, you should test whether the download link is valid before the scheduled time. We can add the --spider parameter to check.

wget --spider URL

If the download link is correct, the following will be displayed:

Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

This ensures that the download can proceed at the scheduled time, but if you provide an incorrect link, the following error will be displayed:

wget --spider url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

You can use the --spider option in the following situations:

• Run a check before a scheduled download
• Periodically check whether a website is available
• Check for dead links on web pages

Increase the number of retries

wget --tries=40 URL

If there is a network issue or you are downloading a large file, the download may fail. By default, wget retries the connection 20 times when downloading a file. If needed, you can use --tries to increase the number of retries.

Download multiple files

wget -i filelist.txt

First, save a file containing the download links:

cat > filelist.txt
url1
url2
url3
url4

Then use this file with the -i option to download them.

Mirror a website

wget --mirror -p --convert-links -P ./LOCAL URL

Download the entire website locally.

• --miror enables mirror downloading.
• -p downloads all files required for the HTML pages to display properly.
• --convert-links converts the links to local ones after downloading.
• -P ./LOCAL saves all files and directories to the specified local directory.

Filter downloads by specific format

wget --reject=gif ur

If you want to download a website but do not want to download images, you can use this command.

Save download information to a log file

wget -o download.log URL

If you don’t want download information to be displayed directly in the terminal but instead in a log file, you can use this.

Limit the total download size

wget -Q5m -i filelist.txt

If you want the download to stop when the files exceed 5M, you can use this. Note: this parameter does not work for downloading a single file; it is only effective for recursive downloads.

Download files of a specified format

wget -r -A.pdf url

This feature can be used in the following situations:

• Download all images from a website.
• Download all videos from a website.
• Download all PDF files from a website.

FTP download

wget ftp-url
wget --ftp-user=USERNAME --ftp-password=PASSWORD url

You can use wget to download from FTP links.

Use wget for anonymous FTP downloads:

wget ftp-url

Use wget for FTP downloads authenticated with a username and password:

wget --ftp-user=USERNAME --ftp-password=PASSWORD url