How to use wget (Direct download files from the internet) – Examples, Scripts

filesguide

wget is a command-line utility for Linux (and other operating systems) that allows you to download files directly from the internet. It’s non-interactive, making it ideal for scripts and automation. It supports protocols like HTTP, HTTPS, and FTP.

Common uses include downloading files, mirroring websites, and recursively downloading content. It’s a go-to tool for shell scripts needing to fetch data from remote servers. wget is available on Linux, macOS, and even Windows (via Cygwin).

Official page: https://www.gnu.org/software/wget/

Written in C, wget is an open-source project maintained by the GNU Project.

Installation

Here’s how to install wget on various operating systems:

Ubuntu/Debian

sudo apt-get install wget

CentOS/RHEL

sudo yum install wget

macOS

brew install wget

Windows (Cygwin)

1. Download and install Cygwin from https://www.cygwin.com/
2. Select wget during the Cygwin installation process.

Example Commands

• Basic Download: wget http://example.com/file.zip
• Background Download: wget -b http://example.com/file.zip
• Rename Download: wget -O myfile.zip http://example.com/file.zip
• Multiple Downloads: wget http://example.com/file1.zip http://example.com/file2.zip
• Resume Download: wget -c http://example.com/file.zip
• Limit Download Speed: wget --limit-rate=100k http://example.com/file.zip
• Download Website (Limited Depth): wget -r -l 3 http://example.com
• Ignore SSL Errors: wget --no-check-certificate https://example.com
• Use Proxy: wget -e use_proxy=yes -e http_proxy=192.168.0.1:8080 http://example.com
• Download PDFs from a Page: wget -r -H -l1 -nd -A.pdf -e robots=off http://example.com
• Quiet Mode (No Output): wget -q http://example.com/file.zip

Example Scripts

Backup Script: Downloads a file with a timestamp for backup purposes.

#!/bin/bash
url="http://example.com/data.zip"
destination="/path/to/backup/folder"
timestamp=$(date +"%Y-%m-%d_%H-%M-%S")
wget -O "${destination}/backup_${timestamp}.zip" $url

Scheduled Download Script: Downloads a daily report using cron.

#!/bin/bash
url="http://example.com/daily-report.csv"
output_directory="/path/to/download/folder"
filename=$(date +"%Y-%m-%d_daily-report.csv")
full_path="${output_directory}/${filename}"
wget -O "$full_path" $url

Website Mirroring Script: Mirrors a website for offline access.

#!/bin/bash
website_url="http://example.com"
output_dir="/path/to/mirror/folder"
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent -P $output_dir $website_url

Bulk Download Script: Downloads files from a list of URLs in a text file.

#!/bin/bash
url_file="urls.txt"
download_dir="/path/to/download/folder"
while IFS= read -r url; do
  wget -P $download_dir $url
done < "$url_file"

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification.
Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.