Tag: Bash

Bash is the shell, or command language interpreter, for the GNU operating system.

AWS dynamic IP ranges

Need to know the dynamic IP ranges used by AWS in a specific region? AWS publishes this as a JSON file.

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes | .[] | select((.region == "eu-west-1") and (.service == "EC2")) | .ip_prefix' | sed 's/"//g'

This filters for EC2 in eu-west-1 (Ireland) and outputs CIDR blocks.

To check another service:

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes | .[] | select(.service == "CLOUDFRONT") | .ip_prefix' | sed 's/"//g'

Useful for firewall rules.

Enjoy!

How to soft reset Linux disks

Sometimes disks stop responding and cause system slowness because the kernel is waiting for a response that never comes. The disk appears 100% utilised in iostat even though nothing is being read or written.

To avoid rebooting, reset the device. First find it:

lsscsi

Then remove and rescan:

echo 1 > /sys/block/sdX/device/delete
echo "- - -" > /sys/class/scsi_host/hostX/scan

Replace sdX with the affected device and hostX with the SCSI host number from lsscsi.

Hope it helps!

Find sdx device/ATAx mappings

When dealing with hardware, you often see log messages like:

ata9: hard resetting link
ata9: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

To map an ata number to the actual sdX device:

ls -l /sys/class/ata_device/ata9/device/host*/target*/*/block

This shows which sdX corresponds to ata9. Check dmesg for the initial discovery mapping too.

Enjoy!

Convert Squid/timestamps to a human readable format

Somebody had the brilliant idea of using Unix timestamps in Squid log files. It makes parsing easy but reading them is not so human-friendly.

This is an elegant way of solving the problem:

cat /var/log/squid/access.log | perl -p -e 's/^([0-9]*)/"[" . localtime($1) . "]"/e'

The Perl one-liner captures the timestamp at the beginning of each line, converts it to a human-readable date with localtime(), and replaces it inline. The /e flag tells Perl to evaluate the replacement as code rather than a string.

Hope it helps!

sed – match and replace BIND/DNS serial number programmatically

I had to make modifications to several BIND zone files. Sed proved itself invaluable once again.

The zone files looked like this:

domain.com. IN SOA ns1.domain.com. postmaster.domain.com. (
               2015021001   ; Serial
               600          ; Refresh (10 minutes)
               1800         ; Retry (30 minutes)
               3600000      ; Expire (1000 hours)
               3600         ; Minimum (1 hour)
               )

To update the serial number on every zone file at once:

sed -i 's/[0-9]\{10\}\s*; Serial/2016020101   ; Serial/' *.zone

This matches a 10-digit number followed by optional whitespace and “; Serial”, then replaces it with the new serial. The -i flag edits files in place — test without it first to see what would change.

Enjoy!

Converting spreadsheets to text files with ssconvert

ssconvert is a utility shipped with Gnumeric, a command-line spreadsheet converter. It is useful when you have data in a spreadsheet and want to parse it from a script.

ssconvert -O 'separator=;' spreadsheet.ods spreadsheet.csv

This converts an ODS file to CSV using a semicolon as separator. ssconvert supports ODS, XLS, XLSX, CSV, HTML, and more.

You can also export specific sheets:

ssconvert -O 'separator=|;format=raw' --export-type=Gnumeric_stf:stf_csv spreadsheet.ods sheet2.csv

The format=raw option prevents ssconvert from reformatting numbers.

Enjoy!

How to update all docker base images with one command

To pull the latest version of all your Docker images at once:

for docker in $(docker images | awk '{print $1,$2}' | sed 's/ /:/g' | grep -v REPOSITORY); do docker pull $docker; done

This lists all images, extracts repository and tag, formats as repository:tag, and pulls each one.

After updating, old image versions stay on disk. To clean them up:

docker image prune -a

This removes all unused images and frees up space.

Enjoy!

How to update all pip python packages with one command

To update all your pip-installed Python packages at once:

sudo pip install -U `pip list -o | cut -d ' ' -f1 | tr '\n' ' '`

This lists outdated packages with pip list -o, extracts their names, joins them with spaces and passes them to pip install -U. A one-liner that saves you from updating each package individually.

NB: One-liners like this are potentially destructive. Read what they do before running them, especially with sudo.

For a safer alternative on modern systems, use pip-review:

pip install pip-review
pip-review --auto

This prompts you before each update.

Enjoy!

Test email from command line

When you set up email forwarding or configure a new MTA, you will want to confirm it is working before relying on it for alerts. A quick test is all you need.

Send a test message

mail -s Test myEmail@gmail.com

If the mail command is not available, install it with apt-get install mailutils. Then check your inbox.

If it does not arrive

Run mailq to see if the message is stuck in the queue. The mail log at /var/log/mail.log will also show the delivery attempt and any errors.

Enjoy!