I’ve gotten tired of googling the same things over and over again.

While loops

Run for period of time

#!/bin/bash

runtime="5 minute"
endtime=$(date -ud "$runtime" +%s)

while [[ $(date -u +%s) -le $endtime ]]
do
    echo "Time Now: `date +%H:%M:%S`"
    echo "Sleeping for 10 seconds"
    sleep 10
done

Resource: https://www.golinuxcloud.com/run-while-loop-until-specific-time-shell-bash/

Infinite loop

item=true
while [ $item = true ]; do echo 'bla'; done

While value is an empty string

value=""
while [[ -z "$value" ]]; do
    echo "value is empty"
    # here's where we break out
    if [[ "$value" ]]; then
        echo "exiting loop because value has been set to $value"
        break
    fi
done

Read each line of a file

while read p; do
  echo "$p"
done <file.txt

Resource: https://stackoverflow.com/questions/1521462/looping-through-the-content-of-a-file-in-bash

One-liner equivalent

while read t; do echo "$t"; done <bbh_targets.txt

Execute command once per line of piped input

command | while read -r line; do command "$line"; done

For example, if you want to run a program and send the output to different files using regular expressions:

recon asn -t recon_targets.txt | \
  while read -r line; do \
    ([[ $line =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/[0-9]{2}$ ]] \
    && echo $line | anew ip_ranges.txt) || ([[ $line =~ ^[0-9]{5}$ ]] && \
      echo $line | anew asns.txt); done

Resources: https://unix.stackexchange.com/questions/7558/execute-a-command-once-per-line-of-piped-input Tool run for the example


For loops

Compute md5sum for every item in a directory

for i in "$(ls)"; do md5sum "${i}"; done | sort
for COUNT in $(seq 1 10); do
  echo $COUNT
  sleep 1
done

Read data into an array and loop over it

mapfile -t buckets < <(aws s3 ls | grep tf | awk '{print $3}' | tr " " "\n")
for b in "${buckets[@]}"; do echo "TF Bucket: $b"; done

Old method that does not spark joy for linters

buckets=($(aws s3 ls |grep tf | awk '{print $3}' | tr " " "\n"))
for b in "${buckets[@]}"; do echo "TF Bucket: $b"; done

SSH

Encrypt Key

openssl rsa -des3 -in key.pem -out encrypted-key.pem
# Enter a passphrase
mv encrypted-key.pem key.pem
chmod 400 key.pem

Decrypt Key

openssl rsa -in key.pem -out key.open.pem
# Enter the passphrase used to encrypt the ke
mv key.open.pem key.pem

Resource: https://security.stackexchange.com/questions/59136/can-i-add-a-password-to-an-existing-private-key

SSH to a host through another

You can run this command:

ssh -J name_of_host_to_jump_through remote_user@remote_host_ip

If you want to do it using ~/.ssh/config:

Host remote-host
  HostName remote_host_ip
  User remote_user
  ProxyJump name_of_host_to_jump_through

Resources:

SSH Config with jumphost

Host jumphost
  HostName jumphost_ip
  User jumphost_user
  # Optional if you are using private keys for auth -
  # this key needs to be on the system you're starting from:
  IdentityFile ~/.ssh/jumphost-ssh-key.pem

Host targethost
  HostName targethost_ip
  User targethost_user
  ProxyJump jumphost
  # Optional if you are using private keys for auth -
  # this key also needs to be on the system you're starting from -
  # it will not work if it's on the jumphost:
  IdentityFile ~/.ssh/targethost-ssh-key.pem

Resource: https://serverfault.com/questions/337274/ssh-from-a-through-b-to-c-using-private-key-on-b

Specify directory for when you ssh in

Simply add the following to your ~/.bashrc or ~/.zshrc, etc:

cd /dir/to/start/in

Proper SSH Permissions

# Directory - 700
chmod 700 ~/.ssh

# Private keys - 600
chmod 600 ~/.ssh/id_rsa

# Config file - 600
chmod 600 ~/.ssh/config

# Public keys - 600 or 644
chmod 600 ~/.ssh/id_rsa.pub

# Make public key readable by others
chmod 644 ~/.ssh/id_rsa.pub

# Make authorized_keys readable by others.
# Should be 600 or 644
chmod 644 ~/.ssh/authorized_keys

Resource: https://unix.stackexchange.com/questions/257590/ssh-key-permissions-chmod-settings

Status of SSH Service

sudo systemctl status sshd.service

Restart SSH Service

sudo systemctl restart sshd.service

Resource: https://www.cyberciti.biz/faq/centos-stop-start-restart-sshd-command/##centos_7_centos_8

Forward local service to remote host

This will forward a service running on localhost:3000 to a remote host on port 3000, and will allow other systems on the network to access that service.

On the remote host, run this command to add a line to the sshd_config:

echo 'GatewayPorts clientspecified' | sudo tee -a /etc/ssh/sshd_config

Next, restart the ssh service:

service ssh restart

Finally, run this command from the system running the service:

ssh -R :3000:localhost:3000 user@$target_server

Resources: https://serverfault.com/questions/861909/ssh-r-make-target-host-accept-connection-on-all-interfaces https://serverfault.com/questions/33283/how-to-setup-ssh-tunnel-to-forward-ssh

Run command over SSH

ssh ubuntu@yoursystem 'sudo apt-get update'

Resource: https://www.cyberciti.biz/faq/unix-linux-execute-command-using-ssh/

Run multiple commands with ssh

ssh -i ${KEY} ${USER}@${IP} << EOF
export AWS_ACCESS_KEY_ID=${ACCESS_KEY}
export AWS_SECRET_ACCESS_KEY=${SECRET_ACCESS_KEY}
export AWS_SESSION_TOKEN=${SESSION_TOKEN}
curl -sLO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
aws eks --region ${AWS_REGION} update-kubeconfig --name ${TARGET_CLUSTER}
./kubectl get pods -n ${TARGET_NAMESPACE}
EOF
}

Resource: https://www.shellhacks.com/ssh-execute-remote-command-script-linux/

Forward remote service to localhost

This example will forward the service running on target.server:8080 to localhost:8080:

ssh -L 8080:localhost:8080 username@target.server

Forward remote service via bastion to localhost

This example will forward the service running on target.server:443 to localhost:4455:

ssh -N -L 4455:target.server:443 username@bastion

Alternatively if you want to use a SOCKS proxy (say you’re using burp for example), you can do the following:

ssh -C -D 8085 username@bastion

and then point your SOCKS proxy to localhost:8085 in order to hit hosts only accessible via the bastion.

Resources: http://www.spencerstirling.com/computergeek/sshtunnel.html https://medium.com/@mccabe615/proxying-burp-traffic-e6e7a8adc101

View images on a remote system

ssh -Y user@server
apt install -y eog
eog pictures/foo.png

Resource: https://superuser.com/questions/557622/how-can-i-view-pictures-via-ssh

Transfer file to host running SSH on a particular port

PORT=2222
scp -P $PORT file user@system

Resource: https://askubuntu.com/questions/182478/ssh-scp-to-copy-file-to-remote-server-port-21

Transfer file on behalf of another system

This is useful if you have two aws instances, and want to transfer a file between them from your laptop.

scp -3 -i $PEM_FILE user1@system_with_file:/file/to/xfer user2@system_that_needs_file:/file/to/xfer

Resource: https://superuser.com/questions/686394/scp-between-two-remote-hosts-from-my-third-pc

Fix SSH Too Many Authentication Failures error

Add this line to your ~/.ssh/config:

Host *
  IdentitiesOnly=yes

Resource: https://www.tecmint.com/fix-ssh-too-many-authentication-failures-error/

SSH master mode

Quick test with password:

sshpass -p 'password123' ssh -o StrictHostKeyChecking=no -N -M -S /tmp/mysock \
  ubuntu@192.168.1.2 &
sleep 2
ssh -S /tmp/mysock ubuntu@192.168.1.2 exit
sleep 2
ssh -S /tmp/mysock -O exit ubuntu@192.168.1.2

Resource: https://unix.stackexchange.com/questions/32984/multiple-ssh-sessions-in-single-command

SCP files using wildcard

Be sure to escape the wildcard, i.e. file-\*

Full example:

scp ubuntu@target:~/.config/cred\*

Resource: https://unix.stackexchange.com/questions/27419/how-to-use-wildcards-when-copying-with-scp

Check if you can ssh to several hosts

hosts.txt:

host1.com
192.168.1.2

test_ssh.sh:

for SERVER in $(cat hosts.txt); do
  ssh -i id_rsa -o StrictHostKeyChecking=no -o BatchMode=yes \
    user@$SERVER exit && echo OK $SERVER || echo ERR $SERVER
done

Resource: https://stackoverflow.com/questions/49564299/script-to-check-if-i-can-access-multiple-servers

Create SSH key script

KEY_NAME=CUSTOM_KEY
ssh-keygen -t ed25519 -C "Key Description" -f "${KEY_NAME}" -N ''

Resources: https://stackoverflow.com/questions/10767488/automate-ssh-keygen-t-rsa-so-it-does-not-ask-for-a-passphrase https://unix.stackexchange.com/questions/48863/ssh-add-complains-could-not-open-a-connection-to-your-authentication-agent/48868

Generate public key from private key

ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

Resource: https://askubuntu.com/questions/53553/how-do-i-retrieve-the-public-key-from-a-ssh-private-key

Show successful ssh logins

grep sshd.\*Accepted /var/log/auth.log

Show failed ssh logins

grep sshd.\*Failed /var/log/auth.log

SSHD with custom file

sshd -f <path/to/sshd_config/file> For example: sshd -f /tmp/sshd_config

The sshd_config file can look roughly like this:

Port 6022
HostKey /tmp/sshd_config/host_rsa
PasswordAuthentication yes
PermitRootLogin no

Fix known_hosts error

You should not use this unless you are dealing with a test host that you created.

TARGET_HOST=some-system
ssh-keygen -R $TARGET_HOST

Resource: https://kinsta.com/knowledgebase/warning-remote-host-identification-has-changed/


Create sudo user non-interactively

Skip the prompts that come with creating a user typically, and give them sudoers privileges. Replace ${USERNAME} with the user you want to create.

# The man page explanation for --gecos isn't that intuitive IMO
# Basically specifying --gecos "" states you don't want to ask
# for real name, phone, etc.
USERNAME=everythingisnormalhere
adduser --disabled-password --gecos "" "${USERNAME}"
touch "/etc/sudoers.d/10_${USERNAME}"
echo "${USERNAME} ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/10_${USERNAME}"

Resources: https://askubuntu.com/questions/94060/run-adduser-non-interactively https://askubuntu.com/questions/420784/what-do-the-disabled-login-and-gecos-options-of-adduser-command-stand

Run function as another user

export -f <function_name>
su <user to run function as> -c "bash -c <function_name>"

Run command as another user

USER=centos
runuser -l $USER -c 'whoami'

Resource: https://devops.stackexchange.com/questions/10402/running-a-command-as-a-specific-user-on-an-ec2-using-ssm

Exit-on-error mode

Put this at the top of your bash script: set -e. If a command returns a nonzero status, the shell will exit.

Resources: https://unix.stackexchange.com/questions/97101/how-to-catch-an-error-in-a-linux-bash-script http://linuxcommand.org/lc3_man_pages/seth.html - man page

See files that would be unzipped without unzipping

unzip -v $FILE_NAME.zip

Unzip to a directory

unzip package.zip -d /opt

Resource: https://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/

Color output in less

FILE=bla.txt
less -r $FILE

YAML templating

template.yml:

---
bobs:
  password: "${bobs_password}"
  username: "${bobs_username}"
  machines: "${bobs_machines}"

Fill in the template and create final.yml:

export sample="Hello Yaml!"
export bobs_machines="10.2.3.4"
export bobs_username="nats"
export bobs_password="password"
rm -f final.yml temp.yml
( echo "cat <<EOF >final.yml";
  cat template.yml;
  echo "EOF";
) >temp.yml
. temp.yml
cat final.yml

Resource: https://www.starkandwayne.com/blog/bashing-your-yaml/

Lint YAML

python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < file.yml

Lint JSON

cat file.json | jq

Code Style

Function declarations:

my_func() {

}

Variable declaration:

cool_var='great'

Source filenames should be lowercase with underscores to separate words, i.e. the_best_script.sh

Resources: https://google.github.io/styleguide/shell.xml##Variable_Names https://dev.to/puritanic/shellscripting-functions-2696 https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization


$HOME vs ~ in scripts

Opt to use $HOME as ~ is expanded by the terminal into the $HOME variable. Subsequently $HOME is going to be a more robust option.

Resource: https://ryankubik.com/blog/tilde-vs-home

Global variables in functions

If you want to set something as a global variable and use it in a function, do the following:

export SOMETHING="variable"

great_function() {
    echo $SOMETHING
}

Parse an env file

export $(egrep -v '^##' .env | xargs)

The env file you want to parse should look something like this:

FOO='bar'
USERNAME='user'

Resource: https://gist.github.com/judy2k/7656bfe3b322d669ef75364a46327836


Send message to all users

echo "hi" | wall

Move file with rsync

rsync --partial --progress filename.txt user@ipaddress_or_hostname:~

Use SSH and rsync together

rsync -azvhe ssh --progress local_dir_to_copy user@192.168.1.2:remote_copy_of_dir

Remote to local:

rsync -azvhe ssh --progress user@192.168.1.2:~/dir_to_copy_to_local local_copy_of_remote_dir

Resource: https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/


Sync two folders

Start with a dry run to ensure that it’s doing what you expect it to do:

rsync -rv --append-verify --dry-run folder_to_sync_from/ /home/ubuntu/folder_to_sync_to

Once you’ve confirmed it’s working as expected, simply remove the --dry-run to execute.

Resource: https://unix.stackexchange.com/questions/487646/copy-changed-new-files-only-to-a-different-directory

Transfer multiple remote directories

This worked well for my MacOS systems:

IP=192.168.1.56
DIRS=(/Users/user/dir1 /Users/user/dir2 /Users/user/.dotfiles /Users/user/dir3)

for DIR in "${DIRS[@]}"; do
    echo "Transferring remote directory ${DIR}, please wait."
    rsync -avzh --progress "user@${IP}:${DIR}" .
    echo "Transfer of remote directory ${DIR} complete!"
done

Resource: https://unix.stackexchange.com/questions/575156/using-rsync-to-back-up-home-folder-with-same-permissions-recursive


Break up VM into multiple 1 GB files

Need to transfer a VM? Try this.

split -b 1000m vm.ova vm.ova.split

Bring it back together

cat vm.ova.splita* > vm.ova

Kill netcat after 3 seconds

timeout 3 nc google.com 80; echo exit=$?

Resource: https://unix.stackexchange.com/questions/96817/how-do-i-get-the-nc-command-to-end-after-2-seconds


Test connectivity to service

Netcat:

HOSTNAME_OR_IP=system.mydomain.com
PORT=5432
# -z: report connectivity status only
# -w1: wait one second and then give up
nc -z -w1 $HOSTNAME_OR_IP $PORT

Telnet:

HOSTNAME_OR_IP=system.mydomain.com
PORT=5432
# -e X: set escape character to X
echo X | telnet -e X $HOSTNAME_OR_IP $PORT

Resources:


Determine if string exists in a file

if grep -q SomeString "$FILE"; then
  # SomeString was found
fi

Resource: https://stackoverflow.com/questions/11287861/how-to-check-if-a-file-contains-a-specific-string-using-bash


Delete files if they match a wildcard

If you have two files, someFile and someFile2, this will delete both of them:

if ls someFile* 1> /dev/null 2>&1; then
  rm someFile*
fi

Resource: https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string


Check if multiple binaries are installed

if command -v cowsay 2>/dev/null && ! command -v gshuf 2>/dev/null; then
  # do things if cowsay is installed but gshuf isn't
fi

Check if file is missing

if [[ ! -f "some/file.txt" ]]; then
  # do things if the file is not there
fi

One-liner to check if file exists

if [ ! -f /tmp/foo.txt ]; then echo "File not found!"; else echo "file found"; fi

Resource: https://unix.stackexchange.com/questions/474244/one-liner-to-check-for-file-exists

Create missing directory

[[ -d dir ]] || mkdir dir

Resource: https://stackoverflow.com/questions/18622907/only-mkdir-if-it-does-not-exist


Number of duplicate lines in a file

FILE='/etc/passwd'
sort "${FILE}" | uniq -c

Resource: https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated


Run command in another directory as another user

This particular example is used to run a command as root in another directory

sudo -u root /bin/bash -c 'pushd "$HOME/.some_directory/"; ls -lart; popd'

Resource: https://stackoverflow.com/questions/10382141/temporarily-change-current-working-directory-in-bash-to-run-a-command

Bypass post-install configurations

If you have to install things via apt and run into blue screens that require human interaction, do this to avoid it:

sudo DEBIAN_FRONTEND=noninteractive apt install -yq [packagename]

Resource: https://serverfault.com/questions/227190/how-do-i-ask-apt-get-to-skip-any-interactive-post-install-configuration-steps

Get count of and list of most frequently used bash commands

history | awk '{a[$2]++;next}END{for (i in a){print i " --> " a[i]}}' \
  | sort -nr -k3

Reinstall a package in debian-based system

sudo apt install --reinstall <package> -y

Unable to fetch some archives, maybe run apt-get update or try with –fix-missing

Run this if apt update --fix-missing isn’t working for you:

rm -rf /var/lib/apt/lists/*; apt update

Alternatively, you can also try:

apt clean; apt update

Resource: https://stackoverflow.com/questions/38743951/unable-to-fetch-some-archives-maybe-run-apt-get-update-or-try-with-fix-missin

Save off auth logs for review

Useful to see who has logged into your system and when

last -F > /tmp/last

https://serverfault.com/questions/375091/getting-login-year-data-with-the-last-command-on-linux

“Incognito mode” for bash

Turn off history for a session.

export HISTFILE=

If you do want a record of your activity in another file, run this command instead:

export HISTFILE=0

This will create a file, 0, with your history for that session.

Resources: https://github.com/mubix/post-exploitation/wiki/Linux-Post-Exploitation-Command-List https://www.maketecheasier.com/linux-command-line-history-incognito/

List NFS mounts

mount |grep nfs

When all users last logged in

lastlog

All previously logged in users

lastlog |grep -v "Never"

Resource: https://www.rebootuser.com/?p=1623

Determine if host is sharing file systems

If there are local mounts, you will be able to see who you’re sharing with.

showmount -e localhost

Hosts with active connections to the system

netstat -pan |grep ESTABLISHED

Check if file has a certain number of lines

If file has 3 lines, tell us about it:

if [[ $(wc -l file) == *3* ]]; then
  echo "There are three lines in file"
fi

Resource: https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string

If wc provides an inaccurate count

Add a newline to the end of the file with:

printf "\n" >> file.txt

Resource: https://stackoverflow.com/questions/12616039/wc-command-of-mac-showing-one-less-result


Strace

General Example

strace /bin/ls

View output of a running process

PID=5
strace -p "${PID}" -e write

Resource: https://unix.stackexchange.com/questions/58550/how-to-view-the-output-of-a-running-process-in-another-bash-session

Filter on write functions

strace -e write /bin/ls

Objdump

Used to get the assembly output of a given binary

BIN="$(which ls)"
objdump -d "${BIN}" | tee binary_output.txt

Get size of all subfolders

du -sh *

Resource: https://www.macobserver.com/tips/mgg-answers/seeing-folders-size-terminal/##:~:text=From%20the%20Terminal%2C%20type%3A%20du,and%20folders%20with%20their%20sizes

List largest directories and files

du -hsx * | sort -rh | head -10

Resource: https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/


Kerberos

List tickets in a keytab

ktutil -kt file.keytab list

Resources: https://kb.iu.edu/d/aumh##list https://community.pivotal.io/s/article/Kerberos-Cheat-Sheet

Create kerberos ticket

kinit <username> -k -t /where/to/store/keytab/nameofkeytab.keytab

Grep

Not matching

This example will find all jobs that don’t contain com:

launchctl list | grep -v com

Multiple not matching

grep -v ".git\|.terraform"

Resource: https://stackoverflow.com/questions/13610642/using-grep-for-multiple-search-patterns

Regex Example

This will get the version of ruby from rvm: rvm list | head -n 1 |grep -Po '\-(.*?)\s' | tail -c +2 The output from that command that it is parsing will look something like this: ruby-2.5.1 [ x86_64 ]

It will return 2.5.1.

Resource: https://askubuntu.com/questions/89995/bash-remove-first-and-last-characters-from-a-string

Negative Match

Match Inversion

grep -Ev 'pattern1|pattern2|pattern3' file

Resource: https://stackoverflow.com/questions/10411616/grep-regex-not-containing-a-string

Egrep Regex and return capture group

This will read a file with a bunch of gmail addresses in it (among other things), and then print the unique email addresses found.

cat file.txt | grep gmail.com | \
  egrep -o '\w+@gmail.com' | uniq -u

Resource: https://stackoverflow.com/questions/18892670/can-not-extract-the-capture-group-with-neither-sed-nor-grep/18892742

Search for string in files and return file name

grep -lrnw "thing to search for" .

Resources:

Output only found files w/ grep

grep -rnlw "stringtofind" .

Resource: https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file

Search for string in specific file type

grep -i -r 'mystring' --include "*.yml"

Resource: https://community.home-assistant.io/t/searching-for-text-in-yaml-and-py-files/181948


Create self-signed cert

openssl req \
    -x509 \
    -nodes \
    -newkey rsa:4096 \
    -keyout server.key \
    -out server.crt \
    -days 3650 \
    -subj "/C=US/ST=Oregon/L=Portland/O=Company Name/OU=Org/CN=www.example.com"Department/CN=*"

Resource: https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl


Archivers

Extract command

This will extract several popular formats for you without needing to worry about the syntax for each:

extract() {
  if [[ -f "${1}" ]] ; then
      case "${1}" in
          *.tar.bz2)   tar xvjf   "${1}"    ;;
          *.tar.gz)    tar xvzf   "${1}"    ;;
          *.bz2)       bunzip2    "${1}"    ;;
          *.rar)       rar x      "${1}"    ;;
          *.gz)        gunzip     "${1}"    ;;
          *.tar)       tar xvf    "${1}"    ;;
          *.tbz2)      tar xvjf   "${1}"    ;;
          *.tgz)       tar xvzf   "${1}"    ;;
          *.zip)       unzip      "${1}"    ;;
          *.Z)         uncompress "${1}"  ;;
          *.7z)        7z x       "${1}"  ;;
          *)           echo "don't know how to extract ${1}..." ;;
      esac
  else
      echo "${1} is not a valid file!"
  fi
}

Extract tar contents to specific directory

tar -xf archive.tar -C /target/directory

Resource: https://askubuntu.com/questions/45349/how-to-extract-files-to-another-directory-using-tar-command

Create tar.gz with var

name="directory_name"
tar -czvf "$name".tar.gz $name

Create encrypted zip

zip -r unenc.zip original_file
openssl enc -in unenc.zip -aes-256-cbc -e > enc.zip
# enter password when prompted

Depending on the situation, you may consider destroying the original unencrypted file:

rm unenc.zip

Decrypt and unzip encrypted zip

openssl enc -in enc.zip -aes-256-cbc -d > unenc.zip
# enter password when prompted
unzip unenc.zip

Remove duplicates from a file

sort inputFile | uniq -u > outputfile

Remove last line from a file

head -n -1 foo.txt > temp.txt ; mv temp.txt foo.txt

Resource: https://stackoverflow.com/questions/4881930/remove-the-last-line-from-a-file-in-bash


Curl

Get status code

BLUE="\033[01;34m"
status=$(curl -s -o /dev/null -w "%{http_code}" http://target.com)
echo -e "${BLUE}$status${RESET}"

Resource: https://superuser.com/questions/272265/getting-curl-to-output-http-status-code

Follow redirect

curl -L http://google.com

Resource: https://stackabuse.com/follow-redirects-in-curl/

Show response headers for request

curl -i http://google.com

Resource: https://stackabuse.com/follow-redirects-in-curl/

Silent mode

Use this when you are leveraging curl in a script:

curl -s http://google.com

Upload a file

To upload a file to an endpoint at /upload with a form field, fileUpload, you can do the following:

curl http://target.com/upload -F "fileUpload=@test.txt" -vvv

Send a POST request with params

curl -X POST http://localhost:4999/target \
  -d "param1=value1&param2=value2"

Download a file

This will also ensure that if a proxy is in place, it will not be used to connect to target.com.

curl http://target.com/puppet-linux \
  -o puppet-linux \
  -vvvv \
  --noproxy target.com

Resource: https://stackoverflow.com/questions/800805/how-do-i-make-curl-ignore-the-proxy

Download a file to a certain folder

TER_VAR=1.1.9
OS=linux
ARCH=amd64
DL_URL="https://releases.hashicorp.com/terraform/${TER_VER}/terraform_${TER_VER}_${OS}_${ARCH}.zip"

curl ${DL_URL}" -L --output "${HOME}/Downloads/terraform_${TER_VER}_linux_amd64.zip"

-L - in case a redirect is found -o - output path for download

Resource: https://askubuntu.com/questions/285976/download-zip-file-with-curl-command

Use proxy

Useful to proxy requests through something like Burp Suite.

curl -X GET http://google.com --proxy http://localhost:8080

Resource: https://forum.portswigger.net/thread/how-do-i-get-burpsuite-to-intercept-my-curl-x-get-requests-that-i-am-launching-from-command-line-bfb630dd

3000 millisecond timeout

curl -m 3 http://www.google.com

Resource: https://ec.haxx.se/usingcurl-timeouts.html


Check if a string is a timestamp

date -r "${STR}"

Example:

date -r 1530279830

Resource: https://medium.com/@amalmurali47/h1-702-ctf-web-challenge-write-up-53de31b2ddce


Kill processes

  • Kill all python and go processes:

    pkill python go
    

Resource: https://www.techwalla.com/articles/how-to-kill-all-python-processes-in-ubuntu

  • Kill act process:

    sudo kill -9 $(pgrep act)
    
  • Kill process by path:

    pkill -9 -f /path/to/app
    
    # Example:
    pkill -9 -f /tmp/Linenum.sh
    

Useful alternative to man pages

Explain Shell


Display Web Content in Terminal

curl http://www.cyberciti.biz/
lynx -dump www.cyberciti.biz
wget -O - http://www.cyberciti.biz

Resource: https://www.cyberciti.biz/faq/unix-linux-get-the-contents-of-a-webpage-in-a-terminal/


Open file, grep it, send output to command

This particular example will search for instances of the ENC string and send them to eyaml decrypt -s:

cat file.txt | grep ENC | xargs --null eyaml decrypt -s

Delete to end of line in terminal

Ctrl+k


Delete to beginning of line in terminal

Ctrl+u

Resource: https://askubuntu.com/questions/269046/bash-delete-from-cursor-till-end-of-line-with-a-keyboard-shortcut


Generate a self-signed cert

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 \
  -nodes -out key.crt -keyout key.key

Run command repeatedly

Option #1

CMD='date'
while true; do
  "${CMD}"; sleep 2; \
done

Option #2

CMD='date'
watch "${CMD}"

Resource: https://www.howtoforge.com/linux-watch-command/


Sleep with timer

Set MIN to the desired number of minutes you want to sleep for:

MIN=1
for i in $(seq $(($MIN*60)) -1 1); do
  echo -n "$i, "; sleep 1; \
done

Resource: https://www.commandlinefu.com/commands/view/5886/countdown-clock


Base64 encode with no word wrap

Useful for long strings that you want to base64 encode.

echo "thing" | base64 -w 0

Resource: https://conf.splunk.com/files/2019/recordings/SEC2286.mp4


Base64 decode string

echo 'stringtodecode' | base64 -d

Resource: https://www.base64decode.net/linux-base64-decode


Test if variable is set

if [[ -z "${1}" ]]; then
    echo -e "No input provided, exiting.\n"
    exit 1
else
  echo "input is ${1}"
fi

Resource: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash


Check if multiple parameters are set

if [[ $## -ne 4 ]]; then
   echo 'Required vars missing'
   return 1
fi

Resource: https://www.unix.com/shell-programming-and-scripting/268257-how-do-i-check-if-all-parameters-set-bash.html


Add ssh fingerprint to known_hosts

Useful if you need this functionality in a script.

IP=192.168.1.6
ssh-keyscan -H "${IP}" >> "${HOME}/.ssh/known_hosts"

Resource: https://www.techrepublic.com/article/how-to-easily-add-an-ssh-fingerprint-to-your-knownhosts-file-in-linux/


Working with input

Check if input is an ip address

if [[ -z $1 ]]; then
  echo "Usage: $0 <ip address>"
  exit
else
  if [[ $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "Valid IP input, doing things with $1"
  else
    echo "Invalid IP input"
    exit
  fi
fi

Resource: https://stackoverflow.com/questions/13777387/check-for-ip-validity

Turn into function

check_input() {
  if [[ -z $1 ]]; then
    echo "Usage: $0 <ip address>"
    exit
  else
    if [[ $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
      echo "Valid IP input, doing things with $1"
    else
      echo "Invalid IP input"
      exit
    fi
  fi
}

# Pass input from $1 to function
check_input $1

Resource: https://unix.stackexchange.com/questions/298706/how-to-pass-parameters-to-function-in-a-bash-script

Check input against multiple cases

if [[ "${1}" != "puppet" ]] && \
   [[ "${1}" != "chef" ]] || \
   [[ -z "${1}" ]]; then
  echo "Usage: $0 <CM target>"
  exit
fi

Resource: https://unix.stackexchange.com/questions/67898/using-the-not-equal-operator-for-string-comparison

Validate script is run as root

if [[ "${EUID}" -ne 0 ]]; then
  echo "Please run as root"
  exit
fi

Resource: https://stackoverflow.com/questions/18215973/how-to-check-if-running-as-root-in-a-bash-script/18216122##18216122


Multiline if with multiple conditions

if [[ -f "${CONTROL_NODE_FILES}/authorized_keys" ]] || \
   [[ -f "${MANAGED_NODE_FILES}/authorized_keys" ]] || \
   [[ -f "${MANAGED_NODE_FILES}/${KEY_NAME}" ]] || \
   [[ -f "${MANAGED_NODE_FILES}/${KEY_NAME}.pub" ]] || \
   [[ -f "${CONTROL_NODE_FILES}/${KEY_NAME}" ]] || \
   [[ -f "${CONTROL_NODE_FILES}/${KEY_NAME}.pub" ]]; then
        echo -e \
          "${YELLOW}
          Found existing SSH key pair, skipping the key pair generation step.
          ${RESET}"
fi

UFW

Show open ports

ufw status

Allow port

Open the firewall for port 46666:

ufw allow 46666/tcp

Allow service

This will work for OpenSSH:

ufw allow 'OpenSSH'

This is for Apache:

ufw allow 'Apache Full'

Disable UFW

ufw disable

Enable UFW

ufw enable

Resource: https://linuxize.com/post/how-to-setup-a-firewall-with-ufw-on-ubuntu-18-04/


Keep NC listener open

This will ensure that netcat doesn’t hang when an incoming connection is established.

PORT_TO_LISTEN_ON=80
nc -lvk "${PORT_TO_LISTEN_ON}"

Resource: https://unix.stackexchange.com/questions/423407/how-can-i-keep-netcat-connection-open

Create random string

LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 13 ; echo ''

Example usage:

wget https://github.com/artyuum/Simple-PHP-Web-Shell/blob/master/index.php \
   -O "/var/www/html/$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo '').php"

Resource: https://unix.stackexchange.com/questions/230673/how-to-generate-a-random-string

Create random number

r1=$((1 + $RANDOM % 10))
echo $r1

Resource: https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash/1195035

Remove newline from end of string

tr -d '\n'

Resource: https://stackoverflow.com/questions/12524308/bash-strip-trailing-linebreak-from-output

Create three directories in tmp

mkdir /tmp/{d1,d2,d3}

Test cron job

# Add this to the user's crontab (crontab -e):
* * * * *   /usr/bin/env > /home/username/tmp/cron-env
# If root user:
* * * * * /usr/bin/env > /root/tmp/cron-env
# Create run-as-cron.sh
echo -e '##!/bin/bash\n/usr/bin/env -i $(cat /home/username/tmp/cron-env) "$@"' \
  > run-as-cron.sh
# Run the script to test it
run-as-cron /the/problematic/script --with arguments --and parameters

Resource: https://unix.stackexchange.com/questions/42715/how-can-i-make-cron-run-a-job-right-now-for-testing-debugging-without-changing

Create daily cron job

This will create a daily cronjob that restarts a systemd job on the system every day at 3:05 am.

create_daily_service_restart() {
  cron_job_loc='/etc/cron.daily/restart_service.sh'
  # Create and set permissions for cron job
  echo "$(which systemctl) restart service" > "${cron_job_loc}"
  chmod +x "${cron_job_loc}"
}

Resources:

Enable logging to /var/log/cron

# Enable cron-specific log
sed -i 's/##cron/cron/' /etc/rsyslog.d/50-default.conf

Crontabs for another user

You need to be root to do this.

List crontabs for a user:

crontab -u $USERNAME -l

You can also edit a crontab by using -e in place of -l.

Copy file to location with random name

# This will remove dashes, newlines and spaces
name=$(date | md5sum | tr -d '-' | tr -d '\n' | tr -d ' ')
if cp -R orig "$name"; then
  echo "Created $name"
fi

Resources: https://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/ https://stackoverflow.com/questions/15801599/shell-script-copy-directory-passed-as-variable-to-another-directory-also-as-var

Copy multiple files to folder

cp {file.txt,file2.txt,file3.txt} dst_folder

Copy multiple folders to folder

cp -r {dir1,dir2,dir3} dst_folder

Show routes

netstat -nrl

Create route

This is useful if you’re having issues with VPN and something like Burp Suite.

sudo route add <target system> <local IP>

Netstat (without actually running it)

awk 'function hextodec(str,ret,n,i,k,c){
    ret = 0
    n = length(str)
    for (i = 1; i <= n; i++) {
        c = tolower(substr(str, i, 1))
        k = index("123456789abcdef", c)
        ret = ret * 16 + k
    }
    return ret
}
function getIP(str,ret){
    ret=hextodec(substr(str,index(str,":")-2,2));
    for (i=5; i>0; i-=2) {
        ret = ret"."hextodec(substr(str,i,2))
    }
    ret = ret":"hextodec(substr(str,index(str,":")+1,4))
    return ret
}
NR > 1 {{if(NR==2)print "Local - Remote";local=getIP($2);remote=getIP($3)}\
  {print local" - "remote}}' /proc/net/tcp

Resource: https://staaldraad.github.io/2017/12/20/netstat-without-netstat/

Show routing rules to reach a destination

ip route get <target>

For example, this will show the route that your system will take to get to 8.8.8.8 (Google’s DNS Server):

ip route get 8.8.8.8

Note that you can also run this command on a mac after you install the iproute2mac package via: brew install iproute2mac

Resource: https://systemoverlord.com/2020/03/22/security-101-virtual-private-networks-vpns.html


AWK

Remove leading and trailing whitespace

trimmed_string=$(echo "no_trimmed_string" | xargs)

Another alternative:

awk '{$1=$1;print}'

Resource: https://unix.stackexchange.com/questions/102008/how-do-i-trim-leading-and-trailing-whitespace-from-each-line-of-some-output

This assumes you have text coming from before the |.

| awk '{$1=""; print $0}'

Resource: https://stackoverflow.com/questions/2961635/using-awk-to-print-all-columns-from-the-nth-to-the-last/2961994

Split on / and print first column

awk -F '/' '{print $0}'

Use in an alias

This particular example will get the IPv6 address for eth0. The trick is remembering to escape the $2:

alias getIP="ifconfig eth0 |grep inet6 |grep global | head -n1 | awk '{print \$2}'"

Resource: https://stackoverflow.com/questions/20111063/bash-alias-command-with-both-single-and-double-quotes

Get string after hyphen

echo "foo-123" | awk -F- '{print $NF}'
# Will output 123

Resource: https://unix.stackexchange.com/questions/193482/get-last-part-of-string-after-hyphen


Get size of current directory

du -hsx

Find

Find files with a specific string and open in VSCode

find . -iname '*.hcl' -exec grep -rnw "prod" {} \; -maxdepth 2 \
  | awk -F : '{print $1}' | xargs code

Find and lint all YAML files

find . -type f -iname '*.yaml' -o -iname '*.yml' -o -iname '*.yml.tpl' \
  | grep -v "submodule_folder\|.terraform" | xargs -I {} cat {} \
  | python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)'

The only issue with this solution is it won’t list the files that it finds. If you would like this information as well, do the following instead:

files=$(find . -type f -iname '*.yaml' -o -iname '*.yml' -o -iname '*.yml.tpl' \
  | grep -v "submodule_folder\|.terraform"); \
  for file in $files; do \
    echo "Testing ${file}" && cat ${file} \
    | python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)'; done

Find and lint all JSON files

find . -type f -iname "*.json" -type f \
  | grep -v "submodule_folder\|.terraform" \
  | xargs -I{} jq . "{}" > /dev/null

If you want the names of each finding, you can adopt the work done for linting YAML above.

We also opt to use > /dev/null so that we only get files with errors.

Resources:

List all file extensions in CWD

find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

Resource: https://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

Grep output of find and save to file

find . -iname "thing to find" -exec grep -rnw "term to search" {} \; | tee filename

As an example, we can search for all python files recursively from the current directory and then grep for the word “git” in those files:

find . -iname "*.py" -exec grep -rnw "git" {} \; | tee git

Resource: https://unix.stackexchange.com/questions/131535/recursive-grep-vs-find-type-f-exec-grep-which-is-more-efficient-faster

Search for content in tar.gz files

One way:

find . -iname "*.tar.gz" -exec zgrep "STRING" {} \;

Another way:

find . -name '*.tar.gz' -print0 | xargs -0 zgrep "STRING"

Resource: https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files

Find and delete all hidden directories matching string

TARGET_DIR='.terragrunt-cache'
find . -iname "${TARGET_DIR}" -exec rm -rf {} +

Resources:

Find and replace all instances of a string

This will find all go files recursively from the current directory and replace all instances of “toreplace” with “replaced”.

Linux:

find . -iname "*.go" | xargs sed -i 's/toreplace/replaced/g'

OS X with gnu-sed (installed via brew install gnu-sed):

find . -iname "*.go" | xargs gsed -i 's/toreplace/replaced/g'

Resource: https://stackoverflow.com/questions/1585170/how-to-find-and-replace-all-occurrences-of-a-string-recursively-in-a-directory-t

Find all bin files and create single file with strings output

find . -iname "*.bin" -exec strings "{}" \; | tee combined_strings_output.txt

Resource: https://superuser.com/questions/566198/linux-command-find-files-and-run-command-on-them

Find directories with a certain name

find . -type d -name "thingtosearchfor*" -print 2>/dev/null

Resource: https://askubuntu.com/questions/153144/how-can-i-recursively-search-for-directory-names-with-a-particular-string-where

Find a specific file

find /etc -name "passwd"

Find all bash_histories and send their contents to a file

find . -iname ".bash_history" -exec cat {} \; | tee ~/bash_histories.txt

Resource: https://stackoverflow.com/questions/864316/how-to-pipe-list-of-files-returned-by-find-command-to-cat-to-view-all-the-files

List all directories the current user has access to

find . -maxdepth 1 -type d -perm -u=rx

Run ls -lart on all dirs current user can access

This will also display the folder at the top of the output because we’ve used +.

find . -maxdepth 1 -type d -perm -u=rx -exec ls -lart {} + 2>/dev/null

Find all bash_history files the current user can read

find . -maxdepth 1 -type d -perm -u=rx -print0 \
  | xargs -0 -I{} find '{}' -readable -iname '.bash_history' 2>/dev/null

Find all .ssh directories that the current user has access to

Clean:

find . -maxdepth 1 -type d -perm -u=rx -print0 \
  | xargs -0 -I{} find '{}' -readable -type d -iname '.ssh' 2>/dev/null

Not quite as clean:

find . -maxdepth 1 -type d -perm -u=rx \
  | xargs ls -lart 2>/dev/null |grep .ssh | grep "^.r..r..r"

Breaking this down:

find . -maxdepth 1 -type d -perm -u=rx - find all directories the current user has access to

| xargs ls -lart 2>/dev/null - send the output of the find command to

ls -lart and suppress the Permission denied messages

|grep .ssh | grep "^.r..r..r" - output all files with ssh in the name with read permissions for all users

Show all occurrences of users running ssh

find . -name .bash_history -print -exec grep ssh {} \;

Find all .go files recursively

find . -name "*.go"

Find bins with SUID or SGID set

Added output to /tmp/out.txt and backgrounded in the event you’re on a crappy shell.

for i in `locate -r "bin$"`; do
  find $i \( -perm -4000 -o -perm -2000 \) -type f 2>/dev/null; done \
  | tee /tmp/out.txt &

Resource: https://www.tecmint.com/how-to-find-files-with-suid-and-sgid-permissions-in-linux/

Search for secrets in config.xml files recursively

In this example, we are searching for patterns in config.xml files matching <secret, <password, or <credential:

find . -name "config.xml" -exec grep -iE "<secret|<password|<credential" {} + \
  | uniq -u

Find all csv files and delete them

find . -name "*.csv" -exec rm {} +

Delete all json files recursively

find . -name "*.json" -exec rm {} \;

Resource: https://www.cyberciti.biz/faq/linux-unix-how-to-find-and-remove-files/

Show all executable files

find . -maxdepth 1 -perm -111 -type f

Resource: https://stackoverflow.com/questions/7812324/finding-executable-files-using-ls-and-grep

Find world writeable directories

find /\(-perm -o w -perm -o x\) -type d 2>/dev/null

Resource: https://delta.navisec.io/privilege-escalation/

Remove any files in a directory

find "/path/to/dir" -type f -exec rm /path/to/dir/* {} \;

Resource: https://www.cyberciti.biz/faq/linux-unix-shell-check-if-directory-empty/

Find files not matching a pattern name

This will list all files in the current directory that do NOT have .template., .settings. in their name.

find . -type f -not -name "*.template.*" -not -name "*.settings.*"

Resource: https://alvinalexander.com/linux-unix/linux-find-files-not-matching-filename-pattern-dont-match/

Look for unique ip addresses with find output

find . -type f -not -name "*.template.*" -not -name "*.settings.*" \
  | xargs grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort -u

Resources:

Find files with multiple extensions

This particular example can be useful when looking for yaml files:

find . -name '*.yaml' -or -name '*.yml'

Resource: https://askubuntu.com/questions/1170231/how-to-use-multiple-criteria-for-find

Find all .js files and tar them

find . -iname '*.js' -print0 | xargs -0 tar -cf /tmp/js.tar

Resource: https://stackoverflow.com/questions/47493109/how-to-find-specific-file-types-and-tar-them

Find all .class and .jar files and tar them

mkdir /tmp/java_files && \
  find . -iname '*.jar' -or -name '*.class' -exec cp {} /tmp/java_files \;
tar -czvf /tmp/java_files.tar.gz /tmp/java_files

Resource:

Decompile all .class and .jar files in a directory

wget https://www.benf.org/other/cfr/cfr-0.151.jar
for file in java_files/*; do
  java -Xmx2048m -jar cfr-0.151.jar $file \
  --outputdir output --caseinsensitivefs true; done

Resource:

Find all js files and cp into a folder

find . -iname "*.js" -exec cp {} ./all_js \;

Find all yaml files

find . -iname '*.yaml' -o -iname "*.yml" -type f

-o: OR in find

Resources:

Recursively search files for string

This will identify all files that have “letsencrypt” in them:

find . -type f -exec grep -l "letsencrypt" {} +

Resource: https://www.cyberciti.biz/faq/howto-recursively-search-all-files-for-words/

Exclude directories found with find

find . -type f -name "*_peaks.bed" ! -path "./tmp/*" ! -path "./scripts/*"

Resource: https://stackoverflow.com/questions/14132210/use-find-command-but-exclude-files-in-two-directories

Using +

The + appends each selected file name to the end of the command.

Resource: https://stackoverflow.com/questions/6085156/using-semicolon-vs-plus-with-exec-in-find

Determine if file has been modified within the last n minutes

minutes=2
find . -mmin "-${minutes}" -type f -print

Resource: https://stackoverflow.com/questions/28337961/find-out-if-file-has-been-modified-within-the-last-2-minutes

Find files that match multiple patterns

find Documents \( -name "*.py" -o -name "*.html" \)

Resource: https://stackoverflow.com/questions/1133698/using-find-to-locate-files-that-match-one-of-multiple-patterns

Great find tutorial

TomNomNom breaks down the find command and makes it very approachable with this tweet: https://twitter.com/TomNomNom/status/1269263730992439296


Add terminal contents to a file

This is very useful if you have a long command that you want to modify quickly.

Type the following to open a file: CTRL + x CTRL + e

Resource: https://stackoverflow.com/questions/657130/fastest-ways-to-move-the-cursor-on-a-terminal-command-line

Get second column of file

cat output.csv | awk '{print $2}'

Make offline copy of a site

wget --mirror            \
     --convert-links     \
     --html-extension    \
     --wait=2            \
     -o log              \
     http://howisoldmybusiness.com

Resource: https://alvinalexander.com/linux-unix/how-to-make-offline-mirror-copy-website-with-wget/

Download entirety of a site

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com

Resource: https://stackoverflow.com/questions/11124292/why-does-wget-only-download-the-index-html-for-some-websites

Sed

Insert text after matched string

This will add retries=2 after [ssh_connection] in ansible.cfg:

sed -i '/\[ssh_connection\]/a retries=2' ansible.cfg

Resource: https://unix.stackexchange.com/questions/121161/how-to-insert-text-after-a-certain-string-in-a-file

Remove last line of a file

sed -i '$ d' foo.txt

Resource: https://stackoverflow.com/questions/4881930/remove-the-last-line-from-a-file-in-bash

Remove lines from file

This particular example will remove lines 5-10 and 12 from file.txt:

sed -e '5,10d;12d' file.txt

Resource: https://stackoverflow.com/questions/2112469/delete-specific-line-numbers-from-a-text-file-using-sed

Find and replace line in file

BSD:

sed -i '' 's/<orig pattern>/<modified>/' file.txt

GNU:

sed -i 's/<orig pattern>/<modified>/' file.txt

BSD and GNU:

sed -i".orig" 's/<orig pattern>/<modified>/' file.txt

Note: This last one will create a .orig file that you’ll probably want to delete later

Resources: https://stackoverflow.com/questions/7573368/in-place-edits-with-sed-on-os-x https://www.javaer101.com/en/article/13833496.html

Find and replace line assigned to variable in file

BSD:

new_str="so new"
sed -i '' "s/staticlineinfile/${new_str}/g" "file/to/change.conf"

GNU:

new_str="so new"
sed -i "s/staticlineinfile/${new_str}/g" "file/to/change.conf"

BSD and GNU:

new_str="so new"
sed -i".orig" "s/staticlineinfile/${new_str}/g" "file/to/change.conf"

Resources: https://stackoverflow.com/questions/4247068/sed-command-with-i-option-failing-on-mac-but-works-on-linux https://askubuntu.com/questions/76808/how-do-i-use-variables-in-a-sed-command

Add prefix to command output

yourcommand1 | sed  's/^/[prefix1]/'

For example, this will get all ip addresses associated with running containers, and add http:// to the front of each of them:

docker ps | awk '{ print $13 }' | grep -o -P "(.*:\d+)" | sed  's/^/http:\/\//'

Resource: https://stackoverflow.com/questions/42482477/add-prefix-in-bash-command-output

Remove nth line of a file

Remove the nth line of a file:

sed 'Nd' file

You have to use -i to modify the file.

For example, to remove the second line from somefile.txt and overwrite it:

sed -i '2d' somefile.txt

Resource: http://www.aodba.com/use-sed-command-delete-lines-linux/

Change shell with sed

This will change the shell from /usr/sbin/nologin to /bin/bash for the www-data user:

sed -i '/www-data/s/\/usr\/sbin\/nologin/\/bin\/bash/g' /etc/passwd

Resource: https://superuser.com/questions/558394/replacing-bin-bash-with-bin-false-in-etc-passwd-file

Get nth line of a file

sed 'NUMq;d' file

For example, to get the second line from somefile.txt:

sed '2q;d' somefile.txt

Resource: https://stackoverflow.com/questions/6022384/bash-tool-to-get-nth-line-from-a-file

Get range of lines from a file

This example will get lines 747-749 from file:

sed -n -e '747,749p' file

Resource: https://unix.stackexchange.com/questions/138398/how-to-get-lines-10-to-100-from-a-200-line-file-into-a-new-file

Insert text at beginning of every line of a file

This example inserts bla at the beginning of every line of filename:

sed 's/^/bla/' filename

Resource: https://stackoverflow.com/questions/4080526/using-sed-to-insert-text-at-the-beginning-of-each-line

Add comma to end of every line of a file but the last

sed '$!s/$/,/' file

Resource: https://stackoverflow.com/questions/35021524/how-can-i-add-a-comma-at-the-end-of-every-line-except-the-last-line/35021663

Avoid escaping slashes

If you have slashes in your replacement string, you can use | instead of / to avoid escaping those slashes. For example:

sed "s|regex|replace|" file.txt

Resource: https://stackoverflow.com/questions/40714970/escaping-forward-slashes-in-sed-command/40715028

Capture control group

This will grab the number (51265) from the string:

echo 'BLA: 51265 - BLA - BLA' | sed -r "s/.*([0-9]{5}).*/\1/"

Resource: https://stackoverflow.com/questions/11650940/sed-how-to-do-regex-groups-using-sed

Replace one character with two

echo "asdlksad ~ adlkajsd ~ 12345" | sed 's/~/~\n/g'

Resource: https://stackoverflow.com/questions/18365482/how-to-replace-one-character-with-two-characters-using-tr

Replace specific lines of file

Print output without changing file:

sed '9,11s/\/\///g' config.js

Make the changes to the file: BSD:

sed -i '' '9,11s/\/\///g' config.js

Resource: https://stackoverflow.com/questions/16202900/using-sed-between-specific-lines-only


Merge two folders

rsync -avh --progress /path/to/source/ /path/to/destination

For example, if you have a recipes folder in /Volumes/stuff/recipes that you want to sync with your local recipes folder:

rsync -avh --progress ~/recipes /Volumes/stuff

Resources: https://unix.stackexchange.com/questions/149965/how-to-copy-merge-two-directories https://unix.stackexchange.com/questions/149965/how-to-copy-merge-two-directories##:~:text=If%20you%20want%20to%20move,are%20on%20the%20same%20filesystem.

Conditional based on output of a command

if echo $(command_to_run) | grep -q "thing to find in output of command"
    then echo "Yay, we found things!"
    else "We didn't find the thing we wanted to find!"; exit 1
fi

Resource: https://stackoverflow.com/questions/3943854/grep-in-if-statement

Setuid

chmod u+s <filename>

Resource: https://www.liquidweb.com/kb/how-do-i-set-up-setuid-setgid-and-sticky-bits-on-linux/

Remove files without an extension

In this case, we’ll rm every file that doesn’t end with the apk file extension:

find . -type f ! -name "*.apk" -exec rm {} \;

Resource: https://stackoverflow.com/questions/41935440/how-to-remove-files-without-certain-extension

Assign output to variable

pid=$(adb shell ps -A | grep processname | awk -F ' ' '{print $2}')

Resource: https://www.cyberciti.biz/faq/unix-linux-bsd-appleosx-bash-assign-variable-command-output/

Get first two characters of a string

Use head -c 2

Example that will get the first two characters of the apk found in /tmp:

find /tmp -name "*.apk*" | head -c 2

Resource: https://stackoverflow.com/questions/1405611/how-to-extract-the-first-two-characters-of-a-string-in-shell-scripting

Break long command down

This is useful if you have a long command and want to be able to lay it out to analyze. Assign the parameter to a variable and add \ for line breaks.

For example:

command="https://google.com/endpoint \
?fields=field1,field2 \
field3&field4=abc&field5=1234xyz"
curl $command

Resource: https://stackoverflow.com/questions/32341091/multiline-curl-command

Get today’s date

today=`date '+DATE:%m%d%y' | cut -d: -f2`

Get the time

time=$(date '+TIME:%H%M%S' | cut -d: -f2)

Get the year

year=$(date | cut -d' ' -f 6)

Get the month and day

month_day=$(date '+DATE:%m%d' | cut -d: -f 2)

Create folder with date and time

mkdir `echo $(date +"%c") | tr -s ' ' | tr ' ' '_'`

Resource: https://stackoverflow.com/questions/13799789/expansion-of-variables-inside-single-quotes-in-a-command-in-bash

Check if today is a weekend

if [[ $(date +%u) -gt 5 ]]; then
    echo 'Sorry, you cannot run this program on the weekend.'
    exit
fi

Resource: https://stackoverflow.com/questions/3490032/how-to-check-if-today-is-a-weekend-in-bash


Systemd

Run script on startup

  1. Create systemd service file called <service name>.service and set the path to where you plan to have the script on disk:

    [Unit]
    After=network.service
    
    [Service]
    ExecStart=/usr/local/bin/<name of script>.sh
    
    [Install]
    WantedBy=default.target
    
  2. Create the script to run on startup (call it whatever you want).

  3. Move the files into place and set the permissions:

    sudo mv <name of script>.sh /usr/local/bin/<name of script>.sh
    sudo mv <service name>.service /etc/systemd/system/<service name>.service
    sudo chmod 744 /usr/local/bin/<name of script>.sh
    sudo chmod 664 /etc/systemd/system/<service name>.service
    
  4. Enable the service:

    sudo systemctl daemon-reload
    sudo systemctl enable <service name>.service
    

Resource: https://linuxconfig.org/how-to-run-script-on-startup-on-ubuntu-20-04-focal-fossa-server-desktop

Show all enabled services

systemctl list-unit-files | grep enabled

Resource: https://askubuntu.com/questions/795226/how-to-list-all-enabled-services-from-systemctl

Get status of a service on the system

systemctl $SERVICE_NAME status

Get status of all user services on the system

XDG_RUNTIME_DIR="/run/user/$UID" systemctl --user status

Resource: https://unix.stackexchange.com/questions/615917/failed-to-get-d-bus-connection-connection-refused

Get status of a specific user service

XDG_RUNTIME_DIR="/run/user/$UID" systemctl --user status file.service

Set env var in systemd service

Create the service like so: /etc/systemd/system/myservice.service:

[Service]
Environment="MY_ENV=abc"
Environment="ANOTHER_ENV=zyx"
ExecStart=/opt/myservice -env=true

Run these commands to update and restart the service:

# Reload units
systemctl daemon-reload
# Start the service
systemctl start myservice
# Confirm everything works as expected
systemctl status myservice

Resource: https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service

Tail -f systemd

journalctl --follow _SYSTEMD_UNIT=gunicorn.service

Resource: https://thierry.marianne.io/view/0ae42ee4d7e86763a0de1765b8ae4ff6


Show what’s listening on a port

sudo lsof -i -P -n | grep LISTEN

Resource: https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/

Exclude file from zip

This will exclude the .git folder, the README.md file and the Makefile:

zip -r output.zip . -x '*.git*' -x README.md -x Makefile

Resources:

Trim whitespace from a bash var

echo "   lol  " | xargs

Resource: https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable

Get open ports

This is one option that has some nice and concise output:

sudo lsof -i -P -n |grep LISTEN

Here’s another for newer versions of linux with a bit more info:

sudo ss -tulwn

Resource: https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/


Handy Shortcuts

This will get the last run command:

!!

So for example, if you run this command:

ls -lart

and then run this command:

!! |grep bash_history

You will be running the equivalent of:

ls -lart |grep bash_history

This will get the second command run in a line of commands:

!:1

For example, if you run this command:

file $(ldd /bin/ls | grep libc.so | cut -d' ' -f3) -L

and then type this in:

!:1

you will get the following command:

$(ldd /bin/ls | grep libc.so | cut -d' ' -f3)

Resource:


Make

@echo Things I want to say!

Resource: https://www.gnu.org/software/make/manual/html_node/Echoing.html

Variable declaration examples

Normal variable assignment:

BOOL_VALUE := false
@echo $(BOOL_VALUE)

Run bash command and assign output to $(BUILD_BUCKET):

BUILD_BUCKET := $(shell terraform output build_bucket | tr -d \")
@echo $(BUILD_BUCKET)

Resources:

Run several actions in a row

# Create phony target to avoid conflict with a file of the
# same name and to improve performance
.PHONY: all
all: build test

# Create phony target to avoid conflict with a file of the
# same name and to improve performance
.PHONY: build
build:
 go build

# Create phony target to avoid conflict with
# a file of the same name and to improve performance
.PHONY: test
test:
 go test -v

To run everything, simply run:

make

Resources:

Bash command substitution

clear_build_bucket:
    - bucket=my-bucket; \
    aws s3api delete-objects --bucket $${bucket} \
      --delete "$$(aws s3api list-object-versions \
      --bucket $${bucket} \
      --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"; \
    aws s3api delete-objects --bucket $${bucket} \
      --delete "$$(aws s3api list-object-versions \
        --bucket $${bucket} --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

Resource:

Use bash for loop

JSON_FILES := $(shell find . -type f -iname "*.json")
lint:
  for json in $(JSON_FILES); do echo "JSON file found: $${json}"; done

Export env var

This example is part of a terragrunt snippet in which the folder is the region associated with the Makefile, which is then used for the AWS_DEFAULT_REGION value.

REGION := $(shell ls | grep us)
export AWS_DEFAULT_REGION ?= $(REGION)

Resource: https://github.com/nzoschke/gofaas/blob/master/Makefile

Loop over array

SHELL=/bin/bash
TF_FILES = .terraform .terragrunt-cache .terraform.lock.hcl

.PHONY: destroy
destroy:
 - terragrunt run-all destroy --terragrunt-non-interactive -lock=false --terragrunt-source-update
 # Clear terragrunt and terraform files
 for file in $(TF_FILES); do find . -name $$file -prune -exec rm -rf {} \; ; done

Resource: https://stackoverflow.com/questions/52282549/for-each-on-target-of-makefile-variable

Run command and continue even w/ error

In this example, the find commands to clear the terragrunt cache will run even if the run-all destroy command has an error:

.PHONY: destroy
destroy:
    - terragrunt run-all destroy --terragrunt-non-interactive
    # Clear terragrunt cache
    find . -type d -name ".terragrunt-cache" -prune -exec rm -rf {} \;
    find . -type d -name ".terraform" -prune -exec rm -rf {} \;

Set default value

Observe $MYVAR in Makefile:

MYVAR ?= default

.PHONY: all

all:
    echo $(MYVAR)

Then:

$ make
echo default
default
$ export MYVAR=notDefault
$ make
echo notDefault
notDefault

Resource: https://stackoverflow.com/questions/53369903/use-environment-variable-if-set-otherwise-use-default-value-in-makefile

If then else

.PHONY: all
all:
ifeq ($(REGION),)
 echo "REGION not set, exiting!"
 exit 1
else
all: init plan apply
endif

Resources:

Run Makefile from another directory

make -C /path/to/makefile

For example, let’s say there’s a Makefile in ${FOLDER}:

FOLDER=/opt/scripts/installer_of_things
make -C "${FOLDER}"

Resource: https://superuser.com/questions/370575/how-to-run-make-file-from-any-directory


Get unique values from bash array

echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '

Resource: https://stackoverflow.com/questions/13648410/how-can-i-get-unique-values-from-an-array-in-bash

Get public IP

Using curl:

# Will get your IPv4 address:
curl ifconfig.me

Using wget:

# Will get your IPv6 address (if applicable):
wget -O - ifconfig.co 2>/dev/null

Show hidden files with tree

tree -a

Expanded view of running processes

Run this to get a sense of processes and their parents

ps axjf

Download file to specific dir with wget

install_dir='/home/ubuntu/install_here'
mkdir $install_dir
wget <file to download> -P $install_dir

Resource: https://www.tecmint.com/wget-download-file-to-specific-directory/

Get newest file matching a string

This will find all files that start with vncpass in /home/ubuntu/vnc and get the most recent one:

ls -t /home/ubuntu/vnc/vncpass* | head -1

Resource: https://stackoverflow.com/questions/5885934/bash-function-to-find-newest-file-matching-pattern

Get all files except ‘.’ and ‘../’

ls -A

Resource: https://stackoverflow.com/questions/22407480/command-to-list-all-files-except-dot-and-dot-dot

Convert cert to single line

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}'  ca.pem

Resource: https://serverfault.com/questions/466683/can-an-ssl-certificate-be-on-a-single-line-in-a-file-no-line-breaks

Get certificate portion of pem file

openssl x509 -in cert.pem -outform der | base64 -w 64

For a full example, run this to generate a self-signed cert:

openssl req \
    -x509 \
    -nodes \
    -newkey rsa:2048 \
    -keyout server.key \
    -out server.crt \
    -days 3650 \
    -subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=yourhostname.com"

and then run:

openssl x509 -in server.crt -outform der | base64 -w 64

Resource: https://stackoverflow.com/questions/40366409/get-just-the-certificate-portion-from-an-openssl-pem-file

Curl with client certificate

This will help you to avoid using the -k option if you’re using a self signed certificate (which you probably shouldn’t unless you’re doing some testing or have another good reason).

Get the client certificate (client.pem):

echo quit | openssl s_client -showcerts -servername yourhostname.com \
  -connect yourhostname.com:443 > cacert.pem

Use it in your curl command:

curl --cacert cacert.pem -X POST https://yoursite.com/endpoint -d 'data=hi'

Resources:

Prevent curl output from being cut off

If curl output is cut off, put the variable in double quotes.

GOOGLE=$(curl -s -i https://google.com)
echo $GOOGLE ## fail
echo "$GOOGLE" ## success

Resource: https://stackoverflow.com/questions/53892741/missing-output-when-storing-curl-output-as-bash-variable

Diff output of two commands

This will diff the output of a find command with a bunch of not statements against a find command that pipes it output to grep, which in turn uses a perl regex to try an find files that match a bunch of numbers.json.

diff <(find . -type f -not -name "*.template.*" \
  -not -name "*.settings.*") <(find . -type f -name "*.json*" \
  | grep -P '.*?\d+.json')

Resources:

Use json for POST body

curl -k -s -i -X POST --compressed \
    -H "Content-Type: application/json" \
    -d @input.json \
    https://target.com/endpoint

Diff two directories

diff -qr directory-1 directory-2

If you want to exclude criteria from the output, you can modify this example:

diff -qr goutils goproject | grep -v -E "(*utils.go|*utils_test.go|.git)"

Resource: https://www.tecmint.com/compare-find-difference-between-two-directories-in-linux/

Diff files and dirs - show diff and unique content

diff -x '.git' -bur directory-1/ directory-2/ | grep -- '---\|Only'

-x: exclude -b: ignore whitespace -u: unified context (3 lines before and after) -r: recursive

Resources:

Get PID’s without ps

find /proc -mindepth 2 -maxdepth 2 -name exe -exec ls -lh {} \; 2>/dev/null

Resource: https://unix.stackexchange.com/questions/115927/find-the-pids-of-all-threads-of-a-process-without-ps-or-pidof


Find PID of process using port

sudo ss -lptn 'sport = :80'

Resource https://unix.stackexchange.com/questions/106561/finding-the-pid-of-the-process-using-a-specific-port


Netstat without netstat

Copy pasta ftw:

awk 'function hextodec(str,ret,n,i,k,c){
    ret = 0
    n = length(str)
    for (i = 1; i <= n; i++) {
        c = tolower(substr(str, i, 1))
        k = index("123456789abcdef", c)
        ret = ret * 16 + k
    }
    return ret
}
function getIP(str,ret){
    ret=hextodec(substr(str,index(str,":")-2,2));
    for (i=5; i>0; i-=2) {
        ret = ret"."hextodec(substr(str,i,2))
    }
    ret = ret":"hextodec(substr(str,index(str,":")+1,4))
    return ret
}
NR > 1 {{if(NR==2)print "Local - Remote";local=getIP($2);
remote=getIP($3)}{print local" - "remote}}' /proc/net/tcp

Resources: https://staaldraad.github.io/2017/12/20/netstat-without-netstat/

Show lines that match in two files

grep -Ff file1.txt file2.txt

Resource: https://unix.stackexchange.com/questions/125155/compare-two-files-for-matching-lines-and-store-positive-results

Show files that have part of a string in them

grep 'string_to_look_for' . -R 2>/dev/null

Resource: https://superuser.com/questions/614526/finding-files-which-contain-a-certain-string-using-find-1-and-grep-1/614538

Cosmetic Color Output

This is an example of how you might consider different colors for output in a bash script:

RED="\033[01;31m"      # Issues/Errors
GREEN="\033[01;32m"    # Success
BLUE="\033[01;34m"     # Heading
YELLOW='\033[0;33m'    # Informational
RESET="\033[00m"       # Normal

echo -e "${BLUE}Running apt-get update, please wait...${RESET}"
apt-get update -y

Resource: https://github.com/TH3xACE/OFFPORT_KILLER/blob/master/OFFPORT_KILLER.sh

CLI input

Boolean Flag Example

sample.sh:

FORCE=false

while getopts ':f' option; do
    case "${option}" in
        'f') FORCE=true ;;
    esac
done
shift $(( OPTIND - 1 ))

main() {
    if [[ $FORCE = true ]]; then
        echo "Option f was supplied on the command line"
    else
        echo "Option f was not supplied on the command line"
    fi
}

main

Example Usage:

bash sample.sh -f

Resources:

Get FQDN

hostname -f

Resource: https://www.tecmint.com/hostname-command-examples-for-linux/##:~:text=To%20view%20the%20name%20of,the%20FQDNs%20of%20the%20machine.&text=To%20display%20the%20alias%20name,%2C%20use%20the%20%2Da%20flag.

Create file with multiple lines

# set for the example below
BLA='hi'

cat <<EOT >> greetings.txt
line 1
"${BLA}"
"$(whoami)"
line 4
EOT

Resource: https://unix.stackexchange.com/questions/77277/how-to-append-multiple-lines-to-a-file

Run command on every line of a file

This example will resolve a list of hosts to their ip addresses:

getent hosts $(<host_list.txt)
dig + short $(<host_list.txt)

Resource: https://stackoverflow.com/questions/13939038/how-do-you-run-a-command-for-each-line-of-a-file

Show long line of ps

If there’s a process that you can’t read, you can run this command to see the whole thing:

ps -ef | more

Resource: https://unix.stackexchange.com/questions/229541/view-full-commands-in-ps-output

Copy multiple files

cp {file1,file2} dest

Resource: https://stackoverflow.com/questions/24206349/copy-multiple-files-from-one-directory-to-another-from-linux-shell

Get active network interface

This script will work on Mac OS or Linux:

ifconfig | awk '/UP/{print $1}' | grep 'eth0:\|en0:' | sed 's/://'

Resources:

Check if script is running on a Mac

if [[ `uname` == 'Darwin' ]]; then
  echo "I am running on MacOS"
else
  echo "I am running on something."
fi

Use placeholder with xargs

If instead of the echo you had a long command that you wanted to pass to sed, this is how you would do it:

echo 'REPLACED' | xargs -I {} sed -i "s/REPLACE_ME/{}/" file.txt

The {} serves as the placeholder for what we’re piping in.

Resource: https://www.shogan.co.uk/how-tos/using-placeholder-templates-with-xargs-in-the-pipeline/

Replace IP Address in file

pub_ip=$(dig @resolver1.opendns.com A myip.opendns.com +short -4)
sed -i -r "s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/$pub_ip/" file.txt

Resource: https://stackoverflow.com/questions/5277156/using-sed-to-search-and-replace-an-ip-address-in-a-file

Find IP addresses in file

This will find invalid IP addresses if those are present:

grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" file.txt

To match only valid ip addresses:

# <!-- markdownlint-disable-next-line -->
grep -oE "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" file.txt

Resources: https://unix.stackexchange.com/questions/296596/how-to-check-if-any-ip-address-is-present-in-a-file-using-shell-scripting match only valid IP addresses

Check if string matches regex

[[ $date =~ ^regex$ ]] && echo "matched" || echo "did not match"

Resource: https://stackoverflow.com/questions/21112707/check-if-a-string-matches-a-regex-in-bash-script/21112809

Check if variable is empty

if [[ -z "$var" ]]
then
      echo "\$var is empty"
else
      echo "\$var is NOT empty"
fi

For example:

if [[ -z $(aws s3 ls |grep -i tf-remote-state) ]]; then
echo 'Remote state not configured! Configuring...';
else echo 'Remote state already configured, moving on...'; fi

Resource: https://www.cyberciti.biz/faq/unix-linux-bash-script-check-if-variable-is-empty/

Check if env var is set

if [[ -z "${DEPLOY_ENV}" ]]; then
  MY_SCRIPT_VARIABLE="Some default value because DEPLOY_ENV is undefined"
else
  MY_SCRIPT_VARIABLE="${DEPLOY_ENV}"
fi

Wait for process to finish execution

This particular example will loop until ansible is finished running. If it is not running at all, it will skip the loop entirely.

while /usr/bin/pgrep ansible >/dev/null; do
        echo "ansible is running"
        sleep 1
done
echo 'ansible is not running!'

Resource: https://stackoverflow.com/questions/38666191/while-loop-in-bash-that-uses-pgrep-to-check-if-service-exists

Regex capture group

users=(bob jane clyde)
regex=".*>\s(.*)\s<.*"

for user in "${users[@]}"; do
  line=$(grep "$user favorite food" userdata.txt)
  if [[ $line =~ $regex ]]; then
    echo "$user has a favorite food and it is: ${BASH_REMATCH[1]}"
  fi
done

Resource: https://stackoverflow.com/questions/1891797/capturing-groups-from-a-grep-regex

Alias an export

alias sroot="export SROOT="\$PWD""
alias drumit="cd \$SROOT/abc/def/drumit"

Resource: https://stackoverflow.com/questions/7747673/how-to-alias-an-export-in-bash

Remove an alias

 unalias gs

Resource: https://askubuntu.com/questions/325368/how-do-i-remove-an-alias

Comma separated string to array

string="bla1,bla2"
IFS=', ' read -r -a blas <<< "$string"
echo ${blas[0]}

Resource: https://stackoverflow.com/questions/10586153/how-to-split-a-string-into-an-array-in-bash

Append value to array

ARRAY=()
## Append foo to $ARRAY:
ARRAY+=('foo')
## Append bar to $ARRAY:
ARRAY+=('bar')

Resource: https://stackoverflow.com/questions/1951506/add-a-new-element-to-an-array-without-specifying-the-index-in-bash

Pass array as value to function

f(){
  b=$1
  shift
  a=("$@")

  for i in "${a[@]}"
  do
    echo $i
  done
  echo $b
}

a=(value1 value2 value3)
b=STANDALONESTRING

f "$b" "${a[@]}"

Resource: https://stackoverflow.com/questions/16461656/how-to-pass-array-as-an-argument-to-a-function-in-bash

Debug bash script

Run this command to get debug output from a bash script:

bash -x name_of_script.sh

Resource: https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html

Exit bash script

exit 0  ## Exits the script completely.  $? contains 0 (success).
exit 1  ## Exits the script completely.  $? contains 1 (failure).

Resource: https://stackoverflow.com/questions/4419952/difference-between-return-and-exit-in-bash-functions

URL Encode a string

This is a lifesaver when working with curl.

urlencode() {

    old_lc_collate=$LC_COLLATE
    LC_COLLATE=C

    local length="${##1}"
    for (( i = 0; i < length; i++ )); do
        local c="${1:$i:1}"
        case $c in
            [a-zA-Z0-9.~_-]) printf '%s' "$c" ;;
            *) printf '%%%02X' "'$c" ;;
        esac
    done

    LC_COLLATE=$old_lc_collate
}
urlencode "a string yay"

Resource: https://gist.github.com/cdown/1163649

Get Public IP address and Geolocation

curl -s https://ipapi.co/$(curl -s ifconfig.me)/json | jq

Resources: https://ipapi.co/##api https://www.tecmint.com/find-linux-server-geographic-location/


Multiline string to a file

Everything in between the EOM delimiters is the string that will go into the file (trust_policy.json in this case).

cat > trust_policy.json <<- EOM
{
  "Version":"2012-10-17",
  "Statement":[
   {
      "Effect":"Allow",
      "Principal":{
      "Service":"ec2.amazonaws.com"
    },
    "Action":"sts:AssumeRole"
    }
  ]
}
EOM

Resource: https://stackoverflow.com/questions/23929235/multi-line-string-with-extra-space-preserved-indentation


Set default value for var

test.sh:

FOO=${1:-bar}
DOG=${2:-yes}

echo ${FOO}
echo ${DOG}

If you run it this way, the output will be will be reflected:

bash test.sh

## Output:
bar
yes

If you run it with arguments, the output will change:

bash test.sh chicken no

## Output:
chicken
no

Resource: https://stackoverflow.com/questions/16319720/bash-command-line-arguments-replacing-defaults-for-variables/16836338

View all arguments submitted via CLI

echo $@

Resource: https://linuxconfig.org/how-do-i-print-all-arguments-submitted-on-a-command-line-from-a-bash-script

Check spelling in files

aspell check --mode=markdown --lang=en README.md

Resource: https://www.manuel-strehl.de/check_markdown_spelling_with_aspell

Run bash commands in script after chroot

chroot /home/ubuntu/chroot/codebase /bin/bash <<"EOT"
cd /tmp/so
ls -l
echo $$
EOT

Delete files with specified extension from directory

DIR=~/Downloads
EXT=jar

ls "${DIR}" | grep "${EXT}" | xargs -I {} rm "${DIR}/{}"

Resource: https://stackoverflow.com/questions/51305706/shell-script-that-does-chroot-and-execute-commands-in-chroot/51312156


HEREDOC with bash variables

kubectl exec -it -f helm_file.yaml -- chroot /mnt /bin/bash <<EOF
./kubeletmein generate
./kubectl --kubeconfig ./kubeconfig.yaml get pods -n ${TARGET_NAMESPACE}
EOF

Resource: https://unix.stackexchange.com/questions/138418/passing-a-variable-to-a-bash-script-that-uses-eof-and-considers-the-variable-a


Bind non priv user to a priv port

setcap 'cap_net_bind_service=+ep' /path/to/program

Resource: https://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-on-linux


Test Web Socket

npm install -g wscat
TARGET="wss://ws-feed.gdax.com"
wscat -c "${TARGET}"

Resource: https://stackoverflow.com/questions/47860689/how-to-read-websocket-response-in-linux-shell


Convert string to lowercase

A few options:

# The strip() at the end removes any trailing newlines
echo "ABC" | python3 -c "print(open(0).read().lower().strip())"
echo "ABC" | sed 's/./\L&/g'
export a="ABC" | echo "${a,,}"

Resource: https://stackoverflow.com/questions/2264428/how-to-convert-a-string-to-lower-case-in-bash

String comparison

#!/bin/bash

read -p "Enter first string: " VAR1
read -p "Enter second string: " VAR2

if [[ "$VAR1" == "$VAR2" ]]; then
    echo "Strings are equal."
else
    echo "Strings are not equal."
fi

Resource: https://linuxize.com/post/how-to-compare-strings-in-bash/

Get architecture string

dpkg --print-architecture

Resource: https://unix.stackexchange.com/questions/180726/easy-command-line-method-to-determine-specific-arm-architecture-string

Reinstall package in /bin

If you accidentally delete a package in /bin, this is how you get it back.

MISSING_PACKAGE=coreutils ## change this based on your situation
cd && apt-get download "${MISSING_PACKAGE}"
dpkg -i *.deb

Resource: https://askubuntu.com/questions/906674/accidentally-removed-bin-how-do-i-restore-it

Get filename and extension from path

filepath='/path/to/file.txt'
extension="${filepath###*.}"
filename="$(basename ${filepath})"

Resources: https://stackoverflow.com/questions/965053/extract-filename-and-extension-in-bash https://www.cyberciti.biz/faq/bash-get-filename-from-given-path-on-linux-or-unix/

Fix centos8 dnf

dnf -y --disablerepo '*' --enablerepo=extras swap centos-linux-repos

Resource: https://stackoverflow.com/questions/70963985/error-failed-to-download-metadata-for-repo-appstream-cannot-prepare-internal


## Get all functions into newline separated string
fstr="$(declare -F \
    | awk '{print $NF}' \
    | sort \
    | egrep -v '^_')"

## This entry also covers converting a multiline string
## to an array:
SAVEIFS=$IFS   ## Save current IFS (Internal Field Separator)
IFS=$'\n'      ## Change IFS to newline char
funcs=($fstr)  ## split `fstr` into an array by the same name
IFS=$SAVEIFS   ## Restore original IFS

echo "This file has the following functions:"
for f in "${funcs[@]}"; do
  echo "${f}"
done

Resources:

Upgrade ubuntu install from LTS

Change the following line in /etc/update-manager/release-upgrades from:

Prompt=lts

to:

Prompt=normal

and run the following to kick the process off:

sudo do-release-upgrade

Resource: https://www.omgubuntu.co.uk/2020/10/how-to-upgrade-to-ubuntu-20-10-from-ubuntu-20-04

Get substring after delimiter

This example will get everything after the first hyphen:

var=one-two-three
cut -d '-' -f2- <<< $var # outputs: two-three
var=four-five
cut -d '-' -f2- <<< $var # outputs: five

Resource: https://linuxhint.com/bash-substring-after-character/

Get OS and Arch

export OS="$(uname | python3 -c 'print(open(0).read().lower().strip())')"
export ARCH="$(uname -a | awk '{ print $NF }')"

Set Static IP on Ubuntu

Open as root /etc/netplan/50-cloud-init.yaml (or whatever file is present in /etc/netplan). Replace the contents of the file with this yaml after updating the various IP addresses for your needs:

---
network:
  ethernets:
    eth0:
      dhcp4: false
      addresses: [192.168.20.124/24]
      routes:
        - to: 0.0.0.0/0
          via: 192.168.20.1
          metric: 100
      nameservers:
        addresses:
          - 1.1.1.1
          - 8.8.8.8
          - 192.168.20.1
  version: 2

Apply it:

netplan --debug apply

The --debug flag provides verbose output - feel free to omit it.

Add the following to /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg:

network: {config: disabled}

Resource: https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-22-04-jammy-jellyfish-desktop-server


Get age of system

sudo dumpe2fs /dev/sda1 | grep 'Filesystem created:'

Resource: https://askubuntu.com/questions/1352/how-can-i-tell-what-date-ubuntu-was-installed

Run command in every directory below the current one

COMMAND='terragrunt init --reconfigure'
for d in ./*/ ; do (cd "$d" && $COMMAND); done

Get architecture

raw_arch=$(uname -m)
case $raw_arch in
    x86_64)
        ARCH="amd64"
        ;;
    arm64)
        ARCH="arm64"
        ;;
    *)
        echo "Unsupported architecture: $raw_arch"
        exit 1
        ;;
esac
export ARCH

Set terminal type

The export TERM=xterm command sets the terminal type to xterm, a common terminal emulator used on Unix-based systems. This can ensure consistent behavior for command-line applications and is a good practice when using a terminal emulator.

export TERM=xterm # resolves "No value for $TERM and no -T specified"