Kubectl
List all applications and services
kubectl get all
List all pods
kubectl get pods
List all containers in all pods
kubectl get pods -o='custom-columns=NameSpace:.metadata.namespace,NAME:.metadata.name,CONTAINERS:.spec.containers[*].name'
List all containers in a pod
kubectl get pods $POD_NAME -o='custom-columns=NameSpace:.metadata.namespace,NAME:.metadata.name,CONTAINERS:.spec.containers[*].name'
Resource: https://serverfault.com/questions/873490/how-to-list-all-containers-in-kubernetes
List pods in a namespace
kubectl get pods -n <namespace>
Get all pods running in all namespaces
kubectl get pods --all-namespaces
Get all container images
kubectl get pods --all-namespaces -o=jsonpath="{..image}
Get all container images filtered by pod label
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=<name>
For example, for a label with the app name 'nginx' created with this tutorial:
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx
Get Pod IP Address
kubectl get pods -l app=<app name> -o yaml |grep podIP
Delete a pod
kubectl delete pod <pod name>
Resource: https://www.fairwinds.com/blog/how-to-create-view-and-destroy-a-pod-in-kubernetes
List all nodes
kubectl get nodes
Get more information about a node
This will have things like the pods that are running on a node.
kubectl describe nodes <node name>
Get information about all deployments
kubectl describe deployments
Get information about a deployment
kubectl describe deployment nginx-deployment
Get shell to first container in a pod
kubectl exec -it $pod_name -- bash
Get shell to specific container in a pod
kubectl exec -it $pod_name --container $container_name -- sh
Secrets
Get a list of secrets
kubectl get secrets
Describe a secret
kubectl describe secret $SECRET_NAME
View a secret
kubectl get secret $SECRET_NAME -o json | jq .
You can also opt to output as yaml as well:
kubectl get secret $SECRET_NAME -o yaml
If a secret is a json blob with multiple key/value pairs (like a kubernetes.io/service-account-token type for example), you can get the associated value from one of the keys like so:
kubectl get secret $SECRET_NAME -o jsonpath='{.data.keyname}' | base64 -d
Resource: https://kubernetes.io/docs/concepts/configuration/secret/
List all services
kubectl get services
Get names of services
kubectl get services --sort-by=.metadata.name
Resource: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
Get more information about a pod
kubectl describe pod $POD_NAME
Alternatively, you can use:
kubectl describe pods/$POD_NAME
Resources:
https://stackoverflow.com/questions/34848422/how-to-debug-imagepullbackoff
https://github.com/cloudnativelabs/kube-router/issues/711
Delete an application
Find the deployment and service beforehand:
kubectl delete deployment.apps/<name> service/<name>
For example, for a service and deployment called 'app':
kubectl delete deployment.apps/app service/app
Alternatively, you could just run this in the directory with all of the files:
kubectl delete -k ./
Resource: https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
Get external IP addresses of all nodes
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'
Resource: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
List all namespaces
kubectl get namespace
Create new namespace
NAMESPACE_NAME=blablabla
kubectl create namespace $NAMESPACE_NAME
Resource: https://jhooq.com/helm-chart-wordpress-installation/
Delete namespace
NAMESPACE_NAME=blablabla
kubectl delete namespace $NAMESPACE_NAME
Show Persistent Volumes
kubctl get pvc
Delete Persistent Volume
PVC_NAME=somepvc
kubectl delete pvc $PVC_NAME
Troubleshooting
This will sort the output for you as well based on when something was created:
kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'
Alternatively, you can just run:
kubectl get events
Resources:
https://serverfault.com/questions/728727/kubernetes-stuck-on-containercreating
https://stackoverflow.com/questions/36377784/pod-in-kubernetes-always-in-pending-state
Copy file from pod to system
kubectl cp $POD_NAME:/run/secrets/kubernetes.io/serviceaccount .
Copy file from system to pod
kubectl cp file $POD_NAME:/tmp
Get all clusters
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
Resource: https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/
List all service accounts
kubectl get serviceaccounts
List all clusterroles
kubectl get clusterrole
Get yaml file from running pod
kubectl get po $POD_NAME -o yaml | tee file.yaml
Resource: https://stackoverflow.com/questions/43941772/get-yaml-for-deployed-kubernetes-services
Check for insecure kubelet API access
From a pod
curl -k https://localhost:10250/pods
Remotely
curl -k https://<target system running kub>:10250/pods
Resource: https://sysdig.com/blog/kubernetes-security-kubelet-etcd/
Kubernetes config file location
env |grep KUBECONFIG
View config
kubectl config view
Use config file
# run before setting the env var
kubectl config view
export KUBECONFIG=/path/to/config/file
# run view config again to see the changes
kubectl config view
You can also run it like this if you don't want to export the environment variable for whatever weird reason you have:
KUBECONFIG=/path/to/config/file kubectl config view
Resource: https://ahmet.im/blog/mastering-kubeconfig/
Access K8s API from inside a Pod
Setup
Set these variables to start:
K8S=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
Health Check
curl -s -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/healthz
Show pods
curl -s -H "Authorization: Bearer $TOKEN" \
--cacert $CACERT $K8S/api/v1/namespaces/$NAMESPACE/pods/
Get network policies
kubectl get networkpolicy
Get information about a particular policy
kubectl describe networkpolicy $NETWORK_POLICY_NAME
Resource: https://www.stackrox.com/post/2020/02/azure-kubernetes-aks-security-best-practices-part-2-of-4/
Default network access policies
If network policies are defined, the Kubernetes default policy is allow. Subsequently, you can talk to networked assets from within a container. To test this, exec into a container:
kubectl exec -it <container name> sh
Once inside, you can try things like querying the AWS metadata service (or another networked resource):
wget -O - -q http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance/
If this works, you can ascertain that an attacker that gains access to a running container gains unfettered network access.
Resolution
Specify network access configurations that minimize ingress and egress access for each pod.
Port forwarding
Service to localhost
This will forward the service running on 8443 to localhost:1234
kubectl port-forward service/<service name> 1234:8443 -n <namespace>
Pod to localhost
This will forward the service running on $pod-port to localhost:$pod-port
kubectl port-forward $POD_NAME $pod-port
Pod to the network
This will expose the service running on $pod-port in $POD_NAME to the system running the kubectl command on $localhost-port to other systems on the network:
kubectl port-forward --address <hostname or IP of system> $POD_NAME $localhost-port:$pod-port
Resource: https://stackoverflow.com/questions/51468491/how-kubectl-port-forward-works
Set cluster
kubectl config use-context <cluster>
Get list of everything a service account can do
kubectl auth can-i --list
Use local image
Add this line to your pod yaml file:
imagePullPolicy: Never
Resource: https://stackoverflow.com/questions/55392014/kubectl-get-pods-shows-errimagepull
Run docker in docker
This is a very bad thing to do from a security standpoint, but when you need it, this is how you do it:
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: dev
imagePullPolicy: Never
name: dev
volumeMounts:
- name: docker-sock-volume
mountPath: "/var/run/docker.sock"
volumes:
- name: docker-sock-volume
hostPath:
# location on host
path: /var/run/docker.sock
Resources:
https://stackoverflow.com/questions/56462126/how-to-add-v-var-run-docker-sock-var-run-docker-sock-when-running-container
https://devops.stackexchange.com/questions/2506/docker-in-kubernetes-deployment
Helm
Debug template
helm template g <template name> -f ./values.yaml --debug
For example, if we deployed the bitnami/ghost template, we could use this command to debug it:
helm template g bitnami/ghost -f ./values.yaml --debug
Resource: https://www.reddit.com/r/kubernetes/comments/j3j3ox/ghost_helm_no_pod_showing_up/
Search for repository by name
This example will search for ghost repositories:
helm search hub ghost --max-col-width=0
Open the desired link to get the helm repo add command to use.
Show local repo list
helm repo list
Remove repo from local repo list
This example will remove the bitnami repo:
helm repo rm bitnami
Add repo to local repo list
This particular example will add the bitnami repo:
helm repo add bitnami https://charts.bitnami.com/bitnami
Search for versions of a chart in local repo
This will return a list of ghost charts:
helm search repo ghost --versions
Get latest version of a chart in local repo
helm search repo ghost --versions | sed -n 2p | awk '{print $2}'
List Releases in all namespaces
helm ls --all-namespaces
# Shorthand:
helm ls -a
Resources:
https://github.com/helm/helm/issues/7527
https://jhooq.com/helm-chart-wordpress-installation/ - tutorial with some great helm examples
https://helm.sh/docs/intro/using_helm/ - helm docs
https://github.com/helm/hub/issues/208 - explanation of how to add a repo found with the search command
Install Plugin
This example will install the helm diff plugin:
helm plugin install https://github.com/databus23/helm-diff
Resources:
https://jhooq.com/helm-chart-plugin/ - tutorial
https://github.com/helm/helm/issues/3156 - initial suggestion
Uninstall Plugin
This example will uninstall the helm diff plugin:
helm plugin uninstall diff