How to get all Logs from Kubernetes Cluster?

Troubleshooting on Kubernetes can quickly turn into a nightmare if there are multiple moving parts in your applications. Running a multi layer application on Kubernetes is complex and troubleshooting or diagnosing for issues on live logs in a live system can quickly become difficult.

So, wouldn’t it be a great idea to get a download of all logs from your Kubernetes cluster and then troubleshoot? I feel it will be great if we think of the following cases:

You are stuck while troubleshooting an issue and you need someone else to have a look at logs for help. So easy sharing!!
Diagnosing logs on your bash/powershell/cmdline can be very tedious and difficult task. So having a download or dump locally can make search fast and easy.

So, inspired Thomas who works at Microsoft, I decided to write down a powershell script for this purpose. The script automates the following tasks:

Get all pod logs (current and previous)
Current -> logs from current pod
Previous -> logs of pod from previous runs. This will show you the logs of the last run of the pod before it crashed. It is a handy feature in case you want to figure out why the pod crashed in the first place.
Describe all pods
Get all events in the Cluster

The script get all the above logs and places them at a particular location on your machine/server.

The script can be found in this GitHub repo.

$root_OutputDir="C:\templogs"
$date=Get-Date
$date = $date.ToString("yyyy-MM-dd")
$output_Namespace_Directory="pod_logs_$(kubectl config current-context)_$date"
$output_Directory="${root_OutputDir}/${output_Namespace_Directory}"
$describe_Directory= "${output_Directory}/describe"
$podlogs_Directory= "${output_Directory}/podlogs"
$eventlogs_Directory= "${output_Directory}/eventlogs"
$extension="log"
echo "Using output dir $output_Directory"
mkdir "$output_Directory"
mkdir "$describe_Directory"
mkdir "$podlogs_Directory"
mkdir "$eventlogs_Directory"

[pscustomobject]$hashtable = kubectl get po -A --no-headers
$hashtable | ForEach-Object {
$count = 0
$_.ToString().Split(" ") | foreach {
$Value1 = $_
if ($Value1 -ne "") {
    $count = $count + 1
    if ($count -eq 1) {
        $namespace=$value1
    }
    if ($count -eq 2) {
        $podname=$value1
    }
}
}
Write-Host $namespace "contains" $podname
$filename = "${describe_Directory}/${namespace}.${podname}.describe"
kubectl describe pod -n "$namespace" "$podname" > "$filename"
foreach($container in (kubectl get po -n "$namespace" "$podname" -o jsonpath="{.spec.containers[*].name}")) {
    $filename_Prefix="$podlogs_Directory"+"/"+"$namespace"+"."+"$podname"+"."+"$container"
    $filename = $filename_Prefix + ".current." + $extension
    kubectl logs -n "$namespace" "$podname" "$container" > "$filename"
    echo "$filename"
    $filename = $filename_Prefix + ".previous." + $extension
    kubectl logs -p -n "$namespace" "$podname" "$container" > "$filename"
    echo "$filename"
}
}
# Dump all events
$filename="$eventlogs_Directory/events.log"
kubectl get events -A > "$filename"

Now lets see the script in action!

I hope you find this helpful and this allows you to troubleshoot and diagnosing your issues faster and easier.