Introducing KubeForenSys: A Kubernetes Forensic Collection Framework for Azure Kubernetes Service (AKS)

Executive Summary

Invictus Incident Response has developed KubeForenSys, an open-source tool designed to strengthen incident response capabilities for organizations using Azure Kubernetes Service (AKS). The result is a practical and effective solution that helps security teams recover essential forensic data and respond more confidently during high-stakes incidents.

One of the biggest challenges during an investigation is building an accurate forensic timeline to understand what happened, when, and how. Teams often struggle with fragmented logging, decentralized data, and the short lifespan of containers, which can lead to critical gaps in evidence when it is needed most.

KubeForenSys addresses this challenge by using the Kubernetes API and a custom-built script to collect and unify key data points. The tool generates a single, comprehensive timeline that brings together events, logs, RBAC changes, and suspicious pod activity, even when proper logging was not fully configured.

Background

Kubernetes is a widely used container orchestration platform, which helps administrators manage their containerized workloads. It automates scalability, flexibility and efficiency through features such as self-healing and Horizontal Pod Autoscaling (HPA).  

The Kubernetes architecture splits the management of the cluster from the actual execution of the workload. The control plane manages the state of the cluster and ensures that it is in-line with the desired state as declared in a YAML file. Execution of workload is done from within containers. Containers are contained in Pods, usually in a one-to-one mapping (but can be many-to-one in case of sidecar containers). Pods are running on nodes, which represent hosts such as virtual machines or physical servers.

Azure provides hosting of Kubernetes clusters through Azure Kubernetes Service (AKS). When using AKS, Azure is responsible for administration of the control plane of the cluster, relieving the administrators of this burden.

The Challenge

When a cluster in AKS is compromised, logs and other data is required to build a forensic timeline, in order to reflect what happened and how. However, the problem within AKS is that this data may be decentralized. Different elements (control plane components, nodes, pods, containers) each provide their own logs which may be interesting in order to build the timeline. Thus, the logs need to be collected in an automatic manner. Furthermore, containers are often short-lived. Kubelets automatically restart containers that are not healthy or have failed. However, if a container is restarted and the logs are not timely captured, these are lost.

Azure provides a solution for this: Container Insights. Container Insights collects logs from containers and cluster components and pushes it to a configured destination, such as a Log Analytics workspace. The Log Analytics workspace data can then be queried using KQL to extract the desired information. However, this feature is opt-in and shall thus not be enabled at every AKS cluster.

Microsoft published the Threat matrix for Kubernetes, which helps organizations in categorizing and understanding threats which their cluster face. The matrix is based upon the original threat matrix, but differs by being specific to containerized environments. It outlines different tactics, such as initial access, execution and persistence and shows techniques to achieve this goal, e.g. using compromised cloud credentials to achieve initial access.

The Solution

The API server exposes an HTTP API as part of the cluster control plane. Through this API, end-users, different components of the cluster and external elements can communicate with each other. Using this API, one can query for several objects within the cluster, such as the events, container logs and Pods. Since this is always enabled (due to being required for functionality of the cluster), it provides a fallback during an incident response case in case Container Insights is not enabled. Also, even if Container Insights is enabled, it provides additional information which is not exposed through Container Insights, such as service accounts in namespaces.

Referring back to the Threat matrix for Kubernetes not all techniques would be made evidently through data provided by Container Insights. For example, Container Insights does not provide insight into commands executed within a container, as referred to by the technique bash/cmd into container. Furthermore, Container Insights does not directly provide a way of detecting hostPath mount, but this data is exposed through the Kubernetes API, which we can leverage to detect abuse.

To utilize this added information and add it to our toolset, a Python script titled KuberForenSys was developed to automatically fetch desired information from the Kubernetes API and push it to a Log Analytics workspace.

The following data is collected using the script:

1. Container logs
2. Kubernetes cluster events
3. Command history from containers
4. Service accounts
5. Suspicious pods based on certain properties
6. RBAC bindings
7. Cronjobs
8. Network policies

Once this data is pushed into the workspace, analysts can correlate it using KQL. The query below unifies events, logs, RBAC changes, suspicious pods, and more into a single timeline object, which can be further filtered according to your needs. It does a union on all tables created by the script and adds a Details column, which contains all columns which are table specific (for instance the message column only exists within the kubelogs_CL table). Analysts can access this within KQL, since Details is a JSON object. Thus, one can narrow their search down based on this field, such as filtering for only a specific container.

Note that the query below needs to be adjusted if resources are named differently. The query assumes default values as specified within KubeForenSys.

let Timeline = union
(
    kubelogs_CL
    | extend source = "kubelogs_CL", EventType = "KubeLog", namespace = tostring(namespace)
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "Message", message,
            "ContainerName", container_name,
            "PodName", pod_name,
            "ContainerImages", containerimages,
            "Labels", labels,
            "Annotations", annotations
        )
),
(
    kubeevents_CL
    | extend source = "kubeevents_CL", EventType = "KubeEvent", namespace = ""
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "action", action,
            "first_timestamp", tostring(first_timestamp),
            "involved_object_name", involved_object_name,
            "involved_object_uid", involved_object_uid,
            "last_timestamp", tostring(last_timestamp),
            "message", message,
            "reason", reason,
            "reporting_component", reporting_component
        )
),
(
    commandhistory_CL
    | extend source = "commandhistory_CL", EventType = "CommandExec"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "pod_name", pod_name,
            "container_name", container_name,
            "command", command
        )
),
(
    serviceaccounts_CL
    | extend source = "serviceaccounts_CL", EventType = "ServiceAccount"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "name", name,
            "automount_service_account_token", automount_service_account_token,
            "image_pull_secrets", image_pull_secrets
        )
),
(
    suspiciouspods_CL
    | extend source = "suspiciouspods_CL", EventType = "SuspiciousPod"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "name", pod_name,
            "issue_type", issue_type,
            "details", details
        )
),
(
    rbacbindings_CL
    | extend source = "rbacbindings_CL", EventType = "RBACBinding"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "binding_type", binding_type,
            "binding_name", binding_name,
            "subject_kind", subject_kind,
            "subject_name", subject_name,
            "subject_namespace", subject_namespace,
            "role_ref_kind", role_ref_kind,
            "role_ref_name", role_ref_name
        )
),
(
    cronjobs_CL
    | extend source = "cronjobs_CL", EventType = "CronJob"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "cronjob_name", cronjob_name,
            "container_name", container_name,
            "image", image,
            "command", command,
            "schedule", schedule
        )
),
(
    networkpolicies_CL
    | extend source = "networkpolicies_CL", EventType = "NetworkPolicy"
    | project
        TimeGenerated,
        source,
        EventType,
        namespace,
        Details = pack(
            "name", name
        )
);
Timeline
| where TimeGenerated > ago(1d)
| sort by TimeGenerated asc
  

Executing the query results in the following (snippet):

It is furthermore important that the script has sufficient permissions to create the new resources within the Azure resource group. The following authorizations are required:

{
  "Name": "<name>",
  "IsCustom": true,
  "Description": "<description>",
  "permissions": [
    {
      "actions": [
        "Microsoft.OperationalInsights/workspaces/write",
        "Microsoft.OperationalInsights/workspaces/tables/write",
        "Microsoft.Insights/dataCollectionEndpoints/write",
        "Microsoft.Insights/dataCollectionRules/write",
        "Microsoft.Insights/dataCollectionRules/read",
        "Microsoft.OperationalInsights/workspaces/sharedKeys/action",
        "Microsoft.ContainerService/managedClusters/read"
      ],
      "notActions": [],
      "dataActions": [
        "Microsoft.Insights/Metrics/Write",
        "Microsoft.Insights/Telemetry/Write"
      ],
      "notDataActions": []
    }
  ],
  "assignableScopes": [
    "/subscriptions/<subscription-id>/resourceGroups/<RP>"
  ]
}
  

You can explore the tool on GitHub, where we’ve published the code, documentation, and usage instructions. Below are screenshots from an example run of KubeForenSys to illustrate what the output looks like in practice.

And pulling data from AKS and pushing it to a Log Analytics workspace:

About Invictus Incident Response

We are an incident response company and we ❤️ the cloud and specialize in supporting organizations in preparing and responding to a cyber attack. We help our clients stay undefeated!

🆘 Incident Response support reach out to cert@invictus-ir.com or go to https://www.invictus-ir.com/24-7