Kubeeye : Audit Tool For Kubernetes

0
45

KubeEye is an inspection tool for Kubernetes to discover Kubernetes resources (by OPA ), cluster components, cluster nodes (by Node-Problem-Detector) and other configurations are meeting with best practices, and giving suggestions for modification.

KubeEye supports custom inspection rules and plugins installation. Through KubeEye Operator, you can view the inspection results and modification suggestions by the graphical display on the web page.

Architecture

KubeEye get cluster resource details by the Kubernetes API, inspect the resource configurations by inspection rules and plugins, and generate inspection results. See Architecture for details.

How to use

  • Install KubeEye on your machine
    • Download pre built executables from Releases.
    • Or you can build from source codeNote: make install will create kubeeye in /usr/local/bin/ on your machine.

git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye
make installke

  • [Optional] Install Node-problem-Detector

Note: This will install npd on your cluster, only required if you want detailed report.

kubeeye install npd

  • Run KubeEye

Note: The results of kubeeye sort by resource kind.

kubeeye audit
KIND NAMESPACE NAME REASON LEVEL MESSAGE
Node docker-desktop kubelet has no sufficient memory available warning KubeletHasNoSufficientMemory
Node docker-desktop kubelet has no sufficient PID available warning KubeletHasNoSufficientPID
Node docker-desktop kubelet has disk pressure warning KubeletHasDiskPressure
Deployment default testkubeeye NoCPULimits
Deployment default testkubeeye NoReadinessProbe
Deployment default testkubeeye NotRunAsNonRoot
Deployment kube-system coredns NoCPULimits
Deployment kube-system coredns ImagePullPolicyNotAlways
Deployment kube-system coredns NotRunAsNonRoot
Deployment kubeeye-system kubeeye-controller-manager ImagePullPolicyNotAlways
Deployment kubeeye-system kubeeye-controller-manager NotRunAsNonRoot
DaemonSet kube-system kube-proxy NoCPULimits
DaemonSet k ube-system kube-proxy NotRunAsNonRoot
Event kube-system coredns-558bd4d5db-c26j8.16d5fa3ddf56675f Unhealthy warning Readiness probe failed: Get “http://10.1.0.87:8181/ready”: dial tcp 10.1.0.87:8181: connect: connection refused
Event kube-system coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9 Unhealthy warning Readiness probe failed: HTTP probe failed with statuscode: 503
Event kube-system vpnkit-controller.16d5ac2b2b4fa1eb BackOff warning Back-off restarting failed container
Event kube-system vpnkit-controller.16d5fa44d0502641 BackOff warning Back-off restarting failed container
Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c Failed warning Failed to pull image “controller:latest”: rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied
Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527 Failed warning Error: ImagePullBackOff
Role kubeeye-system kubeeye-leader-election-role CanDeleteResources
ClusterRole kubeeye-manager-role CanDeleteResources
ClusterRole kubeeye-manager-role CanModifyWorkloads
ClusterRole vpnkit-controller CanImpersonateUser
ClusterRole vpnkit-controller CanDeleteResources

What KubeEye can do

  • KubeEye inspects cluster resources according with Kubernetes best practices, to make cluster stable.
  • KubeEye can find problems of your cluster control plane, including kube-apiserver/kube-controller-manager/etcd, etc.
  • KubeEye helps you detect all kinds of cluster nodes problems, including memory/cpu/disk pressure, unexpected kernel error logs, etc.

Checklist

YES/NOCHECK ITEMDescriptionLevel
PrivilegeEscalationAllowedPrivilege escalation is alloweddanger
CanImpersonateUserThe role/clusterrole can impersonate other userwarning
CanModifyResourcesThe role/clusterrole can delete kubernetes resourceswarning
CanModifyWorkloadsThe role/clusterrole can modify kubernetes workloadswarning
NoCPULimitsThe resource does not set limits of CPU in containers.resourcesdanger
NoCPURequestsThe resource does not set requests of CPU in containers.resourcesdanger
HighRiskCapabilitiesHave high-Risk options in capabilities such as ALL/SYS_ADMIN/NET_ADMINdanger
HostIPCAllowedHostIPC Set to truedanger
HostNetworkAllowedHostNetwork Set to truedanger
HostPIDAllowedHostPID Set to truedanger
HostPortAllowedHostPort Set to truedanger
ImagePullPolicyNotAlwaysImage pull policy not alwayswarning
ImageTagIsLatestThe image tag is latestwarning
ImageTagMissThe image tag do not declaredanger
InsecureCapabilitiesHave insecure options in capabilities such as KILL/SYS_CHROOT/CHOWNdanger
NoLivenessProbeThe resource does not set livenessProbewarning
NoMemoryLimitsThe resource does not set limits of memory in containers.resourcesdanger
NoMemoryRequestsThe resource does not set requests of memory in containers.resourcesdanger
NoPriorityClassNameThe resource does not set priorityClassNameignore
PrivilegedAllowedRunning a pod in a privileged mode means that the pod can access the host’s resources and kernel capabilitiesdanger
NoReadinessProbeThe resource does not set readinessProbewarning
NotReadOnlyRootFilesystemThe resource does not set readOnlyRootFilesystem to truewarning
NotRunAsNonRootThe resource does not set runAsNonRoot to true, maybe executed run as a root accountwarning
CertificateExpiredPeriodCertificate expiration date less than 30 daysdanger
EventAuditEvent auditwarning
NodeStatusnode status auditwarning
DockerStatusdocker status auditwarning
KubeletStatuskubelet status auditwarning

Add your own inspection rules

Add custom OPA rules

  • create a directory for OPA rules

mkdir opa

  • Add custom OPA rules files

Note: the OPA rule for workloads, package name must be kubeeye_workloads_rego for RBAC, package name must be kubeeye_RBAC_rego for nodes, package name must be kubeeye_nodes_rego

  • Save the following rules to rule file such as imageRegistryRule.rego to check the image registry address complies with rules.

package kubeeye_workloads_rego
deny[msg] {
resource := input
type := resource.Object.kind
resourcename := resource.Object.metadata.name
resourcenamespace := resource.Object.metadata.namespace
workloadsType := {“Deployment”,”ReplicaSet”,”DaemonSet”,”StatefulSet”,”Job”}
workloadsType[type]
not workloadsImageRegistryRule(resource)
msg := {
“Name”: sprintf(“%v”, [resourcename]),
“Namespace”: sprintf(“%v”, [resourcenamespace]),
“Type”: sprintf(“%v”, [type]),
“Message”: “ImageRegistryNotmyregistry”
}
}
workloadsImageRegistryRule(resource) {
regex.match(“^myregistry.public.kubesphere/basic/.+”, resource.Object.spec.template.spec.containers[_].image)
}

  • Run KubeEye with custom rules

Note: Specify the path then Kubeeye will read all files in the directory that end with .rego.

root:# kubeeye audit -p ./opa
NAMESPACE NAME KIND MESSAGE
default nginx1 Deployment [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot]
default nginx11 Deployment [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot]
default nginx111 Deployment [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]

KubeEye Operator

What is KubeEye Operator

KubeEye Operator is an inspection platform for Kubernetes, manage KubeEye by operator and generate inspection result.

What KubeEye Operator can do

  • KubeEye Operator provides management functions through web page.
  • KubeEye Operator recode inspection results by CR, can view and compare cluster inspection results by web page.
  • KubeEye Operator provides more plugins.
  • KubeEye Operator provides more detailed modification suggestions.

deploy Kubeeye

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

get the inspection results

apiVersion: v1
items:
apiVersion: kubeeye.kubesphere.io/v1alpha1
kind: ClusterInsight
metadata:
name: clusterinsight-sample
namespace: default
spec:
auditPeriod: 24h
status:
auditResults:
auditResults:
resourcesType: Node
resultInfos:
namespace: “”
resourceInfos:
items:
level: warning
message: KubeletHasNoSufficientMemory
reason: kubelet has no sufficient memory available
level: warning
message: KubeletHasNoSufficientPID
reason: kubelet has no sufficient PID available
level: warning
message: KubeletHasDiskPressure
reason: kubelet has disk pressure
name: kubeeyeNode

LEAVE A REPLY

Please enter your comment!
Please enter your name here