Introduction
As containerized environments and Kubernetes orchestration gain popularity, so do concerns around security. Containers bring a unique set of challenges in security that differ from traditional monolithic infrastructure, especially given the distributed, ephemeral nature of these environments. In this article, we’ll dive into container security basics, a threat model, and introduce a selection of open-source tools for vulnerability scanning, compliance, and configuration checking. These tools provide robust methods to scan and identify risks in containerized environments. Believe me that is very important topic, quite often treated in a low-key manner.
Container security involves safeguarding containerized applications from vulnerabilities, misconfigurations, and runtime threats. Unlike virtual machines, containers share the host operating system kernel, which increases their dependency on OS security and creates new attack vectors. The adoption of Kubernetes, a powerful container orchestration tool, adds further complexity but also introduces built-in security measures.
Despite these measures, securing the lifecycle of a container – image building, deployment, runtime – requires specialized tools for vulnerability scanning and compliance checks to ensure that images are free from critical vulnerabilities and configured securely.
Container Threat Model
In the context of containers, there is no one model which covers all potential threats. It depends on your environment and the software installed on your host. However, it’s possible to create a threat model by identifying the most common vulnerabilities. When securing containers it’s helpful to understand a basic threat model that outlines common vulnerabilities and attack vectors:
- Container Image Risks: Misconfigurations or vulnerabilities in base images can propagate through multiple deployments. In case of creating own application – vulnerable code (libs/bins) as well.
- Configuration Missteps: Insecure default configurations, secret exposure, improper entry point setup can lead to privilege escalations.
- Host Exploits: Since containers share the host kernel, any kernel vulnerability can impact the container ecosystem. Insecure networking or gaps in host’s software also are treated as considerations here.
- Runtime Risks: Attackers may exploit running containers to move laterally or escalate privileges within the Kubernetes environment. Badly configured runtime/orchestrator can be easy bite.
Given this landscape, let’s look at some essential open-source tools for vulnerability scanning and configuration checking that mitigate these risks.
Security Open-Source Map
The open-source container security ecosystem offers an impressive array of tools for different stages of the security lifecycle. Here, we’ll cover tools that focus specifically on vulnerability scanning, compliance, and configuration checking. You can find them pointed in red.
- vulnerability scanning: Trivy, Grype and Syft
- compliance and configuration: Kube-Bench and Hadolint
Vulnerability Scanner
1. Trivy
Trivy by Aqua Security is a popular open-source vulnerability scanner for container images, filesystems and Git repositories. It scans images for vulnerabilities in OS packages, application dependencies, and other components.
I have actually a 3-node Kubernetes cluster up and running all the time so I will use in my case this one to test the vulnerability scanner against some YAML manifest files but also will present how to scan the image and a whole repository. Let’s start and make hands dirty!
Installation is described here: https://aquasecurity.github.io/trivy/v0.57/getting-started/installation/. It is very simple and adjusted to your needs – you can install it as script/binary or from package manager – depends on you. Once you have it installed, you can validate by running:
$ trivy --versionNow, let’s go into the vulnerability scanning. As mentioned above Trivy gives us a possibility to perform multiple kind of scanning. I will present following:
- single YAML manifest file
- multiple YAML manifest files (into single folder)
- Github repository scanning (supported public and private repos)
- container image scanning
Config Files Scanning
I start with the simplest example – single Kubernetes Deployment file that is by intention not secure. I created the manifest file with command:
$ kubectl create deployment nginx-test --image nginx --replicas 3 --dry-run=client -o yaml > nginx-test.yamlNext I want to scan this definition, so I run the command:
$ trivy config nginx-test.yamlThe result of it is as you can see on the screenshot below.
I receive the overall summary on the top where you can find how many tests were performed and how many are passed and how many fails. For each finding you have a description, small summary and vulnerability code. Each vulnerability has its own code, i.e. “AVD-KSV-003” that is for not having a DROP: ALL for capabilities. You also have a specific URL to the vulnerability and description of it as well – just in case you need dig into the specific finding.
The full output of all vulnerabilities is quite descriptive and it is hard to read all of the results so if you want to have a little bit more readable output I encourage you to use also grep command to parse it, or just use another format like JSON to for instance parse it up to your needs:
$ trivy config --format json nginx-test.yaml
$ trivy config nginx-test.yaml | grep AVDLet’s now create multiple files, maybe more secure in terms of Kubernetes best practices. Here are 2 YAML manifest that I used for scanning:
- sec-pod1.yaml: ideal Pod in terms of security – all K8s best practices applied
- sec-pod2.yaml: almost ideal config, I intentionally missed
runAsNonRootoption and also do not apply any seccomp profile
As you see I have added few things for both configuration files, like security contexts, requests and limits, seccomp profile, cut all capabilities and more security options for both Pod and container sections.
apiVersion: v1
kind: Pod
metadata:
name: sec-pod1
spec:
containers:
- name: sec-container
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h" ]
securityContext:
allowPrivilegeEscalation: false
runAsUser: 20000
runAsGroup: 20000
readOnlyRootFilesystem: true
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
apiVersion: v1
kind: Pod
metadata:
name: sec-pod2
spec:
containers:
- name: sec-container
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h" ]
securityContext:
allowPrivilegeEscalation: false
runAsUser: 20000
runAsGroup: 20000
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
I created a dedicated folder to keep both files.
$ mkdir my-pods
$ ls -la my-pods
Now I can scan the entire folder content for vulnerability:
$ trivy config my-podsAs you can see on below screen, the output is pretty satisfying.
Tests: 93 (SUCCESSES: 90, FAILURES: 3)
Failures: 3 (UNKNOWN: 0, LOW: 1, MEDIUM: 2, HIGH: 0, CRITICAL: 0)Only 3 failures (which I expect) and 90 passed items. That’s cool.
Git Repository Scanning
Next option is to scan the entire repository and look for the vulnerability inside the repo code and files. Git scan option is really awesome because it can be used when you have a project and multiple teams are working together using same repo. I will give you example how to scan public repository with one of my course content.
The crucial thing here is that Trivy is not downloading the repo locally – it is reading all the content and analyze the vulnerability fully remotely, which in my opinion is good thing.
In terms of using the private repo for scanning you need to define the environment variable called GITHUB_TOKEN or GITLAB_TOKEN depends which Git repo you are using and store you repo token in there. Simplest way to do so is just execute:
$ export GITHUB_TOKEN="your_private_github_token"
or
$ export GITLAB_TOKEN="your_private_gitlab_token"In my case I’m presenting the public example. In terms of command it doesn’t matter which type of repo you are using – it is simply the same:
$ trivy repo <repo_url>
And and voilà – repository is scanned and I see that it detects some private key issues as well as tokens that are part of normal text files.
Next option that is also valuable and I want to mention here is to scan the repo and find any misconfiguration within the repository files. To use this we need to add --scanners config to the previous command:
$ trivy repo --scanners config <repo_url>
Of course as an example I scan Github repo but same way you can for instance scan your private onprem repository like Harbor. I think it is good to mention that Trivy is now included with Harbor as the default scanner, which of course make life a little bit easier.
And that is all if we touch the configuration part. Let’s move to the container scanning.
Container Image Scanning
I start with a general scanning of the image. I will choose alpine:latest as an first image.
$ trivy image alpine:latest
As we can see it detects only 2 Severity LOW issues with some libraries. I think that is quite good results. AS you can see the container scanning output is also in table format by default and is providing all required information for any vulnerability found inside the container.
To compare this with less secure container I will choose nicolaka/netshoot image.
$ trivy image nicolaka/netshootnicolaka/netshoot (alpine 3.19.1)
Total: 151 (UNKNOWN: 4, LOW: 22, MEDIUM: 72, HIGH: 44, CRITICAL: 9)
As you can see it is really buggy image (but we use it to troubleshoot containers quite often). Despite that we need to say it is very insecure. But we can use that image to actually test some additional parameters for container scanning command:
--input: scan a container image from a tar archive--severity: filter by severities--ignore-unfixed: ignore unfixed/unpatched vulnerabilities--format json --output result.json: generate JSON result and save it with specific filename
All command examples are provided below:
$ trivy image --input ruby-3.1.tar
$ trivy image --severity HIGH,CRITICAL nicolaka/netshoot
$ trivy image --ignore-unfixed nicolaka/netshoot
$ trivy image --format json --output result.json nicolaka/netshootExample output for nicolaka/netshoot analysis for only HIGH and CRITICAL severity vulnerabilities listing only those which we can fix increase the scan result little bit.
$ trivy image --severity HIGH,CRITICAL --severity HIGH,CRITICAL nicolaka/netshoot
And that concludes the Trivy tool. You should now know how to deal with the config, container and repository scanning with Trivy.
2. Grype and Syft
Grype
Grype by Anchore is a powerful vulnerability scanner that works with images, filesystems, and SBOM (Software Bill of Materials) produced by Syft. It identifies vulnerabilities and integrates well with CI/CD workflows.
Official website: https://github.com/anchore/grype
Installation can be done with oneliner:
$ curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sudo sh -s -- -b /usr/local/binTo start the vulnerability scanning, execute following command. I’m using same alpine:latest and nicolaka/netshoot image to have also a possibility to compare with Trivy tool.
$ grype alpine:latest
As you can see above – results are identical with the output we received from the Trivy. That is good, it means that this image is really secure and has only 2 vulnerabilities.
Let’s validate second image that has more vulnerabilities detected:
$ grype nicolaka/netshoot
Ok, we have it. As you can see, output contains a lot of reported issue and security gaps. I would say even more than reported by Trivy, and actually we have a big mismatch here. Grype reported many more issues than Trivy.
Trivy reported: Total: 53 (HIGH: 44, CRITICAL: 9)
Grype reported: 33 critical, 167 high, 147 medium, 14 low, 0 negligible (8 unknown)
✔ Scanned for vulnerabilities [369 vulnerability matches]
├── by severity: 33 critical, 167 high, 147 medium, 14 low, 0 negligible (8 unknown)
└── by status: 324 fixed, 45 not-fixed, 0 ignoredOf course you can also use fancy parameters here to limit the severity. For example, here’s how you could trigger a CI pipeline failure if any vulnerabilities are found in the image with a severity of “medium” or higher:
$ grype <image_name> --fail-on mediumLike Trivy, Grype can generate reports in various formats:
$ grype <image_name> -o json > grype_report.jsonSyft
Syft by Anchore complements Grype by generating SBOMs for container images, which can then be used to assess vulnerabilities in each layer.
Official website: https://github.com/anchore/syft
Installation is very easy, also with one line command execution:
$ curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sudo sh -s -- -b /usr/local/bin
Once installed – we are ready to generate an SBOM for a container image (I will again use alpine:latest image). To do so simply execute:
$ syft alpine:latest
SBOM is ready, but actually in the real production world we are not doing this in that way, default output is not used. Our aim is on create SBOM as JSON file. In order to generate it as a JSON file simply add -o json=<filename> to the command:
$ syft alpine:latest -o json=alpine_sbom.jsonGenerated file is a pure JSON file having SBOM items inside it – you can validate that and check the structure (don’t be afraid as this is all written in one line):
Generated SBOMs can be also used for even faster vulnerability scanning in Grype. To use it execute grype sbom command and provide the filename. I will provide the file I generated for alpine:latest conatiner image:
$ grype sbom:./alpine_sbom.json
The output is (and actually it has to be) the same as we execute grype alpine:latest command. The only difference here is that grype isn’t looking for an image remotely and isn’t parsing it from cataloged contents remotely but rely only on provided SBOM file as the only source of truth.
As you see already, those two tools (Grype and Syft) are fully compliant with themselves and hence they are usually used together providing both SBOMs and vulnerability features all in one.
Configuration & compliance
To ensure that containers are compliant with best practices and configuration guidelines, Kube-Bench and Hadolint tools can provide extensive insights into vulnerabilities stemming from misconfigurations. They are really useful when you want to take care about the image beforehand. What I saw from experience – developers usually build an image first, test it and then later on they harden them to meet some security requirements. The perfect situation is to think about security while creating a container and not after.
1. Kube-Bench
Kube-Bench by Aqua Security checks if Kubernetes is configured securely, following the CIS Kubernetes Benchmark standards.
Official website: https://github.com/aquasecurity/kube-bench.
Installation of kube-bench is officially done via defining a Kubernetes Job object that is scanning the node he runs on. I adjusted the Job template and prepared customized DaemonSet object that runs on al Kubernetes cluster nodes and gives you the report per node. This is the content of kube-bench.yaml manifest file:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-bench
labels:
kubernetes.io/security: kube-bench
spec:
selector:
matchLabels:
kubernetes.io/security: kube-bench
template:
metadata:
labels:
kubernetes.io/security: kube-bench
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- command: ["sh", "-c", "while true; do sleep 3600; done"]
image: docker.io/aquasec/kube-bench:v0.9.1
name: kube-bench
volumeMounts:
- name: var-lib-cni
mountPath: /var/lib/cni
readOnly: true
- mountPath: /var/lib/etcd
name: var-lib-etcd
readOnly: true
- mountPath: /var/lib/kubelet
name: var-lib-kubelet
readOnly: true
- mountPath: /var/lib/kube-scheduler
name: var-lib-kube-scheduler
readOnly: true
- mountPath: /var/lib/kube-controller-manager
name: var-lib-kube-controller-manager
readOnly: true
- mountPath: /etc/systemd
name: etc-systemd
readOnly: true
- mountPath: /lib/systemd/
name: lib-systemd
readOnly: true
- mountPath: /srv/kubernetes/
name: srv-kubernetes
readOnly: true
- mountPath: /etc/kubernetes
name: etc-kubernetes
readOnly: true
- mountPath: /usr/local/mount-from-host/bin
name: usr-bin
readOnly: true
- mountPath: /etc/cni/net.d/
name: etc-cni-netd
readOnly: true
- mountPath: /opt/cni/bin/
name: opt-cni-bin
readOnly: true
hostPID: true
volumes:
- name: var-lib-cni
hostPath:
path: /var/lib/cni
- hostPath:
path: /var/lib/etcd
name: var-lib-etcd
- hostPath:
path: /var/lib/kubelet
name: var-lib-kubelet
- hostPath:
path: /var/lib/kube-scheduler
name: var-lib-kube-scheduler
- hostPath:
path: /var/lib/kube-controller-manager
name: var-lib-kube-controller-manager
- hostPath:
path: /etc/systemd
name: etc-systemd
- hostPath:
path: /lib/systemd
name: lib-systemd
- hostPath:
path: /srv/kubernetes
name: srv-kubernetes
- hostPath:
path: /etc/kubernetes
name: etc-kubernetes
- hostPath:
path: /usr/bin
name: usr-bin
- hostPath:
path: /etc/cni/net.d/
name: etc-cni-netd
- hostPath:
path: /opt/cni/bin/
name: opt-cni-bin
Once we have the file ready, let’s run it and wait until it will be in a Running state on all nodes:
$ kubectl apply -f kube-bench.yamlValidate all Pods are running:
$ kubectl get pods -o wide
To run configuration and compliance scan for a full CIS benchmark we need to first connect to the particular Pod and then execute scanning command:
$ kubectl exec -it kube-bench-gdrx4 -- sh
$ kube-bench
As output you will see the security controls summary and then later on remediation actions necessary to fix the vulnerabilities.
Whole scanning is divided into 5 chapters:
- Control Plane Security Configuration
- Etcd Node Configuration
- Control Plane Configuration
- Worker Node Security Configuration
- Kubernetes Policies
Please note that if you run a Pod within data plane node (worker node) then first three chapters of the scanning are skipped as simply they are not applicable.
After remediations, each section ends with the summary. Here we can see in total how many checks are passed or failed. For detailed information see below screenshot.
At the end of the whole report you have summary for all tests and checks.
If you require a report – export results to JSON and save it as a file:
$ kube-bench --json > master-report.jsonAnd that’s it if we talk about the kube-bench tool. Generated report is self-explanatory and does not require further description. Once you have a CIS item and quick summary as well as remediation you should be have any additional questions or problems to deal with them and fixed all findings.
2. Hadolint
Hadolint is a Dockerfile linter that helps identify best practice violations and configurations that could lead to security issues. Hadolint checks your Dockerfile for possible errors, security vulnerabilities, and performance problems.
Official website: https://github.com/hadolint/hadolint
Installation:
$ wget -O hadolint https://github.com/hadolint/hadolint/releases/download/v2.12.0/hadolint-Linux-x86_64
$ sudo mv hadolint /usr/local/bin/hadolint
$ sudo chmod +x /usr/local/bin/hadolintTo lint a Dockerfile we need to first create one. I will use my simple example of ubuntu container:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y curl
RUN groupadd -g 2500 appgroup && useradd -m -u 2500 -g appgroup appuser
WORKDIR /home/appuser
ENTRYPOINT ["/bin/bash"]
CMD ["-c", "tail -f /dev/null"]Let’s run Hadolint against the above Dockerfile:
$ hadolint DockerfileYou will get the following output with a Warning and Info message with the rule numbers and remediation info as shown in the image below.
Hadolint outputs errors and warnings directly to the terminal by default, but results can be piped to a file for later use:
$ hadolint Dockerfile > hadolint-report.txtThe severity of the issues is categorized as follows:
- Info: we receive suggestions for improvements, considered as less severe,
- Style: related to formatting of the Dockerfile or structure, like using wrong indentation or long single lines, etc.,
- Warning: reports non critical issues and minor security vulnerabilities,
- Error: real issues with our Dockerfile, potentially relates to security vulnerabilities or major best practice violations.
As we see on our results we are having only 2 Warning and 3 Info messages. Let’s take a look on them quickly:
DL3007: that warning means that we should use specific tag/image version instead oflatestDL3009: info that after using apt tool we should perform a cleanup of temporary date and cached dataDL3059: it tells us that we should singleRUNcommand with&&instead of repeating it in multiple linesDL3008: warning to install always a specific version of the package, not latest availableDL3015: information that flag--no-install-recommendsshould be used to avoid installation of not required dependencies
After short analysis, to be honest we can really quickly improve our Dockerfile to be more concrete and meet all container best practices. I will present now new “updated” version of the same Dockerfile. I named it Dockerfile-new and it contains fixes and looks as follows:
FROM ubuntu:22.04
RUN apt-get update && \
apt-get install -y curl=8.4.0 --no-install-recommends && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
RUN groupadd -g 2500 appgroup && useradd -m -u 2500 -g appgroup appuser
WORKDIR /home/appuser
ENTRYPOINT ["/bin/bash"]
CMD ["-c", "tail -f /dev/null"]There are scenarios where you might not want to remediate all the recommendations. In such cases, you ignore specific rules using the --ignore flag.
For example, if I want to ignore DL3008 and DL3007 that recommends specifying versions of software packages and container image, I can use the following command.
$ hadolint --ignore DL3008 --ignore DL3007 Dockerfile
Then we can see that warnings we disable are not listed. We can go even further and specify the config file for Hadolint. Configuration file standardize all the configuration and rules for linting all Dockerfiles, so we can customize how Hadolint handles the output and recommendations using our specific configuration file.
You can refer to the official documentation for all the supported parameters.
Here is a sample config with all the key parameters. You can save the file with whatever name you prefer. My filename is equal hadolint-config.yaml:
failure-threshold: warning
format: tty
ignored:
- DL3007
override:
error:
- DL3015
warning:
- DL3015
info:
- DL3008
style:
- DL3015
trustedRegistries:
- docker.io
- "*.gcr.io"
- quay.ioThen to run the scan again using the above config file:
$ hadolint --config hadolint-config.yaml Dockerfile
As you can see on the output – I overwrite the DL3015 to be reported on as a info but as an error. This is because I defined that on my own using the specific config file. And this is the power of this file. We can mark info messages as mandatory for us, ignore some that are not important for us or apply only images from particular repositories excluding any public images. Configuration can be really different and fully adjusted to your case – this is the aim of using the configuration file.
Conclusion
Securing containers and Kubernetes environments is essential in a rapidly expanding cloud-native ecosystem. Vulnerability scanning tools like Trivy, Grype, and Syft allow teams to identify risks within container images, while configuration checks with kube-bench and Hadolint ensure best practices are followed in container and Kubernetes setups. These tools together provide a comprehensive security strategy for containerized applications. By integrating these open-source solutions into development and deployment workflows, teams can proactively defend against container-specific threats and enhance the overall resilience of their applications.