The state of container security: 5 key steps to lock down releases

Container technologies are rapidly transforming application development and deployment practices at many organizations. But they also present a minefield of security risks for the growing number of organizations using the technology to package and deploy modern, microservices-based applications.

Containers are highly portable because they include everything that an application needs to run, such as dependences and configuration files. So, for development teams, containers offer a way to build an application once and deploy it across on-premises, multicloud, and virtualized environments. Container orchestration software such as Docker Swarm or Kubernetes has made it relatively easy for development teams to centrally deploy and manage containerized apps.

In development environments, containers have made it easier for developers to automate the pipeline and more efficiently move applications from testing to production. Many organizations are taking advantage of container technologies to migrate internally developed applications to the cloud or to build cloud-native applications from scratch. Server consolidation and multicloud adoption have also contributed to container adoption.

Primary risks include those related to vulnerable container images, poor access controls, overly permissive privileges, exposed attack surfaces, and misconfigured runtimes. Over the last couple of years, the rise in software supply chain attacks has increased container security risks — and heightened the need for organizations to deploy controls for managing and mitigating those risks.

As containers have become fundamental to modern software development and deployment, it is vital to secure them, said Patrick Tiquet, vice president for security and architecture at Keeper Security.  

"Securing containers involves implementing a combination of best practices, tools, and processes to protect containerized applications and their environments."
—Patrick Tiquet

Here are five essential best practices — and recommendations for modern application security tooling — to lock down your organization's containers and ensure secure software releases. 

[ Special Report: The State of Software Supply Chain Security (SSCS) 2024 | Download Report: State of SSCS ]

1. Add controls for containers accessing sensitive data

Users, applications, and devices should be allowed to communicate with and access only those resources that are required within their role. The best way to ensure that is to follow the principles of zero trust at the user, application, and network layers, said Anthony Tam, manager of security engineering at Tigera. In addition, ensure that all sensitive data is encrypted at rest and in transit, regardless of whether the destination is internal or external. Ensure that you use the latest industry-standard crypto algorithms, Tam said.

Kong Yew Chan, director of product management, container security at Qualys, said key encryption should be secured with external key management, so that only authorized individuals can retrieve the encryption key to encrypt or decrypt the sensitive data.

"Data encryption is a mandatory requirement for customer PII data to meet compliance requirements such as GDPR. Use network segmentation to limit only authorized containers to access sensitive data volumes, and use the principle of least privilege to limit the number of authorized users who have access to sensitive data."
—Kong Yew Chan

KC Berg, chief architect at the API security testing firm StackHawk, said administrators should treat all containers as if they contain personally identifiable information (PII). For containers that need extra security controls, implement a separate namespace with role-based access control (RBAC) and even more limited access.

Berg said to leverage the Init Containers feature in Kubernetes to unmount/unset configuration files and/or secrets once the primary process has loaded them.

"Run a sidecar or use a service such as a reverse proxy in front of the container's exposed network service port that monitors for PII data patterns and generates alerts if unexpected access is detected."
—KC Berg

2. Use runtime security to detect anomalous behaviors

Continuously monitor and analyze application behavior during runtime to identify and respond to threats in real time. Runtime tools can help reduce false positives by helping security teams detect attacks, risks, and anomalous behaviors that are specific to an organization's infrastructure, said Tam.

"They can also help security teams respond to incidents in real time through actions like blocking network traffic to the affected container, quarantining the container, or other methods of preventing the spread of an attack."
—Anthony Tam

Runtime security products also leverage automation for rapid investigation and response, and many can take containment actions such as killing processes or isolating workloads with a single click, Tam said.

Qualys' Chan said organizations should use technology based on extended Berkeley Packet Filter (eBPF) to monitor and protect containers during runtime. With eBPF-based technology, runtime behaviors, including file, process, and networking events, can be monitored and protected.

"For example, any unintended or unauthorized file access to specific system files such as 'etc/hosts' can be considered malicious. Users want to prevent malicious file access with container runtime so they can prevent potential threat risks and meet compliance requirements such as PCI-DSS."
—Kong Yew Chan

Create eBPF-based rules to protect against known threats. For example, treat any processes that are created from in-memory data as fileless malware. Alternatively, administrators can create whitelist rules that allow specific known behaviors while blocking unknown ones.

"This approach ensures that all other events besides the well-known behaviors will not trigger potential attacks."
—Kong Yew Chan

3. Mitigate dependency-related security risks

Maintaining visibility into dependencies for container environments can be challenging, but organizations can take certain measures to mitigate risk from open-source and other third-party code dependencies.

Chan said organizations should use minimal base images or distroless images to build their container images. "The minimal base images will have limited dependencies, reducing the number of potential vulnerabilities," Chan says.

"This approach will simplify base image dependency management and reduce the complexity of managing vulnerabilities of the base image."
—Kong Yew Chan

It's also a good idea to create a software bill of materials (SBOM) for container images. Open-source tools such as Dependency Tracker allow admins to maintain visibility into the software package dependencies within container images, Chan said.

In addition, use Kubernetes inventories to maintain visibility for container images in use. Running containers have specific images with specific dependencies. "Therefore, users can identify vulnerabilities within those running containers and prioritize fixes related to exploitable vulnerabilities with critical severity," Chan said.

StackHawk's Berg said other measures organizations can take to maintain visibility into dependencies in container environments include enabling container scanning in the Docker registry, running a container registry as a cache and then scanning third-party containers, scanning containers for vulnerable packages on every pull request, and using vulnerability-testing tools.

"Engineering teams should review vulnerabilities as part of a regular cadence. Once a good state has been reached, switch this to be pull request check-driven and treat vulnerable packages like any other software defect."
—KC Berg

4. Control supply chain risks in container ecosystems

Vulnerabilities in the software supply chain pose a major security risk in container environments. But there are several measures organizations can take to mitigate these risks, said Qualys' Chan. These include using base images only from trusted sources so images can be reproduced and verified and scanning images with tools such as software composition analysis to identify known vulnerabilities and potential license issues and such as binary analysis to identify malware. Chan said organizations should also scan images with for secrets, to prevent any from being leaked within the container images.

"Sign container images to ensure the authenticity of the image source. Signed container images will prevent tampering with the immutable images, preventing malicious packages from being injected into the image."
—Kong Yew Chan

It's also a good idea to deploy container runtime security tools such as binary analysis to detect configuration drifts and malicious threats in the runtime environments, Chan said.

"Deploy container runtime security tools to detect configuration drifts, malicious threats in the runtime environments. This approach can detect and prevent threats in the runtime environment, thereby maintaining software supply chain integrity."
—Kong Yew Chan

Chris Romeo, co-founder and CEO of the threat modeling firm Devici, said that, ultimately, containers are nothing more than a stack of operating system and application packages bundled together that are vulnerable to security issues just as in any software environment.

"The attacker's focus on the software supply chain is the cause of the bulk of threats against container security in recent years." 
—Chris Romeo 

5. Use controls to protect against lateral movement

Tam pointed to a recent survey that Tigera conducted of more than 1,200 users who are actively using Calico open-source networking and security tools in their container and Kubernetes environments. The survey found that 61% of Calico users were using workload access policies to limit pod-to-pod communication, 41% were using secure egress policies, and 24% were using microsegmentation to limit pod-to-pod communication.

"An overwhelming 85% of users said they needed to achieve network segmentation and protect east-west traffic. IT leaders need enhanced security controls at the workload level to reduce the risk of lateral movement of threats."
—Anthony Tam

Berg outlined four measures organizations can take to ensure isolation, prevent lateral movement, and detect anomalous behavior in container environments. First, ensure that containers run with the least privileges and that file and network access is secure. For instance, run the primary process as a non-root user with limited access to container assets and mount all configurations as read only.

Second, ensure that containers run one primary process as far as possible, and make sure a container's network access is limited to required resources.

"Use your cloud providers IAM and SDN capabilities in combination with Kubernetes' RBAC to run containers with least privilege access to assets like databases and other services. Use service mesh and networking tools like istio or traefik to control K8s service-to-service connections."
—KC Berg

Third, monitor containers via node/host services that can expose telemetry. Organizations need to understand that just because an application runs in a container does not necessarily mean it is automatically secure, said Keeper Security's Tiquet. 

"The same discipline and best practices for scanning and managing vulnerabilities in on-prem and cloud server instances can and should be applied to containers."
—Patrick Tiquet

Rethinking container security requires the right tools

Build pipeline attacks are on the rise, and software supply chain security is front and center. With the potential for attackers to inject malware, tamper, or compromise signing, the focus for security teams needs to shift beyond vulnerabilities. To ensure container security, you need to know if someone has changed or introduced malware in your container images — just like your code.

Choosing the right tool to run within the container to monitor for compromise and evaluate the current security posture is critical. So too is education and awareness for developers about the inherent risks in using containers, orchestration, and images for application development and deployment, Romeo said.

Romeo recommended that development and security teams take the time to learn about tools that are currently available to help organizations secure container environments. They need to understand the capabilities of these tools and recognize the differences in the risks that container environments pose compared to typical app development and deployment pipelines.

"Answer the questions: Are we scanning our containers for vulnerabilities? Are we monitoring running containers for compromise? Scope the depth of the problem, understand it, and then implement solutions."
—Chris Romeo

Lisa Azevedo, CEO of container security firm Containn, said one big limitation with many current container security products and services is that they are reactive, designed to detect after-the-fact security vulnerabilities. Many container security products allow organizations to scan for and detect known security issues but do little to prevent them from happening in the first place. Most tools, at best, allow organizations to get a point-in-time assessment of security vulnerabilities in the container environment, she said.

Currently available container security tools generally are good at detecting existing vulnerabilities, providing a remediation report, and pushing the work of fixing the issues back to the development team. A growing number use machine learning to predict vulnerabilities in software under development. But they don’t give security teams an opportunity to stay ahead of the curve, because by the time organizations have a chance to remediate the detected issues, new ones likely have surfaced, Azevedo said.

The key is to ensure container security by pushing it further left during the build process, Azevedo said. Organizations should be thinking about how to implement container security at scale from the beginning and finding ways to maintain control of container deployments and state. The focus should be on shrinking the attack surface while maintaining control of deployments and container state.

Such capabilities are critical because many organizations are on the cusp of moving away from manual tools to intelligent tools for container development and deployment, Azevedo noted. The goal is to be able to spin up containers that are standardized for specific environments and integrate security and compliance features such as those required under various industry regulations and national data security and privacy mandates.

Matt Rose, field CISO at ReversingLabs, said new tools are needed to take on the rise of software supply chain attacks, and containers are an essential component. He said binary analysis, which was recently recommended by the Enduring Security Framework (ESF) working group, is key. Binary code analysis can help organizations evaluate and verify the security of both internally developed software and third-party commercial software before it is released into their environment, Rose said.

"It is the final examination of a package for software supply chain risk, which allows for trust in that piece of software that you are either developing for your customers or that you are buying to help operate your business."
—Matt Rose