According to Gartner, more than 75% of businesses will be running containerized applications this year. Kubernetes (K8s) is essential for managing these environments, but until now, security has been an afterthought.
Managing containers at scale makes it difficult to capture security events due to their volume and complexity. K8s are typically implemented to help manage and scale containers across an environment. Containers are complex because of their ephemeral nature; when managed at scale, the complexity increases. This has led to a sense of hesitancy: more than half of businesses have delayed deploying K8s due to security concerns.
A key issue arises when organizations prioritize pre-deployment or build time/configurations to secure their K8s and container environments while overlooking the importance of runtime visibility. Pre-deployment ensures that workloads use the least amount of privileges. While this is key for minimizing the K8s attack surface, the size of the potential breach and the ability for a threat to move laterally, it’s only part of the picture.
Companies also need runtime visibility to understand manual changes and how K8s clusters and container configurations might drift from their initial state. Runtime detection is fundamental for enabling security teams to detect cyber-threats, vulnerabilities, and misconfigurations at all layers of their K8s and container environment.
So, how can you monitor and secure your K8s environments in runtime? Here are five tips.
1) Start with Network Connections
To meet K8s compliance requirements and detect threats, network connections must be monitored and mapped. This is critical for understanding how K8s workloads and namespaces interact with each other and knowing what external resources, such as cloud services, are being accessed.
Importantly, for a runtime solution to detect normal behavior, it needs to understand what normal network traffic looks like. This way, it can detect operational issues that lead to increased errors in internal traffic or if there are too many calls to an external API blocked by the provider. Abnormal network connections can also be caused by a threat to the environment.
Access to any known malicious IPs and domains must be monitored as many attacks toward K8s have resulted in crypto miner software being installed in containers to send information back to the attacker. Cyber-criminals can also use misconfigurations to access sensitive production services.
2) Monitor Ingress Endpoints
A top concern for DevOps teams is accidentally exposing an internal service to the internet. Adding an external load balancer or ingress in K8s could unintentionally leave a service lacking the proper authentication and authorization open to hackers. Any addition of this kind should be logged and validated.
Connections from public IPs to a workload should also be raised to ensure that accidental exposure does not occur. Organizations should use the network map to determine if any of the public services have direct access to sensitive resources, such as internal databases or vaults.
3) Keep Track of Audit Logs
K8s best practice is using infrastructure as code (IaC) to define environments before deployment. Manual activities, such as logging into a container, should be minimized, and actions like the manual deletion of resources should be classed as forbidden activities.
It’s impossible to tie manual changes back to a user when observing a K8s cluster configuration. While the solution involves enabling the K8s audit logs that record all K8s API calls, millions of these logs can be generated daily for internal activities. Organizations need a solution to pull out interesting events and highlight unusual activities.
Changes to least-privileged Kubernetes role-based access control are difficult to track outside the audit logs, and spotting changes to existing roles is extremely challenging. It’s simpler to keep an audit trail of these changes and validate them later.
4) Understand New K8s Components and Workloads
K8s breaches are often a result of rogue containers being deployed in the environment. In a lot of cases, these remain undiscovered for months. This is largely because configuration-based security solutions detect unsafe attributes being used by workloads, meaning a rogue container using none of these privileged accesses in K8s will go undetected.
Workload runtime monitoring must be able to pinpoint new registries and repositories in use. This involves detecting workloads created by new users outside of continuous deployment workflows. It also means spotting out-of-band deployments of workloads, containers or clusters that may not have been validated by the regular continuous integration workflow, potentially bypassing security checks implemented earlier in the application lifecycle.
5) Implement the Right Tools
Lastly, using automated, machine-based threat detection to identify threats within the sea of events generated across K8s, containers and workloads is crucial for providing always-on protection and reducing the burden on security teams.
Data-driven tools determine and baseline what is “normal” by monitoring the activity in K8s and container environments. Anomaly detection can then identify deviations from this baseline that could represent potential threats. Monitoring network connections would show new IPs or domain connections, possibly some already known as malicious.
Holistic Protection
Containers and K8s are key to enhancing innovation and helping organizations create a competitive advantage today, but to ensure success and reduce risk, security must be a central priority. Vitally, this needs to extend well beyond pre-deployment to encompass the environment in runtime – something that can only be achieved with the right monitoring and threat detection techniques and tools.