In a previous post, I discuss using “IT-Tools” to help prevent data leaks from within Engineering teams.

Today, we will deploy IT-Tools and demonstrate the steps needed for an Engineer to “security harden” a workload in Kubernetes - specifically an web application which IT-Tools fits in nicely. This can be used whether you’re just securing a standalone app or delivering software as part of a wider team

Note

• You’ll need a version of Kubernetes that supports Seccomp and AppArmour (v1.19+).
• Security outside the workload i.e. cluster (RBAC, Node hardening, Networks i.e. Routing, mTLS) won’t be covered.
• Knowing some web development basics can help with hardening

DevSecOps, how it fits in

Think of DevSecOps as a evolved version of DevOps. The same principles, culture and tools of DevOps still apply, and we are still delivering software quicker, more reliably, with better quality. In DevSecOps, quality now encompasses security as a integral part of the process. This means integrating security practices from the outset, ensuring that software is not only functional but also resilient against potential threats.

Prior to this, security was often an afterthought in the engineering process, addressed reactively rather proactively. Agile focused on speed and collaboration and didn’t fully integrate security or caused the late discovery of vulnerabilities and slow remediation of those.

Formal wider IT Governance, Risk Management, and Compliance (GRC) frameworks recognize the importance of integrating security into the software lifecycle. Some even mention DevSecOps by name while others mention principles that align with it’s objectives. Like NIST Cybersecurity Framework (CSF), ISO/IEC 27001 (e.g. principle of secure coding), upcoming PCI DSS 4.0.

Because IT-Tools is an open source application, we cannot apply all DevSecOps principles as we don’t have full control over it’s lifecycle (such as plan and code). Despite this we can ‘adopt’ then implement measures to reduce risks on some parts that are within our remit like build, release, run etc.

Hardening is further made important because the potential additional risks of using an externally developed application, which we will explore further.

For a wider guide on DevSecOps I really recommend you read https://platingnum-official.medium.com/success-in-devsecops-requires-manpower-not-just-technology-62178855054c

Can hardening be automated with AI? (will it replace me?)

Somewhat, but for now let’s rest our paranoid minds because robust security hardening still needs well informed Engineers, contextual understanding, a degree of collaboration with developers, and testing.

I don’t think this area can be completely ChatGPT’d just yet. It’s an area that is pretty difficult to do right. At least from my copilot testing - It often breaks apps with poor quality suggestions. Boo! 🤖👎

You’re the final checkpoint

I have been there too. The ever growing backlog, a weekly sprint, you’re seemingly blocking everyone including the kitchen-sink-svc.yaml and you’re the only DevOps person on the project.

It’s easy to forget you play an important role as the last engineer in the development process before the application gets passed to the QA/release teams and then production.

Unlike our IT security counterparts. DevOps teams will use Agile methodologies to deliver software, it might be a challenge to meet deadlines while providing quality. Take time when writing your security configuration and manage your teams expectations, don’t hesitate to push back if necessary [1]. Creating hardened security configurations can vary from being straightforward to requiring in-depth investigations, it depends. Get it right because:

1. Changing/improving security configuration later or retrospectively increases development and testing time, might introduce regression, or lead to increased support overhead
2. Technical stakeholders are available and issues can be addressed during project sprints
3. Enforcement of ‘least-privileged’ rules in CI identifies issues faster
4. Deployment configurations are often forgotten / not disturbed after release (“if it ain’t broke”)

Finally, it’s your problem. While DevSecOps promotes security as everyone’s responsibility, the incumbent Engineer ultimately takes ownership. Rushing through projects could have significant consequences for your team and reputation in the long run.

[1] Read soft skills for Engineers: https://www.thirdrepublic.com/blog/soft-skills-devops/

Know your attack surface

The attack surface of a web application is the combination of all potential security vulnerabilities, backdoors, and other attack vectors in the application and its infrastructure. It can include not just unpatched software and firmware but also unsafe configurations, insecure access to data and applications, default or hard-coded logins and passwords, lack of suitable encryption for data in transit and/or at rest, and so on.

Apart from reducing the risk of malware attacks and other security threats, minimizing the attack surface also brings a host of other benefits. Hardened systems are easier to maintain because they have fewer active components. Hardening can also improve performance by eliminating unnecessary functionality that might otherwise drain valuable resources.

You’ll need a technical understanding of the app, so you’ll need to study it closely which might involve looking through documentation, high/low level design, running a a local development environment and reading the code.

I have taken the time to do this with IT-Tools and it works purely inside the users browser, and acts like a single page app (SPA).

Here’s are 7 hardening tips that I found suitable

1. Content Security Policy (CSP)

CSP was introduced in 2012, and surprisingly it’s still relatively unused in my experience.

A scenario where this could be used is if an web app had a client-side vulnerability like XSS. Without CSP an attacker could execute/load arbitrary scripts and to give them a greater attack surface.

My favorite use of this attack would be to replace the page with a login screen. Imagine being sent a legitimate link to tools.yourorg.com and seeing a login screen. Made worse if your password manager has auto-complete enabled.

Other things which may be possible:

• Exploit Same Origin Policy to other sub-domains
• Trick the user to download a file originating from the tools.yourorg.com domain
• Steal Cookies or other information if the application was more complex

Good Application Security Engineer’s will know how to configure a CSP rule to mitigate this while preserving core functionality. Here’s a nice generator tool you can use to help you https://report-uri.com/home/generate.

Content-Security-Policy: default-src 'self'

The policy tells the browser, anything via self can be loaded. Anything outside of this rule gets blocked.

2. Kubernetes NetworkPolicy

The Dockerfile, (https://github.com/CorentinTh/it-tools/blob/main/Dockerfile) indicates the application is nothing more than a static website.

It’s base image is nginx, naturally the pod should never initiate an external connection under normal circumstances. We’ll use NetworkPolicy rule to enforce this.

3. Build from source & version pinning

Third-party software vendors pose a very attractive target to attackers as they provide a pretty easy way into many organizations at once, also known as a supply chain attack.

Using any external application requires a degree of trust. When an enterprise uses a third-party app, some due diligence is expected by the software vendor like code signing, which provides some assurances the code has not been tampered with after build and is from an official source. This does not change with open source applications, rather the opposite, the risks can be even more heightened as explained below.

The NCSC has a nice guide on supply chain security: https://www.ncsc.gov.uk/collection/supply-chain-security and explains the various attacks possible.

Open-source apps can be modified by anyone, or the developer might claim plausible deniability if malicious code made it’s way into the codebase. Not to mention the thousands of dependencies a typical app utilizes these days - each a potential attack vector.

It feels like an impossible task to guard against. But there are some things we can do to mitigate some of the risks posed here.

An area open-source has an advantage is the code is viewable and auditable by anyone, try looking at the source of an binary executable for a proprietary app! Good luck! We can run scans at different levels to identify and flag potential issues. Oh boy how lucky you are, there was a time you need to manually inspect PHP source files line by line and hope you wouldn’t miss a one liner shell.

Some automated things we can do:

1. Static Application Security Testing (SAST)
2. Dynamic Application Security Testing (DAST)
3. Interactive Application Security Testing (IAST)
4. Software Composition Analysis (SCA)

IT-Tools provides an image ready to use, the images does not give us much transparency or flexibility for running the above, it also poses another risk entirely - how do you know the image pushed to the image repo has not modified away from it’s original source? More about container security here

Ultimately we have to trust something at some stage somewhere. I think a pragmatic approach would be to build our own image from a commit hash - pinning the code and dependencies (package.lock.json) preventing introduction of new supply chain attacks but at a cost of new features.

4. Seccomp

Seccomp stands for Secure Computing and it’s a technology that makes up the building blocks of the Linux containerization process, together with Namespaces, cgroups and SELinux. It is a Linux feature used to restrict the set of system calls that an application is allowed to make.

While Docker is no longer the underlying container runtime for Kubernetes. The default Seccomp profile for Docker disables around 44 syscalls (over 300 are available). Each runtime have a default Seccomp profile which can differ. Default profiles are pretty easy to find and useful when determining your workloads compatibility.

• containerd (https://github.com/containerd/containerd/blob/f0a32c66dad1e9de716c9960af806105d691cd78/contrib/seccomp/seccomp_default.go#L51)
• cri-o (https://github.com/cri-o/cri-o/blob/main/internal/config/seccomp/seccomp.go)
• Docker (https://github.com/microsoft/docker/blob/master/docs/security/seccomp.md)

Interestingly default policies are not enabled by default in Kubernetes however you can easily set this up after testing compatibility with your workloads. (Requires v1.27+)

Why do we need Seccomp here?

We’re going to use nginx as the base container image and fortunately because of it’s prevalence, Seccomp profiles are readily available from trusted sources.

If you can’t find a Seccomp profile, you will need to generate your own

5. AppArmour

AppArmor is a crucial component of Linux security, providing mandatory access control (MAC) by confining individual programs within defined security policies. In the context of containerized environments like Kubernetes.

AppArmor profiles define what actions an application can perform, including file access, network communication, and system calls. By restricting these actions to only those necessary for the application’s intended functionality, AppArmor helps mitigate the impact of potential security vulnerabilities and unauthorized access attempts.

We can create a custom AppArmor profile tailored specifically for the nginx container. This profile would define strict rules governing the nginx process’s behavior, ensuring that it only accesses the necessary files and resources required to serve static web content.

Here’s a simplified example of an AppArmor profile for an nginx container:

apparmor

#include <tunables/global>

profile nginx_profile {
  # Allow read access to static web content
  /var/www/html/** r,

  # Allow read access to nginx configuration files
  /etc/nginx/nginx.conf r,
  /etc/nginx/conf.d/** r,

  # Allow network access for serving HTTP traffic
  network inet tcp,

  # Allow DNS resolution for hostname resolution
  network inet domain,

  # Allow access to necessary system libraries
  /usr/lib/** mr,

  # Allow access to essential system resources
  capability sys_chroot,
  capability setgid,
  capability setuid,

  # Deny access to potentially dangerous capabilities
  deny capability dac_override,
  deny capability dac_read_search,
  deny capability fowner,
  deny capability fsetid,
  deny capability kill,
  deny capability mknod,
  deny capability sys_admin,
  deny capability sys_boot,
  deny capability sys_module,
  deny capability sys_ptrace,
  deny capability sys_rawio,
  deny capability sys_time,
  deny capability sys_tty_config
}

This profile allows read access to the nginx configuration files (/etc/nginx/nginx.conf and /etc/nginx/conf.d/) and the static web content directory (/var/www/html/). It also permits network access for serving HTTP traffic and DNS resolution.

Also granting access to essential system libraries (/usr/lib/**) and capabilities necessary for nginx operation, such as sys_chroot, setgid, and setuid. Meanwhile, it explicitly denies access to potentially dangerous capabilities that are unnecessary for nginx, such as sys_admin, sys_rawio, and sys_ptrace.

6. No root privileges

Adhering to the principle of least privilege ensures containers run with the minimum permissions necessary to perform their intended tasks.

Let’s consider an example of deploying a simple web application using nginx in a Kubernetes pod, with the application running as a non-root user:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      readOnlyRootFilesystem: true
    ports:
    - containerPort: 80

In this example, we have a Kubernetes pod (nginx-pod) running a single container (nginx-container) based on the nginx:latest image. To adhere to the principle of least privilege, we’ve configured the container with the following security context:

runAsNonRoot: true: This setting ensures that the nginx container runs as a non-root user, rather than with root privileges. By running as a non-root user, the container’s access to system resources and potentially sensitive files is restricted, reducing the impact of any security vulnerabilities or exploits.

runAsUser: 1000: We specify a specific non-root user ID (UID) for the container to run as. This further limits the container’s access rights and ensures that it operates within a confined environment.

readOnlyRootFilesystem: true: Setting the root filesystem to read-only prevents any writes to the root filesystem within the container. This adds an extra layer of security by mitigating the risk of unauthorized modifications to critical system files or configurations.

By configuring the nginx container with these security settings, we ensure that it operates with the minimum privileges necessary to serve web content. This reduces the likelihood of potential security vulnerabilities or breaches, enhancing the overall security posture of the Kubernetes environment.

7. TLS

Our app will be exposed to our internal users and we might not be using an VPN. TLS in this case becomes necessary to prevent MITM attacks. We can achieve this a number of ways but the easiest would be to use an Ingress controller with Let’s Encrypt support.

In our case the app works in-browser and does not send sensitive data back to a server to be processed, an attacker wouldn’t be able to intercept data however injection might change this and let the attacker exfiltrate data and do various other things.

Summary

In another post I will walkthrough and publish a repo with all the mentioned rules!

I really hope you enjoyed reading that as much as I enjoyed writing it! Please drop me a line with your feedback via e-mail.