MIT 6.033 (Computer System Engineering) covers 4 parts: Operating Systems, Networking, Distributed Systems, and Security.

This is the course note for Part IV: Security. And in this section, we mainly focus on common pitfalls in the security of computer systems, and how to combat them.

To build a secure system, we need to be clear about two aspects:

1. security policy (goal)
2. threat model (assumptions on adversaries)

Authentication

In this section, we authenticate users through username and password.

• Security Policy: provide authentication for users
• Threat Model: adversary has access to the entire stored username-password table and get password.

One solution is to use hash functions $H$, which take an input string of arbitary size and output a fixed-length string:

• $H$ is deterministic: if $x_1 = x_2$, then $H(x_1) = H(x_2)$
• $H$ is collision-resistant: if $x_1 \neq x_2$, then the probability of $H(x_1)=H(x_2)$ is virtually $0$.
• $H$ is one-way: given $x$, it is easy to compute $H(x)$; given $H(x)$ without knowing $x$, it is virtually impossible to determine $x$.

But the adversary can still use Rainbow Table to precompute hashes to determine password. This issue can be mitigated by slow hash functions with salt (a random info stored in plaintext), making it infeasible to determine password, especially without knowing salt.

Another solution is to limit transmission of passwords, because transmitting password frequently opens a user up to other attacks outside our current threat model.

• Session Cookies allow users to authenticate themselves for a period of time, without repeatedly transmitting their passwords.

sequenceDiagram
    title: Figure 1. Session Cookies
    actor User
    participant Server

    User->>+Server: username/password
    Server-->>-User: cookie
    User->>Server: cookie

• Challenge-Response Protocols authenticate users without ever transmitting passwords.

sequenceDiagram
    title: Figure 2. Challenge-Response Protocols
    actor User
    participant Server

    Server->>User: 658427(random number)
    User-->>Server: H(password | 658427)

However, there are always trade-offs, many other measures do add security, but often add complexity and decrease usability.

Low-Level Exploits

In this section, our threat model is that the adversary has the ability to run code on that machine, and the goal of adversary is to input a string that overwrites the saved instruction pointer so that the code jumps to the target function to open a shell.

There is no perfect solution for this issue. Modern Linux has protections(NX, ASLR, etc.) to prevent attacks, but there are also some counter-attacks(return-to-libc, heap-smashing, pointer-subterfuge, etc.) to those protections. And Bound-checking is also a solution, but it ruins the ability to generate compact C code.(Note the trade-offs of security vs. performance)

The Ken Thompson Hack (in essay Reflections on Trusting Trust, Thompson hacked compiler, so that it will bring backdoors to UNIX system and all subsequent versions of the C compiler) tells us that, to some extents, we cannot trust the code we didn't write ourselves. It also advocates policy-based solutions, rather than technology-based.

Secure Channels

Secure Channels protect packet data from an adversary observing data on the network.

• Security Policy: to provide confidentiality (adversary cannot learn message contents) and integrity (adversary cannot tamper with packets and go undetected).
• Threat Model: adversary can observe and tamper with packet data.

sequenceDiagram
    title: Figure 3. TLS handshake
    participant Client
    participant Server

    Client->>Server: ClientHello
    Server-->>Client: ServerHello
    Server-->>Client: {Server Certificate, CA Certificates}
    Server-->>Client: ServerHelloDone
    Note over Client: Verifies authenticity of server
    Client->>Server: ClientKeyExchange
    Note over Server: computes keys
    Client->>Server: Finished
    Server-->>Client: Finished

Encrypting with symmetric keys provides secrecy, and using Message Authentication Code (MAC) provides integrity. Diffie-Hellman key exchange lets us exchange the symmetric key securely. (The reason we use symmetric key to encrypt/decrypt data is that it is faster.)

To verify identities, we use public-key cryptography and cryptographic signatures. We often distribute public keys with certificate authorities (CA).

Note that the secure channel alone only provides confidentiality and integrity of packet data, but not for packet header.

Tor

Tor provides some level of anonymity for users, preventing an adversary from linking senders and receivers.

• Security Policy: provide anonymity (only the client should know that it is communicating with the server)
• Threat Model: packet header exposes to the adversary that is A is communicating with B.

However, there are still ways to attack Tor, e.g., correlating traffic analysis from various points in the network.

DDoS

Distributed Denial of Service (DDoS) is a type of cyber attack that prevents legitimate access to the Internet.

• Security Policy: maintain availability of the service.
• Threat Model: adversary controlls a botnet (large collection of compromised machines), and prevents access to a legitimate service via DDoS attacks.

Network-Intrusion Detection Systems (NIDS) may help to mitigate DDoS attacks, but are not perfect, because DDoS attacks are sophisticated and can miminc legitimate traffic.