Some notes and command dumps of interesting Isovalent Labs
https://labs-map.isovalent.com/
root@server:~/learning-ebpf/chapter2# cat hello.py
#!/usr/bin/env python3
from bcc import BPF
import sys
#The eBPF program that will run in the kernel is stored as a string in a variable called `program`. The program looks like a C function called `hello()`, and it simply writes a line of tracing before returning:
program = r"""
int hello(void *ctx) {
bpf_trace_printk("Hello World!");
return 0;
}
"""
#The rest of the file is Python code that compiles the eBPF program, loads it into the kernel, and attaches it to a kprobe that will be hit whenever the `execve` system call runs. That system call is used to execute a program, so every time a new program runs on this virtual machine, the `hello()` eBPF program will get triggered.
b = BPF(text=program)
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")
try:
b.trace_print()
except KeyboardInterrupt:
sys.exit(0)
root@server:~/learning-ebpf/chapter2# ./hello.py
b' <...>-3149 [001] ....1 925.136050: bpf_trace_printk: Hello World!'
# List all running ebpf programs
bpftool prog list
# Show map information
bpftool map show id $MAP_ID
# dump map entries
bpftool map dump id $MAP_ID
# Update map entries
bpftool map update id $MAP_ID key 5 0 0 0 0 0 0 0 value 0 0 0 0 0 0 0 1
root@server:~# bpftool map dump id $MAP_ID
[{
"key": 989,
"value": 224
},{
"key": 101,
"value": 114
},{
"key": 5,
"value": 72057594037927936
},{....
}]
# list all network-related eBPF programs
Visualize the TCP traffic sent by the frontend-service pod in the tenant-a namespace with:
hubble observe --from-pod tenant-a/frontend-service --protocol tcp
You should see a list of logs, each with:
->, <-, or at times <> if the direction could not be determined)to-endpoint, to-stack, to-overlay)FORWARDED or DROPPED)UDP, TCP), optionally with flagshubble observe --from-pod tenant-a/frontend-service --protocol tcp
Jun 10 14:53:47.316: tenant-a/frontend-service:53238 (ID:13131) -> tenant-a/backend-service:80 (ID:12518) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jun 10 14:53:47.318: tenant-a/frontend-service:53238 (ID:13131) -> tenant-a/backend-service:80 (ID:12518) to-endpoint FORWARDED (TCP Flags: ACK)
Jun 10 14:54:01.744: tenant-a/frontend-service (ID:13131) <> 10.96.52.146:80 (world) pre-xlate-fwd TRACED (TCP)
Jun 10 14:54:01.744: tenant-a/frontend-service (ID:13131) <> tenant-b/backend-service:80 (ID:5989) post-xlate-fwd TRANSLATED (TCP)
Jun 10 14:54:01.744: tenant-a/frontend-service:41234 (ID:13131) -> tenant-b/backend-service:80 (ID:5989) to-endpoint FORWARDED (TCP Flags: SYN)
Jun 10 14:54:01.744: tenant-a/frontend-service:41234 (ID:13131) -> tenant-b/backend-service:80 (ID:5989) to-endpoint FORWARDED (TCP Flags: ACK)
Jun 10 14:54:01.744: tenant-a/frontend-service:41234 (ID:13131) -> tenant-b/backend-service:80 (ID:5989) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Find out which Tetragon pod is running on the kind-worker node:
kubectl -n tetragon get po -l app.kubernetes.io/name=tetragon \
--field-selector spec.nodeName=kind-worker -o name
Next, inspect the Tetragon logs and find the events relating to /v1/exhaust-port and the tiefighter-7f64dfb56d-sh49k pod.
We will look for occurrences of /v1/exhaust-port in the Tetragon logs on the node, then pipe the resulting JSON logs into the tetra CLI provided in the image in order to display a compact and colored view of the logs (instead of raw JSON) for better readability:
kubectl -n tetragon exec -ti pod/tetragon-r62v5 -c tetragon -- \
sh -c 'cat /var/run/cilium/tetragon/tetragon*.log | \
grep /v1/exhaust-port | \
tetra getevents -o compact --pods tiefighter-7f64dfb56d-sh49k'
You should see 4 lines of logs, showing the start and end of the process:
🚀 process endor/tiefighter-76d85c5887-tvknv /usr/bin/curl -s -XPUT deathstar.endor.svc.cluster.local/v1/exhaust-port
🔌 connect endor/tiefighter-76d85c5887-tvknv /usr/bin/curl tcp 10.244.2.240:41724 -> 10.96.176.4:80
🧹 close endor/tiefighter-76d85c5887-tvknv /usr/bin/curl tcp 10.244.2.240:41724 -> 10.96.176.4:80
💥 exit endor/tiefighter-76d85c5887-tvknv /usr/bin/curl -s -XPUT deathstar.endor.svc.cluster.local/v1/exhaust-port 0
These logs include details such as:
process, network connect, network close, process exit etc.)process, network flow details for connect, etc.)