Pod events before container startup are not visible in Coder UI #119
Description
Problem
When using coder-logstream-kube to stream pod logs/events to the Coder UI, events that occur before the container comes up are not visible. This includes critical errors like volume mount failures that prevent the workspace from starting properly.
Example
In the case below, NFS mount failures are occurring but are not visible in the Coder UI:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m55s default-scheduler Successfully assigned coder/coder-user-amber-mollusk-73-home-5565ff9496-djql4 to ec2.internal
Normal SuccessfulAttachVolume 2m52s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-a76f6f03-d40c-4b94-91dc-92056fd367de"
Normal SuccessfulAttachVolume 2m52s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-388bdf7f-adc9-4d74-84b8-7b0d235aba9e"
Warning FailedMount 47s (x9 over 2m55s) kubelet MountVolume.SetUp failed for volume "coder-nfs-c688d342-0dec-fdc0-a740-012311857993" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o ro aaaa.com:/doesnt/exist /var/lib/kubelet/pods/6c36ad5e-da07-4d63-95cf-2ba6d67cfdba/volumes/kubernetes.io~nfs/coder-nfs-c688d342-0dec-fdc0-a740-012311857993
Output: mount.nfs: Failed to resolve server aaaa.com: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 42s (x9 over 2m54s) kubelet MountVolume.SetUp failed for volume "coder-nfs-7ce329e5-c54c-d8da-8e2b-c4aeb69a70bf" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o ro working-server.com:/AAAAAAAAAAAAAAAAAA /var/lib/kubelet/pods/6c36ad5e-da07-4d63-95cf-2ba6d67cfdba/volumes/kubernetes.io~nfs/coder-nfs-7ce329e5-c54c-d8da-8e2b-c4aeb69a70bf
Output: mount.nfs: access denied by server while mounting working-server.com:/AAAAAAAAAAAAAAAAAA
Impact
Users cannot see why their workspace failed to start when the issue occurs during the pre-container phase (volume mounting, image pulling, etc.). This makes debugging workspace startup issues significantly more difficult.
Expected Behavior
Pod events that occur before container startup should be visible in the Coder UI to help users understand why their workspace failed to start.
Environment
- Using Kubernetes deployment with
wait_for_rollout = false - NFS volumes configured in the workspace template
- Events visible via
kubectl describe podbut not in Coder UI