kubectl logs displays only 'API server listening at: [::]:40000' when remote debugging with dlv is enabled - How do I get my logs back?

2 min read 05-10-2024

kubectl logs displays only 'API server listening at: [::]:40000' when remote debugging with dlv is enabled - How do I get my logs back?

"API server listening at: [::]:40000" - The Mystery of Missing Logs in Kubernetes with DLV Debug

You've got your Go application running inside a Kubernetes pod, you've enabled remote debugging with Delve, and you're eager to dive into the code. But when you try to inspect the logs using kubectl logs, all you get is a cryptic message: "API server listening at: [::]:40000". Frustrating, right? This seemingly innocuous message is actually a red herring that hides the real culprit: Delve's debug server intercepting your logs.

The Scenario and Code:

Here's a typical scenario:

# Deploy your application with debug enabled
kubectl apply -f deployment.yaml

# Start a Delve debug session
dlv debug --listen=:2345 --headless --api-version=2 --accept-multilang your-app-name

# Access the logs 
kubectl logs your-pod-name

This results in:

API server listening at: [::]:40000

Understanding the Issue

When you enable remote debugging with Delve, you essentially create a separate process within your container that intercepts the standard output (stdout) and standard error (stderr) streams. This is why kubectl logs is showing this debug message instead of the actual application logs.

How to Get Your Logs Back

Here are several solutions to reclaim your precious application logs:

Disable Delve's Debug Server: The most straightforward solution is to temporarily disable Delve's debug server during log inspection. You can achieve this by modifying your dlv command:
```
dlv debug --headless --api-version=2 --accept-multilang your-app-name --listen=:0
```
Setting --listen=:0 tells Delve to bind to a random port, effectively disabling its debug server.
Use kubectl's -f option: Another effective approach is to directly target the application's log file within the pod using kubectl logs with the -f flag:
```
kubectl logs your-pod-name -f /path/to/your/application.log
```
Replace /path/to/your/application.log with the actual path to your application's log file within the container.
Redirect Logs to a Separate File: Modify your application to write logs to a dedicated file within the container. This can be a temporary solution, but it allows you to isolate your logs from Delve's interference.
Use a Dedicated Logging System: For more robust logging, integrate a logging system like Fluentd or ElasticSearch within your Kubernetes cluster. These systems can collect logs from multiple sources, including your application and Delve, enabling centralized monitoring and analysis.

Additional Tips and Considerations:

Examine Delve's Configuration: Delve allows you to customize its behavior with various options. You can explore the documentation and consider adjusting settings related to logging to better manage log output.
Understand Debugging Tools: While Delve is a popular choice, other debugging tools may offer more flexible logging options. Research and compare different tools to find the best fit for your specific needs.

Conclusion

Facing "API server listening at: [::]:40000" when using Delve for remote debugging is a common hurdle. By understanding how Delve interacts with logging and applying the solutions outlined in this article, you can reclaim your logs and get back to debugging your application. Remember, effective debugging requires the right tools and the ability to navigate potential conflicts.

References:

Delve Documentation: https://dlv.io/
Kubernetes Logging: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-applications/
Fluentd: https://www.fluentd.org/
ElasticSearch: https://www.elastic.co/