Kubernetes: How to setup Promtail as a sidecar in order to read custom log paths

ONLINECAST

3 min read

·

05-10-2024

9

0

Share

Logging Your Custom Paths: Setting Up Promtail as a Kubernetes Sidecar

Kubernetes has become the de facto standard for container orchestration, offering powerful tools for managing and scaling your applications. But managing logs across distributed systems can be a challenge. This is where Prometheus and its log ingestion component, Promtail, come in.

Promtail can be deployed as a sidecar container alongside your application, giving you a centralized logging solution. This article will guide you through setting up Promtail as a sidecar container within your Kubernetes environment to read logs from custom paths within your application pods.

The Problem: Managing Application Logs

Imagine you have a complex application running within a Kubernetes cluster. This application might generate logs in multiple locations:

• Standard Output (stdout): The default location for container logs.
• Custom Log Files: Your application might write logs to specific files within the container, like /var/log/myapp.log.
• Multiple Locations: You might have different log files for different components within your application.

Kubernetes offers basic logging functionality, but managing logs from various sources across your cluster can become cumbersome.

The Solution: Promtail as a Sidecar

Promtail provides a powerful solution by acting as a log collector within your application pods. It can read logs from different sources, including:

• Standard Output: Promtail can easily gather logs from the container's stdout.
• Custom File Paths: You can configure Promtail to read logs from specific files within your container.
• Multiple Sources: Promtail can be configured to read logs from multiple sources simultaneously.

Let's dive into a practical example:

Scenario: You have a Node.js application that generates logs to a custom file, /var/log/myapp.log.

Original Code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080
        command: ["npm", "start"]
        resources:
          limits:
            memory: "128Mi"
          requests:
            memory: "64Mi"

Promtail Configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: promtail-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
      - name: promtail
        image: grafana/promtail:latest
        resources:
          limits:
            memory: "128Mi"
          requests:
            memory: "64Mi"
        volumeMounts:
        - name: myapp-logs
          mountPath: /var/log/myapp
        ports:
        - containerPort: 9080
        command: ["promtail", "-config.file=/etc/promtail/promtail.yaml"]
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
      volumes:
      - name: config
        configMap:
          name: promtail-config
      - name: myapp-logs
        emptyDir: {}

Promtail ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    scrape_configs:
    - job_name: 'myapp_logs'
      static_configs:
      - targets: ['localhost:9080']
      labels:
        job: 'myapp_logs'
      scrape_interval: 5s
      scrape_timeout: 10s
    positions:
    - filename: "/tmp/promtail.positions"
    client:
      url: http://prometheus-server:9090
    server:
      http_listen_port: 9080
    logs:
    - paths:
        - /var/log/myapp/myapp.log
      json:
        keys:
          - level
          - timestamp
          - message

Explanation:

1. Promtail Deployment: We deploy a Promtail container as a sidecar alongside the myapp-deployment.
2. Volume Mounts: We mount the myapp-logs volume to provide access to the application's log files. We also mount a volume containing the Promtail configuration (promtail.yaml).
3. Promtail ConfigMap: The ConfigMap defines the Promtail configuration. We specify the scrape interval, scrape timeout, and the location of the log files within the container (/var/log/myapp/myapp.log).

Key Considerations:

• Permissions: Ensure Promtail has read permissions to access your custom log files.
• Log Rotation: Implement log rotation policies within your application to prevent log files from growing indefinitely.
• Logging Levels: Configure your application and Promtail to log messages at appropriate levels (e.g., debug, info, warning, error).
• Prometheus Integration: Integrate Promtail with your Prometheus server to collect and analyze the ingested logs.

Conclusion

Setting up Promtail as a sidecar in Kubernetes provides a robust and efficient way to manage logs from custom paths within your application pods. This approach offers centralized logging, improved observability, and simplifies log management in your distributed environment. By leveraging Promtail and its powerful features, you can gain valuable insights into your application's behavior and ensure efficient troubleshooting.