Loki json logs filter by detected fields from grafana

ONLINECAST

2 min read

·

05-10-2024

10

0

Share

Filtering Loki JSON Logs with Detected Fields from Grafana

Problem: You're using Loki for logging and Grafana for visualization, but finding specific logs within a sea of JSON data can be challenging. You want to leverage Grafana's powerful querying capabilities to filter Loki logs based on detected fields within your JSON data.

Rephrased: Imagine you have a bunch of logs stored in Loki, all in JSON format. You want to use Grafana to quickly search and analyze these logs, but you need a way to filter them based on the specific information inside each JSON object.

Scenario:

Let's say you have a simple web application generating logs like this:

{
  "timestamp": "2023-10-26T12:00:00Z",
  "event": "user_login",
  "user_id": "12345",
  "status": "success"
}

You want to filter these logs in Grafana based on the user_id field.

Original Code:

In Grafana, you might try a basic query like:

{job="webapp"}

This will fetch all logs from the webapp job, but won't filter by user_id.

Solution:

Here's how to achieve this filtering:

1. Enable JSON Parsing in Loki:

   • Ensure your Loki configuration includes json_parsing: true. This instructs Loki to parse JSON data and make fields accessible for querying.

2. Use Grafana's Query Language:

   • Within your Grafana dashboard, create a query using Grafana's powerful querying language. In this case, you'd use the json function:

     {job="webapp"} | json | user_id="12345"
     

   • Breakdown:

     • {job="webapp"}: This filters for logs from the webapp job.
     • | json: This applies the json function, making JSON fields accessible.
     • user_id="12345": This filters the results to only include logs where the user_id field is equal to "12345".

Additional Insights:

• Field Discovery: Grafana can automatically discover fields within your JSON logs. This allows you to easily build queries without needing to know the exact field names beforehand.

• Regex: Use regular expressions within the json function to filter based on more complex patterns.

• Multiple Filters: Combine multiple filters by using the | operator. For instance, you could filter by both user_id and event:

  {job="webapp"} | json | user_id="12345" | event="user_login"
  

Benefits:

• Targeted Analysis: Filter logs based on specific data within your JSON objects for more focused insights.
• Enhanced Visualization: Use filtered logs to create dashboards that display relevant information clearly.
• Improved Debugging: Quickly find logs related to specific events or users for easier troubleshooting.

Resources:

• Loki Documentation
• Grafana Query Language

Conclusion:

Filtering Loki JSON logs in Grafana based on detected fields is a powerful technique for gaining insights from your application logs. By leveraging Grafana's querying capabilities, you can effectively analyze and visualize your data, leading to better troubleshooting and decision-making.