Loki parse JSON and filter by field

2 min read 05-10-2024

Parsing and Filtering JSON Data with Loki: A Comprehensive Guide

Loki, a popular open-source log aggregation and analysis platform, can be a powerful tool for handling JSON data. But how can you extract specific information from your JSON logs and filter them based on desired criteria? This article will walk you through parsing and filtering JSON data with Loki, empowering you to analyze your logs with greater precision.

The Scenario: Understanding the Challenge

Imagine you have a system that logs events as JSON objects, similar to this example:

{
  "timestamp": "2023-10-26T10:00:00Z",
  "event": "user_login",
  "user_id": "12345",
  "status": "success",
  "location": "New York"
}

You need to analyze only the events where status is "failed" and extract information like the timestamp and user_id. How can you achieve this using Loki?

Leveraging Loki's JSON Parsing Capabilities

Loki provides the __json operator, which allows you to access fields within JSON objects. Let's break down the process:

Parsing the JSON: Loki can automatically parse JSON logs. You can use the __json operator to access specific fields:
```
{job="my-app"} |= "event=user_login" and __json.status == "failed"
```
- |= "event=user_login" filters the logs for events with event equal to "user_login".
- __json.status == "failed" checks the status field for the value "failed".
Extracting Information: You can use the __json operator to extract fields you need:
```
{job="my-app"} |= "event=user_login" and __json.status == "failed" | __json.timestamp, __json.user_id
```
This query will now show only the timestamp and user_id for failed login events.

Additional Considerations

Nested JSON: Loki allows you to access nested fields using dot notation: __json.nested.field.
String Matching: For flexible string matching, use the ~ operator instead of ==. For example, __json.location ~ "New York" will match logs where the location field contains "New York".

Example Application: Analyzing User Login Errors

Let's say you want to monitor user login errors in your application. Here's how you can use Loki:

Configure your application to log user login events as JSON.
Configure Loki to receive these logs.

Create a Grafana dashboard:

Create a graph panel with the query:
```
{job="my-app"} |= "event=user_login" and __json.status == "failed" | count by __json.user_id
```
This will show you the number of failed login attempts per user.

Create another graph panel to track failed login attempts over time:

{job="my-app"} |= "event=user_login" and __json.status == "failed" | count by __json.user_id, __json.timestamp

Set alerts based on specific criteria: You can set up alerts to notify you when there are a high number of failed login attempts from a particular user or over a specific period.

Conclusion

By utilizing Loki's powerful JSON parsing capabilities, you can gain valuable insights from your JSON logs. Whether you need to filter based on specific fields, extract key information, or analyze trends, Loki empowers you to make informed decisions about your system's performance and security.

Remember, this article only scratches the surface of what's possible with Loki and JSON parsing. Explore Loki's documentation and community resources for more advanced techniques and use cases.

Resources

Loki Documentation: https://grafana.com/docs/loki/
PromQL Documentation: https://prometheus.io/docs/prometheus/latest/querying/basics/
Grafana Dashboarding: https://grafana.com/docs/grafana/latest/dashboards/