Log Processors
Every pipeline includes a chain of processors that define the transformations it will apply to logs. When a log matches a pipeline's filter, it is transformed by each processor in the pipeline one by one.
The following log transformation processors are available for defining pipelines.
Regex
The Regex processor can be used to extract information out of text using regular expressions.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Pattern | The regex pattern to be used. Must include atleast one named capture group |
| Parse From | The log text field to parse from. Eg: body or attributes.sessionInfo |
| Parse To | The path to parse to. Eg: If set to attributes, a capture group like (?P<userId>.+) in the regex pattern would get stored in attributes.userId |
| On Error | What to do if the processor fails. Options are to drop the log or send it to the next processor |
Grok
The Grok processor can be used to extract information out of text
using grok patterns.
Grok is a regular expression dialect with convenient aliases for commonly used expressions.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Pattern | The grok pattern to be used. Must include atleast one named capture group |
| Parse From | The log text field to parse from. Eg: body or attributes.sessionInfo |
| Parse To | The path to parse to. Eg: If set to attributes, a capture group like %{WORD:userId} in the grok pattern would get stored in attributes.userId |
| On Error | What to do if the processor fails. Options are to drop the log or send it to the next processor |
By default, values extracted using grok patterns are strings.
For example, parsing status: 202 with the pattern status: %{INT:status_code} will extract status_code as a string with value "202".
However, it is possible to extract float or int typed values by adding a 3rd part to grok capture groups.
For example, parsing status: 202 with the pattern status: %{INT:status_code:int} will extract status_code as an integer with value 202.
This can enable the use of numeric operators (>, < etc) on the extracted values and unlock features like using the values as metrics in dashboards.
JSON Parser
The JSON parsing processor can be used to parse serialized JSON text into log attributes.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Parse From | The log field containing serialized JSON text. Eg: body or attributes.sessionInfo |
| Parse To | The path to parse to. Eg: If set to attributes, parsing the JSON text '{ "userId": 8888 }' would set attributes.userId to 8888 |
Trace Parser
The trace processor can be used to populate trace id, span id and trace flags for a log.
Populating trace identifiers in logs allows navigation to and from corresponding traces for correlation.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Parse Trace Id From | The log field containing otel Trace Id. Eg: attributes.myTraceId Value at the specified path must be an even length string of hex characters |
| Parse Span Id From | The log field containing otel Span Id. Eg: attributes.mySpanId Value at the specified path must be an even length string of hex characters |
| Parse Trace Flags From | The log field containing otel Trace Flags. Eg: attributes.myTraceFlags Value at the specified path must be an unsigned int |
At least one field among Parse Trace Id From, Parse Span Id From and Parse Trace Flags From must be specified.
Timestamp Parser
The timestamp parsing processor can be used to parse log timestamp out of a log field.
Processor Fields
| Field | Description |
|---|---|
| Name of Timestamp Parsing Processor | A descriptive name for the processor. |
| Parse Timestamp Value From | The log field containing timestamp value. Eg: attributes.timestamp |
| Timestamp Format Type | Type of timestamp value to be parsed. epoch can be used for parsing unix time and strptime can be used for parsing human readable values using ctime-like directives such as %Y (4-digit year) and %H (2-digit hour). |
| Timestamp Format | Format for parsing timestamp value. For example %Y-%m-%d can be used for parsing values like 2023-12-06 when Timestamp Format Type is strptime, or seconds.milliseconds can be used for parsing unix time values like 1701869406.245 when Timestamp Format Type is epoch |
Severity Parser
The severity parsing processor can be used to parse log severity out of a log field.
Processor Fields
| Field | Description |
|---|---|
| Name of Severity Parsing Processor | A descriptive name for the processor. |
| Parse Severity Value From | The log field to parse severity from. For example attributes.log_level |
| Values for level TRACE | Comma separated list of values that should be mapped to level TRACE. For example 0, trace |
| Values for level DEBUG | Comma separated list of values that should be mapped to level DEBUG. For example debug, 2xx |
| Values for level INFO | Comma separated list of values that should be mapped to level INFO. For example info, 3xx |
| Values for level WARN | Comma separated list of values that should be mapped to level WARN. For example warning, 4xx |
| Values for level ERROR | Comma separated list of values that should be mapped to level ERROR. For example error, 5xx |
| Values for level FATAL | Comma separated list of values that should be mapped to level FATAL. For example panic, -1 |
Severity level values are case insensitive.
As a special case, 2xx, 3xx, 4xx and 5xx can be used to map number ranges to severity levels. This can be useful for mapping HTTP status codes. For example 5xx can be used to parse numbers between 500-599 into level ERROR.
Add
The add processor can be used to add a field to the log.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Field | Path of the field to be added. Must be of the form attributes.* or resource.* |
| Value | The value to be set in the specified field |
The value field can be set to an expression which will get evaluated for each entry to set the value.
For example the value can be set to EXPR(attributes.subtotal + attributes.taxes) to add a new field for total.
Value expressions are also useful for accessing array items that can't be referenced with field paths in operators like COPY and MOVE, for example EXPR(attributes.locale[0]).
Remove
The remove processor can be used for removing unwanted log fields such as PII.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| Field | Path of the field to be removed. Must be of the form attributes.* or resource.* |
Move
The move processor can be used to move or rename a log field.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| From | Path of the field to be moved. Must be of the form attributes.* or resource.* |
| To | Path to move the field to. Must be of the form attributes.* or resource.* |
Copy
The copy processor can be used to copy log fields.
Processor Fields
| Field | Description |
|---|---|
| Name | A descriptive name for the processor. |
| From | Path of the field to be copied. Must be of the form attributes.* or resource.* |
| To | Path to copy the field to. Must be of the form attributes.* or resource.* |