nOutput NDJson
Outputs current values, warnings and/or warnings to a local file in the NDJSON (New Line delimited JSON) format for processing by other tools. This output will also automatically gzip older files and delete old files in order to maintain a defined retention period. Optionally these local NDJSON files can be pushed to a S3 compatible object storage (retention in the object storage bucket won’t be enforced by nAttrMon).
Example of use of the execArgs
output:
name : Output ndjson
chSubscribe: nattrmon::cvals::buffer
execFrom : nOutput_NDJson
execArgs :
folder : /tmp/logs
#dontCompress : true
#filenameTemplate : .ndjson
#fileDateFormat : "yyyy-MM-dd"
fileDateFormat : "yyyyMMdd-HHmm"
#backupFilenameTemplate: "\\d{4}-\\d{2}-\\d{2}\\.ndjson"
backupFilenameTemplate: "\\d{4}\\d{2}\\d{2}-\\d{2}\\d{2}\\.ndjson.gz"
#howLongAgoInMinutes : 7200
#includeValues : true
#includeWarns : false
#includeLValues : false
#includeAttrs : false
#include :
#- Some/Attribute 1
#exclude :
#- Some/Attribute 2
#considerSetAll : true
s3sync :
secFile : /secrets/secrets.yaml
secBucket: nattrmon
secKey : s3
#url : http://127.0.0.1:9000
#accessKey: minio
#secret : minio123
bucket : logs
prefix : "monitoring/logs/{{owFormat_fromDate now 'yyyyMMdd'}}/"
Description of execArgs
execArgs | Type | Mandatory | Description |
---|---|---|---|
folder | String | No | The folder path where the ndjson files will be dumped. Defaults to output_ndjson |
dontCompress | Boolean | No | Controls if the generated ndjson files should be compressed when not actively being updated. Defaults to false |
filenameTemplate | String | No | Template of the filename to generate (uses ”{{timedate}}” to place the time & date string determined by “fileDateFormat”). Defaults to ”{{timedate}}.ndjson” |
fileDateFormat | String | No | Format to use for time & date like ‘yyyyMMdd-HHmm” (see more ow.format.toDate in OpenAF). Defaults to “yyyy-MM-dd”. |
backupFilenameTemplate | String | No | Regular expression used to determine the file candidates to be deleted after the period howLongAgoInMinutes. Keep in mind that the selection of files to delete are based on the each file modified date and not on any date & time on the filename. Defaults to “\d{4}-\d{2}-\d{2}\.ndjson” |
howLongAgoInMinutes | Number | No | This is the value, in minutes, for which files to be deleted, using the backupFilenameTemplate, will be identified using the file’s modified date. Defaults to 7200 minutes. |
includeValues | Boolean | No | Boolean flag to indicate with current values/metrics should be included. Defaults to true. |
includeWarns | Boolean | No | Boolean flag to indicate if warnings should be included. Defaults to false. |
includeLValues | Boolean | No | Boolean flag to indicate if previous values/metrics should be included. Defaults to false. |
includeAttrs | Boolean | No | Boolean flag to indicate if attribute/metric definitions should be included. Defaults to false. |
include | Array | No | An array of attribute names or warning titles regex to include. |
exclude | Array | No | An array of attribute names or warning titles regex to exclude. |
considerSetAll | Boolean | No | The nOutput_NDJson can be based on channel subscription (chSubscribe) so it can potential receive a setAll operation. In case you use nAttrMon channel buffers you need to set this argument to true. Otherwise you are probably fine with false. |
s3sync | Map | No | Map of extra arguments to configure the push of the ndjson files to a S3 compatible bucket (see below for a more complete description) |
Don’t forget to change backupFilenameTemplate whenever the fileDateFormat is updated.
Description of the s3sync argument map
execArgs | Type | Mandatory | Description |
---|---|---|---|
s3sync.url | String | Yes | The S3 compatible service URL |
s3sync.accessKey | String | No | The S3 compatible service access key (for S3 IAM role authentication or equivalent leave it unset). |
s3sync.secret | String | No | The S3 compatible service secret (for S3 IAM role authentication or equivalent leave it unset). |
s3sync.region | String | No | The S3 compatible service region to use (if any). |
s3sync.bucket | String | No | The S3 compatible service bucket to use. |
s3sync.prefix | String | No | The S3 compatible service prefix to use to push the NDJSON files on the configured bucket. Since this is also a template you can use now, folder, region and bucket as data (e.g. “logs/{{owFormat_fromDate now ‘yyyyMMdd’}}/”) |
You can also use s3sync.secFile, s3sync.secBucket and s3sync.secKey to use a secrets file, for example. This file can contain the ulr, accessKey, secret and region, for example. This is a good security practice to keep all access related credentials on a specific secrets file with controlled/restricted access.
Tips
When nAttrMon is running on a container
Containers are limited on the amount of ephemeral storage available. You should configure:
- the howLongAgoInMinutes parameter more aggressively to save on space
- change the fileDateFormat to store per minute files (e.g. “yyyyMMdd-HHmm”)
- change the backupFilenameTemplate according with then changes to fileDateFormat (e.g “\d{4}\d{2}\d{2}-\d{2}\d{2}\.ndjson”)
- configure the s3sync parameter to store the NDJSON files in a S3 compatible storage bucket
Storing a file per minute will generate several files (1440 files per day) but, on the other hand, it will ensure that if the container, where nAttrMon is running, fails only a minute of metrics will be lost. Do balance this according with your capturing needs. Setting fileDateFormat to “yyyyMMdd-HHm”, for example, will just keep a file every 10 minutes which results in 144 files per day while losing 10 minutes of metrics in the event of a failure.
When nAttrMon is running outside a container
Be mindfull of the folder you set to ensure enough space is available. The default howLongAgoInMinutes of 7200 minutes will keep a 5 days retention on the NDJSON files which should be enough for most cases. You can, of course, increase it.
Generic notes
- Choose the fileDateFormat wisely to avoid large amounts of files retained. If necessary using the s3sync is a better alternative than leaving the files on a regular filesystem.
- Keep the default option for compression. NDJSON files traditionally have a high degree of compression.
- When pushing files using s3sync it’s a good practice to set a retention period on the bucket + path used.
- You can use the templating functionality for the prefix to change it daily (e.g. “logs/{{owFormat_fromDate now ‘yyyyMMdd’}}/”) or even use environment variables to set it (e.g. “{{$env ‘LOGFOLDER’}}/”)