• View
• Edit
• History
• Print

PrettyTools

Editing Using The Python Line/Pretty Tools

In the past njmon has output pretty JSON which means human friend and indented as below {

    "cpu_total": {
        "guest": 0.0,
        "guestnice": 23.0,
        "hardirq": 0.0,
        "idle": 98.6,
        "iowait": 0.083,
        "nice": 0.0,
        "softirq": 0.0,
        "steal": 0.0,
        "sys": 0.383,
        "user": 0.867
    },

etc.

This is often called by the Python programmers pretty print format.

Then in the output file these JSON records were separate by a comma and the whole lots surrounded by curly braces {}

{

  {
    JSON
  }, 
  {
    JSON
  }

}

Unfortunately this has a few problems.

1. It is not a recognised multiple JSON records in a file format
   • There is no formal standard for JSON in a file
   • About the best documentation is here https://en.wikipedia.org/wiki/JSON_streaming
   • In this there are four popular schemes - we have move to the Newline Delimited scheme
2. It results in having to read the njmon output file or stream one line at a time looking for the concluding "}" and removing the extra { } and comma
3. This slows down parsing the file

Newline Delimited puts the whole JSON record on one line (JSON does not require the "white space" or indentation) and is also acceptable in Python reading in to dictionary data structures. So the above snippet looks like

{ "cpu_total": {"guest": 0.0, "guestnice": 23.0, "hardirq": 0.0, "idle": 98.6, "iowait": 0.083, "nice": 0.0, "softirq": 0.0, "steal": 0.0, "sys": 0.383, "user": 0.867 }, etc. ... } NEWLINE_CHARACTER

A whole njmon record is many KB in size and the njmon output is practically unreadable and not editable as a single line with cover the whole screen many times over.

To work around this we have Python filters To convert the new njmon 41 output to pretty print:

• ./line2pretty.py < myfile.JSON > myfile_pretty.py

or if you prefer

• cat myfile.json | ./line2pretty.py > myfile_pretty.py

The JSON data is exactly the same but now it is "Concatenated JSON" format The second JSON record just starts immediately after the closing } of the previous one.

If there is a problem with the JSON data you can now find it, fix it but before you inject it convert it back to line delimited format as below.

To convert the pretty print/Concatenated JSON back to line delimited:

• ./pretty2line.py <myfile_edited.py >myfile_fixed.json

or if you prefer

• cat myfile_edited.json | ./pretty2line.py > myfile_fixed.py

or if you prefer straight into the injector

• cat myfile_edited.json | ./pretty2line.py | injector41.py

If you have older files from njmon before version 40 you can convert the pretty format (with extra bits) to line delimited with the following

• ./njmonold2line.py <myfile_old.py >myfile_ready.json

or if you prefer

• cat myfile_old.json | ./njmonold2line.py > myfile_ready.py

or if you prefer straight into the injector

• cat myfile_old.json | ./njmonold2line.py | injector41.py

The line delimited advantages are:

1. Faster injection as the injector can use a function readline() to get each whole json record for higher efficiency than searching for ending "}" characters.
2. Smaller json files and smaller data transfers over pipes and sockets
3. Time-Series databases that deal with log files like elastic/elasticsearch/ELK and Splunk (I think) prefer a record as a line for faster input
4. One side effect is that the JSON stats in pretty format are all listed in alphabetical order - making names easier to find.