Docker Logging

So Docker captures logs to stdout/stderr. It can format them as json by providing json-file (link) as the logging driver type. Unfortunately, it logs things like this:

{
  "log": "{\"timestamp\":\"2023-12-15T17:25:40.133352Z\",\"level\":\"INFO\",\"fields\":{\"message\":\"[ClusterDb]: Using configuration SphereMeta {\\n    db_url: Some(\\n        \\\"postgresql://SECRET\\\",\\n    ),\\n    db_max_connections: 100,\\n    blocks_db_url: Some(\\n        \\\"gs://SECRET2/\\\",\\n    ),\\n    blocks_db_service_account_path: None,\\n}\"},\"target\":\"clusterdb::cluster_db\"}\n",
  "stream": "stdout",
  "time": "2023-12-15T17:25:40.136384092Z"
}

This double escaped json is not ideal. At Subconscious Networks, our systems also happen to use Google’s ops agent which will do log munging. Unfortunately, it doesn’t seem to have a facility to say “unescape the json at location X”.

There’s a seemingly undocumented feature of ops agent where you can parse a specific field like this:

  escaped_json:
    type: parse_json
    field: log
    time_key: timestamp
    time_format: %Y-%m-%dT%H:%M:%S.%f%z

Unfortunately, you can’t maintain the docker json wrapper, due to a bug in their implementation.