select - Get last unique JSON inputs in bash

Question

I'm looking to pull the latest (by timestamp) source and message from some JSON data. I tried to get jq to pull only the latest unique message based on the source but can't quite figure it out.

Example source data is:

{"envelope":{"source":"+13101234567","sourceDevice":1,"relay":null,"timestamp":1610256979995,"dataMessage":{"timestamp":1610256979995,"message":"Yes","expiresInSeconds":0,"reaction":null,"quote":null,"mentions":[],"attachments":[],"groupInfo":null},"syncMessage":null,"callMessage":null,"receiptMessage":null}}
{"envelope":{"source":"+13102345678","sourceDevice":1,"relay":null,"timestamp":1610256985623,"dataMessage":{"timestamp":1610256985623,"message":"1","expiresInSeconds":0,"reaction":null,"quote":null,"mentions":[],"attachments":[],"groupInfo":null},"syncMessage":null,"callMessage":null,"receiptMessage":null}}
{"envelope":{"source":"+13102345678","sourceDevice":1,"relay":null,"timestamp":1610256987736,"dataMessage":{"timestamp":1610256987736,"message":"3","expiresInSeconds":0,"reaction":null,"quote":null,"mentions":[],"attachments":[],"groupInfo":null},"syncMessage":null,"callMessage":null,"receiptMessage":null}}
{"envelope":{"source":"+13101234567","sourceDevice":1,"relay":null,"timestamp":1610256990731,"dataMessage":{"timestamp":1610256990731,"message":"4","expiresInSeconds":0,"reaction":null,"quote":null,"mentions":[],"attachments":[],"groupInfo":null},"syncMessage":null,"callMessage":null,"receiptMessage":null}}
{"envelope":{"source":"+13105678901","sourceDevice":1,"relay":null,"timestamp":1610256990731,"dataMessage":{"timestamp":1610256990731,"message":"4","expiresInSeconds":0,"reaction":null,"quote":null,"mentions":[],"attachments":[],"groupInfo":null},"syncMessage":null,"callMessage":null,"receiptMessage":null}}

For this example, the result should be:

[{"source":"+13102345678","message":"3"},{"source":"+13101234567","message":"4"},{"source":"+13105678901","message":"4"}]

Using .envelope.source,.envelope.dataMessage.message gets the information without the headers but piping it into unique[] says "Cannot iterate over string" on jqplay.

question from:https://stackoverflow.com/questions/65650703/get-last-unique-json-inputs-in-bash

Answer 1

As I understand the question, a reasonable solution would be as follows, assuming the -n command-line option is used:

[inputs.envelope
 | {source, 
    message: .dataMessage.message,
    timestamp: .dataMessage.timestamp}]
| group_by(.source)
| [ .[] | max_by(.timestamp) | {source, message} ]

This at least produces the expected results, and pays attention to the "timestamps".