I’m once again wondering if there is any way to get more value out of dumping our log drains into a third-party application (e.g., Datadog, Sumologic). Basically because each line is sent via https over as a separate post, there is no way to put the humpty dumpty back together again. From Sumologic’s website:
Mutliline detection on an HTTP source only works within the confines of a single HTTP request. If you send multiple multiline messages within a single HTTP post request the multiline options will apply to those messages. If you send a multiline message as separate POST requests the multiline options do not apply.
So that seems to signal that the problem has to be addressed on the generation side of the log output, rather than after it arrives at the third party site. Has anyone had any success in this regard? Our app is rails served by puma, and I’d love to find a way to wrangle either of those outputs in such a way that we can get more useful chunks of data out of them rather than just the spitfire of separate lines.
I recognize that this isn’t an easy problem, but then again, if the line break is currently determining where to terminate each post request, shouldn’t it be in theory possible to use some OTHER token to make that decision?
Or am I looking at the problem backwards? Meaning I should focus on adding content to my output that THEN I know what to look for (as single lines) once it comes over the log drain?
Advice appreciated.
Thanks,
Jessica