[infra][qa] multiline tracebacks not being indexed anymore
On Tue, Nov 5, 2019, at 8:56 AM, Matt Riedemann wrote:
> We used to be able to query for things like this:
> message:"in reserve_block_device_name" AND message:"MessagingTimeout"
> AND tags:"screen-n-api.txt"
> to fingerprint a traceback in logstash like this  but that no longer
> works. The multiline logstash filter is at  but doesn't seem to be
> getting applied anymore.
> I asked about this in -infra today and fungi said:
> "(4:44:00 PM) fungi: mriedem: i suspect that coincided with switching
> away from osla, we may need some means of parsing tracebacks out of logs
> in the indexer"
> I don't know what that means (what's osla? is  no longer used?) but
> if someone could point me at some things to look at I could see if I can
> generate a fix.
os-loganalyze, https://opendev.org/openstack/os-loganalyze, was in use on the old log server to do filtering of severity and related manipulation. One thing it would do is collapse lines that didn't have a timestamps or severity prefix. However I think that may have only been for the html rendering which logstash didn't use. I'm not sure this is the issue.
As for debugging this you can grab a log file and send it through logstash locally and fiddle with the rules until you get what you want. I'd help but currently at the summit and not in a good spot to do so.