elasticsearch - How to parse multi line XML in logstash?

Question

I have multiline XML files (~800 lines) in my s3 bucket and i want to index them in Elasticsearch but I can't parse them in logstash. Fields are sometimes empty so it's impossible to manually parse files.

My xml looks like:

<ServiceSalesClosed>
   <ErrorLevel>0</ErrorLevel>
   <ErrorMessage/>
   <LaborSaleCustomerPay>50.00</LaborSaleCustomerPay>`
   ... 

In my input I have the config:

codec => multiline
{ 
pattern => "<ServiceSalesClosed.*"
what => next
}

In my filter the following config:

multiline { 
pattern => [""]
what => next
} 

Answer 1

Ok, so it looks like the problem is, you've got confused about your multiline codec and your XML filter.

Can I suggest you set your multiline up:

codec => multiline {
     pattern => "<ServiceSalesClosed>" 
     negate => "true"
     what => "previous"
}

This will take any line that doesn't contain this tag, and keep it with the previous line(s). This should group your XML stanzas into parsable chunks. You should see the results of this in _source.

Then in your filter:

filter {
  xml => {
    source => "message"
    target => "xml_content"
    xpath => [ "//ErrorLevel", "error_level" ] 
  }
}

This should then parse your XML, create fields in the elasticsearch DB for "xml_content" (including your parsed XML) but also specifically extract ErrorLevel into a field of it's own.