python - Writing with lxml emitting no whitespace even when pretty_print=True

Question

I'm using the lxml library to read an xml template, insert/change some elements, and save the resulting xml. One of the elements which I'm creating on the fly using the etree.Element and etree.SubElement methods:

tree = etree.parse(r'xml_archiveemplatesmetadata_template_pts.xml')
root = tree.getroot()

stream = []
for element in root.iter():
    if isinstance(element.tag, basestring):
        stream.append(element.tag)

        # Find "keywords" element and insert a new "theme" element
        if element.tag == 'keywords' and 'theme' not in stream:
            theme = etree.Element('theme')
            themekt = etree.SubElement(theme, 'themekt').text = 'None'
            for tk in themekeys:
                themekey = etree.SubElement(theme, 'themekey').text = tk
            element.insert(0, theme)

prints to the screen nicely print etree.tostring(theme, pretty_print=True):

<theme>
  <themekt>None</themekt>
  <themekey>Hydrogeology</themekey>
  <themekey>Stratigraphy</themekey>
  <themekey>Floridan aquifer system</themekey>
  <themekey>Geology</themekey>
  <themekey>Regional Groundwater Availability Study</themekey>
  <themekey>USGS</themekey>
  <themekey>United States Geological Survey</themekey>
  <themekey>thickness</themekey>
  <themekey>altitude</themekey>
  <themekey>extent</themekey>
  <themekey>regions</themekey>
  <themekey>upper confining unit</themekey>
  <themekey>FAS</themekey>
  <themekey>base</themekey>
  <themekey>geologic units</themekey>
  <themekey>geology</themekey>
  <themekey>extent</themekey>
  <themekey>inlandWaters</themekey>
</theme>

However, when using etree.ElementTree(root).write(out_xml_file, method='xml', pretty_print=True) to write out the xml, this element gets flattened in the output file:

<theme><themekt>None</themekt><themekey>Hydrogeology</themekey><themekey>Stratigraphy</themekey><themekey>Floridan aquifer system</themekey><themekey>Geology</themekey><themekey>Regional Groundwater Availability Study</themekey><themekey>USGS</themekey><themekey>United States Geological Survey</themekey><themekey>thickness</themekey><themekey>altitude</themekey><themekey>extent</themekey><themekey>regions</themekey><themekey>upper confining unit</themekey><themekey>FAS</themekey><themekey>base</themekey><themekey>geologic units</themekey><themekey>geology</themekey><themekey>extent</themekey><themekey>inlandWaters</themekey></theme>

The rest of the file is written nicely, but this particular element is causing (purely aesthetic) trouble. Any ideas of what I'm doing wrong?

---

Below is a snippet of markup from the template xml file (save this as "template.xml" to run with code snippet at bottom). The flattening of tags only occurs when I parse an existing file and insert a new element, not when the xml is created from scratch using lxml.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="fgdc_classic.xsl"?>
<metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://water.usgs.gov/GIS/metadata/usgswrd/fgdc-std-001-1998.xsd">
    <keywords>
       <theme>
            <themekt>ISO 19115 Topic Categories</themekt>
            <themekey>environment</themekey>
            <themekey>geoscientificInformation</themekey>
            <themekey>inlandWaters</themekey>
        </theme>
        <place>
            <placekt>None</placekt>
            <placekey>Florida</placekey>
            <placekey>Georgia</placekey>
            <placekey>Alabama</placekey>
            <placekey>South Carolina</placekey>
        </place>
    </keywords>

</metadata>

Below is a snippet of code to be used with the snippet of markup (above):

# Create new theme element to insert into root
themekeys = ['Hydrogeology', 'Stratigraphy', 'inlandWaters']

tree = etree.parse(r'template.xml')
root = tree.getroot()

stream = []
for element in root.iter():
    if isinstance(element.tag, basestring):
        stream.append(element.tag)

        # Edit theme keywords
        if element.tag == 'keywords':
            theme = etree.Element('theme')
            themekt = etree.SubElement(theme, 'themekt').text = 'None'
            for tk in themekeys:
                themekey = etree.SubElement(theme, 'themekey').text = tk
            element.insert(0, theme)

# Write XML to new file
out_xml_file = 'test.xml'
etree.ElementTree(root).write(out_xml_file, method='xml', pretty_print=True)
with open(out_xml_file, 'r') as f:
    lines = f.readlines()

with open(out_xml_file, 'w') as f:
    f.write('<?xml version="1.0" encoding="UTF-8"?>
')
    for line in lines:
        f.write(line)

Answer 1

If you replace this line:

tree = etree.parse(r'template.xml')

with these lines:

parser = etree.XMLParser(remove_blank_text=True)
tree = etree.parse(r'template.xml', parser)

then it will work as expected. The trick is to use an XMLParser that has the remove_blank_text option set to True. Any existing ignorable whitespace will be removed and will therefore not disrupt the subsequent pretty-printing.