python - pylint 1.4 reports E1101(no-member) on all C extensions

Question

We've been long-time fans of pylint. Its static analysis has become a critical part of all our python projects and has saved tons of time chasing obscure bugs. But after upgrading from 1.3 -> 1.4, almost all compiled c extensions result in E1101(no-member) errors.

Projects that previously run perfectly clean through pylint 1.3 now complain about almost every C extension member with E1101. We've been forced to disable E1101 errors, but this materially detracts from the usefulness of pylint.

For example, this perfectly valid use of the lxml package

r"""valid.py: demonstrate pylint 1.4 error"""
from lxml import etree
print etree.Element('mydoc')

Run this through pylint, and it reports:

$ pylint -rn valid.py
No config file found, using default configuration
************* Module valid
E:  3, 6: Module 'lxml.etree' has no 'Element' member (no-member)

But it is perfectly valid:

$ python valid.py
<Element mydoc at 7fddf67b1ba8>

Here's where it gets really weird. A very small handful of C extensions seem to work just fine through pylint, e.g.:

r"""valid2.py: this one works fine"""
import sqlite3
print sqlite3.version

$ pylint -rn valid2.py
No config file found, using default configuration

My question is, has anyone else witnessed this? And if so, would you be willing to share your workaround/solution?

We've experimented with trying to create plugins to suppress these warnings (http://docs.pylint.org/plugins.html#enter-plugin), but we're having difficulty making heads or tails of the docs -- and the astroid base class is uber-complex and has defied our attempts to grok it.

For real bonus points (and our eternal gratitude) we'd love to understand what changed in pylint. We'd be happy to fix the code (or at least publish a best practice document for C extension authors) that would satisfy pylint.

Platform details

$ pylint --version
No config file found, using default configuration
pylint 1.4.0,
astroid 1.3.2, common 0.63.2
Python 2.7.5 (default, Jul  1 2013, 18:09:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]

Answer 1

Shortly after posting my question, I found the answer. The change was in fact done on purpose as a security measure. Pylint imports modules to effectively identify valid methods and attributes. It was decided that importing c extensions that are not part of the python stdlib is a security risk and could introduce malicious code.

This was done in the release of Astroid 1.3.1 https://mail.python.org/pipermail/code-quality/2014-November/000394.html

Only C extensions from trusted sources (the standard library) are loaded into the examining Python process to build an AST from the live module.

There are four solutions if you want to use pylint on projects that import non-stdlib c extensions.

1) Disable safety using the --unsafe-load-any-extension=y command line option. This feature is undocumented and classified as a hidden option (https://mail.python.org/pipermail/code-quality/2014-November/000439.html).

2) Disable safety using the pylint.rc setting unsafe-load-any-extensions=yes. This is recommended over option 1 and includes full documentation in the default pylint.rc file (created with --generate-rcfile).

3) Specifically list packages or modules names that you trust to be loaded by pylint in the pylint.rc file using the extension-pkg-whitelist= option.

4) Create a plugin to manipulate the AST (I have no idea how to effect this -- but it's regularly discussed on on the pylint mailing list).

We opted for Option 3. We added the following line to our project pylint.rc file:

extension-pkg-whitelist=lxml