regex - Python regular expressions with Foreign characters in python PyQT5

Question

This problem might be very simple but I find it a bit confusing & that is why I need help.

With relevance to this question I posted that got solved, I got a new issue that I just noticed.

---

Source code:

from PyQt5 import QtCore,QtWidgets

app=QtWidgets.QApplication([])

def scroll():
    #QtCore.QRegularExpression(r''+'cat'+'')
    item = listWidget.findItems(r'cat', QtCore.Qt.MatchRegularExpression)
    for d in item:
        print(d.text())

window = QtWidgets.QDialog()
window.setLayout(QtWidgets.QVBoxLayout())
listWidget = QtWidgets.QListWidget()
window.layout().addWidget(listWidget)


cats = ["love my cat","catirization","cat in the clouds","cat??"]

for i,cat in enumerate(cats):
    QtWidgets.QListWidgetItem(f"{i}  {cat}", listWidget)

btn = QtWidgets.QPushButton('Scroll')
btn.clicked.connect(scroll)
window.layout().addWidget(btn)
window.show()
app.exec_()

Output GUI:

---

Now as you can see I am just trying to print out the text data based on the regex r"cat" when I press the "Scroll" button and it works fine!

Output:

0  love my cat
2  cat in the clouds
3  cat??

However... as you can see on the #3, it should not be printed out cause it obviously does not match with the mentioned regular expression which is r"cat". However it does & I am thinking it has something to do with that special foreign character ?? that makes it a match & prints it out (which it shouldn't right?).

I'm expecting an output like:

0  love my cat
2  cat in the clouds

Researches I have tried

I found this question and it says something about this p{L} & based on the answer it means:

If all you want to match is letters (including "international" letters) you can use p{L}.

To be honest I'm not so sure how to apply that with PyQT5 also still I've made some tries & and I tried changing the regex to like this r''+r'p{cat}'+r''. However I got this error.

QString::contains: invalid QRegularExpression object
QString::contains: invalid QRegularExpression object
QString::contains: invalid QRegularExpression object
QString::contains: invalid QRegularExpression object

Obviously the error says it's not a valid regex. Can someone educate me on how to solve this issue? Thank you!

question from:https://stackoverflow.com/questions/65600139/python-regular-expressions-with-foreign-characters-in-python-pyqt5

Answer 1

In general, when you need to make your shorthand character classes and word boundaries Unicode-aware, you need to pass the QRegularExpression.UseUnicodePropertiesOption option to the regex compiler. See the QRegularExpression.UseUnicodePropertiesOption reference:

The meaning of the w, d, etc., character classes, as well as the meaning of their counterparts (W, D, etc.), is changed from matching ASCII characters only to matching any character with the corresponding Unicode property. For instance, d is changed to match any character with the Unicode Nd (decimal digit) property; w to match any character with either the Unicode L (letter) or N (digit) property, plus underscore, and so on. This option corresponds to the /u modifier in Perl regular expressions.

In Python, you could declare it as

rx = QtCore.QRegularExpression(r'cat', QtCore.QRegularExpression.UseUnicodePropertiesOption)

However, since the QListWidget.findItems does not support a QRegularExpression as argument and only allows the regex as a string object, you can only use the (*UCP) PCRE verb as an alternative:

r'(*UCP)cat'

Make sure you define it at the regex beginning.