parsing - Why does the order of ANTLR4 tokens matter?

Question

I have a simple grammar that will eventually parse YANG source. When I make when seem to be an arbitrary change the location of the MODULE token the IntelliJ ANTLR4 Plugin can/cannot parse my input.

The input string to be parsed:

module x { }

Here is the grammar that works without any error:

grammar Yang ;

yang: module_open module_close;

module_open : MODULE ID BRACKET_OPEN ;

module_close: BRACKET_CLOSE ;

MODULE: 'module' ;

ID: ([A-Za-z][A-Za-z0-9_-]*) ;
BRACKET_OPEN: '{' ;
BRACKET_CLOSE: '}' ;

WS: [ 
]+ -> skip ;

Here is the grammar that fails:

grammar Yang ;

yang: module_open module_close;

module_open : MODULE ID BRACKET_OPEN ;

module_close: BRACKET_CLOSE ;

ID: ([A-Za-z][A-Za-z0-9_-]*) ;

MODULE: 'module' ;

BRACKET_OPEN: '{' ;
BRACKET_CLOSE: '}' ;

WS: [ 
]+ -> skip ;

All I'm doing is cutting-pasting the MODULE token definition before/after the ID token, and it always fails if the MODULE definition is after the ID definition.

What am I missing? I see no discussion of order of tokens in the docs!

EDIT: @BartKiers Related Post... ANTLR4 lexer rules don't work as expected

Answer 1

It fails if module is after ID because the text 'module' is also a valid 'ID'. If the ID rule appears first, then it has precedence. That's when the order of lexer rules matters, when two or more lexer rules can match the same input. In this case, the one appearing first trumps those that follow; it has precedence.

Your excellent test case here is a perfect and exemplary illustration of this behavior at work.

There used to be in the ANTLR4 documentation here a great article by none other than Sam Harwell that explained this perfectly, but I can no longer find it.