parsing - Using regex in StandardTokenParsers

Question

I'm trying to use regex in my StandardTokenParsers based parser. For that, I've subclassed StdLexical as follows:

class CustomLexical extends StdLexical{
  def regex(r: Regex): Parser[String] = new Parser[String] {
    def apply(in:Input) = r.findPrefixMatchOf(in.source.subSequence(in.offset, in.source.length)) match {
      case Some(matched) => Success(in.source.subSequence(in.offset, in.offset + matched.end).toString,
        in.drop(matched.end))
      case None => Failure("string matching regex `" + r + "' expected but " + in.first + " found", in)
    }
  }

  override def token: Parser[Token] =
    (   regex("[a-zA-Z]:\\[\w\\?]* | /[\w/]*".r)     ^^ { StringLit(_) }
      | identChar ~ rep( identChar | digit )               ^^ { case first ~ rest => processIdent(first :: rest mkString "") }
      | ...

But I'm a little confused on how I would define a Parser that takes advantage of this. I have a parser defined as:

def mTargetFolder: Parser[String] = "TargetFolder" ~> "=" ~> mFilePath

which should be used to identify valid file paths. I tried then:

def mFilePath: Parser[String] = "[a-zA-Z]:\\[\w\\?]* | /[\w/]*".r

But this is obviously not right. I get an error:

scala: type mismatch;
 found   : scala.util.matching.Regex
 required: McfpDSL.this.Parser[String]
    def mFilePath: Parser[String] = "[a-zA-Z]:\\[\w\\?]* | /[\w/]*".r
                                                                          ^

What is the proper way of using the extension made on my StdLexical subclass?

Answer 1

If you really want to use token based parsing, and reuse StdLexical, I would advise to update the syntax for "TargetFolder" so that the value after the equal sign is a proper string literal. Or in other words, make it so the path should be enclosed with quotes. From that point you don't need to extends StdLexical anymore.

Then comes the problem of converting a regexp to a parser. Scala already has RegexParsers for this (which implicitly converts a regexp to a Parser[String]), but unfortunately that's not what you want here because it works on streams of Char (type Elem = Char in RegexParsers) while you are working on a sttream of tokens. So we will indeed have to define our own conversion from Regex to Parser[String] (but at the syntactic level rather than lexical level, or in other words in the token parser).

import scala.util.parsing.combinator.syntactical._
import scala.util.matching.Regex
import scala.util.parsing.input._

object MyParser extends StandardTokenParsers {
  import lexical.StringLit
  def regexStringLit(r: Regex): Parser[String] = acceptMatch( 
    "string literal matching regex " + r, 
    { case StringLit( s ) if r.unapplySeq(s).isDefined => s }
  )
  lexical.delimiters += "="
  lexical.reserved += "TargetFolder"
  lazy val mTargetFolder: Parser[String] = "TargetFolder" ~> "=" ~> mFilePath
  lazy val mFilePath: Parser[String] = regexStringLit("([a-zA-Z]:\\[\w\\?]*)|(/[\w/]*)".r)  
  def parseTargetFolder( s: String ) = { mTargetFolder( new lexical.Scanner( s ) ) }
}

Example:

scala> MyParser.parseTargetFolder("""TargetFolder = "c:Dir1Dir2" """)
res12: MyParser.ParseResult[String] = [1.31] parsed: c:Dir1Dir2

scala> MyParser.parseTargetFolder("""TargetFolder = "/Dir1/Dir2" """)
res13: MyParser.ParseResult[String] = [1.29] parsed: /Dir1/Dir2

scala> MyParser.parseTargetFolder("""TargetFolder = "Hello world" """)
res14: MyParser.ParseResult[String] =
[1.16] failure: identifier matching regex ([a-zA-Z]:\[w\?]*)|(/[w/]*) expected
TargetFolder = "Hello world"
           ^

Note that also fixed your "target folder" regexp here, you had missing parens around the two alternative, plus unneeded spaces.