rust - How do I return an Iterator that's generated by a function that takes &'a mut self (when self is created locally)?

Question

Update: The title of the post has been updated, and the answer has been moved out of the question. The short answer is you can't. Please see my answer to this question.

I'm following an Error Handling blog post here (github for it is here), and I tried to make some modifications to the code so that the search function returns an Iterator instead of a Vec. This has been insanely difficult, and I'm stuck.

I've gotten up to this point:

fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str)
    -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>,
                        FnMut(Result<Row, csv::Error>)
                            -> Option<Result<PopulationCount, csv::Error>>>,
              CliError>  {
    let mut found = vec![];
    let input: Box<io::Read> = match *file_path {
        None => Box::new(io::stdin()),
        Some(ref file_path) => Box::new(try!(fs::File::open(file_path))),
    };

    let mut rdr = csv::Reader::from_reader(input);
    let closure = |row: Result<Row, csv::Error>| -> Option<Result<PopulationCount, csv::Error>> {
        let row = match row {
            Ok(row) => row,
            Err(err) => return Some(Err(From::from(err))),
        };
        match row.population {
            None => None,
            Some(count) => if row.city == city {
                Some(Ok(PopulationCount {
                    city: row.city,
                    country: row.country,
                    count: count,
                }))
            } else {
                None
            }
        }
    };
    let found = rdr.decode::<Row>().filter_map(closure);

    if !found.all(|row| match row {
        Ok(_) => true,
        _ => false,
    }) {
        Err(CliError::NotFound)
    } else {
        Ok(found)
    }
}

with the following error from the compiler:

src/main.rs:97:1: 133:2 error: the trait `core::marker::Sized` is not implemented for the type `core::ops::FnMut(core::result::Result<Row, csv::Error>) -> core::option::Option<core::result::Result<PopulationCount, csv::Error>>` [E0277]
src/main.rs:97 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>>>, CliError>  {
src/main.rs:98     let mut found = vec![];
src/main.rs:99     let input: Box<io::Read> = match *file_path {
src/main.rs:100         None => Box::new(io::stdin()),
src/main.rs:101         Some(ref file_path) => Box::new(try!(fs::File::open(file_path))),
src/main.rs:102     };
                ...
src/main.rs:97:1: 133:2 note: `core::ops::FnMut(core::result::Result<Row, csv::Error>) -> core::option::Option<core::result::Result<PopulationCount, csv::Error>>` does not have a constant size known at compile-time
src/main.rs:97 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>>>, CliError>  {
src/main.rs:98     let mut found = vec![];
src/main.rs:99     let input: Box<io::Read> = match *file_path {
src/main.rs:100         None => Box::new(io::stdin()),
src/main.rs:101         Some(ref file_path) => Box::new(try!(fs::File::open(file_path))),
src/main.rs:102     };
                ...
error: aborting due to previous error

I've also tried this function definition:

fn search<'a, P: AsRef<Path>, F>(file_path: &Option<P>, city: &str)
    -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, F>,
              CliError>
    where F:  FnMut(Result<Row, csv::Error>)
                  -> Option<Result<PopulationCount, csv::Error>> {

with these errors from the compiler:

src/main.rs:131:12: 131:17 error: mismatched types:
 expected `core::iter::FilterMap<csv::reader::DecodedRecords<'_, Box<std::io::Read>, Row>, F>`,
 found    `core::iter::FilterMap<csv::reader::DecodedRecords<'_, Box<std::io::Read>, Row>, [closure src/main.rs:105:19: 122:6]>`
(expected type parameter,
found closure) [E0308]
src/main.rs:131         Ok(found)

I can't Box the closure because then it won't be accepted by filter_map.

I then tried this out:

fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &'a str)
    -> Result<(Box<Iterator<Item=Result<PopulationCount, csv::Error>> + 'a>, csv::Reader<Box<io::Read>>), CliError> {
    let input: Box<io::Read> = match *file_path {
        None => box io::stdin(),
        Some(ref file_path) => box try!(fs::File::open(file_path)),
    };

    let mut rdr = csv::Reader::from_reader(input);
    let mut found = rdr.decode::<Row>().filter_map(move |row| {
        let row = match row {
            Ok(row) => row,
            Err(err) => return Some(Err(err)),
        };
        match row.population {
            None => None,
            Some(count) if row.city == city => {
                Some(Ok(PopulationCount {
                    city: row.city,
                    country: row.country,
                    count: count,
                }))
            },
            _ => None,
        }
    });

    if found.size_hint().0 == 0 {
        Err(CliError::NotFound)
    } else {
        Ok((box found, rdr))
    }
}

fn main() {
    let args: Args = Docopt::new(USAGE)
                            .and_then(|d| d.decode())
                            .unwrap_or_else(|err| err.exit());


    match search(&args.arg_data_path, &args.arg_city) {
        Err(CliError::NotFound) if args.flag_quiet => process::exit(1),
        Err(err) => fatal!("{}", err),
        Ok((pops, rdr)) => for pop in pops {
            match pop {
                Err(err) => panic!(err),
                Ok(pop) => println!("{}, {}: {} - {:?}", pop.city, pop.country, pop.count, rdr.byte_offset()),
            }
        }
    }
}

Which gives me this error:

src/main.rs:107:21: 107:24 error: `rdr` does not live long enough
src/main.rs:107     let mut found = rdr.decode::<Row>().filter_map(move |row| {
                                    ^~~
src/main.rs:100:117: 130:2 note: reference must be valid for the lifetime 'a as defined on the block at 100:116...
src/main.rs:100     -> Result<(Box<Iterator<Item=Result<PopulationCount, csv::Error>> + 'a>, csv::Reader<Box<io::Read>>), CliError> {
src/main.rs:101     let input: Box<io::Read> = match *file_path {
src/main.rs:102         None => box io::stdin(),
src/main.rs:103         Some(ref file_path) => box try!(fs::File::open(file_path)),
src/main.rs:104     };
src/main.rs:105     
                ...
src/main.rs:106:51: 130:2 note: ...but borrowed value is only valid for the block suffix following statement 1 at 106:50
src/main.rs:106     let mut rdr = csv::Reader::from_reader(input);
src/main.rs:107     let mut found = rdr.decode::<Row>().filter_map(move |row| {
src/main.rs:108         let row = match row {
src/main.rs:109             Ok(row) => row,
src/main.rs:110             Err(err) => return Some(Err(err)),
src/main.rs:111         };
                ...
error: aborting due to previous error

Have I designed something wrong, or am I taking the wrong approach? Am I missing something really simple and stupid? I'm not sure where to go from here.

Answer 1

Returning iterators is possible, but it comes with some restrictions.

To demonstrate it's possible, two examples, (A) with explicit iterator type and (B) using boxing (playpen link).

use std::iter::FilterMap;

fn is_even(elt: i32) -> Option<i32> {
    if elt % 2 == 0 {
        Some(elt)
    } else { None }
}

/// (A)
pub fn evens<I: IntoIterator<Item=i32>>(iter: I)
    -> FilterMap<I::IntoIter, fn(I::Item) -> Option<I::Item>>
{
    iter.into_iter().filter_map(is_even)
}

/// (B)
pub fn cumulative_sums<'a, I>(iter: I) -> Box<Iterator<Item=i32> + 'a>
    where I: IntoIterator<Item=i32>,
          I::IntoIter: 'a,
{
    Box::new(iter.into_iter().scan(0, |acc, x| {
        *acc += x;
        Some(*acc)
    }))
}

fn main() {
    // The output is:
    //  0 is even, 10 is even, 
    //  1, 3, 6, 10, 
    for even in evens(vec![0, 3, 7, 10]) {
        print!("{} is even, ", even);
    }
    println!("");

    for cs in cumulative_sums(1..5) {
        print!("{}, ", cs);
    }
    println!("");
}

You experienced a problem with (A) -- explicit type! Unboxed closures, that we get from regular lambda expressions with |a, b, c| .. syntax, have unique anonymous types. Functions require explicit return types, so that doesn't work here.

Some solutions for returning closures:

• Use a function pointer fn() as in example (A). Often you don't need a closure environment anyway.
• Box the closure. This is reasonable, even if the iterators don't support calling it at the moment. Not your fault.
• Box the iterator
• Return a custom iterator struct. Requires some boilerplate.

You can see that in example (B) we have to be quite careful with lifetimes. It says that the return value is Box<Iterator<Item=i32> + 'a>, what is this 'a? This is the least lifetime required of anything inside the box! We also put the 'a bound on I::IntoIter -- this ensures we can put that inside the box.

If you just say Box<Iterator<Item=i32>> it will assume 'static.

We have to explicitly declare the lifetimes of the contents of our box. Just to be safe.

This is actually the fundamental problem with your function. You have this: DecodedRecords<'a, Box<Read>, Row>, F>

See that, an 'a! This type borrows something. The problem is it doesn't borrow it from the inputs. There are no 'a on the inputs.

You'll realize that it borrows from a value you create during the function, and that value's lifespan ends when the function returns. We cannot return DecodedRecords<'a> from the function, because it wants to borrow a local variable.

Where to go from here? My easiest answer would be to perform the same split that csv does. One part (Struct or value) that owns the reader, and one part (struct or value) that is the iterator and borrows from the reader.

Maybe the csv crate has an owning decoder that takes ownership of the reader it is processing. In that case you can use that to dispel the borrowing trouble.