Dart ffi - Reading emojis from file with C

Question

I am saving a File with UTF-8 encoding which contains some information including a name for a button from Dart side with the following code:

file.writeAsString([
          name.length.toString(),
          name + Constants.nativeFileDelimeter,
          ids.length.toString(),
          ids.join(" "),
        ].join(" "));

// Constants.nativeFileDelimeter is "|", it is used so that user can enter a name with whitespaces

I read the same file with C and use FFI to pass data between C and Dart.

        FILE *file;

        file = fopen(filePath, "r");

        if (!file) {
            LOGE("Could not open %s!", filePath);
            *operationState = MediaLoadState::FAILED_TO_LOAD;
            goto cleanup;
        }

        int32_t size;

        if(fscanf(file, "%d ", &size) != 1){
            LOGE("fscanf can not assign variables %s!", filePath);
            *operationState = MediaLoadState::FAILED_TO_LOAD;
            goto cleanup;
        }

        // +1 because C strings ends with /0
        *namePtr = new char[size + 1];

        if (size != 0){
            if(fscanf(file, "%[^|]|", *namePtr) != 1){
                LOGE("fscanf can not assign variables %s!", filePath);
                *operationState = MediaLoadState::FAILED_TO_LOAD;
                goto cleanup;
            }
        }

Dart code that reads the pointer saved by C:

  Pointer<Pointer<Utf8>> _namePtrPtr;
  String get name => Utf8.fromUtf8(_namePtrPtr.value);

My problem is this code works with 0 bugs it even works with japanese, russion characters but when emojis are introduced thing get weird. When I save a file containing emojis and I try to read it with C and Dart ffi I get strange errors thrown by Utf8.fromUtf8. for example:

Unfinished UTF-8 octet sequence (at offset 48)

Sometimes the same code it works and renders the emojis but later on the app crashes randomly. The exceptions thrown seems to be different each time I read the file, sometimes I get no exception but later a crash! It is not consistent. I have no idea what I am doing wrong, I expected it to work with emojis. can anyone help me solve this issue?

Answer 1

In Dart, String.length returns the number of UTF-16 code units. For reading UTF-8 in C, you need to know the number of UTF-8 bytes instead. Therefore, output utf8.encode(name).length instead of name.length in the Dart code (and import dart:convert). Exceptions and crashes may be because of undefined behavior triggered by too short size.

Storing the data size separately in a text format is error-prone. Better use this approach:

It seems you are using C++. There, you can just open the file as std::ifstream, create a std::string name; and use std::getline(file, name, '|'); to read the name with dynamic size. You can use *namePtr = strdup(name.c_str()) to create a plain C string out of the std::string.