python - What causes my segmentation fault in tensorflow?

Question

In Tensorflow I am currently working on a program that creates a 4D numpy matrix and output a saved tensorflow dataset. I don't really care about the amount of memory it is going to take as it at least will fit on 80gb of ddr4 ram. My matrix has shape [14540,512,128,4] and contains np.float32 values. I want to store this in the following way:

matrix = np.load(output_path+".npy")
    if seed is None:
        seed = np.random.randint(1e6)

    print("unshuffled")
    if output_path is None:
            return tf.data.Dataset.from_tensor_slices(matrix )
    else:
            tf.data.experimental.save(tf.data.Dataset.from_tensor_slices(matrix),output_path+"_unshuffled",compression="GZIP")

The seed is for later shuffling the dataset, I am storing two version an unshuffled and a shuffled version. Rather I get a "segmentation fault error". My memory climbs to ~22gb and then the fault occurs, any ideas why this might be? Thanks in advance!

question from:https://stackoverflow.com/questions/65877216/what-causes-my-segmentation-fault-in-tensorflow

Answer 1

I think to have found the answer. I went back and changed:
matrix = np.load(output_path+".npy")
to:
matrix = np.load(output_path+".npy").astype(np.float64)
This way I was able to save the Tensorflow dataset. Appareantly Tensorflow.data does not like it when I try to store a matrix as np.float32... This seems to be a bug, I'll post this on the Github.