Keras to_categorical results in different memory allocation error every re-run

I'm working on a semantic segmentation model through U-net for classifying 11 categories. After splitting my image and label data into training and testing arrays, I turn the arrays into Unsigned Integer 8 so I have

labels = labels.astype(np.uint8, copy=False) array([[[[2], [2], [2], ..., [9], [9], [9]]]], dtype=uint8)

I then use to_categorical to make them into categorical shapes for Keras

labels_cat = to_categorical(labels)

However, I get

---> categorical = np.zeros((batch_size, num_classes)) MemoryError: Unable to allocate ___ GiB for an array with shape (_) and data type __

Moreover, every time I re-run the cell block, I get a different value for either the memory, shape, or data type. the memory would sometimes be 26 GiB or 1 GiB or even 100 Mb, or the shape would be something like (276824064, 13) or (100, 512, 512, 3). The datatype would be float64, float32, uint8.

What could be the reason for these fluctuating errors, and how do I fix it?