python - program freezes on creating a large numpy array

Question

I am making a Tkinter based project where array size can sometimes get as high as 10^9 or even more(although quite minimal chances of more).
Earlier I used a simple array using loops but it took a lot of time in an array of size of order 10^6 or more, so I decided to switch my approach to NumPy and in most cases, it gave far better results, but at the above-mentioned condition(size>=10^9), the program just freezes(sometimes even the computer also freezes leaving no option other than force restart), unlike the simple looping approach which gave the result even for higher sizes of the list, but however, took a whole lot of time.
I looked this but it involved terminologies like using heap memory, stack size and I know little about these things.

I am not quite used to the stack platform, so any advice would be appreciated.

Update: I am adding the chunk of code where I tried replacing normal list with numpy one. Commented lines are the ones I used earlier with simple list.

    def generate(self):
        # t is number of times we need to generate this list
        for i in range(self.t):
            self.n = randint(self.n_min, self.n_max)  # constraints
            # self.a = [0] * self.n
            self.a = np.random.randint(low=self.a_min, high=self.a_max, size=self.n)
            # for j in range(self.n):
            #     self.a[j] = randint(self.a_min, self.a_max)
            

and then insert all these values in the output screen of Tkinter GUI,
here 'n' i.e size of NumPy array can take very high values sometimes.

I am on dual boot (win+ubuntu) and the current situation is faced on ubuntu in which I have allocated 500 GB storage and my laptop's RAM is 8 GB.

Answer 1

You ran out of memory most likely, for a 1e9 element float64 array in numpy, that would be 8GB alone. Also if you are naively looping over that array (something like):

for item in big_numpy_arrray: 
   do_something(item)

That is going to take forever. Avoid doing that, and use numpy's vector operations when possible.