Skip to content

Instantly share code, notes, and snippets.

@alexandru-dinu

alexandru-dinu/over_allocation.py

Last active August 29, 2021 20:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save alexandru-dinu/a38976ca656b98ccc41aa4498483e19e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save alexandru-dinu/a38976ca656b98ccc41aa4498483e19e to your computer and use it in GitHub Desktop.
Download ZIP
Python over allocation
Raw
over_allocation.py
import os
import sys
import psutil
UNIT = 2 ** 20 # MB
print("All size units are in M\n")
proc = psutil.Process(pid=os.getpid())
m0 = proc.memory_info().rss
size = int(sys.argv[1]) * (2 ** 20)
print(f"Creating an int list `xs` of size {size // UNIT}.")
obj = []
for i in range(size):
obj.append(i)
obj_size_M = sys.getsizeof(obj) / UNIT
print(f"\tgetsizeof(xs) = {obj_size_M:.3f}")
m1 = proc.memory_info().rss
print(f"\tresident mem difference (creating `xs`) = {(m1 - m0) / UNIT:.3f}")
print()
print("Copying `xs` to `ys`")
copy = obj[:]
copy_size_M = sys.getsizeof(copy) / UNIT
print(f"\tgetsizeof(ys) = {copy_size_M:.3f}")
m2 = proc.memory_info().rss
print(f"\tresident mem difference (creating `ys` copy) = {(m2 - m1) / UNIT:.3f}")
@alexandru-dinu
Copy link
Author

alexandru-dinu commented Aug 29, 2021 •
edited
Loading

All size units are in M

list-3M

Creating an int list xs of size 3.

getsizeof(xs) = 26.936
resident mem difference (creating `xs`) = 121.988

Copying xs to ys

getsizeof(ys) = 24.000
resident mem difference (creating `ys` copy) = 23.977

Unpickling int list xs of size 3 (~15M on disk)

getsizeof(xs) = 25.749
resident mem difference (creating `xs`) = 121.914

list-128M

Creating an int list xs of size 128.

getsizeof(xs) = 1037.682
resident mem difference (creating `xs`) = 5187.039

Copying xs to ys

getsizeof(ys) = 1024.000
resident mem difference (creating `ys` copy) = 1024.086

Unpickling int list xs of size 128 (~641M on disk)

getsizeof(xs) = 1117.807
resident mem difference (creating `xs`) = 5186.930

numpy-128M

Creating an int32 numpy array xs of size 128.

getsizeof(xs) = 512.000
resident mem difference (creating `xs`) = 512.109

Copying xs to ys

getsizeof(ys) = 512.000
resident mem difference (creating `ys` copy) = 512.055

Unpickling int32 numpy array xs of size 128 (~513M on disk)

resident mem difference = 511.984375

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment