這篇文章原文出自kaggle，文中給出了reduce_mem_usage方法可以用來自動縮減dataframe占用空間

這篇notebook展示了通過使用更合理的數據類型來減少dataframe的內存使用量

方法如下：

迭代每一個column
檢查column是否為數字型
檢查column是否可以用integer表示
找出column下的最大值和最小值
選擇適用于數據范圍的最合適的數據類型
通過以上步驟處理后將一份測試數據從1.3GB減少到466MB

?

import numpy as np # linear algebra import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)def reduce_mem_usage(props):start_mem_usg = props.memory_usage().sum() / 1024**2 print("Memory usage of properties dataframe is :",start_mem_usg," MB")NAlist = [] # Keeps track of columns that have missing values filled in. for col in props.columns:if props[col].dtype != object: # Exclude strings# Print current column typeprint("******************************")print("Column: ",col)print("dtype before: ",props[col].dtype)# make variables for Int, max and minIsInt = Falsemx = props[col].max()mn = props[col].min()# Integer does not support NA, therefore, NA needs to be filledif not np.isfinite(props[col]).all(): NAlist.append(col)props[col].fillna(mn-1,inplace=True) # test if column can be converted to an integerasint = props[col].fillna(0).astype(np.int64)result = (props[col] - asint)result = result.sum()if result > -0.01 and result < 0.01:IsInt = True# Make Integer/unsigned Integer datatypesif IsInt:if mn >= 0:if mx < 255:props[col] = props[col].astype(np.uint8)elif mx < 65535:props[col] = props[col].astype(np.uint16)elif mx < 4294967295:props[col] = props[col].astype(np.uint32)else:props[col] = props[col].astype(np.uint64)else:if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:props[col] = props[col].astype(np.int8)elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:props[col] = props[col].astype(np.int16)elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:props[col] = props[col].astype(np.int32)elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:props[col] = props[col].astype(np.int64) # Make float datatypes 32 bitelse:props[col] = props[col].astype(np.float32)# Print new column typeprint("dtype after: ",props[col].dtype)print("******************************")# Print final resultprint("___MEMORY USAGE AFTER COMPLETION:___")mem_usg = props.memory_usage().sum() / 1024**2 print("Memory usage is: ",mem_usg," MB")print("This is ",100*mem_usg/start_mem_usg,"% of the initial size")return props, NAlist

原文鏈接

總結

以上是生活随笔為你收集整理的【Python学习】 - 解决DataFrame占用内存过大问题的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。