keras核心已轉(zhuǎn)儲(chǔ)

This article is the follow up from Part 1. Here, I will compare the tf.data and Keras.ImageDataGeneratoractual training times using the mobilenetmodel.

本文是第1部分的后續(xù)文章。在這里，我將使用mobilenet模型比較tf.data和Keras.ImageDataGenerator實(shí)際訓(xùn)練時(shí)間。

YouTube VideoYouTube視頻

In Part 1, I showed that loading images using tf.data is approximately 5 times faster in comparison toKeras.ImageDataGenerator. The dataset considered was Kaggle- dogs_and_cats (217 MB) having 10000 images distributed among 2 different classes.

在第1部分中，我展示了使用tf.data加載圖像的速度比Keras.ImageDataGenerator快5倍。 所考慮的數(shù)據(jù)集是Kaggledogs_and_cats (217 MB)，具有10000張分布在2個(gè)不同類別中的 圖像 。

In this Part 2, I have considered a bigger dataset which is commonly used for image classification problems. The dataset chosen is COCO2017 (18 GB) having 117266 images distributed among 80 different classes. Various versions of COCO datasets are freely available to try and test at this link. The reason for choosing the bigger dataset of 18 GB is to have better comparison results. For a practical image classification problem, datasets can be even bigger ranging from 100 GB (gigabytes)to a few TB (terabytes). In our case, 18 GB of data is enough to understand the comparison as using the dataset in TB would significantly increase the training times and computational resources.

在第2部分中，我考慮了一個(gè)較大的數(shù)據(jù)集，該數(shù)據(jù)集通常用于圖像分類問題。 選擇的數(shù)據(jù)集為COCO2017 (18 GB) ，其中117266張圖像分布在80個(gè)不同的類別中。 可通過此鏈接免費(fèi)獲得各種版本的COCO數(shù)據(jù)集進(jìn)行嘗試和測試。 選擇更大的18 GB數(shù)據(jù)集的原因是為了獲得更好的比較結(jié)果。 對(duì)于實(shí)際的圖像分類問題，數(shù)據(jù)集甚至可能更大，范圍從100 GB(千兆字節(jié))到幾TB(TB)。 在我們的案例中，18 GB的數(shù)據(jù)足以理解比較，因?yàn)樵赥B中使用數(shù)據(jù)集會(huì)大大增加訓(xùn)練時(shí)間和計(jì)算資源。

培訓(xùn)時(shí)間結(jié)果(提前) (Training times Results (in advance))

The above results are compared on a workstation having 16-GB RAM, 2.80 GHz with Core i7 using GPU version of TensorFlow 2.x. The dataset considered is COCO2017 (18 GB) having 117266 images distributed among 80 different classes.

以上結(jié)果在使用GPU版本的TensorFlow 2.x，具有Core i7的具有16 GB RAM，2.80 GHz的工作站上進(jìn)行了比較。 所考慮的數(shù)據(jù)集為COCO2017 (18 GB) ，其中117266張圖像分布在80個(gè)不同的類別中。

• When using Keras.ImageDataGenerator, it took approximately 58 minutes during training for each epoch using COCO2017 dataset.

  使用Keras.ImageDataGenerator ，使用COCO2017數(shù)據(jù)集訓(xùn)練每個(gè)歷時(shí)大約需要58分鐘 。

• When using tf.data with variablecache=True , the program crashes. The reason behind this crash is that the dataset considered (size 18 GB) is bigger than the RAM of the workstation. With cache=True the program starts storing the images in the RAM for quick access and when this surpasses the RAM (16 GB in our case), program crashes. I have tested the same option for a smaller dataset Kaggle- dogs_and_cats and it worked fine. This shows that we shouldn’t use cache=True option when the size of the considered dataset is bigger than the RAM.

  當(dāng)將tf.data與變量cache=True ，程序崩潰。 發(fā)生此崩潰的原因是，所考慮的數(shù)據(jù)集( 大小為 18 GB)大于工作站的RAM 。 當(dāng)cache=True ，程序開始將圖像存儲(chǔ)在RAM中以進(jìn)行快速訪問，當(dāng)超過RAM( 在我們的情況下為16 GB)時(shí)，程序崩潰。 我已經(jīng)對(duì)較小的數(shù)據(jù)集Kaggledogs_and_cats測試了相同的選項(xiàng)，并且效果很好。 這表明當(dāng)所考慮的數(shù)據(jù)集的大小大于RAM時(shí)，我們不應(yīng)使用cache=True選項(xiàng)。

• When using tf.data with variablecache=False , the program takes approximately 23 minutes, which is 2.5 times faster in comparison to Keras.ImageDataGenerator.

  當(dāng)使用帶有變量cache=False tf.data時(shí)，該程序大約需要23分鐘，比Keras.ImageDataGenerator 快2.5倍 。

• When using cache='some_path.tfcache' , during the first epoch tf.data will make a dump of the dataset/images in your computer directory. This is why it is slower during the first epoch and takes around 42 minutes. In successive epochs, it doesn’t have to store the images again on the computer but use the already created dump during the first epoch, which ultimately speeds up the training times. During successive epochs, it took only 14 minutes for each epoch. Creating the dump is only 1-time process. For the hyperparameter tuning, it takes around 14 minutes in comparison to 58 minutes with Keras , which is approximately 4.14 times faster.

  當(dāng)使用cache='some_path.tfcache' ，在第一個(gè)時(shí)期tf.data將轉(zhuǎn)儲(chǔ)計(jì)算機(jī)目錄中的數(shù)據(jù)集/圖像。 這就是為什么它在第一個(gè)時(shí)期比較慢并且需要大約42分鐘的原因 。 在連續(xù)的時(shí)期中，它不必將圖像再次存儲(chǔ)在計(jì)算機(jī)上，而是在第一個(gè)時(shí)期中使用已經(jīng)創(chuàng)建的轉(zhuǎn)儲(chǔ)，從而最終加快了訓(xùn)練時(shí)間。 在連續(xù)的時(shí)期中， 每個(gè)時(shí)期僅花費(fèi)14分鐘 。 創(chuàng)建轉(zhuǎn)儲(chǔ)只是1次過程。 對(duì)于超參數(shù)調(diào)整，大約需要14分鐘 ，而使用Keras則需要58分鐘 ，這大約快了4.14倍。

Note: The dump created in the memory when using cache='some_path.tfcache'is approximately of size 90 GB which is actually far greater than the original size (18 GB) of the dataset. I couldn’t exactly understand the reason for this as there is no clear documentation from TensorFlow about this. I hope this is the glich which will be sorted out in the future.

注意：使用cache='some_path.tfcache'時(shí)在內(nèi)存中創(chuàng)建的轉(zhuǎn)儲(chǔ)大小約為90 GB，實(shí)際上遠(yuǎn)大于數(shù)據(jù)集的原始大小(18 GB)。 我無法完全理解其原因，因?yàn)門ensorFlow沒有明確的文檔。 我希望這是將來可以解決的問題。

For smaller datasets like Kaggle- dogs_and_cats which is only 217 MB, there will not be a noticeable difference in speed or training times with tf.data and Keras as the RAM is big enough to store all the images at once.

對(duì)于較小的數(shù)據(jù)集像Kaggle- dogs_and_cats這是只有217 MB，不會(huì)有與速度或訓(xùn)練次數(shù)明顯的區(qū)別tf.data和Keras的RAM足夠大到所有圖像保存一次。

訓(xùn)練模型代碼 (Training model code)

Here, the initial code for creating and training the model is shown. Pretrained model mobilenet is used to perform the transfer learning with base layers of pretrained model are frozen.

在這里，顯示了用于創(chuàng)建和訓(xùn)練模型的初始代碼。 預(yù)訓(xùn)練模型mobilenet用于凍結(jié)已預(yù)訓(xùn)練模型基礎(chǔ)層的轉(zhuǎn)移學(xué)習(xí)。

所有的代碼在一起 (All the Code together)

Here I have combined the code from both Part 1 and Part 2. The comments and docstrings for functions will help in understanding the code.

在這里，我結(jié)合了第1部分和第2部分中的代碼。函數(shù)的注釋和文檔字符串將有助于理解代碼。

結(jié)論 (Conclusion)

This article shows thattf.data is 2.5(when using cache=False) to 4.14 (when using cache=`some_path.tfcache`) times faster in comparison to Keras.ImageDataGenerator for any practical image classification problem. I think it is worth to try tf.data.

本文表明， tf.data為2.5 (when using cache=False) 至4.14 (when using cache= ` some_path.tfcache`)相比倍的速度 Keras.ImageDataGenerator為任何實(shí)用的圖像分類問題。 我認(rèn)為值得嘗試tf.data 。

翻譯自: https://medium.com/@sunnychugh/dump-keras-imagedatagenerator-start-using-tensorflow-tf-data-part-2-fba7cda81203

keras核心已轉(zhuǎn)儲(chǔ)

總結(jié)

以上是生活随笔為你收集整理的keras核心已转储_转储Keras-ImageDataGenerator。 开始使用TensorFlow-tf.data（第2部分）的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。