照顧好自己才能照顧好別人

I am pretty sure that on your data journey you came across some courses, videos, articles, maybe use cases where someone takes some data, builds a classification/regression model, shows you great results, you learn how that model works and why it works that way and not another and everything seems to be fine. You think you just learned a new thing (and you did), you are happy about that (yes, you are ! I am not kidding around here, you’re doing great!) and you continue to the next piece of content.

我很確定在您的數(shù)據(jù)之旅中，您遇到了一些課程，視頻，文章，也許是用例，其中有人獲取了一些數(shù)據(jù)，建立了分類/回歸模型，向您展示了出色的結(jié)果，您了解了該模型的工作原理以及工作原理這樣，而不是另一種，一切似乎都很好。 您以為自己剛剛學(xué)到了新事物(并且確實做到了)，對此感到很高興(是的，您是！我在這里不是在開玩笑，您做得很好！)，并且您繼續(xù)閱讀下一篇內(nèi)容。

But later on you start to ask additional questions (everyone has different length of that “l(fā)ater on”), like: where did that data come from? and if I have more data, will that model run so smoothly as it did during the demonstration? does the data in real world exist in such format? can I get similar data and if I can will it be so easy to process? what did the results of that model mean? can I present that data in prettier way? and so on and so on and so on.

但是稍后，您開始提出其他問題(每個人的“以后”的時長都不同)，例如：數(shù)據(jù)來自何處？ 如果我有更多數(shù)據(jù)，該模型是否會像演示期間那樣平穩(wěn)運行？ 現(xiàn)實世界中的數(shù)據(jù)是否以這種格式存在？ 我可以得到類似的數(shù)據(jù)嗎？如果可以的話，它會很容易處理嗎？ 該模型的結(jié)果是什么意思？ 我可以用更漂亮的方式顯示這些數(shù)據(jù)嗎？ 等等，依此類推。

When I started to learn about data analytics, data science, world of data in general I was always amused by the results people will get after processing some piece of data, or after running a machine learning model or after getting keys from word buckets etc. But every time I would try to do something on my own it will always appear a new obstacle: the data I would like to analyze is too much or not enough, one model will run with one piece of data, but it won’t with another etc etc.

當(dāng)我開始學(xué)習(xí)數(shù)據(jù)分析，數(shù)據(jù)科學(xué)，一般的數(shù)據(jù)領(lǐng)域時，我總是對人們在處理某些數(shù)據(jù)，運行機器學(xué)習(xí)模型或從單詞存儲桶中獲取鍵等后所獲得的結(jié)果感到很滿意。但是每次我嘗試自己做某事時，總會出現(xiàn)新的障礙：我想分析的數(shù)據(jù)太多或不夠，一個模型將只處理一個數(shù)據(jù)，而不會其他等等

After having all these difficulties and learning to deal with them the hard way I would like to share with the essential 5 Vs of data that you have to have taken care of before you start your data project/solution.

在經(jīng)歷了所有這些困難并學(xué)會了用困難的方式解決這些問題之后，我想與您在開始數(shù)據(jù)項目/解決方案之前必須要處理的5個基本數(shù)據(jù)進行分享。

1st V —音量 (1st V — Volume)

When we talk “volume” in regards of data we have to be aware of amount of data that has to be handled in the project — should we use several servers to handle that volume and distribute the load between them? or maybe our computer with our own hard disk is quite enough to solve the problem?

當(dāng)我們談?wù)摂?shù)據(jù)的“卷”時，我們必須知道項目中必須處理的數(shù)據(jù)量-我們是否應(yīng)該使用多個服務(wù)器來處理該卷并在它們之間分配負載？ 還是我們擁有自己的硬盤的計算機足以解決問題？

2nd V —速度 (2nd V — Velocity)

Velocity is the speed with which data travels through our model/project/solution. The speed with which it is ingested, processed and delivered to the end client. We have to be aware if this is real-time data, near real-time or maybe this is just historic data which is not going anywhere soon and we can talk her out slowly and efficiently 😉

速度是數(shù)據(jù)在我們的模型/項目/解決方案中傳播的速度。 攝取，處理并交付給最終客戶端的速度。 我們必須知道這是實時數(shù)據(jù)還是接近實時數(shù)據(jù)，或者這僅僅是歷史數(shù)據(jù)，很快就不會流傳了，我們可以慢慢有效地與她交談talk

3rd V —綜藝 (3rd V — Variety)

Data comes from various sources, in various types, structured, semi-structured and not structured at all (officially unstructured XD) and boy, I’ve got burned on it a lot. When my pipeline will expect one data type (because I tested it with the sample and it worked) and then it will give me an error because there is additional type or structure that is not yet supported by my solution. This kind of things has to be defined in the beginning, you have to know the levels of variety of the data you are working with.

數(shù)據(jù)來自各種類型，結(jié)構(gòu)化，半結(jié)構(gòu)化和根本沒有結(jié)構(gòu)化(官方非結(jié)構(gòu)化XD)的各種來源，而我對此非常著迷。 當(dāng)我的管道需要一種數(shù)據(jù)類型時(因為我已經(jīng)用示例對其進行了測試并且可以工作)，然后它會給我一個錯誤，因為我的解決方案尚不支持其他類型或結(jié)構(gòu)。 這類事情必須在一開始就定義好，您必須知道所使用的各種數(shù)據(jù)的級別。

4th V —準確性 (4th V — Veracity)

Is the data I am working with is worth trusting? Is it trustworthy? Is it still correct after all the manipulations and cleanings? Was the pipe of transformation correct? These are the questions we ask when we talk about the veracity of the data. We can collect all the data we need and it won’t be that difficult, but will it be accurate and consistent, won’t it be falsely altered — that’s another challenge. We all aware that in order to get insights from the data we have to perform a little of preprocessing and we have to make sure that process does not skew the data.

我正在使用的數(shù)據(jù)值得信任嗎？ 值得信賴嗎？ 經(jīng)過所有的操作和清潔后，它仍然正確嗎？ 轉(zhuǎn)型的管道正確嗎？ 這些是我們談?wù)摂?shù)據(jù)準確性時要提出的問題。 我們可以收集所需的所有數(shù)據(jù)，并不會那么困難，但是它將是準確且一致的，不會被錯誤地更改，這是另一個挑戰(zhàn)。 我們都知道，為了從數(shù)據(jù)中獲得見解，我們必須執(zhí)行一些預(yù)處理，并且必須確保過程不會使數(shù)據(jù)傾斜。

5th V —價值 (5th V — Value)

And the last V goes for value. Because in the end of the day the whole point of all this is to get value from data. That includes creating reports and dashboards, finding useful insights that can improve business, highlighting critical areas to make more informed decisions.

最后一個V表示價值。 因為歸根結(jié)底，這一切的全部目的都是從數(shù)據(jù)中獲取價值。 這包括創(chuàng)建報告和儀表板，發(fā)現(xiàn)可以改善業(yè)務(wù)的有用見解，突出顯示關(guān)鍵區(qū)域以做出更明智的決策。

You may object that those are 5 Vs of big data and you will be right. Yes, those are 5 Vs of big data, but not only. Any data project has to deal with these 5 Vs. Big data project will have it more complicated to handle, small data project will be just easier to manage all 5 of Vs.

您可能會反對說那是大數(shù)據(jù)的5 V，您將是對的。 是的，那是大數(shù)據(jù)的5 V，但不僅如此。 任何數(shù)據(jù)項目都必須處理這5個V。 大數(shù)據(jù)項目將使其更復(fù)雜，小數(shù)據(jù)項目將更易于管理所有5個V。

For example, I was working on a data solution for the HR department and in the beginning we had to address the 5 Vs of the data. Even though we didn’t have terabytes of data, we had a lot of small Excel files were the data was previously stored and distributed (volume). There were 3 different sources of data to collect from: Excel files, corporate DB and corporate CRM (variety). The data would be updated on a daily basis and users would want the actual data as quickly as possible with a maximum delay of 30 minutes — it’s not even close to real-time, but we still have to make sure that the pipeline is executed fast enough (velocity). Data coming from Excels would be always altered by the human at some point of time and there is always a dispute which actualization goes first, so we had to deal with that too (veracity). And in order to get value from the data we had to find a way to visualize it and create a possibility for the end user to explore it and make their own conclusions (value).

例如，我正在為人事部門開發(fā)數(shù)據(jù)解決方案，一開始我們必須處理5 V數(shù)據(jù)。 即使我們沒有太字節(jié)的數(shù)據(jù)，我們還是有很多小的Excel文件，這些文件是以前存儲和分發(fā)(卷)的數(shù)據(jù)。 有3種不同的數(shù)據(jù)來源可供收集：Excel文件，公司DB和公司CRM(品種)。 數(shù)據(jù)將每天進行更新，并且用戶希望盡可能快地獲取實際數(shù)據(jù)，最大延遲為30分鐘-甚至不接近實時，但我們?nèi)匀槐仨毚_保管道能夠快速執(zhí)行足夠(速度)。 來自Excel的數(shù)據(jù)將始終在某個時間點被人類更改，并且始終存在首先實現(xiàn)的爭議，因此我們也必須處理(準確性)。 為了從數(shù)據(jù)中獲得價值，我們必須找到一種可視化的方法，并為最終用戶創(chuàng)造一種探索它并得出自己的結(jié)論(價值)的可能性。

We invested our time in the beginning to find the solutions for every V with our data and having done that we were able to finish our project just in time — even with lovely documentation.

我們從一開始就投入了時間，使用我們的數(shù)據(jù)為每個V查找解決方案，并且這樣做即使沒有精美的文檔也能及時完成我們的項目。

So even though you are just going to process Titanic datasets, think of 5 Vs, it will take you 2 minutes, but you will be ready for the unpredictable. despite you know who’s gonna die there XD.

因此，即使您只是要處理Titanic數(shù)據(jù)集，以5 V為例，它也將花費您2分鐘的時間，但您已經(jīng)為不可預(yù)測的事情做好了準備。 盡管您知道誰會在那里死XD。

Originally published at https://sergilehkyi.com on August 10, 2020.

最初于 2020年8月10日 發(fā)布在 https://sergilehkyi.com 上。

翻譯自: https://medium.com/swlh/5-essential-data-vs-you-have-to-take-care-of-b4e03e8964c1

照顧好自己才能照顧好別人

總結(jié)

以上是生活随笔為你收集整理的照顾好自己才能照顾好别人_您必须照顾的5个基本数据的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。