Iyini i-AI Dataset?

Uma wakha, uthenga, noma uhlola amasistimu e-AI, uzongena embuzweni owodwa olula ngokukhohlisa & yini idathasethi ye-AI futhi kungani ibaluleke kangaka? Inguqulo emfushane: uphethiloli, incwadi yokupheka, futhi ngezinye izikhathi ikhampasi yemodeli yakho.

Izihloko ongase uthande ukuzifunda ngemva kwalesi:

🔗 I-AI ibikezela kanjani izitayela
Ihlola ukuthi i-AI ihlaziya kanjani amaphethini ukuze ibikezele izehlakalo zesikhathi esizayo nokuziphatha.

🔗 Ungakala kanjani ukusebenza kwe-AI
Amamethrikhi nezindlela zokuhlola ukunemba, ukusebenza kahle, nokuthembeka kwemodeli.

🔗 Ungakhuluma kanjani ne-AI
Isiqondiso sokwenza ukusebenzisana okungcono ukuze kuthuthukiswe izimpendulo ezikhiqizwa yi-AI.

🔗 Iyini i-AI ekhuthazayo
Uhlolojikelele lwendlela ukwaziswa okulolonga ngayo imiphumela ye-AI kanye nekhwalithi yokuxhumana iyonke.

Iyini i-AI Dataset? Incazelo esheshayo 🧩

Iyini isethi yedatha ye-AI? Iqoqo lezibonelo imodeli yakho efunda kuzo noma ehlolwa kuzo. Isibonelo ngasinye sinalokhu:

Okokufaka - izici ezibonwa imodeli, njengamazwibela ombhalo, izithombe, umsindo, imigqa yethebula, ukufundwa kwenzwa, amagrafu.
Okuqondiwe - amalebula noma imiphumela imodeli okufanele iyibikezele, njengezigaba, izinombolo, ububanzi bombhalo, izenzo, noma ngezinye izikhathi lutho nhlobo.
Imethadatha - umongo ofana nomthombo, indlela yokuqoqa, izitembu zesikhathi, amalayisense, ulwazi lwemvume, namanothi ngekhwalithi.

Kucabange njengebhokisi lokudla lasemini elipakishwe ngokucophelela lemodeli yakho: izithako, amalebula, amaqiniso omsoco, futhi yebo, inothi elinamathelayo elithi “ungayidli le ngxenye.” 🍱

Emisebenzini egadiwe, uzobona okokufaka kubhangqwe namalebula asobala. Ngemisebenzi engagadiwe, uzobona okokufaka ngaphandle kwamalebula. Ukuze ufunde ukuqinisa, idatha ivamise ukubukeka njengeziqephu noma ama-trajectories anezifunda, izenzo, imiklomelo. Ngomsebenzi we-multimodal, izibonelo zingahlanganisa umbhalo + isithombe + umsindo kurekhodi elilodwa. Kuzwakala kumnandi; ikakhulukazi amapayipi.

Ama-primer kanye nemikhuba ewusizo: we-Datasheets wama-Datasets usiza amaqembu ukuchaza okungaphakathi nokuthi kufanele kusetshenziswe kanjani [1], kanye nama-Model Cards agcwalisa amadokhumenti edatha ohlangothini lwemodeli [2].

Yini eyenza i-AI Dataset enhle ✅

Masikhulume iqiniso, amamodeli amaningi aphumelele ngoba idathasethi ibingemabi. Isethi yedatha "enhle" ithi:

Omele amacala okusetshenziswa kwangempela, hhayi nje izimo zelebhu.
Ilebulwe ngokunembile, enemihlahlandlela ecacile nokwahlulela ngezikhathi ezithile. Amamethrikhi esivumelwano (isb, izilinganiso zesitayela se-kappa) asiza ingqondo-ukuhlola ukuvumelana.
Kuphelele futhi kulinganisela ngokwanele ukugwema ukwehluleka buthule emigqeni emide. Ukungalingani kuyinto evamile; ukunganaki akujwayelekile.
Sula ekuqaleni, ngemvume, ilayisense, nezimvume ezibhaliwe. Amaphepha ayisicefe avimbela amacala ajabulisayo.
Kubhalwe kahle kusetshenziswa amakhadi edatha noma ama-database achaza ukusetshenziswa okuhlosiwe, imikhawulo, nezindlela zokwehluleka ezaziwayo [1]
Kulawulwa ngokwenziwa kwezinguqulo, ama-changelog, kanye nokuvunyelwa. Uma ungakwazi ukuphinda ukhiqize isethi yedatha, awukwazi ukuphinda ukhiqize imodeli. Isiqondiso esivela ku -NIST's AI Risk Management Framework siphatha ikhwalithi yedatha kanye nemibhalo njengezinto ezikhathazayo zesigaba sokuqala [3].

Izinhlobo ze-AI Datasets, ngalokho okwenzayo 🧰

Ngomsebenzi

Ukuhlelwa - isb, ugaxekile uma uqhathaniswa nogaxekile, izigaba zezithombe.
Ukwehla - bikezela inani eliqhubekayo njengentengo noma izinga lokushisa.
Ukulebula okulandelanayo - izinhlangano eziqanjwe igama, izingxenye zenkulumo.
Isizukulwane - isifinyezo, ukuhumusha, amazwibela esithombe.
Isincomo - umsebenzisi, into, ukuxhumana, umongo.
Ukutholwa okungaqondakali - izehlakalo ezingavamile ochungechungeni lwesikhathi noma amalogi.
Ukufunda kokuqinisa - isimo, isenzo, umvuzo, ukulandelana kwesifunda esilandelayo.
Ukubuyisa - imibhalo, imibuzo, izahlulelo ezifanele.

Ngokwemodi

Ithebula - amakholomu afana neminyaka, imali engenayo, i-churn. Ingaphansi, isebenza ngesihluku.
Umbhalo - imibhalo, izingxoxo, ikhodi, okuthunyelwe kwenkundla, izincazelo zomkhiqizo.
Izithombe - izithombe, ukuskena kwezokwelapha, amathayili wesathelayithi; noma ngaphandle kwemaski, amabhokisi, amaphuzu angukhiye.
Umsindo - amafomu amaza, okulotshiwe, omaka besipikha.
Ividiyo - ozimele, izichasiselo zesikhashana, amalebula esenzo.
Amagrafu - ama-node, imiphetho, izimfanelo.
Uchungechunge lwesikhathi - izinzwa, ezezimali, i-telemetry.

Ngokugadwa

Ilebulwe (igolide, isiliva, ifakwe ilebula ngokuzenzakalelayo), inelebula elibuthakathaka, elingalebulanga, elenziwe. Ingxube yekhekhe ethengwe esitolo ingaba ehloniphekile-uma ufunda ibhokisi.

Ngaphakathi kwebhokisi: isakhiwo, ukuhlukana, nemethadatha 📦

Isethi yedatha eqinile ngokuvamile ihlanganisa:

I-schema - izinkambu ezithayiphiwe, amayunithi, amanani avunyelwe, ukuphatha okuyize.
Ukuhlukana - isitimela, ukuqinisekiswa, ukuhlolwa. Gcina idatha yokuhlola ivaliwe-yiphathe njengocezu lokugcina lukashokoledi.
Uhlelo lwamasampula - ukuthi udwebe kanjani izibonelo kubantu; gwema amasampula alula endaweni eyodwa noma idivayisi.
Ama-Augmentations - ama-flips, izitshalo, umsindo, ama-paraphrases, imaski. Kuhle uma uthembekile; eziyingozi uma zisungula amaphethini angenzeki endle.
Ukuguqulwa - isethi yedatha v0.1, v0.2… kanye nama-changelog achaza ama-delta.
Amalayisense nemvume - amalungelo okusebenzisa, ukusabalalisa kabusha, nokugeleza kokususa. Izilawuli zikazwelonke zokuvikela idatha (isb, i-ICO yase-UK) zihlinzeka ngohlu lokuhlola olusebenzayo, lokucubungula okusemthethweni [4].

Umjikelezo wokuphila wesethi yedatha, isinyathelo ngesinyathelo 🔁

Chaza isinqumo - imodeli izonquma ini, nokuthi kuzokwenzekani uma kungalungile.
Izici zobubanzi namalebula - kuyalinganiseka, kuyaphawuleka, kuyafaneleka ukuqoqwa.
Idatha yomthombo - amathuluzi, amalogi, izinhlolovo, izinkampani zomphakathi, ozakwethu.
Imvume kanye nezomthetho - izaziso zobumfihlo, ukukhetha ukuphuma, ukunciphisa idatha. Bheka isiqondiso somlawuli se-"kungani" kanye "kanjani" [4].
Qoqa futhi ugcine - isitoreji esivikelekile, ukufinyelela okusekelwe endimeni, ukuphatha kwe-PII.
Ilebula - izichasiselo zangaphakathi, i-crowdsourcing, ochwepheshe; phatha ikhwalithi ngemisebenzi yegolide, ukucwaninga, namamethrikhi esivumelwano.
Hlanza futhi wenze kube ngokwejwayelekile - dedupe, phatha ukushoda, linganisa amayunithi, lungisa umbhalo wekhodi. Umsebenzi oyisicefe, nobuqhawe.
Hlukanisa futhi uqinisekise - vimbela ukuvuza; hlela lapho kufanele khona; khetha ukuhlukaniswa okwazi isikhathi kudatha yesikhashana; futhi usebenzise ukuqinisekiswa okuphambene ngokucabangisisa ngezilinganiso eziqinile [5].
Idokhumenti - idatha yedatha noma ikhadi ledatha; ukusetshenziswa okuhlosiwe, izexwayiso, imikhawulo [1].
Gada futhi ubuyekeze - ukutholwa kokukhukhuleka, ukuvuselela ijubane, izinhlelo zokushona kwelanga. I-AI RMF ye-NIST ihlela lolu hlaka lokuphatha oluqhubekayo [3].

Ithiphu elisheshayo, elimise okomhlaba wangempela: amaqembu ngokuvamile "awine idemo" kodwa akhubeke ekukhiqizeni ngoba idathasethi yawo inyakazisa imigqa yomkhiqizo emisha buthule, inkambu eqanjwe kabusha, noma inqubomgomo eshintshile. I-changelog elula + iphasi yesichasiselo esiphindaphindwayo igwema iningi lalobo buhlungu.

Ikhwalithi yedatha nokuhlola - hhayi njengokufiphala njengoba kuzwakala 🧪

Ikhwalithi ine-multi-dimensional:

Ukunemba - ingabe amalebula alungile? Sebenzisa amamethrikhi esivumelwano nokwahlulela ngezikhathi ezithile.
Ukuphelela - hlanganisa izinkambu namakilasi owadinga ngempela.
Ukuvumelana - gwema amalebula aphikisanayo okokufaka okufanayo.
Isikhathi - idatha endala idala ukuqagela.
Ukulunga kanye nokubandlulula - ukumbozwa kuzo zonke izinhlobo zabantu, izilimi, amadivayisi, izindawo; qala ngokuhlolwa okuchazayo, bese kuba ukuhlolwa kokucindezeleka. Imibhalo - kuqala imikhuba (amashidi edatha, amakhadi amamodeli) yenza lokhu kuhlolwa kubonakale [1], kanti izinhlaka zokuphatha zigcizelela lokhu njengokulawula ubungozi [3].

Ukuze uthole ukuhlolwa kwemodeli, sebenzisa ukuhlukaniswa okufanele futhi ulandelele kokubili ama-metrics amaphakathi namamethrikhi eqembu elibi kakhulu. Isilinganiso esicwebezelayo singafihla i-crater. Okuyisisekelo kokuqinisekisa okuphambanayo kuhlanganiswe kahle kumadokhumenti ajwayelekile wamathuluzi e-ML [5].

Izimiso zokuziphatha, ubumfihlo, kanye nokunikezwa kwelayisensi - i-Guardrails 🛡️

Idatha yesimilo ayiyona i-vibe, yinqubo:

Imvume kanye nomkhawulo wenhloso - cacisa ngokusetshenziswa kanye nezisekelo zomthetho [4].
Ukuphathwa kwe-PII - nciphisa, zenze mbumbulu, noma uveze igama elingaziwa ngendlela efanele; cabanga ngobuchwepheshe bokuthuthukisa ubumfihlo lapho izingozi ziphezulu.
Ukunikezwa kanye namalayisense - hlonipha imikhawulo yokwabelana kanye nokusetshenziswa kwezentengiselwano.
Ubandlululo kanye nokulimala - ukuhlola ukuhlobana okungamanga (“ukukhanya kwasemini = okuphephile” kuzodideka kakhulu ebusuku).
Ukulungisa - yazi ukuthi ungayikhipha kanjani idatha lapho ucelwa nokuthi ungawabuyisela kanjani amamodeli aqeqeshwe kuyo (bhala lokhu kudatha yakho yedatha) [1].

Ingakanani enkulu ngokwanele? Ukulinganisa nokukhombisa-emsindweni 📏

Umthetho oyisisekelo: izibonelo eziningi zivame ukusiza uma zifaneleka futhi zingacishe zifane nezifanekiso. Kodwa ngezinye izikhathi kungcono ukuthi ube ambalwa, ahlanzekile, futhi anelebula engcono kunezintaba eziningi ezingcolile.

Bukela:

Ama-curve okufunda - ukusebenza kwesakhiwo vs. usayizi wesampula ukuze ubone ukuthi uboshelwe yidatha noma imodeli.
Ukumbozwa komsila omude - amakilasi angajwayelekile kodwa abalulekile ngokuvamile adinga iqoqo eliqondiwe, hhayi nje inqwaba eyengeziwe.
Lebula umsindo - isilinganiso, bese unciphisa; okuncane kuyabekezeleleka, igagasi lolwandle alikwazi.
Ukushintsha kokusatshalaliswa - idatha yokuqeqeshwa isuka endaweni ethile noma isiteshi ingase ingafinyeleleki kwesinye; qinisekisa kudatha yokuhlola efana nethagethi [5].

Uma ungabaza, gijima abashayeli bezindiza abancane futhi wandise. Kufana nesinongo-engeza, ukunambitha, ukulungisa, ukuphinda.

Lapho ungathola khona futhi uphathe idathasethi 🗂️

Izinsiza ezidumile kanye namathuluzi (asikho isidingo sokukhumbula ama-URL njengamanje):

I-Hugging Face Datasets - ukulayisha ngohlelo, ukucubungula, ukwabelana.
I-Google Dataset Search - i-meta-search kuwebhu yonkana.
I-UCI ML Repository - ezakudala ezikhethiwe zezisekelo nokufundisa.
I-OpenML - imisebenzi + idathasethi + isebenza ngokuqala.
I-AWS Open Data / I-Google Cloud Public Datasets - isingathwe, izinkampani ezinkulu.

Icebiso lochwepheshe: ungalandi nje kuphela. Funda ilayisensi kanye neshidi ledatha, bese ubhala phansi ikhophi yakho enezinombolo zenguqulo kanye nemvelaphi [1].

Ukulebula nesichasiselo - lapho kuxoxwa khona ngeqiniso ✍️

Isichasiselo kulapho umhlahlandlela welebula yakho yetiyori ubambana neqiniso:

Idizayini yomsebenzi - bhala imiyalelo ecacile ngezibonelo nezibonelo eziphikisayo.
Ukuqeqeshwa kwe-Annotator - imbewu enezimpendulo zegolide, gijima imijikelezo yokulinganisa.
Ukulawulwa kwekhwalithi - sebenzisa amamethrikhi esivumelwano, izindlela zokuvumelana, nokuhlolwa kwezikhathi ezithile.
Amathuluzi - khetha amathuluzi aphoqelela ukuqinisekiswa kwe-schema kanye nemigqa yokubuyekeza; ngisho namaspredishithi angasebenza ngemithetho namasheke.
Izihibe zempendulo - bamba amanothi esichasiselo namaphutha emodeli ukuze ucwenge umhlahlandlela.

Uma kuzwakala njengokuhlela isichazamazwi nabangani abathathu abaphikisana ngokhefana… kujwayelekile lokho. 🙃

Ukubhalwa kwedatha - ukwenza ulwazi oluyimfihlo lube sobala 📒

Imininingwane engasindi noma ikhadi ledatha kufanele lifake:

Iqoqwe ngubani, kanjani, futhi ngani.
Ukusetshenziswa okuhlosiwe kanye nokusetshenziswa ngaphandle kobubanzi.
Izikhala ezaziwayo, ukuchema, nezindlela zokwehluleka.
Iphrothokholi yokulebula, izinyathelo ze-QA, nezibalo zesivumelwano.
Ilayisense, imvume, ukuthintana ngezinkinga, inqubo yokususa.

Izifanekiso nezibonelo: Ama-Datasheets Amasethi Edatha Namakhadi Amamodeli asetshenziswa kabanzi izindawo zokuqala [1].

Yibhale usakha, hhayi ngemva kwalokho. I-Memory iyindawo yokugcina engaqinile.

Ithebula Lokuqhathanisa - izindawo zokuthola noma zokusingatha idatha ye-AI 📊

Yebo, lokhu kunombono omncane. Futhi amagama awalingani ngenhloso. Kulungile.

Ithuluzi / Repo	Izithameli	Intengo	Kungani kusebenza ngokoqobo
Amasethi Wedatha Wobuso Obugonayo	Abacwaningi, onjiniyela	Isigaba samahhala	Ukulayisha okusheshayo, ukusakaza, imibhalo yomphakathi; amadokhumenti amahle kakhulu; amasethi edatha ahumushiwe
Usesho lwe-Google Dataset	Wonke umuntu	Mahhala	Indawo ebanzi; kuhle ukutholwa; kwesinye isikhathi imethadatha engahambelani tho
I-UCI ML Repository	Abafundi, othisha	Mahhala	Ama-classics akhethiwe; encane kodwa ihlanzekile; kuhle kwesisekelo kanye nokufundisa
I-OpenML	Abacwaningi be-Repro	Mahhala	Imisebenzi + amasethi edatha + isebenza ndawonye; izitayela ezinhle zokugqoka
I-AWS Vula Irejista Yedatha	Onjiniyela bedatha	Ikakhulukazi mahhala	ukusingathwa kwe-Petabyte-scale; ukufinyelela komdabu wasefu; bukela izindleko zokuphuma
Kaggle Datasets	Abasebenzi	Mahhala	Ukwabelana okulula, imibhalo, imincintiswano; izimpawu zomphakathi zisiza ukuhlunga umsindo
I-Google Cloud Public Datasets	Abahlaziyi, amaqembu	Mahhala + ifu	Ibanjwe eduze kwe-compute; Ukuhlanganiswa kwe-BigQuery; qaphela ngokukhokhisa
Izingosi zezemfundo, amalebhu	Ochwepheshe be-Niche	Kuyahlukahluka	Okukhethekile kakhulu; kwesinye isikhathi akubhalwa phansi-kusafanele ukuzingelwa

(Uma iseli libukeka linengxoxo, lokho kuyinhloso.)

Ukwakha eyokuqala - ikhithi yokuqalisa esebenzayo 🛠️

Ufuna ukusuka kokuthi "iyini idathasethi ye-AI" uye kokuthi "Ngenzile, iyasebenza." Zama le ndlela encane:

Bhala isinqumo kanye nemethrikhi - isb, yehlisa izindlela ezingalungile zosekelo ngokubikezela iqembu elilungile. Imethrikhi: macro-F1.
Bhala izibonelo ezi-5 ezinhle nezi-5 ezimbi - bonisa amathikithi angempela; ungazenzi.
Yenza umhlahlandlela welebula - ikhasi elilodwa; imithetho ecacile yokufaka/yokukhishwa.
Qoqa isampula encane, yangempela - amathikithi angamakhulu ambalwa kuzo zonke izigaba; susa i-PII ongayidingi.
Hlukanisa ngokuhlolwa kokuvuza - gcina yonke imilayezo evela kukhasimende elifanayo ekuhlukaniseni okukodwa; sebenzisa ukuqinisekiswa okuphambene ukuze ulinganisele ukuhluka [5].
Chasisa nge-QA - izichasiselo ezimbili kusethi encane; xazulula ukungezwani; buyekeza umhlahlandlela.
Qeqesha isisekelo esilula - ukuhamba kuqala (isb, amamodeli aqondile noma ama-compact transformer). Iphuzu liwukuhlola idatha, hhayi ukuwina izindondo.
Buyekeza amaphutha - lapho yehluleka futhi kungani; buyekeza idathasethi, hhayi imodeli kuphela.
Idokhumenti - idatha yedatha encane: umthombo, isixhumanisi senkomba yelebula, ukuhlukaniswa, imikhawulo eyaziwayo, ilayisensi [1].
Hlela ukuvuselela - izigaba ezintsha, isitsotsi esisha, izizinda ezintsha ziyafika; ukuhlela okuncane, izibuyekezo ezivamile [3].

Uzofunda okuningi kule luphu kunokuthatha eziyinkulungwane ezishisayo. Futhi, gcina izipele. Ngiyacela.

Izingibe ezivamile ezehlela amaqembu 🪤

Ukuvuza kwedatha - impendulo ingena ezicini (isb, ukusebenzisa izinkambu zokuxazulula le nkinga ukubikezela imiphumela). Kuzwakala njengokukopela ngoba kunjalo.
Ukwehlukahlukana okungajulile - indawo eyodwa noma idivayisi izenza imbulunga yonke. Ukuhlolwa kuzoveza ukujika kwesakhiwo.
Ukuzulazula kwelebula - imigomo iyashintsha ngokuhamba kwesikhathi kodwa umhlahlandlela welebula awushintshi. Bhala phansi futhi uguqule i-ontology yakho.
Izinhloso ezingacaciswanga kahle - uma ungakwazi ukuchaza isibikezelo esibi, idatha yakho nayo ngeke ichaze.
Amalayisense angcolile - ukukhuhla manje, ukuxolisa kamuva, akulona isu.
I-Over-augmentation - idatha yokwenziwa efundisa ama-artifact angelona iqiniso, njengokuqeqesha umpheki ngesithelo sepulasitiki.

Imibuzo Evame Ukubuzwa Esheshayo mayelana nebinzana ngokwalo ❓

Ingabe "Kuyini isethi yedatha ye-AI?" kuyinto nje yokuchaza? Ngokuvamile, kodwa futhi kuyisibonakaliso sokuthi uyakhathalela izinto eziyisicefe ezenza amamodeli athembeke.
Ingabe ngihlala ngidinga amalebula? Cha. Ukungagadiwe, ukuzigada, kanye nokusetha kwe-RL kuvame ukweqa amalebula asobala, kodwa ukucushwa kusebalulekile.
Ngingasebenzisa idatha yomphakathi kunoma yini? Cha. Hlonipha amalayisensi, imigomo yenkundla, nezibopho zobumfihlo [4].
Okukhulu noma okungcono? Kokubili, ngokufanelekile. Uma kufanele ukhethe, khetha okungcono kuqala.

Amazwi Okugcina - Ongakuthwebula isithombe-skrini 📌

Uma othile ekubuza ukuthi iyini isethi yedatha ye-AI, yithi: iyiqoqo lezibonelo ezikhethiwe nezilotshiwe ezifundisa futhi zivivinya imodeli, ezihlanganiswe nokuphatha ukuze abantu bathembe imiphumela. Amasethi edatha amahle kakhulu amele, anelebula elihle, ahlanzekile ngokomthetho, futhi agcinwa njalo. Okunye imininingwane - imininingwane ebalulekile - mayelana nesakhiwo, ukuhlukaniswa, nazo zonke lezo zithiyo ezincane ezivimbela amamodeli ukuthi angahambi emgwaqweni. Ngezinye izikhathi inqubo izwakala njengokulima ngamaphepha okushicilela; ngezinye izikhathi njengokwelusa amaphikseli. Noma ngabe yikuphi, tshala imali kudatha, futhi amamodeli akho azokwenza izinto ngendlela engajwayelekile. 🌱🤖

Izinkomba

[1] Amashidi edatha wamasethi edatha - Gebru et al., arXiv. Isixhumanisi
[2] Amakhadi Emodeli Okubika Amamodeli - Mitchell et al., arXiv. Isixhumanisi
[3] Uhlaka Lokuphathwa Kwengozi Ye-NIST Artificial Intelligence (AI RMF 1.0). Isixhumanisi
[4] Isiqondiso nezinsiza ze-UK GDPR - Ihhovisi LikaKhomishana Wolwazi (ICO). Isixhumanisi
[5] Ukuqinisekiswa Okuphambene: ukuhlola ukusebenza komqageli - scikit-learn Umhlahlandlela Womsebenzisi. Isixhumanisi

Thola i-AI Yakamuva Esitolo Esisemthethweni Somsizi we-AI

Mayelana NATHI

Buyela kubhulogi