🪄 Enhance/fix data quality w. openai's API 🦾

adriens - Mar 30 '23 - - Dev Community

❔ About

🤔 Sometimes you face lack of data or data quality issues that prevent you from producing insights.

💡 Whatif you could call AI to the rescue to fix/enhance some data

I first started some Prompt engineering on chatGPT:

Image description

☝️ Notice

Notice that guessing gender on firstnames can seem useless or a bit dumb (or nerdy). Yes,but...

  • 🗺️ This work relies on openAI... which acts as a universal language firstname parser
  • 💡 This work is just an illustration of how prompt engineering and OpenAPI'API can help review/fix any kind of data quality issues... and makes a concrete illustration on how you may enrich your enterprise data pipeline

🎯 Target

The purpose of this article is to see how openai's API can help on a very specific testable dataset.

Image description

📝 Kaggle Notebook

This short notebook I will:

  1. 📥 Download data
  2. 🐼 Load data in pandas
  3. 🦾 Call openai's API to guess firstname's gender
  4. ⚖️ Compare guessed vs. real data

Image description

🍿 Demo

🗃️ Input Dataset

I have used the top-10-prenoms-a-noumea-depuis-1860 open dataset from data.gouv.nc:

Top 10 des Prénoms à Nouméa depuis 1860 — Open Data NC

Ce jeu de données présente la liste des dix prénoms les plus donnés à Nouméa, depuis 1860, d'après le registre de l'état civil. Fréquence de mise à jour : Annuelle

favicon data.gouv.nc

🤖 The text-davinci-003 model

I have used text-davinci-003 from GPT-3.5 models as they can:

"understand and generate natural language or code."

Image description

📊 Results 👏

Image description

☝️ Notice

Notice that I have put the guessed value in a dedicated structure... so we can easily flag it as AI generated when reporting its metadatas:

💰 Gains

  • 📈 Data quality
  • 💡 Better decisions & opportunities
  • 💸 Puts the cost of the lack of data quality in evidence (API calls are not free)
  • 🧠 Create more intelligence

👨‍🔬 Further optimizations

  • Benchmark models to spend as less money as possible while getting the best results as possible

🔭 News & perpsectives

Image
Top Tools Every Software Developer Should Know
Image
How to Enable WebSocket Protocol in Windows 11?

windowsfeatures, enablewebsocketproto, howtoenableinwindows, windows11
Image
Levelling up as Developers

certification, learning
Image
The Only Guide You Need to Pass the CKAD (Certified Kubernetes Application Developer) Exam

kubernetes, ckad, exam, container
Image
‘hi’ shows it’s colors

monitoring, linux, cli, bash
Image
This Language has just Solved Concurrency in your Backend Services.

elixir, webdev, backenddevelopment, rust
Image
Dive into the World of Data Mining with Google's Comprehensive Course 🚀

getvm, universitycourses, programming, tutorial
Image
Bootstrap 5: New Features and Updates

css, sass, scss, tailwindcss
Image
How to find WiFi Password in Windows 11?

windows11, tips, tutorial, beginners
Image
Happy Birthday message

codepen
Image
Should You Refinance Your Mortgage? What to Consider

mortgage
Image
ChatGPT vs Sista AI: A Revolutionary Comparison

ai, react, javascript, typescript
Image
Min Stack | LeetCode | Java

java, leetcode, beginners, algorithms
Image
Scan me you will love it <3

codepen
Image
Explore JavaScript Data Types with getType Function

labex, javascript, coding, programming
Image
Hello guys and gals.
Image
美国学位文凭,WFU毕业证成绩单+Q微信234178458维克森林大学学位证成绩单*办理WFU真实学历认证Wake Forest University
Image
Key traits of a brilliant private dentist:

dentist
Image
美国学位文凭,Tufts毕业证成绩单+Q微信234178458塔夫斯大学学位证成绩单*办理Tufts真实学历认证Tufts University
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player