Why is the cost of AI data collection so high?

[Lunaproxy]

I don't know why, but I feel that the cost of my data collection every time is extremely high. It may be largely related to the fact that I don't have clear requirements. Also, I haven't made good use of data sharing and open-source data. Can anyone give me better suggestions regarding the data collection process? Currently, I'm collecting data related to e-commerce. Or if you have better methods to save costs and improve efficiency, your guidance is most welcome, and I would be extremely grateful.

 

[sharonmsom]

Hi! First of all, I really appreciate your honesty — identifying the issue is already a big step forward.


Since you’re collecting data related to e-commerce, here are a few tips to reduce costs and improve efficiency:

1. Define Clear Objectives First
   Before collecting any data, ask: What exactly do I want to learn? Define the KPIs or business questions you’re trying to answer — this will help you avoid wasting time and resources on irrelevant data.
2. Use Open Datasets
   Platforms like Kaggle, Google Dataset Search, and data.gov offer free e-commerce-related datasets. For example, product reviews, transaction simulations, pricing models, etc. You might find what you need without starting from scratch.
3. Web Scraping with Limits
   If you scrape websites like Amazon or Shopify stores, use tools like Scrapy or Octoparse with rate-limiting and specific filters to avoid over-collecting data you don’t use.
4. Collaborate or Share Data
   Look for Slack groups, LinkedIn communities, or research forums in the e-commerce space. Many data enthusiasts are willing to collaborate and share datasets.
5. Automate Smartly
   If you’re manually collecting data, try automating the process using Python scripts or tools like Make.com or Zapier. Set conditions to collect only new or updated data.

If you’d like, feel free to share more details about what type of e-commerce data you’re working on — maybe I can point you to something even more specific!