Things I’ve learned from applying GenAI at a traditional company in 2024

TL’DR: Cautiously optimistic on using GenAI at traditional industries.

I work in data science & machine learning engineering team at a challenger energy company in the U.K. Here is a list of things I wish I knew before using GenAI in production.

1. For most companies, ROI of GenAI apps remain spurious.

There is a high chance that you’re doing GenAI projects because:

• Someone high up the corporate ladder has been reading about AI in the Economist/Forbes, and has asked the company to “look into it”
• Someone on your team wants to put GenAI on their CV. This could be yourself.

In 2024, many GenAI projects tend to be PR and marketing spiel, instead of actually impacting the business in significant ways. This is especially true if your company is not the incumbent in your industry (e.g. challengers or those who are going through digital transformation).

2. You’re going to need to retrain your in-house team. This is because GenAI related skills and best practices require a vast domain knowledge and as this is now a new branch of data science, creating the new 4 pillars of data science:

• causal inference & A/B testing (“What is statistical significance?”)
• Traditional ML (Boosted…