Marcin: Hi, my name is Marcin Kokott and welcome to Product Odyssey, the podcast where we try to find out ways to build products fundamentally better. Today with me are Mike and Maciej.

Mike: Hi.

Maciej: Hi.

Marcin: How are you?

Mike: Good, thanks.

Marcin: Today, let's dive into the topic, a hype topic, a hot topic, which is AI. A very loaded word, right? But let's look at it from the perspective of a CEO or CTO who might be interested in AI or building AI solutions. They are thinking about how to organize and build the solution without taking too much risk on the cost. How not to bankrupt by going with AI. But let's start with some basics, right? AI is such a hot topic. I cannot believe that we can discuss anything without assuming that in two weeks, it can change a lot.

Marcin: We went through 2023 where AI was a marketing thing. If you have a solution, it needs to have AI to sell. If you have an AI logo or a topic, everybody buys that and invests. But still, we haven't seen any huge practical implementation or solution built on top of that. That's probably where we are going with this in 2024. When we think about AI, we look at multiple things behind that, and everyone seems to be talking about chatbots. Is it so that all the potential behind AI is just chatbots?

Maciej: I understand where this is coming from, especially that for the last two years, AI is basically synonymous with chatbots. But AI as a discipline goes back to the 1950s. This is not a new thing; there has been a lot of progress since then. For the last two years, AI has gone mainstream mostly because of ChatGPT. It's mass media, it's all around us. It's a very hyped topic. So, if you're building some product in IT, you want to have everything connected with AI, right? Anything that we do, everybody is saying, "Can we add AI inside?" We are building a calculator, and can we have AI inside?

Marcin: Exactly. It seems like a hype topic. It's not just big tech companies entering the floor like Google and Apple. Every big company is trying to experiment with open AI topics. That's why it's in the mainstream. Everyone is talking about chatbots, typing a prompt, and getting an answer. But is this the only thing we should look for? Is there any bigger potential or examples of what we can do with AI beyond just writing?

Maciej: I think this is the current state. When we think AI nowadays, it's mostly text interfaces, mostly chatbots. But I believe that 2024 will bring a paradigm shift in this regard. Multimodality will be the hottest topic this year. We are recording this in early 2024, so it's hard to predict what's going to happen later this year. But I guess that multimodality and interacting with AI in other means than just text will be the main thing we discuss at the end of the year.

Maciej: We got a preview of this with all the hype that Sora generated. Mainstream is now aware that AI is not only text; it's also video. Tools like MidJourney or DALL-E are common nowadays to generate images using AI. This is nothing new. Most people who use AI daily already know that you can have multimodality support. Even in ChatGPT, you can upload images and ask about them. You get the image as input and text as output. This is just the beginning. These are still the early, humble beginnings of a completely new industry where all those modalities will mix and blend. Nowadays, text is basically a medium, a glue that connects everything, at least in the products we interact with daily. This will change this year from a technical standpoint. Based on the research papers, there is no need to use text as an intermediary. But as tool makers, we, who create consumer-facing products, are not aware of this yet. This is the biggest change that my prediction is for this year, that people and products will make use of it. If you're not doing this right now, it's a good moment to actually get into the game.

Marcin: That's a good message. I guess everybody is feeling that you need to get in the game. Maybe most of the cases if I'm a small company, a midsize company, or a startup. But you've mentioned just to move back to the multimodality, to check if I understand it correctly. The aspect is that text is something we are familiar with, so I'm putting text inside and getting text outside. But what you have mentioned is that we are going far beyond that. Examples are shown that it's going beyond the form. Image transformation to video, or video to image, or scanning my food and getting a number of calories in it, or any kind of different form, like voice to text, which is known. Text to speech is known, but there is far more.

Mike: Yes, there are many proof of concepts. Not many shipped market products yet, but many proof of concepts are really interesting. For example, people talk with an AI agent on a computer to create a complete web app without touching the keyboard at all.

Marcin: I've seen that, it's crazy. You don't need a development team. You write one sentence, and the AI creates the code, resulting in a fully working application in minutes.

Mike: Exactly. This is also a paradigm shift for how we deliver products to the market. Code is just a means to achieve the goal, not the goal itself. You need to know how to talk with the AI agent to achieve what you want. This may change with AGI and other progress, but currently, AI is less context-aware than a typical developer. You need to be more descriptive. These proof of concepts allow quick prototyping and testing with early adopters to ensure traction.

Marcin: It seems we are in the same situation but with far greater speed and advancement like with the internet and approach to building solutions. Now it's not a question if something is possible to be built; we can build anything, even beyond software to hardware. The question is only if you have time, money, and the team. With AI, we have advanced far faster. It's like a new internet. You've mentioned two interesting things: faster and cheaper prototyping but creating a competitive advantage and scaling up is still challenging. What should a CEO look for as a risk when approaching new AI projects?

Mike: There are multiple risks. First, we need to have the right skill sets in-house. If we haven't built AI products before, we likely don't have anyone capable of delivering such products. We need to hire either in-house or use consultancy. In the past, we thought about data engineers and data scientists for machine learning. Now, we need people with model thinking, prompt engineering, etc.

Maciej: Prompt engineering will become a basic skill everyone needs. Multimodalities may change this, but currently, prompt engineering is the next coding. Everyone should be able to do it at a basic level to be efficient in the workplace. Skill set gaps are one issue. Another often overlooked is cost. AI solutions can be very expensive if not done correctly. New startups often integrate with proprietary foundation models via APIs like GPT models from OpenAI or Vertex from Google. This is fine for quick prototyping and going to market, but once shipped, costs can be higher than anticipated. Development costs are usually fixed, but AI integration is pay-per-usage, similar to serverless. This can quickly escalate if not managed properly.

Marcin: I can imagine situations where a programming bug causes an infinite loop, ramping up costs quickly. Is there a simple fix like putting a cap on the number of prompts or budgets to secure against this?

Mike: Yes, creating caps or alerts can help manage costs and prevent runaway usage. But if the product is extremely successful, sudden high usage can still cause issues. You want success but not at the cost of hitting caps and stopping new users from accessing your product.

Maciej: Exactly. You can't create hard caps that risk cutting off users. Large enterprises or VC-backed companies might absorb the cost difference, but better solutions involve managing user journeys and calculating typical usage to ensure costs are controlled. This requires expertise and proper planning.

Marcin: Regarding costs, besides usage, what other aspects should we consider, like infrastructure or daily operations?

Maciej: If integration is basic, you pay by token, but there are ways to optimize this. If training your own models or feeding models with your own data, other costs come into play. The easiest way is using your internal knowledge and injecting it into the model's context. This requires storage and vector databases. Tools can connect to internal databases, retrieve data, and make it usable for the model.

Mike: Fine-tuning models is another approach, more expensive initially but potentially more cost-effective long-term. High-context models like Google Gemini Pro with a million-token context might change things, but they will be more costly. You either adapt existing models with your data or fine-tune them to reduce long-term costs.

Marcin: So, either use models as they are, connecting them to your data, or invest in training models to reduce data retrieval costs. What about unstructured data?

Maciej: Models can extract knowledge from unstructured data, but a data engineer can ensure meaningful output. Businesses sitting on valuable data might not even be aware of it. Multimodalities include unstructured data like logs, user interactions, and events. AI can find patterns and context in this data, making it useful.

Marcin: So, AI models are good at understanding patterns. Connecting them to your data can reveal insights. But it's a big effort, so costs are a factor. Should we drop everything into the model or be selective?

Maciej: It depends on the use case. Connecting all internal knowledge is not ideal due to security and compliance. AI models can be hacked, and prompt injections can extract sensitive data. Careful management of what data is used is crucial.

Marcin: Building a prototype is easy, but scaling it with security and accuracy is where costs appear. You mentioned creating competitive advantages. What should we look for in this context?

Mike: Data is king. If you can quickly integrate with public APIs and go to market, anyone can. You have no competitive advantage. Proprietary data is the best moat. If you've been in the market for years, look into your datasets. You might have valuable data you're not aware of. External experts can help identify this value.

Marcin: It's past the R&D phase for AI; it's mainstream. We need to be aware of costs and avoid getting the latest models without returns. Is it too late if I'm not on the AI wave yet?

Maciej: The best moment to get involved was years ago; the second best is now. We're still on the rising tide of AI involvement. It's not too late to catch up and make use of advantages. AI is an efficiency multiplier. If you have a product, use AI to get up to speed quickly. Not using AI now is not a wise business decision.

Marcin: We've covered a broad area regarding AI costs. The main takeaways for a CEO are the costs of hiring specific skill sets, the processing and training costs, and the importance of competitive advantages. Think about specific use cases and involve experts. AI is here to stay. Thank you, Mike and Maciej, for the insights. We'll continue this discussion with our guests in future episodes. Thank you for watching Product Odyssey. Remember to give us feedback, ask questions, and keep exploring. See you next time.