AI Frequently Asked Questions

To deploy AI models in a cloud environment effectively, it’s important to first establish a strong understanding of the AI deployment lifecycle. The lifecycle should contain model training, deployment, validation, and monitoring, among other things. The lifecycle and deployment strategy should be aligned to ensure a robust and safe deployment process. Monitoring and observability should be established early and provide awareness of the overall health of the AI model.

To support GenAI innovation and growth, establish a dedicated AI Center of Excellence (CoE) that combines technical expertise with business acumen. Create cross-functional teams that include data scientists, engineers, domain experts, and business strategists to ensure AI initiatives align with enterprise goals. Implement a hub-and-spoke model where the central CoE provides guidance, best practices, and oversight, while individual business units have embedded AI teams for specific applications. Foster a culture of continuous learning and experimentation, encouraging knowledge sharing and upskilling across the organization. Establish clear governance structures and ethical guidelines to ensure responsible AI development and deployment while maintaining agility for innovation.

We feel you should meet your organization where it is in its AI journey. This can start with a maturity assessment that will clearly establish how mature you are when it comes to leveraging AI. Given the relative “newness” of this technology, we encourage individuals and organizations to establish a learning and experimentation environment. This environment will help provide your organization with a higher degree of confidence when navigating what AI solutions to build, and which ones to buy.

We strongly encourage our partners to lean on the same core governing principles they may have applied in other areas of their organization (e.g. Data Governance). This includes knowing who owns the AI solution, and who is accountable for it. It’s also tremendously important to establish a robust and responsive incident response protocol. Additionally, it’s useful to create a well-written set of supporting documentation. This documentation should be as intuitive and clear as possible.

A great place to start is looking at developer productivity, one of the fastest-growing and most mature marketplaces for AI-powered tools. We recommend you identify some specific workflows that are good candidates for AI integration (i.e. generate unit tests, research, documentation, etc.). With the current state of your workflow as your baseline, you can begin to integrate AI tool(s) into your workflow. It’s important to establish clear performance indicators first, as this will provide you with a more objective assessment of the tool(s). Here are a few of Ippon’s favorite tools:

• AWS Bedrock
• Langchain
• Github CoPilot
• Snowflake Cortex

Some of the most widely used benchmarking is centered around the latency and throughput of an AI model. These two metrics can provide a great deal of insight into the performance of the AI model. It can be difficult to establish good objective key results when working with AI models, and these two metrics can provide further insight into the end user’s experience. There are many additional benchmarking techniques available that can help when working with Large Language Models (LLM). These techniques include the Abstraction and Reasoning Corpus benchmark used to more accurately assess the reasoning capabilities of an LLM.

Often the most overlooked part of building good AI solutions, data preparation should be given careful consideration and care. Organizations that work with highly sensitive data (e.g. Personal Identifiable Information) must ensure that existing data governance is being followed while preparing data for use for AI. As a general rule, we place a stronger emphasis on the quality of data over quantity to begin with. Furthermore, it’s important to understand that data preparation is NOT a one-time task. It should be a continuous practice designed to allow for iterative improvements.

While this question sounds open-ended, it is one we often encounter. More broadly speaking, any initiative to train a custom foundational AI model is going to be the most cost-intensive. This could be for either an LLM focused on more broad GenAI solutions, or a specific logistic regression algorithm designed to provide better fraud detection. The largest contributor to cost in this case will be the amount of training data needed to train the model. Organizations that are instead looking to leverage existing foundational AI models in the marketplace, often are focused on latency and throughput.

Key Performance Indicators (KPIs) for model success depend on the specific use case but generally include accuracy metrics such as latency and throughput. While these two metrics are related they are not simply an inverse of the other. In particular, when an AI model is handling concurrent requests it is crucial to balance the throughput and latency. It can be helpful to plot the Latency vs the Throughput on a graph. From here you can draw a “best fit” line, and determine where exactly you will begin to see diminishing returns on your performance. This is represented by a fairly distinct exponential increase in latency alongside minimal throughput increases, often called the “knee”. While this isn’t an exact science, it’s a great starting point to begin to understand how to better monitor performance.

Consider cloud-based platforms like AWS, Snowflake, Google Cloud, or Azure as they provide a suite of different AI-oriented tools. Given the high degree of importance of the data, we recommend focusing on data ingestion, cleaning, and preparation tools (e.g. AWS Kinesis, Google Vertex AI, Snowflake, etc.)