The Forrester Wave™: AI Foundation Models For Language, Q2 2024

AI Foundation Models For Language Vendor Product Comparisons 2024

Key features and pros and cons of top AI Foundation Models For Language products:

Databricks - Leaders

Pros:

Takes a distinct approach offering both its own pretrained model (DBRX) and support for clients pretraining or fine-tuning their own models
Strong capabilities in application development, governance, and security
Strong capabilities in managing models for training and deployment
DBRX is open weight and nicely integrated with Databricks' Data Intelligence Platform
Well-positioned to capture the growing sector of smaller and more domain-specific models

Cons:

Needs to bring multilingual capabilities up to par
Needs to add multimodality
Needs more transparency to its data for training the model
Has a comprehensive feature-focused vision and roadmap but needs to do more to help typical enterprises innovate and implement AI-FMLs within their applications

Google - Leaders

Pros:

Gemini has the largest context window of all vendors evaluated: one million tokens today and two million tokens just announced
One of the few commercially available multimodal large language models (LLMs)
Substantial multilingual capabilities across 37 languages, higher than any other provider
Strong capabilities across governance and security, model management, and resilience and scalability
Uniquely differentiated in the market, especially in multimodality and context length
Interconnectivity with the broader ecosystem of complementary cloud services
Enormous AI infrastructure capacity, very deep bench of AI researchers, and growing number of enterprise customers in Google Cloud

Cons:

Can become even more attractive to customers if it articulates a broader, transformative AI vision to enterprises versus highlighting single use cases

NVIDIA - Leaders

Pros:

Very strong multilingual capabilities
Offers multimodal interactivity through its Megatron model
Provides a robust set of filters and curation layers that can be applied to the data to produce high-quality training data to increase model performance
NeMo Framework allows it, and other AI-FML providers, to build AI-FML models more quickly and efficiently on its platform
Offers AI infrastructure (hardware) that companies across the world rely on to train and use AI models
Nemotron family of models is formidable in its own right

Cons:

Reference customers did not respond to Forrester's outreach for this evaluation
Needs to establish a roadmap that includes more enterprise tooling
Needs to articulate a vision of enterprise transformation rather than technical prowess to compete more effectively for enterprise customers

Anthropic - Contenders

Pros:

Very strong language capabilities in its core model
One of the longest context windows currently on the market
Significant work to align its model family to its Constitutional AI approach during pretraining
Carved unique place in the AI-FML market by introducing the concept of 'Constitutional AI' as a guiding principle for aligning models to enterprise needs

Cons:

Needs to offer more significant capabilities for enterprises to build applications and manage models
Today, enterprises leveraging Anthropic largely rely on third-party services or AI/ML platforms to support integrating the models into applications
Remains to be seen whether there is enough differentiation in the market to sustain the high cost of hyper-premium AI-FMLs
Declined to participate in the full Forrester Wave evaluation process

Cohere - Contenders

Pros:

Strengths in core model capabilities for language and reasoning
Some of the most notable multilingual capabilities with pretraining data from a variety of languages, including specific optimizations to perform well in some of the most common business languages
Offers some of the longer context windows available
Approach will appeal to enterprises that want to build RAG-based knowledge retrieval applications but aren't equipped to build the supporting embedding creation and retrieval mechanisms

Cons:

Must expand its partner ecosystem
Must invest in enterprise governance tooling that goes beyond API calls
Has work to do to catch up in its application development and model management tools
Needs to develop its multimodal capabilities
Declined to participate in the full Forrester Wave evaluation process

IBM - Strong Performers

Pros:

Granite family of models provides some of the most robust and transparent insights into the underlying training data
Significant and powerful capabilities to align its models to enterprise needs
Governance structures to enable the monitoring and management of models
Enterprise customers get some of the best enterprise tooling designed specifically for developing highly governable AI solutions
100% vendor indemnification from model training data
Reference customers appreciate IBM's emphasis on trusted AI, implementation services, and the sophisticated tools within watsonx to build AI solutions

Cons:

Lacking in context window
Lacking in multimodal capabilities
Lacking in some core model capabilities
Has to take its 'built for business' vision to the next level to outmaneuver the competition
Needs to prove itself in the market

Microsoft - Strong Performers

Pros:

Phi models leverage large amounts of synthetically generated content in addition to real content, allowing training with a smaller dataset that can be more tightly curated
Deeply invested in building out tooling to enable technical and non-technical users to work on aligning models to business use cases
Azure AI services surrounding the Phi family provide strong capabilities for aligning model behavior to enterprise needs
Small size and tightly curated training dataset is a core value proposition for enterprises looking to deploy in constrained computing environments or that want a model with tightly focused behavior

Cons:

Phi models are less capable than many of the others in this market
Significantly behind in the scope of their training corpus
Significantly behind in core model language capabilities
Significantly behind in context windows
Declined to participate in the full Forrester Wave evaluation process

Mistral AI - Challengers

Pros:

Strong core language capabilities
Uses a mixture of experts (MoE) approach, which enables them to achieve higher accuracy while using fewer computing resources at inference time
Offers open-weight models that make strong appearances in model performance leaderboards

Cons:

Must quickly add sales, marketing, platform tooling development, and partner operations to compete with the growing number of players
Roadmap takes its cues from customer requests and needs but lacks a cohesive and coherent approach on how it will address the needs of enterprise users
Has a lot of catching up to do in capabilities for governance and security, application development, model management, and reliance and scalability
Did not provide reference customers for this evaluation

OpenAI - Strong Performers

Pros:

Models are some of the most capable in the market
One of the few providers offering multimodal LLMs
Strengths in core model capabilities, including code generation, multilingual capabilities, context window, and the scope of its training data
Very strong research team
Reference customers appreciate OpenAI's swift innovation and the availability of GPT-4 directly from OpenAI and from the Azure OpenAI service

Cons:

Has a lot to do in developing tooling and support for enterprise developers who want to work with the models within OpenAI's platform
Must improve its capabilities for enterprise application development, model management, and latency
Ever-widening landscape of commercial and open-source competition will present a challenge in the long run
Enterprise customers also need premium tooling to build and scale applications, not just great models

Amazon Web Services - Strong Performers

Pros:

Strong capabilities in many of the surrounding support tools that its Bedrock service provides, such as model alignment, governance and security, and application development
Vast existing customer base will have easy access to the Titan family of models and all other models in AWS's Amazon Bedrock model service
AWS customers appreciate that they can stay within the AWS cloud environment to experiment not only with Titan models but with other models in Bedrock
Marketplace approach will be appealing to AWS customers

Cons:

Amazon Titan models have not seen as significant adoption among customers as some of its competitors' models
AWS has not articulated a clear vision about when customers should choose to use Titan models versus other models available in Bedrock such as Anthropic's Claude
Lacking in Titan's core capabilities for natural language understanding and generation, context window length, and code generation
Will need to focus on improving the Titan family of models if it wishes its models to continue to be a significant player
Declined to participate in the full Forrester Wave evaluation process

FAQs

Q: Which vendors are Leaders in 2024?

A: Databricks, Google, NVIDIA

Q: What does "Leader" mean in Forrester Wave terms?

A: Leaders in the Forrester Wave for AI Foundation Models for Language are vendors that offer world-class models combined with mature enterprise capabilities. Leaders demonstrate strong performance across core model capabilities (language understanding, multimodality, context windows, multilingual support), enterprise tooling (governance, security, application development, model management), and infrastructure (resilience, scalability, low latency). They have clearly articulated roadmaps fine-tuned to enterprise needs, provide tools to configure and govern models to reduce hallucinations and align with brand values, respect IP rights, and can scale with low latency and high availability. Leaders combine cutting-edge AI research with comprehensive enterprise services and have significant customer adoption.

Q: How do I interpret a Forrester Wave?

A: https://www.forrester.com/policies/forrester-wave-methodology/

Report:

Top Vendor Products for AI Foundation Models For Language in 2024

AI Foundation Models For Language Vendor Product Comparisons 2024

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

FAQs

Q: Which vendors are Leaders in 2024?

Q: What does "Leader" mean in Forrester Wave terms?

Q: How do I interpret a Forrester Wave?

Reference