Managing Model Endpoints
Use model endpoints to send inference requests to pretrained, custom, or imported models hosted on dedicated AI clusters in OCI Generative AI.
A model endpoint is a separately named access point where a model accepts inference requests and returns responses. You create a model endpoint on a hosting dedicated AI cluster. A model hosted in on-demand mode doesn't require a model endpoint on a dedicated AI cluster.
How Model Endpoints Use a Hosting Cluster
A hosting dedicated AI cluster must have at least one model replica. Model replicas use dedicated AI units and provide the compute resources for inference. Model endpoints use the model and compute resources provisioned on the cluster.
Creating additional model endpoints doesn't add model replicas or increase the inference throughput of the cluster. To increase throughput, add model replicas to the hosting cluster.
When you create a hosting cluster, you select a pretrained base model, the base model for a custom model, or a specific imported model. The model used by an endpoint must be compatible with that cluster selection and its unit shape. Unrelated models can't share the same cluster. For model-specific endpoint and cluster compatibility, see the applicable model page.
Hosting clusters for imported models don't require the 744-unit-hour minimum commitment that applies to hosting clusters for OCI Generative AI pretrained and custom models. This billing difference doesn't remove the requirement to create a model endpoint to access an imported model. See Hardware Unit Shapes for Imported Models.
Endpoint Capacity
By default, a hosting dedicated AI cluster can have up to 50 model endpoints. On the dedicated AI cluster details page, Remaining endpoint capacity shows how many additional model endpoints the cluster can support under its current limit. For example, a cluster with a capacity of 50 and one existing endpoint shows a remaining endpoint capacity of 49.
You can create multiple endpoints as aliases for a compatible model and use them for different users, applications, or purposes. Where supported, each endpoint can have its own endpoint configuration, such as public or private network access and guardrail settings. The endpoints share the compute resources and model replicas provisioned on the cluster.
If you no longer need an endpoint, delete the endpoint so that you can use the cluster's available endpoint capacity for another endpoint.
For more than 50 endpoints on a hosting cluster, request a service limit increase for
endpoint-per-dedicated-unit-count. See Service Limits for Generative AI.Model Endpoints and Private Endpoints
A model endpoint and a private endpoint serve different purposes. A model endpoint provides access to a model hosted on a dedicated AI cluster. A private endpoint provides a private network path to OCI Generative AI through a virtual cloud network (VCN).
To access a model on a dedicated AI cluster through a private network, first create a private endpoint. Then, when you create the model endpoint on the cluster, select the private endpoint for network access.
Management Tasks
After you create an active endpoint for a custom or imported model, the model is available in the playground's model list.
You can perform the following tasks to create and manage model endpoints: