xorbits.deploy.kubernetes.client.new_cluster#

xorbits.deploy.kubernetes.client.new_cluster(kube_api_client: None, worker_cpu: int, worker_mem: str, image: str | None = None, supervisor_num: int = 1, supervisor_cpu: int = 1, supervisor_mem: str = '4G', worker_num: int = 1, worker_spill_paths: List[str] | None = None, worker_cache_mem: str | None = None, min_worker_num: int | None = None, pip: str | List[str] | None = None, conda: str | List[str] | None = None, timeout: int | None = None, cluster_type: str = 'auto', external_storage: str | None = None, **kwargs) → KubernetesClusterClient[source]#

The entrance of deploying xorbits cluster.

Parameters:

  • kube_api_client – Kubernetes API client, required, can be created with new_client_from_config

  • worker_cpu – Number of CPUs for every worker, required

  • worker_mem – Memory size for every worker, required

  • image – Docker image to use, xprobe/xorbits:<xorbits version> by default

  • supervisor_num – Number of supervisors in the cluster, 1 by default

  • supervisor_cpu – Number of CPUs for every supervisor, 1 by default

  • supervisor_mem – Memory size for every supervisor, 4G by default

  • worker_num – Number of workers in the cluster, 1 by default

  • worker_spill_paths – Spill paths for worker pods on host

  • worker_cache_mem – Size or ratio of cache memory for every worker

  • min_worker_num – Minimal ready workers, equal to worker_num by default

  • pip –

    Either a list of pip requirements specifiers, or a string containing the path to a pip requirements.txt file. None by default. Both supervisor and worker will install the specified pip packages. Examples:

    • pip=["requests==1.0.0", "aiohttp"]

    • pip="/path/to/requirements.txt"

  • conda –

    Either a string containing the path to a conda environment.yml file, or a list of conda packages used for conda install command. None by default. Both supervisor and worker will install the specified conda packages. When this parameter is list type, install the conda packages from the default channel. When this parameter is string type, the name attribute in the environment.yml file will not take effect, since all package changes will occur in the base conda environment where Xorbits exists. Examples:

    • conda=["tensorflow", "tensorboard"]

    • conda="/path/to/environment.yml"

  • timeout – Timeout in seconds when creating clusters, never timeout by default

  • cluster_type – K8s cluster type, auto, kubernetes or eks supported, auto by default. auto means that it will automatically detect whether the kubectl context is eks. You can also manually specify kubernetes or eks in some special cases.

  • external_storage – You can specify an external storage. Currently, only juicefs is supported as one of our storage backend.

  • kwargs – Extra kwargs

Returns:

a KubernetesClusterClient instance

Return type:

KubernetesClusterClient

On this page

This Page