如何堆叠两台Ascend GX10进行AI推理(1)

发布于 — 2026 年 07 月 31 日
#推理 #DGX Spark #GX10

两台GX10组成推理资源池进行并行推理

背景

买了两台ASUS Ascend GX10,用的芯片与DGX Spark一样,都是GB10,1PFlops算力,单台统一内存128GB,足够运行很多大模型了。但两台如何连起来更强大?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0  On |                  N/A |
| N/A   44C    P8              4W /  N/A  | Not Supported          |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

环境配置

环境连线

参考官方faq:https://www.asus.com/tw/support/faq/1056547/

vllm源码编译安装

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
tanwubin@gx10-work:~/projects$ git clone https://github.com/vllm-project/vllm.git
Cloning into 'vllm'...
remote: Enumerating objects: 239872, done.
remote: Counting objects: 100% (92/92), done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 239872 (delta 56), reused 42 (delta 31), pack-reused 239780 (from 2)
Receiving objects: 100% (239872/239872), 229.91 MiB | 1.09 MiB/s, done.
Resolving deltas: 100% (187199/187199), done.

anwubin@gx10-work:~/projects$ cd vllm/

tanwubin@gx10-work:~/projects/vllm$ uv venv --python 3.12 --seed
Using CPython 3.12.3 interpreter at: /usr/bin/python3.12
Creating virtual environment with seed packages at: .venv
 + pip==26.1.2
Activate with: source .venv/bin/activate
tanwubin@gx10-work:~/projects/vllm$ source .venv/bin/activate
(vllm) tanwubin@gx10-work:~/projects/vllm$ 

(vllm) tanwubin@gx10-work:~/projects/vllm$ uv pip install -r requirements/build/cuda.txt 
Resolved 41 packages in 2.55s
Prepared 40 packages in 38m 37s
Installed 40 packages in 64ms
Prepared 1 package without build isolation in 6m 46s
Installed 1 package in 94ms
 + build==1.5.1
 + cmake==4.4.0
 + cuda-bindings==13.3.1
 + cuda-pathfinder==1.5.6
 + cuda-toolkit==13.0.2
 + filelock==3.29.7
 + fsspec==2026.6.0
 + jinja2==3.1.6
 + markupsafe==3.0.3
 + mpmath==1.3.0
 + networkx==3.6.1
 + ninja==1.13.0
 + nvidia-cublas==13.1.0.3
 + nvidia-cuda-cupti==13.0.85
 + nvidia-cuda-nvrtc==13.0.88
 + nvidia-cuda-runtime==13.0.96
 + nvidia-cudnn-cu13==9.19.0.56
 + nvidia-cufft==12.0.0.61
 + nvidia-cufile==1.15.1.6
 + nvidia-curand==10.4.0.35
 + nvidia-cusolver==12.0.4.66
 + nvidia-cusparse==12.6.3.3
 + nvidia-cusparselt-cu13==0.8.0
 + nvidia-nccl-cu13==2.28.9
 + nvidia-nvjitlink==13.0.88
+ nvidia-nvshmem-cu13==3.4.5
 + nvidia-nvtx==13.0.85
 + packaging==26.2
 + protobuf==7.35.1
 + pyproject-hooks==1.2.0
 + regex==2026.7.10
 + semantic-version==2.10.0
 + setuptools==80.10.2
 + setuptools-rust==1.13.0
 + setuptools-scm==10.2.0
 + sympy==1.14.0
 + torch==2.11.0
 + triton==3.6.0
 + typing-extensions==4.16.0
 + vcs-versioning==2.2.2
 + wheel==0.47.0
(vllm) tanwubin@gx10-work:~/projects/vllm$ python -c "import torch; print(torch.__version__)"
/home/tanwubin/projects/vllm/.venv/lib/python3.12/site-packages/torch/_subclasses/functional_tensor.py:307: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
2.11.0+cu130
(vllm) tanwubin@gx10-work:~/projects/vllm$ uv pip install numpy
Resolved 1 package in 1.04s
Prepared 1 package in 13.27s
Installed 1 package in 8ms
 + numpy==2.5.1
(vllm) tanwubin@gx10-work:~/projects/vllm$ python -c "import torch; print(torch.__version__)"
2.11.0+cu130

(vllm) tanwubin@gx10-work:~/projects/vllm$ export TORCH_CUDA_ARCH_LIST="12.0+PTX"
# export TORCH_CUDA_ARCH_LIST="12.1a"
# export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
(vllm) tanwubin@gx10-work:~/projects/vllm$ 
(vllm) tanwubin@gx10-work:~/projects/vllm$ export CUDA_HOME=/usr/local/cuda
(vllm) tanwubin@gx10-work:~/projects/vllm$ 

(vllm) tanwubin@gx10-work:~/projects/vllm$ uv pip install --no-build-isolation -e .
# 如果要看详细编译过程,加上-v参数:uv pip install --no-build-isolation -e . -v --force-reinstall
Resolved 192 packages in 2.68s
Uninstalled 1 package in 0.78ms
Installed 4 packages in 7ms
      Built vllm @ file:///home/tanwubin/projects/vllm
Prepared 1 package without build isolation in 36m 10s
Installed 1 package in 1ms
 ~ nvidia-cusparselt-cu13==0.8.0
 + torchaudio==2.11.0
 + torchcodec==0.14.0
 + torchvision==0.26.0
 + vllm==0.23.1rc1.dev1034+g1bd8f80a6 (from file:///home/tanwubin/projects/vllm)
(vllm) tanwubin@gx10-work:~/projects/vllm$ 
(vllm) tanwubin@gx10-work:~/projects/vllm$ vllm --version
0.23.1rc1.dev1034+g1bd8f80a6
(vllm) tanwubin@gx10-work:~/projects/vllm$ 

算力集群配置

1、确保两台机器terminal显示的用户名一致 2、在Node1(主节点)生成并分发SSH密钥,实现节点间免密连接

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ssh-keygen -t rsa -b 4000
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tanwubin/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/tanwubin/.ssh/id_rsa
Your public key has been saved in /home/tanwubin/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:yaq3OZnErquE1VRHrO9ysZCxxU5B/yVbTLUHCsaOi+Q tanwubin@gx10-home-master
The key's randomart image is:
+---[RSA 4000]----+
|      .++.o   o..|
|     . .oo.. + ..|
|    .  o +. o +..|
|   o  +.=... =  .|
|  . .+ XS.  o    |
| o    E.=        |
|. .  o.= o       |
| .   .B.+        |
|  ..++o=         |
+----[SHA256]-----+
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ssh-copy-id tanwubin@169.254.118.128
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys
tanwubin@169.254.118.128's password: 

Number of key(s) added: 2

Now try logging into the machine, with:   "ssh 'tanwubin@169.254.118.128'"
and check to make sure that only the key(s) you wanted were added.

(.venv) tanwubin@gx10-home-master:~/projects/vllm$ 

3、安装ray vllm我们已经源码编译OK,所以再安装下ray。在vllm的venv下安装就行。

1
2
# 在两个node都记得要安装哦
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ pip install "ray[default]"

4、服务启动

Node1(主控,master):169.254.206.234

Node2(接收,worker):169.254.118.128

export VLLM_HOST_IP=169.254.206.234

export VLLM_HOST_IP=169.254.118.128

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
# 在Node1启动ray服务
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ray start --head --node-ip-address=169.254.206.234 --port=6379
Enable usage stats collection? This prompt will auto-proceed in 10 seconds to avoid blocking cluster startup. Confirm [Y/n]: 
Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.

Local node IP: 169.254.206.234

--------------------
Ray runtime started.
--------------------

Next steps
  Note: The following commands are intended for use on
  the head node or within the cluster network.
  # 这里也给出了添加集群算力节点的示例
  To add another node to this Ray cluster, run
    ray start --address='169.254.206.234:6379'
  
  To connect to this Ray cluster:
    import ray
    ray.init(_node_ip_address='169.254.206.234')
  
  To submit a Ray job using the Ray Jobs CLI:
    RAY_API_SERVER_ADDRESS='http://127.0.0.1:8265' ray job submit --working-dir . -- python my_script.py
  
  See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html 
  for more information on submitting Ray jobs to the Ray cluster.
  
  To terminate the Ray runtime, run
    ray stop
  
  To view the status of the cluster, use
    ray status
  
  To monitor and debug Ray, view the dashboard at 
    127.0.0.1:8265
  
  If connection to the dashboard fails, check your firewall settings and network configuration.
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ 
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ray status
======== Autoscaler status: 2026-07-12 10:35:44.735219 ========
Node status
---------------------------------------------------------------
Active:
 1 node_75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Total Usage:
 0.0/20.0 CPU
 0.0/1.0 GPU
 0B/79.53GiB memory
 0B/34.08GiB object_store_memory

From request_resources:
 (none)
Pending Demands:
 (no resource demands)

(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ray list nodes

======== List: 2026-07-12 10:39:14.882201 ========
Stats:
------------------------------
Total: 1

Table:
------------------------------
    NODE_ID                                                   NODE_IP          IS_HEAD_NODE    STATE    STATE_MESSAGE    NODE_NAME        RESOURCES_TOTAL                  LABELS
 0  75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce  169.254.206.234  True            ALIVE                     169.254.206.234  CPU: 20.0                        ray.io/accelerator-type: GB10
                                                                                                                                          GPU: 1.0                         ray.io/node-id: 75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce
                                                                                                                                          accelerator_type:GB10: 1.0
                                                                                                                                          memory: 79.530 GiB
                                                                                                                                          node:169.254.206.234: 1.0
                                                                                                                                          node:__internal_head__: 1.0
                                                                                                                                          object_store_memory: 34.084 GiB

# 在Node2启动ray,加入集群
(vllm) tanwubin@gx10-work:~/projects/vllm$ ray start --address='169.254.206.234:6379' --node-ip-address=169.254.118.128
Local node IP: 169.254.118.128

--------------------
Ray runtime started.
--------------------

To terminate the Ray runtime, run
  ray stop
(vllm) tanwubin@gx10-work:~/projects/vllm$ 

# 加入集群后

(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ray list nodes

======== List: 2026-07-12 10:41:24.434768 ========
Stats:
------------------------------
Total: 2

Table:
------------------------------
    NODE_ID                                                   NODE_IP          IS_HEAD_NODE    STATE    STATE_MESSAGE    NODE_NAME        RESOURCES_TOTAL                  LABELS
 0  100e10685f9991b162336d38327d3e8d26b6f4bb43a3b5557cdcc27a  169.254.118.128  False           ALIVE                     169.254.118.128  CPU: 20.0                        ray.io/accelerator-type: GB10
                                                                                                                                          GPU: 1.0                         ray.io/node-id: 100e10685f9991b162336d38327d3e8d26b6f4bb43a3b5557cdcc27a
                                                                                                                                          accelerator_type:GB10: 1.0
                                                                                                                                          memory: 82.508 GiB
                                                                                                                                          node:169.254.118.128: 1.0
                                                                                                                                          object_store_memory: 35.361 GiB
 1  75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce  169.254.206.234  True            ALIVE                     169.254.206.234  CPU: 20.0                        ray.io/accelerator-type: GB10
                                                                                                                                          GPU: 1.0                         ray.io/node-id: 75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce
                                                                                                                                          accelerator_type:GB10: 1.0
                                                                                                                                          memory: 79.530 GiB
                                                                                                                                          node:169.254.206.234: 1.0
                                                                                                                                          node:__internal_head__: 1.0
                                                                                                                                          object_store_memory: 34.084 GiB

(.venv) tanwubin@gx10-home-master:~/projects/vllm$ 

(.venv) tanwubin@gx10-home-master:~/projects/vllm$ ray status
======== Autoscaler status: 2026-07-12 10:42:10.745052 ========
Node status
---------------------------------------------------------------
Active:
# 从这里可以看到集群有2个node了
 1 node_75b11172ec44444928f76aa6d18fc4894e025ec71cc7e223e13a9cce
 1 node_100e10685f9991b162336d38327d3e8d26b6f4bb43a3b5557cdcc27a
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
# 从这里可以看到CPU/GPU/memory资源都聚合成集群的资源了
Total Usage:
 0.0/40.0 CPU
 0.0/2.0 GPU
 0B/162.04GiB memory
 0B/69.45GiB object_store_memory

From request_resources:
 (none)
Pending Demands:
 (no resource demands)
(.venv) tanwubin@gx10-home-master:~/projects/vllm$ 

# 本地的dashboard
View the dashboard at http://127.0.0.1:8265