Benchmark performance on 70B 8bit TPS
Created by @reikai0
| Brand | GPU Name | GPU Price | Count | Form Factor | Prompt | Eval | VRAM | VRAM Speed | Tflops | Total Price |
|---|---|---|---|---|---|---|---|---|---|---|
| AMD | Epyc 77xx | $1470 | 1 | Desktop | 48.1 | 1.5 | 128 | 191 | 53 | $1470 |
| Nvidia | P40 | $595 | 2 | Desktop | 21.8 | 2.8 | 48 | 654.2 | 24 | $1190 |
| Nvidia | 2080Ti 22G | $937 | 2 | Desktop | 207.1 | 4.9 | 44 | 1232 | 228 | $1874 |
| AMD | 7900XT | $1060 | 2 | Desktop | 187.1 | 6.4 | 40 | 1600 | 206 | $2119 |
| AMD | 7900XTX | $1335 | 2 | Desktop | 223.5 | 7.6 | 48 | 1920 | 246 | $2670 |
| Nvidia | RTX 3090 | $1433 | 2 | Desktop | 258 | 7.5 | 48 | 1872.4 | 284 | $2865 |
| Nvidia | P40 | $814 | 4 | Workstation | 43.6 | 5.5 | 96 | 1388.4 | 48 | $3255 |
| Nvidia | RTX 3070Ti | $945 | 4 | Workstation | 323.4 | 9.7 | 64 | 2433.2 | 356 | $3780 |
| Nvidia | 2080Ti 22G | $1081 | 4 | Workstation | 414.3 | 9.8 | 88 | 2464 | 456 | $4323 |
| AMD | 7900XT | $1197 | 4 | Workstation | 374.3 | 12.7 | 80 | 3200 | 412 | $4788 |
| AMD | 7900XTX | $1641 | 4 | Workstation | 447 | 15.3 | 96 | 3840 | 492 | $6565 |
| Nvidia | RTX 3090 | $1728 | 4 | Workstation | 516.1 | 14.9 | 96 | 3744.8 | 568 | $6910 |
| AMD | MI100 | $2068 | 4 | Workstation | 672.3 | 19.6 | 128 | 4920 | 740 | $8270 |
| Nvidia | RTX 4090 | $2485 | 4 | Workstation | 1199.2 | 16 | 96 | 4032 | 1320 | $9940 |
| Nvidia | RTX 8000 | $3220 | 4 | Workstation | 239.8 | 10.7 | 192 | 2688 | 264 | $12880 |
| Nvidia | A6000/A40 | $4208 | 4 | Workstation | 563.2 | 12.2 | 192 | 3072 | 620 | $16830 |
| Nvidia | RTX 3090 | $2228 | 8 | Server | 1032 | 29.8 | 192 | 7489.6 | 1136 | $17820 |
| AMD | AMD MI100 | $2448 | 8 | Server | 1344.5 | 39.2 | 256 | 9840 | 1480 | $19580 |
| Nvidia | RTX 8000 | $3508 | 8 | Server | 479.7 | 21.4 | 384 | 5376 | 528 | $28060 |
| Nvidia | A6000/A40 | $4571 | 8 | Server | 1126.5 | 24.4 | 384 | 6144 | 1240 | $36570 |
| AMD | MI210 | $8168 | 8 | Server | 1315.4 | 52.2 | 512 | 13120 | 1448 | $65340 |
| AMD | MI300A | $30960 | 4 | Big Iron | 3561.1 | 104.2 | 512 | 26200 | 3920 | $123840 |
| AMD | MI250 | $18023 | 8 | Big Iron | 2783.5 | 104.4 | 1024 | 26240 | 3064 | $144180 |
| Nvidia | A100 | $18168 | 8 | Server | 2267.5 | 49.5 | 640 | 12440 | 2496 | $145340 |
| Nvidia | A100 SXM | $25000 | 8 | Big Iron | 2718.1 | 63.7 | 640 | 16000 | 2992 | $200000 |
| AMD | MI300X | $38010 | 8 | Big Iron | 9447.9 | 327.9 | 1536 | 82400 | 10400 | $304080 |
| Nvidia | B100 | $48580 | 8 | Big Iron | 12718.3 | 254.6 | 1536 | 64000 | 14000 | $388640 |
| Nvidia | B200 | $55710 | 8 | Big Iron | 16352.1 | 254.6 | 1536 | 64000 | 18000 | $445680 |