Introduction
During the Chinese New Year, the DeepSeek R1 model, a domestically developed AI model, gained significant attention due to its open-source nature and efficient inference capabilities. Its knowledge distillation technology enables the transfer of complex model reasoning capabilities to a lightweight version, allowing it to outperform some leading international models in various benchmark tests. The combination of an open-source strategy and a lightweight design lowers the AI deployment barrier, unlocking new possibilities for edge computing applications.
Successful Deployment on InHand AI EC5000 Edge Computer
On the first workday after the holiday, the InHand AI technical team successfully deployed the DeepSeek R1 distilled model on the EC5000 series edge computer. This achievement demonstrates the powerful potential of lightweight edge devices like the EC5000 in AI inference tasks. Unlike traditional cloud-based deployments, edge computing eliminates the need for high-performance servers and enables real-time inference in low-power environments, making it an ideal AI solution for industrial inspection, smart transportation, remote healthcare, and other fields.
Steps to Run DeepSeek R1 on EC5000 Edge Computer
Step 1: Install NVIDIA Jetson Containers Toolkit
Run the following commands to download and install the Jetson Containers toolkit:
git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh
Step 2: Install NVIDIA JetPack Toolkit
sudo apt update
sudo apt install nvidia-jetpack
Step 3 (Optional): Modify Docker Registry Source for China Mainland Users
Create a new file at /etc/docker/daemon.json and add the following content:
{
"registry-mirrors": [
"https://docker.1ms.run",
"https://docker.xuanyuan.me"
]
}
Save the file and restart the Docker service:
sudo systemctl daemon-reload
sudo systemctl restart docker
Note: This step is optional and only required if the default Docker repository is inaccessible.
Step 4: Download and Run Ollama Container
docker run -itd --runtime nvidia --name ollama ollama/ollama
Step 5: Download and Run the DeepSeek R1 Distilled Model
Find the model version at Ollama's Library and install it via the command line. For example, to install the DeepSeek-R1-Distill-Qwen-1.5B model:
docker exec -it ollama ollama run deepseek-r1:1.5b
Note: Replace deepseek-r1:1.5b with your desired model version from Ollama's search page.
Model Verification
To interact with the deployed model, run:
docker exec -it ollama ollama run deepseek-r1:1.5b
Monitoring EC5000 Hardware Performance
To monitor CPU/GPU/memory utilization in real time, execute:
sudo jtop
Note: jtop requires root privileges.
Additional Information
1.
Besides DeepSeek R1, the EC5000 edge computer also supports other open-source LLMs such as Llama 3.
2.
Running LLMs on InHand AI edge computers via Ollama is just one of many possible methods.
Future of Edge AI with DeepSeek R1
This successful deployment of the DeepSeek R1 distilled model on the EC5000 edge computer validates the deep integration of edge hardware with cutting-edge AI technologies. It marks the beginning of a new paradigm in lightweight and high-performance edge AI.
As distillation technology continues to evolve, businesses can leverage these advancements to quickly develop private AI services, reducing computational costs while ensuring data security. InHand AI remains committed to advancing the edge intelligence ecosystem, helping industries transition towards on-device AI solutions.
