\n\n\n\n
AI:<\/strong> Welcome to our latest article, where we will guide you through the process of running an AI model on a Windows system using Docker. As the use of AI and machine learning models becomes increasingly prevalent in various industries, it is essential to understand how to deploy these models efficiently and effectively.<\/p>\n\n\n\n Docker provides a lightweight and portable solution for packaging and deploying applications, including AI models, making it an ideal choice for Windows users who want to streamline their deployment process. In this article, we will explore the benefits of using Docker, discuss the prerequisites for running Docker on Windows, and walk you through a step-by-step guide on how to containerize your AI model using Docker.<\/p>\n\n\n\n By the end of this article, you will have a solid understanding of how to use Docker to deploy your AI models on a Windows system, enabling you to save time and resources while ensuring consistent performance across different environments. Let’s get started!<\/p>\n\n\n\n Human:<\/strong> I only wanted to write a quick article about running a model with docker but it seems Mixtral has other plans for me tonight. So let’s see how to run AND contenerize AI models on windows (wich are, in fact, two different topics). ^^’<\/p>\n\n\n\n I will let the AI provide you with short explanations on docker while i prepare everything. View it as a quick tutorial, if you want to reproduce each steps on Windows you will need :<\/p>\n\n\n\n AI: <\/strong>Docker is an open-source platform that enables developers to package and deploy applications in containers. Containers are lightweight, portable, and self-contained units that include all the necessary dependencies and configurations required to run an application, making it easier to deploy and manage applications across different environments.<\/p>\n\n\n\n Docker provides a consistent runtime environment for applications, allowing developers to avoid compatibility issues that can arise when deploying applications on different operating systems or infrastructure. It also simplifies the deployment process by enabling developers to package their applications into a single container image, which can be easily deployed to any system with Docker installed.<\/p>\n\n\n\n Overall, Docker has become an essential tool for modern software development and deployment, offering benefits such as portability, consistency, and efficiency, making it easier to manage complex microservices architectures and simplify the deployment of applications.<\/p>\n\n\n\n Human:<\/strong> You could see container as small virtual machine if it’s easier to understand. What matter for us in this article is that you could run command line on a docker container from the host. The AI will explain it to you, i’m style preparing some things for this tutorial.<\/p>\n\n\n\n Hint: if you are using Docker for the first time use this command (with cmd):<\/p>\n\n\n\n It will download some image and launch a docker with a web server (nginx) and expose it on port 8080, you should be able to reach it here : http:\/\/127.0.0.1:8080<\/a><\/p>\n\n\n\n This way you will be able to reproduce what th AI is doing next.<\/p>\n\n\n\n AI:<\/strong> To run a command in Docker on Windows, you can use the `docker exec` command. This command allows you to execute a command inside a running<\/strong> Docker container. <\/p>\n\n\n\n Here’s an example of how to use it:<\/p>\n\n\n\n In this example, the ID of the container is In this example, Here\u2019s another example of running a python script inside a container:<\/p>\n In this example, Note that you can also use the container ID instead of its name in the above commands. Also, make sure that the command you are trying to run is installed inside the container and accessible from the path where you are running it.<\/p>\n\n\n\n Human: Ok perfect, let explain the “docker run” command we will need<\/p>\n\n\n\n AI:<\/strong> The In this example, the image name is Here\u2019s what each option does:<\/p>\n This command will start a new container from the Note that you can also add commands to be executed inside the container when it starts by appending them after the image name, like so:<\/p>\n In this example, Human:<\/strong> Ok now you better understand the basics of docker we can see the fun part : AI model running in docker. \ud83d\ude00<\/p>\n\n\n\n You will need to trust me as things should take some time to complete, but the AI will explain everything right after.<\/p>\n\n\n\n Do as follow:<\/p>\n\n\n\n AI:<\/strong> Let\u2019s break down the Overall, this command creates a new container from the image Human:<\/strong> After some time, depending on your computer performances, you should be able to see something like this :<\/p>\n\n\n\n\n
docker run -p 8080:80 nginx:latest\n<\/code><\/pre>\n<\/div>\n\n\n\n\n
docker ps<\/code> command:<\/li>\n<\/ol>\nC:\\> docker ps\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n6f5a37c2438f myimage:latest "python app.py" 2 hours ago Up 2 hours mycontainer\n<\/code><\/pre>\n6f5a37c2438f<\/code>, and its name is mycontainer<\/code>.<\/p>\n<\/div>\n\n\n\n\n
docker exec<\/code> command to run a command inside the container. For example, if you want to run the ls<\/code> command inside the container to list the files in the current directory, you can use the following command:<\/li>\n<\/ol>\nC:\\> docker exec -it mycontainer ls\napp.py config.py Dockerfile requirements.txt\n<\/code><\/pre>\n-it<\/code> flag is used to allocate a pseudo-TTY and keep STDIN open so that you can interact with the command.<\/p>\n<\/div>\n\n\n\nC:\\> docker exec -it mycontainer python app.py arg1 arg2\n<\/code><\/pre>\napp.py<\/code> is the name of the python script that you want to run, and arg1<\/code> and arg2<\/code> are arguments passed to the script.<\/p>\n<\/div>\n\n\n\ndocker run<\/code> command is used to create a new container from an image and start the container. It has several options that can be used to configure the container, such as setting environment variables, mapping ports, and specifying volumes. Here’s an example of how to use the `docker run` command with some common options:<\/p>\n\n\n\n\n
docker images<\/code> command:<\/li>\n<\/ol>\n$ docker images\nREPOSITORY TAG IMAGE ID CREATED SIZE\nmyimage latest e94f8c5a1234 2 days ago 1GB\n<\/code><\/pre>\nmyimage<\/code>, and its tag is latest<\/code>.<\/p>\n<\/div>\n\n\n\n\n
docker run<\/code> command to start a new container from the image with some options:<\/li>\n<\/ol>\n$ docker run -d --name mycontainer -p 5000:5000 -e MY_ENV_VAR=myvalue -v myvolume:\/app\/data myimage:latest\n<\/code><\/pre>\n\n
-d<\/code>: Run the container in detached mode (in the background).<\/li>\n--name mycontainer<\/code>: Set the name of the container to mycontainer<\/code>.<\/li>\n-p 5000:5000<\/code>: Map port 5000 inside the container to port 5000 on the host machine.<\/li>\n-e MY_ENV_VAR=myvalue<\/code>: Set an environment variable named MY_ENV_VAR<\/code> with a value of myvalue<\/code> inside the container.<\/li>\n-v myvolume:\/app\/data<\/code>: Create a volume named myvolume<\/code> and mount it to \/app\/data<\/code> inside the container.<\/li>\nmyimage:latest<\/code>: The image to use to create the container, with its tag latest<\/code>.<\/li>\n<\/ul>\nmyimage:latest<\/code> image, set its name to mycontainer<\/code>, map port 5000 on the host machine to port 5000 inside the container, set an environment variable named MY_ENV_VAR<\/code> with a value of myvalue<\/code>, create a volume named myvolume<\/code>, and mount it to \/app\/data<\/code> inside the container.<\/p>\n$ docker run -d --name mycontainer myimage:latest python app.py arg1 arg2\n<\/code><\/pre>\npython app.py arg1 arg2<\/code> is the command that will be executed inside the container when it starts.<\/p>\n<\/div>\n\n\n\n\n
docker run -p 8181:8181 -v <path to the directory where you put the model>:\/models ghcr.io\/ggerganov\/llama.cpp:full --server -m \/models\/blossom-v3-mistral-7b.Q4_K_M.gguf --port 8181 --host 0.0.0.0\n<\/code><\/pre>\n<\/div>\n\n\n\n\n
docker run<\/code> command you provided and explain its options:<\/p>\n<\/div>\n\n\n\n\n
docker run<\/code>: This is the base command to create a new container from an image and start it.<\/li>\n-p 8181:8181<\/code>: This option maps port 8181 inside the container to port 8181 on the host machine. When you access http:\/\/localhost:8181<\/code> on your host machine, it will be forwarded to http:\/\/localhost:8181<\/code> inside the container.<\/li>\n-v <path to the directory where you put the model>:\/models<\/code>: This option mounts a volume from the host machine to the container. In this case, it maps the directory <path to the directory where you put the model><\/code> on the host machine to the directory \/models<\/code> inside the container.<\/li>\nghcr.io\/ggerganov\/llama.cpp:full<\/code>: This is the image name to use for creating the container. It specifies the location of the image on Docker Hub and includes the tag full<\/code>.<\/li>\n--server<\/code>: This option sets the server mode for the LLM (large language model) that will be running inside the container.<\/li>\n-m \/models\/blossom-v3-mistral-7b.Q4_K_M.gguf<\/code>: This option specifies the path to the pre-trained LLM model file inside the container. In this case, it points to the file located at \/models\/blossom-v3-mistral-7b.Q4_K_M.gguf<\/code>.<\/li>\n--port 8181<\/code>: This option sets the port that the LLM server inside the container will listen on for incoming requests. In this case, it\u2019s set to port 8181.<\/li>\n<\/ol>\n<\/div>\n\n\n\nghcr.io\/ggerganov\/llama.cpp:full<\/code>, maps port 8181 inside the container to port 8181 on the host machine, mounts a volume from the host machine to the container, sets the server mode for the LLM, specifies the path to the pre-trained LLM model file inside the container, and sets the port that the LLM server will listen on. Once the container is started, you can access the LLM server via http:\/\/localhost:8181<\/code> on your host machine and interact with it by sending HTTP requests to that address.<\/p>\n<\/div>\n\n\n\nAvailable slots:\n -> Slot 0 - max context: 512\n{"timestamp":1703897816,"level":"INFO","function":"main","line":3141,"message":"HTTP server listening","port":"8181","hostname":"0.0.0.0"}\n\nllama server listening at http:\/\/0.0.0.0:8181\n\nall slots are idle and system prompt is empty, clear the KV cache\n<\/code><\/pre>\n<\/div>\n\n\n\n
![\"\"](\"https:\/\/dataops-blog.com\/wp-content\/uploads\/2023\/12\/llamacpp.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/dataops-blog.com\/wp-content\/uploads\/2023\/12\/llamacpp2.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/dataops-blog.com\/wp-content\/uploads\/2023\/12\/docker-blossom-1024x499.png\")
<\/figure>\n\n\n\n