Running & shipping AI model on Windows using Docker

docker run -p 8080:80 nginx:latest

1. First, you need to find the ID or name of your Docker container using the docker ps command:

C:\> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6f5a37c2438f myimage:latest "python app.py" 2 hours ago Up 2 hours mycontainer

In this example, the ID of the container is 6f5a37c2438f, and its name is mycontainer.

2. Next, you can use the docker exec command to run a command inside the container. For example, if you want to run the ls command inside the container to list the files in the current directory, you can use the following command:

C:\> docker exec -it mycontainer ls
app.py config.py Dockerfile requirements.txt

In this example, -it flag is used to allocate a pseudo-TTY and keep STDIN open so that you can interact with the command.

Here’s another example of running a python script inside a container:

C:\> docker exec -it mycontainer python app.py arg1 arg2

In this example, app.py is the name of the python script that you want to run, and arg1 and arg2 are arguments passed to the script.

1. First, you need to find the image you want to run using the docker images command:

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
myimage latest e94f8c5a1234 2 days ago 1GB

In this example, the image name is myimage, and its tag is latest.

2. Next, you can use the docker run command to start a new container from the image with some options:

$ docker run -d --name mycontainer -p 5000:5000 -e MY_ENV_VAR=myvalue -v myvolume:/app/data myimage:latest

Here’s what each option does:

• -d: Run the container in detached mode (in the background).
• --name mycontainer: Set the name of the container to mycontainer.
• -p 5000:5000: Map port 5000 inside the container to port 5000 on the host machine.
• -e MY_ENV_VAR=myvalue: Set an environment variable named MY_ENV_VAR with a value of myvalue inside the container.
• -v myvolume:/app/data: Create a volume named myvolume and mount it to /app/data inside the container.
• myimage:latest: The image to use to create the container, with its tag latest.

This command will start a new container from the myimage:latest image, set its name to mycontainer, map port 5000 on the host machine to port 5000 inside the container, set an environment variable named MY_ENV_VAR with a value of myvalue, create a volume named myvolume, and mount it to /app/data inside the container.

Note that you can also add commands to be executed inside the container when it starts by appending them after the image name, like so:

$ docker run -d --name mycontainer myimage:latest python app.py arg1 arg2

In this example, python app.py arg1 arg2 is the command that will be executed inside the container when it starts.

docker run -p 8181:8181 -v <path to the directory where you put the model>:/models ghcr.io/ggerganov/llama.cpp:full --server -m /models/blossom-v3-mistral-7b.Q4_K_M.gguf --port 8181 --host 0.0.0.0

AI: Let’s break down the docker run command you provided and explain its options:

1. docker run: This is the base command to create a new container from an image and start it.
2. -p 8181:8181: This option maps port 8181 inside the container to port 8181 on the host machine. When you access http://localhost:8181 on your host machine, it will be forwarded to http://localhost:8181 inside the container.
3. -v <path to the directory where you put the model>:/models: This option mounts a volume from the host machine to the container. In this case, it maps the directory <path to the directory where you put the model> on the host machine to the directory /models inside the container.
4. ghcr.io/ggerganov/llama.cpp:full: This is the image name to use for creating the container. It specifies the location of the image on Docker Hub and includes the tag full.
5. --server: This option sets the server mode for the LLM (large language model) that will be running inside the container.
6. -m /models/blossom-v3-mistral-7b.Q4_K_M.gguf: This option specifies the path to the pre-trained LLM model file inside the container. In this case, it points to the file located at /models/blossom-v3-mistral-7b.Q4_K_M.gguf.
7. --port 8181: This option sets the port that the LLM server inside the container will listen on for incoming requests. In this case, it’s set to port 8181.

Overall, this command creates a new container from the image ghcr.io/ggerganov/llama.cpp:full, maps port 8181 inside the container to port 8181 on the host machine, mounts a volume from the host machine to the container, sets the server mode for the LLM, specifies the path to the pre-trained LLM model file inside the container, and sets the port that the LLM server will listen on. Once the container is started, you can access the LLM server via http://localhost:8181 on your host machine and interact with it by sending HTTP requests to that address.

Available slots:
 -> Slot 0 - max context: 512
{"timestamp":1703897816,"level":"INFO","function":"main","line":3141,"message":"HTTP server listening","port":"8181","hostname":"0.0.0.0"}

llama server listening at http://0.0.0.0:8181

all slots are idle and system prompt is empty, clear the KV cache

AI: Here’s an example Python code using the requests library to interact with the LLM (large language model) running inside the Docker container:

import requests

# Set the URL for the LLM server running inside the Docker container
url = "http://localhost:8181/completion"

# Define a prompt to send to the LLM server
prompt = "What is the capital of France?"

# Send a POST request to the LLM server with the prompt as JSON data
response = requests.post(url, json={"prompt": prompt})

# Extract the generated text from the response
generated_text = response.json()["content"]

# Print the generated text
print("Generated text:", generated_text)

In this example code, we first set the URL for the LLM server running inside the Docker container to http://localhost:8181. We then define a prompt to send to the LLM server as a string. In this case, we ask “What is the capital of France?”

Next, we use the requests library to send a POST request to the LLM server with the prompt as JSON data in the body of the request. The LLM server will generate text based on the prompt and return it as a response.

Finally, we extract the generated text from the response using the json() method and print it to the console. In this example, we expect the generated text to be “The capital of France is Paris.”

Note that you can modify the prompt string to ask any question or make any statement you want to generate text for. Additionally, you can adjust the requests.post() function call to include additional parameters or headers as needed for your specific use case.

AI: A Dockerfile is a text file that contains instructions for building a Docker image. It specifies the base image to use, any necessary dependencies or libraries to install, and how to run the application inside the container. The Dockerfile serves as a blueprint for creating a customized Docker image tailored to your specific application needs.

Here are some examples of common instructions found in a Dockerfile:

• FROM: This instruction specifies the base image to use for the new image being built. For example, FROM python:3.9-slim-buster would use the official Python 3.9 slim version as the base image.
• WORKDIR: This instruction sets the working directory inside the container for subsequent instructions. For example, WORKDIR /app would set the working directory to /app inside the container.
• COPY: This instruction copies files or directories from the host machine to the container. For example, COPY requirements.txt . would copy the requirements.txt file from the host machine to the current directory (.) inside the container.
• RUN: This instruction runs a command inside the container. For example, RUN pip install -r requirements.txt would run the pip install -r requirements.txt command inside the container to install any necessary Python packages.
• CMD: This instruction specifies the default command to run when a container is launched from the image. For example, CMD ["python", "app.py"] would run the python app.py command by default when launching a container from the image.

To build a Docker image from a Dockerfile, you can use the following command:

$ docker build -t myimage:latest .

In this example command, docker build is used to build an image using the Dockerfile in the current directory (.). The -t flag sets the tag for the new image to myimage:latest, indicating that it is the latest version of the myimage image.

During the image building process, Docker will execute each instruction in the Dockerfile sequentially to create a new layer on top of the previous one. Each layer contains only the changes made by the corresponding instruction, making the final image smaller and faster to download or upload. Once all instructions have been executed, the resulting image can be used to launch containers with the desired application and dependencies pre-installed.

Overall, Dockerfiles provide a convenient way to automate the process of building customized Docker images for your applications. By specifying the exact steps required to build an image in a text file, you can easily share and reproduce consistent builds across different machines and environments.

(MKDIR to create the directory from the windows command line, cd /D E:\directory if you want to change disk)

Now create a file named model.Dockerfile and edit it.

FROM ghcr.io/ggerganov/llama.cpp:full
RUN mkdir -p /models
COPY blossom-v3-mistral-7b.Q4_K_M.gguf /models/.
EXPOSE 8181/tcp
ENTRYPOINT ["/app/.devops/tools.sh", "--server", "-m", "/models/blossom-v3-mistral-7b.Q4_K_M.gguf", "--port", "8181", "--host", "0.0.0.0"]

docker build -t blossom:v1 -f blossom.Dockerfile .

docker run -p 8181:8181 blossom:v1