LLM Part 6 | Dockerize Your Career Coach: Deploying an LLM App with Ollama and LangChain

AI generated by Thierry Moreau and Ajeet Singh Raina

On LLM Series:

• LLM Series Part 1 | Cloud Journey — Part 11 | LLMs in Finance using FinGPT
• LLM Series Part 2 | Unleashing the Power of Language Models: Disrupting Retail Banking with LLM Technology | Platformization of Banking — Part 6
• LLM Series Part 3 | Make AI-Driven Decisions: Financial Statement Analysis with LangChain & OpenAI
• LLM Series Part 4 | Level Up Your Developer Experience (DevEx) with GenAI
• LLM Series Part 5 | How LLMs Can Chatify Your Database
• LLM Part 6 | Dockerize Your Career Coach: Deploying an LLM App with Ollama and LangChain

This blog post, part 6 of the LLM Series, dives into containerizing Ollama and LangChain web applications for deployment. We’ll explore the benefits of containerization and provide a step-by-step guide using a real-world example: WillowPath.ai, an LLM for career coaching.

What is WillowPath.ai ?

Feeling lost in the labyrinth of career choices? Unsure of the skills and knowledge needed to land your dream job? WillowPath.ai is your friendly LLM (Large Language Model) career coach, here to guide you through the ever-evolving job market. Whether you’re seeking a complete career shift or looking to refine your current skillset, WillowPath.ai offers a personalized roadmap to success.

This innovative platform leverages the power of artificial intelligence to analyze your goals and aspirations, recommending tailored learning paths, books, and resources to bridge the gap between where you are and where you want to be. WillowPath.ai doesn’t stop there.

It also empowers you with valuable insights into potential job opportunities, including salary ranges, helping you make informed career decisions with confidence. So, take the first step towards your dream career and explore the possibilities with WillowPath.ai!

Dockerize your LangChain App

Step 0. LangSmith

Make you setup your LangSmith as well:

os.environ['LANGCHAIN_TRACING_V2'] = "True"
os.environ['LANGCHAIN_ENDPOINT'] = "https://api.smith.langchain.com"
os.environ['LANGCHAIN_API_KEY'] = ""
os.environ['LANGCHAIN_PROJECT'] = ""

If you also integrate the streamlit-feedback to it, it can give you good traceability matters.

Step 1. Dockerfile

FROM ubuntu:latest
LABEL authors="chrisshayan"

FROM python:3.12-slim
WORKDIR /app/
RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    software-properties-common \
    git \
    gh \
    && rm -rf /var/lib/apt/lists/*
#ensure your GITHUB_TOKEN is set if you wanna use gh
RUN gh repo clone chrisshayan/  hr-coach .
RUN pip install -r requirements.txt
EXPOSE 80
HEALTHCHECK CMD curl --fail http://localhost:80/_stcore/health
ENTRYPOINT ["streamlit", "run", "main.py", "--server.port=80", "--server.address=0.0.0.0"]

Step 2. Build Your Image

docker build -t hr-coach .
docker images //to verify your image is there

Step 3. Pull your Ollama Docker Image

You can find details in here.

Step 4. Build your docker-compose.yml

Let’s now build a docker-compose configuration file, to define the network of the hr-coach application and the Ollama container, so that they can interact with each other.

version: '3'
services:
  ollama-container:
    image: ollama/ollama
    volumes:
      - ./data/ollama:/root/.ollama
    ports:
      - 11434:11434
  streamlit-app:
    image: hr-coach:latest
    ports:
      - 80:80

If you are using Nvidia GPU then you need to run:

$  nvidia-smi

Result of nvidia-smi

Then add to your compose this section:

You also need to pull the model you need:

docker exec -it hr-coach_ollama-container_1 ollama run <modelname>

Step 5. Bringing up

Simply run:

docker-compose up