LLM chatbot — a business assistant for a major manufacturer and distributor of electrical installation products

To address the tasks set before us, together with the client, we decided to develop a chatbot based on artificial intelligence. This is a chatbot capable of communicating with users in natural human language. Through interaction with various APIs, the bot always knows stock availability and can provide all the necessary information about the products.

The language model can autonomously select APIs to interact with, retrieve user conversation history, and order information.

For the project development, we utilized the Microsoft Azure ecosystem. We deployed a secure dedicated server, connected and trained GPT-4, configured modules for working with the vector database using Azure AI search, connected the DBMS, and integrated with the client's API—data from the website about call center operations and their contacts. To select the information source, we orchestrated the language model using LangChain.

Typical interaction scenarios with users are stored in the vector database. Thus, we achieved a standardized response format and provided the language model with the necessary context for responses. The relational database stores the conversation history with the user and information that is periodically updated (e.g., operating hours of trading points).

Quality control and adherence to communication standards with users are evaluated by the discriminator (GPT agent). An administrative panel is provided for outputting analytical data, debugging the chatbot, and communicating with users. The quality assessment of the chatbot's responses is implemented using a feedback system.

1. The user interacts with the chatbot UI and submits a query.

2. The backend service processes the query: the user's request and conversation history are sent to the language model.

3. The language model selects APIs and/or accesses the Embedding model based on the user's query theme and primary classification.

4. The language model retrieves necessary information from sources and generates a response.

5. The user's query is filtered by the GPT agent for legality and/or to check for question correctness (e.g., questions about competitors and prices).

6. The response is returned to the backend and recorded in the database to save the user conversation.

7. The response is delivered to the user.

8. The user provides feedback, which is recorded in the relational database and displayed in the admin panel.

Integrated into the client's website as a web widget, User Interface enables access to support services. Developed using React, HTML 5, CSS 3, it utilizes webhooks to communicate with the backend.

The backend houses the core logic of the service. Integrated with the database of user conversations and orders, as well as the language model, it processes user requests, manages user conversation history, and interacts with the language model. Load balancer and instance replication are utilized. The backend is designed to enable the creation of replicas of existing instances and distribute the load through the load balancer. This ensures stable chatbot operation on the website during peak times.

For the development of the relational database, we used PostgreSQL. This approach has several advantages:

• Ability to implement ACID transactions and rollbacks. The entire user interaction history is preserved even in case of connection loss.
• Scalability: Postgres can easily be distributed across multiple servers to handle large volumes of data.
• Security: Postgres ensures a high level of security through flexible access control, authentication and encryption mechanisms, cryptography, auditing, and backup.

There are two options for developing the chatbot's language model:

1. Using the commercial GPT-4 Turbo API.
2. Deploying the GPT-4 model on a private Azure server.

The first option has several drawbacks:

• The GPT API does not guarantee 100% uptime and may restrict access during overloads, unlike a dedicated GPT model.
• GPT-4 has a limit on daily requests. The limit increases over time, so initially, the chatbot will have a limited number of interactions.

We chose the second option with a local model because it has several advantages:

• Provides greater stability for the chatbot since the model is deployed on its own server. It eliminates risks associated with changes in the terms of use of third-party APIs.
• Offers full control over computational resources and performance.
• Guarantees the confidentiality and security of personal data, which are not transmitted to third parties.
• Provides greater flexibility in configuring and scaling the model architecture to meet project requirements.

This model translates natural language queries into vector representations. It is directly integrated with the semantic search engine for use in the vector database. We used Azure OpenAI Embeddings Ada as the embedding model.

For semantic search, we used Azure AI Search. The service is a leading information retrieval platform for creating RAG models. In addition to configuring semantic search, the service allows integration between LLM, embedding models, the semantic search engine, and the vector database within the Microsoft Azure ecosystem.

The vector database is populated with standard communication scenarios, general information about project solutions, and product usage regulations. The chatbot accesses the vector database through the Azure AI Search semantic search engine.

To enable the model to automatically determine which API to use, we employed LangChain orchestration.

1. Upon receiving a user request, the first Large Language Model (LLM), acting as an intent classifier, determines the theme of the question—whether it involves technical specifications of electrical installation products or commercial information.
2. Depending on the theme, the second LLM (router) selects the appropriate API from the available options (e.g., ERP API or website API).
3. The third LLM (response generator) shapes the final response by accessing the previously selected API and utilizing the obtained data.
4. The response is returned to the user.

The GPT agent serves as a discriminator, processing the final response from the language model immediately before delivering it to the user. The discriminator verifies that the model has not included information about prices and competitors in its response.

Users instantly receive the necessary information about the company's products. The chatbot addresses 80% of customer inquiries, and only in 1 out of 5 cases do users request assistance from an operator.