As Business Analytics students from The George Washington University, our project involves building a chatbot and text analytics model for our client, the World Wildlife Fund.
Team members: Pedro Chames Carvalho, Ke Chai, Varsha Katam, Hao Ren, Priscilla Sit, Alejandra Taboada
- Executive Summary
- Problem Understanding
- Data Collection and Pre-processing
- The Text Analytics Model
- The Chatbot
- Recommended Next Steps
- Risk Considerations
Our project aimed to develop an interactive chatbot platform for WWF that leverages natural language processing (NLP) and machine learning (ML) algorithms to streamline customer query resolution and reduce the workload of the Customer Service team. By mapping customer queries to one of ten possible topics and providing accurate solutions quickly, the platform enhances customer service and satisfaction for WWF.
In our research, we identified extended wait times, inaccurate query resolution, and difficulty navigating the website as WWF’s key challenges. To address these issues, we developed the chatbot platform that can answer customer queries with high accuracy and includes a live scoring mechanism to provide feedback on the accuracy of queries.
WWF needed a comprehensive solution to improve their customer query resolution process and ease the workload of their Customer Service team. We developed a chatbot and a text analytics model that could accurately forecast customer queries and display their accuracy percentage. WWF faced significant challenges in managing customer queries and resolving issues in a timely manner due to the reliance on phone and email support. This approach made it difficult to identify recurring queries and pain points of customers, resulting in dissatisfaction and frustration.
The objective of the project was to build an interactive chatbot platform that reduces the number of recurring queries and provides valuable insights into customer needs and preferences. The identified pain points included extended wait times, inaccurate query resolution, and difficulties in navigating the website. Addressing these issues could enhance customer satisfaction and loyalty, leading to an improvement in WWF's overall reputation.
1. Data Collection:
The dataset of customer inquiries was provided by the World Wildlife Fund in a file named TextAnalytics.csv. It included 15,219 rows and 5 columns, such as the inquiry's full description, submission date and time, unique identifier, and general topic group (10 were given by the client).
Data Cleaning: We used Python to clean the data, ensuring it was consistent, relevant, and error-free. The cleaning process removed irrelevant data (such as greetings and thank-you messages), standardized data by changing capitalization and punctuation, and removed noise (such as stop words, special characters, and HTML). We used 're' and 'nltk' packages for data cleaning.
2. Topic Extraction:
After cleaning the data, we used SAS Enterprise Miner to identify the top 25 topics in the dataset. These topics formed the basis for developing the interactive chatbot platform. We are continuously refining the topic extraction process to improve the quality of the topics identified. The top five significant areas we identified include
- Cancel monthly donation
- Order status
- Cancel membership payment
- Monthly payment status
- Remove customer mailing list
3. Data Pre-processing:
We pre-processed the data by tokenizing sentences into individual words, performing lemmatization, and removing stop words. This transformed the raw text data into a more structured format suitable for analysis.
Technologies Used for Text Analytics Model
We then proceeded to create the Text Analytics model. We want to create model that accurately predict the topic of a customer inquiry and assigns it one. To do this, we used SAS Enterprise Miner. Secondly, we used the Model Comparison feature of SAS so we could find the most accurate model. Finally, we also utilized the Score feature of the software so the model displays the probability that a customer inquiry is about a specific topic and assigns it the most probable one.
Results for the Text Analytics Model
After topic extraction and matching our collected 25 topics with the 10 general ones given by the client, we compared a neural network, decision tree, and regression model to see which would more reliably predict the topic of a customer inquiry based on its contents. Out of those three, regression was the most accurate model, featuring a KS-statistic of 0.133 and a KS probability cutoff of 0.295.
The text analytics model also uses live scoring to identify the topic of a customer inquiry. The full documentation of the model and the live scoring results can be found in the Appendix. Full integration of the text analytics model into the chatbot is a work in progress.
- Webstorm Integrated Development Environment (IDE).
- Node JS framework, which is an open-source framework that allows developers to build scalable and high-performance applications using JavaScript.
- NPM (Node Package Manager) to manage and install all the dependencies required for our chatbot.
- JavaScript with the ReactJS library, which is a popular front-end development library that provides a component-based approach to building user interfaces.
- Customized and configured the react-chatbot-kit, which is an open-source library that provides a framework for building chatbots using ReactJS. This allowed us to build a chatbot platform that was both user-friendly and effective in resolving customer queries.
- Go to the official Node.js website
- Select the appropriate installer for your operating system (Windows, macOS, or Linux) and download it.
- Once the installer has finished downloading, run it and follow the on-screen instructions to install Node.js.
- After the installation is complete, open a terminal or command prompt and type "node -v" to verify that Node.js has been installed correctly. You should see the version number of Node.js printed to the console.
- Next, type "npm -v" to verify that npm (Node Package Manager) has also been installed. You should see the version number of npm printed to the console.
- Clone the repo using HTTPS or SSH
$ git clone https://github.com/katamvarsha/chatbot.git $ git clone git@github.com:katamvarsha/chatbot.git
- Install NPM packages and node modules
$ npm install
- Install react-chatbot-kit
$ npm install react-chatbot-kit
For further details and information regarding React Chatbot Kit, please refer to the official documentation.
The parse function of MessageParser is utilized to process each user message.
This function includes a set of rules that determine the action to be initiated.
const MessageParser = ({ children, actions }) => {
const parse = (message) => {
message = message.toLowerCase()
if (message.includes('hello')) {
actions.handleHello()
}
};
return (
<div>
{React.Children.map(children, (child) => {
return React.cloneElement(child, {
parse: parse,
actions,
});
})}
</div>
);
};
After defining the ruleset, the next step is to develop an action that can be activated. It should be designed to carry out a specific task or response based on the user's input. Depending on the context and purpose of the program, the action can take various forms such as a displaying a message or executing a function.
const ActionProvider = ({ createChatBotMessage, setState, children }) => {
const updateState = (botMessage) => {
setState((prev) => ({
...prev,
messages: [...prev.messages, botMessage],
}));
}
const handleHello = () => {
const botMessage = createChatBotMessage('Hello. Nice to meet you.');
updateState(botMessage)
};
return (
<div>
{React.Children.map(children, (child) => {
return React.cloneElement(child, {
actions: {
handleHello
},
});
})}
</div>
);
};
To start the interactive chatbot, simply run $ npm start. This will open the development server at http://localhost:3000/.
$ cd wwf
$ npm start
| Key Words | Example Prompts from WWF Dataset | Group |
|---|---|---|
| (refund AND (membership OR donat(ion/ing))) | Recently cancelled monthly donation and asked for a refund however havent yet received money back | Refund |
| ((change OR update) AND (mail OR address OR name)) | I am trying to donate online. My name has recently changed from Lewis to Maloney. My email address has remained the same but the website won't accept this as a valid email address even though I received my donation request through that email address | Change Personal Information |
| (donation OR adopt) | Good morning, I've made a donation a few days ago, I would like to know which carrier do you guys use so I can check with the people from the receiving department since I got it ship to my job. Thanks | Donation Question |
| (update OR remove OR (take AND off)) AND mail | Hi. We very much appreciate the work you're doing, and we do donate. Would you please take us off the paper mailing list in order to save paper | Change Solicitation Preferences |
| cancel AND (subscription OR membership or donation) | Hi, I need to cancel my membership at this time. I am currently donating $15/month.Please advise me when this is done | Sustainer Request |
| renew | I sent a membership renewal on the 17th, the wwf website hungup at the end of transaction. I am not sure if it went through. Can you help? | Renewal Issue |
| (receive OR send OR forward) AND (mail OR receipt OR invoice OR address) | Please forward a copy of invoice with mailing address to example@gmail. | Mailed Info |
| member OR contribution | I purchased membership a while ago for my two young sons and they requested the socks but never received them. It has been a long time and they are now disappointed as if their membership doesn't count. How can I correct this? | Member Inquiry |
| ((update OR incorrect OR status) AND (order OR address OR mail)) | I recently ordered this but the mailing address is incorrect. The correct mailing address is 123 Example St | Benefit Update |
| (not received OR never received) AND (order OR mail OR receipt) | Hello, i still have not received my gift bag and i ordered it before christmas. just wondering what is taking so long | Benefit Issue |
There are a few suggested next steps to take the development of our chatbot and model further.
1. Conduct user testing to gather feedback and improve the chatbot's functionality
The topic model was built and tested on the dataset provided. However, as we know the English language is fluid and there are multiple variations in the use of the language, it would be helpful to release the chatbot as a trial to collect real time customer input to test its response and accuracy. User testing will also enable WWF to understand what the chatbot may potentially struggle to understand and respond appropriately, e.g. certain languages, dialects, abbreviations, etc. so as to incorporate these considerations into the further development of the chatbot.
**2. Integrate chatbot with text analytics model
To take the chatbot development to the next phase, it is suggested to connect the text analytics model to the chatbot for it to provide live scoring. Having the accuracy score will be a great addition.
3. Integrate with different messaging platforms to reach a wider audience
We understand WWF is also on social media platforms like Facebook and Instagram where there are messaging platforms attached. Integrating the chatbot to these platforms enables WWF to solve customer enquiries not only with its official website but on multiple channels. This allows WWF to reach a wider span of audience and further enhance their customer service quality.
4. Add more features to the chatbot to enhance user experience
In addition to text messages, customers could record voice messages on their enquiries. This would leverage voice recognition and natural language processing in the model’s further development. Also, the use of emojis are not uncommon nowadays and WWF can utilize that for sentiment analysis. It can be useful to distinguish positive and negative comments from customers to cater to their immediate needs and for the chatbot to react more appropriately if negative sentiments are detected.
5. Connect the chatbot to WWF’s database
To continuously build the chatbot’s knowledge base, it will be useful to connect the chatbot to WWF’s database and store all incoming customer enquiries there. This practice will not only apply to the chatbot but all customer enquiries from various existing customer service channels, e.g. email, phone, etc.. This will support the model’s continuous learning to better automate responses based on accurate and up-to-date information.
1. Data privacy: Data leak and/or data breach caused by malicious attacks or human mistakes are possible. WWF will need a secure database to store these data to ensure they are encrypted and properly stored. Irrelevant personal information should be erased from the database.
2. Chatbot response accuracy: As the chatbot requires and depends on learning from the knowledge base to provide its responses, its accuracy is highly dependent on the quality and completeness of the information provided to it. There is a risk of the chatbot providing inaccurate information or inappropriate responses if the data collected is not monitored, reviewed and cleaned. Inaccurate or inappropriate responses could lead to misunderstandings and confusion, harm the brand reputation. A possible solution to this would be adding a troubleshoot option in the chatbot to inform the IT team promptly.
3. Limited Ability to Handle Complex Situations: Chatbots may have difficulty dealing with complex customer queries or issues that require human intervention and problem-solving skills. Similar to point 2, thsi could be solved by adding a "chat with customer service representative" button in the chatbot for customer service staff to take on the queries.
4. Change management: Chatbots require regular maintenance and updates to ensure they continue to function properly and provide accurate responses. Failure to do so can lead to a breakdown in the system and a negative impact on the customer experience. Another consideration is that inconsistencies/descrepancies could exist between frontend and backend. or issues from API, throughout the maintenance process, which may cause infunctionality.