Keywords:-

Keywords: Artificial Intelligence, Chatbot, Digital Banking, Google Gemini API, FAQ Knowledge Base, Chat Log Analytics

Article Content:-

Abstract

This study designs and evaluates a web-based customer service chatbot for Islamic banking services. The chatbot integrates the Google Gemini API with a validated Frequently Asked Questions knowledge base to generate responses based on institutional service information. The system was developed using the Flask framework and evaluated through Black Box Testing, FAQ-based response evaluation, and chat log analytics. The evaluation used 310 validated FAQ records and 1,000 anonymized raw chat log records, which were filtered into 360 valid categorized records for interaction-pattern analysis. Based on 310 FAQ-based test questions, the chatbot produced 287 correct responses and 23 incorrect responses. The results show an accuracy of 92.58%, precision of 95.67%, recall of 96.63%, F1-score of 96.14%, success rate of 92.58%, and an average response time of 1.84 seconds. Chat log analysis further indicates that Mobile Banking was the most frequent service category requested by users. These findings suggest that combining Generative AI with a structured FAQ knowledge base can support fast and relevant customer service responses in Islamic banking.

References:-

References

C. Ledro, A. Nosella, and A. Vinelli, “Artificial intelligence in customer relationship management: literature review and future research directions,” Dec. 19, 2022, Emerald Group Holdings Ltd.

doi: 10.1108/JBIM-07-2021-0332.

S. G. Ayinaddis, B. A. Taye, and B. G. Yirsaw, “Examining the effect of electronic banking service quality on customer satisfaction and loyalty: an implication for technological innovation,” J. Innov. Entrep., vol. 12, no. 1, Dec. 2023,

doi: 10.1186/s13731-023-00287-y.

D. Leocádio, L. Guedes, J. Oliveira, J. Reis, and N. Melão, “Customer Service with AI-Powered Human-Robot Collaboration (HRC): A Literature Review,” in Procedia Computer Science, Elsevier B.V., 2024, pp. 1222–1232. doi: 10.1016/j.procs.2024.01.120.

L. Jenneboer, C. Herrando, and E. Constantinides, “The Impact of Chatbots on Customer Loyalty: A Systematic Literature Review,” Mar. 01, 2022, MDPI. doi: 10.3390/jtaer17010011.

L. Banh and G. Strobel, “Generative artificial intelligence,” Electronic Markets, vol. 33, no. 1, Dec. 2023, doi: 10.1007/s12525-023-00680-1.

Z. Bin Akhtar, “Unveiling the evolution of generative AI (GAI): a comprehensive and investigative analysis toward LLM models (2021–2024) and beyond,” Journal of Electrical Systems and Information Technology, vol. 11, no. 1, Jun. 2024, doi: 10.1186/s43067-024-00145-1.

T. R. McIntosh et al., “From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape,” Feb. 01, 2025, Multidisciplinary Digital Publishing Institute (MDPI).

doi: 10.3390/technologies13020051.

L. A. L. F. da Costa et al., “Advancing Chatbot Conversations: A Review of Knowledge Update Approaches,” Journal of the Brazilian Computer Society, vol. 30, no. 1, pp. 55–68, Mar. 2024,

doi: 10.5753/jbcs.2024.2882.

H. Ho, S. M. Han, J. Cha, and L. Pham, “Mobile Banking Customer Satisfaction and Loyalty: The Roles of Technology Readiness,” Journal of Risk and Financial Management, vol. 18, no. 7, Jul. 2025,

doi: 10.3390/jrfm18070403.

K. W. Church, J. Sun, R. Yue, P. Vickers, W. Saba, and R. Chandrasekar, “Emerging trends: A gentle introduction to RAG,” Nat. Lang. Eng., vol. 30, no. 4, pp. 870–881, Jul. 2024,

doi: 10.1017/S1351324924000044.

K. Kuligowska and M. Stanusch, “Commercial chatbot monitoring: Approaches focused on automated conversation analysis,” Humanities & Social Sciences Reviews, vol. 12, no. 2, pp. 54–60, Sep. 2024, doi: 10.18510/hssr.2024.1227.

S. Woollaston, B. Flanagan, P. Ocheja, Y. Toyokawa, and H. Ogata, “ARCHIE: Exploring Language Learner Behaviors in LLM Chatbot-Supported Active Reading Log Data with Epistemic Network Analysis,” in 15th International Conference on Learning Analytics and Knowledge, LAK 2025, Association for Computing Machinery, Inc, Mar. 2025, pp. 642–654. doi: 10.1145/3706468.3706556.

G. Caldarini, S. Jaf, and K. McGarry, “A Literature Survey of Recent Advances in Chatbots,” Information (Switzerland), vol. 13, no. 1, Jan. 2022, doi: 10.3390/info13010041.

S. U. Singh and A. S. Namin, “A survey on chatbots and large language models: Testing and evaluation techniques,” Natural Language Processing Journal, vol. 10, Mar. 2025, doi: 10.1016/j.nlp.2025.100128.

M. Abbasian et al., “Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI,” NPJ Digit. Med., vol. 7, no. 1, Dec. 2024, doi: 10.1038/s41746-024-01074-z.

T. K. Q. Nguyen, “E-Banking Service Quality and Customer Satisfaction with Moderator Factor,” Emerging Science Journal, vol. 9, no. 2, pp. 725–740, Apr. 2025, doi: 10.28991/ESJ-2025-09-02-012.

T. B. Mashigo, W. Dinath, and S. Khumalo, “Chatbot evaluation for effectiveness in customer query resolution,” S. Afr. J. Inf. Manag., vol. 27, no. 1, Sep. 2025, doi: 10.4102/sajim.v27i1.1963.

Downloads

Citation Tools

How to Cite
Ahlam, N., & Efgivia, M. (2026). Evaluating FAQ Based Generative AI Chatbot Performance in Islamic Banking Customer Service. International Journal Of Mathematics And Computer Research, 14(07), 6653-6659. https://doi.org/10.47191/ijmcr/v14i7.04