Keywords:-

Keywords: Large Language Model, Public Service Chatbot, Query Analytics, Artificial Intelligence, Expert Based Evaluation.

Article Content:-

Abstract

Although Conversational AI significantly improves the delivery of public information, Large Language Models (LLMs) frequently face challenges in adhering strictly to Standard Operating Procedures (SOPs) and rarely leverage interaction logs for strategic analysis. To overcome these limitations, this study presents a novel police public service chatbot built on the Laravel framework, integrating the Google Gemini API with Information Retrieval (IR) methodologies and an official SOP database. The system processes user inquiries through text preprocessing, TF-IDF weighting, and Cosine Similarity to extract relevant documents, guaranteeing responses that are both contextually accurate and SOP-compliant. Furthermore, it systematically records conversation logs to analyze public needs and trends. Comprehensive evaluation via Black Box testing, IR metrics, and qualitative reviews demonstrated superior performance. The proposed system achieved an accuracy of 95.42%, outperforming rule-based (82.14%), standard TF-IDF (89.72%), and baseline Gemini (93.41%) approaches. It also received excellent satisfaction ratings from domain experts (4.83/5) and a cohort of 100 citizens (4.73/5). In conclusion, this architecture effectively combines the conversational fluency of LLMs with the strict procedural precision necessary for robust public service administration.

References:-

References

M. Salar Khan and E. Arledge, “How to Promote AI in the US Federal Government: Insights from Policy Process Frameworks,” 2023.

T. Chen, M. Gascó-Hernandez, and M. Esteve, “The Adoption and Implementation of Artificial Intelligence Chatbots in Public Organizations: Evidence from U.S. State Governments,” Am. Rev. Public Adm., vol. 54, no. 3, pp. 255–270, Apr. 2024, doi: 10.1177/02750740231200522.

D. Baur, J. Ansorg, C. E. Heyde, and A. Voelker, “Development and Evaluation of a Retrieval-Augmented Generation Chatbot for Orthopedic and Trauma Surgery Patient Education: Mixed-Methods Study,” JMIR AI, vol. 4, 2025,

doi: 10.2196/75262.

E. Halford and A. Webster, “Using chat GPT to evaluate police threats, risk and harm,” Int. J. Law Crime Justice, vol. 78, Sep. 2024,

doi: 10.1016/j.ijlcj.2024.100686.

S. Alon-Barkat and M. Busuioc, “Human-AI Interactions in Public Sector Decision Making: ‘Automation Bias’ and ‘Selective Adherence’ to Algorithmic Advice,” Journal of Public Administration Research and Theory, vol. 33, no. 1, pp. 153–169, Jan. 2023,

doi: 10.1093/jopart/muac007.

S. Mellouli, M. Janssen, and A. Ojo, “Introduction to the Issue on Artificial Intelligence in the Public Sector: Risks and Benefits of AI for Governments,” Digital Government: Research and Practice, vol. 5, no. 1, Mar. 2024, doi: 10.1145/3636550.

T. S. Gesk and M. Leyer, “Artificial intelligence in public services: When and why citizens accept its usage,” Gov. Inf. Q., vol. 39, no. 3, Jul. 2022,

doi: 10.1016/j.giq.2022.101704.

D. H. Hagos, R. Battle, and D. B. Rawat, “Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives,” Aug. 2024, [Online]. Available: http://arxiv.org/abs/2407.14962

A. Androniceanu, “Generative artificial intelligence, present and perspectives in public administration,” Administratie si Management Public, vol. 2024, no. 43, pp. 105–119, 2024,

doi: 10.24818/amp/2024.43-06.

M. Babšek, D. Ravšelj, L. Umek, and A. Aristovnik, “Artificial Intelligence Adoption in Public Administration: An Overview of Top-Cited Articles and Practical Applications,” Mar. 01, 2025, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/ai6030044.

A. F. Vatamanu and M. Tofan, “Integrating Artificial Intelligence into Public Administration: Challenges and Vulnerabilities,” Adm. Sci., vol. 15, no. 4, Apr. 2025, doi: 10.3390/admsci15040149.

S. Wang et al., “GPT, large language models (LLMs) and generative artificial intelligence (GAI) models in geospatial science: a systematic review,” 2024, Taylor and Francis Ltd.

doi: 10.1080/17538947.2024.2353122.

H. Naveed et al., “A Comprehensive Overview of Large Language Models,” Oct. 2024, [Online]. Available: http://arxiv.org/abs/2307.06435

C. Munyao and J. Ndia, “Natural Language Processing with Transformer-Based Models: A Meta-Analysis,” Journal on Artificial Intelligence, vol. 7, no. 1, pp. 329–346, 2025,

doi: 10.32604/jai.2025.069226.

G. Feretzakis, K. Papaspyridis, A. Gkoulalas-Divanis, and V. S. Verykios, “Privacy-Preserving Techniques in Generative AI and Large Language Models: A Narrative Review,” Nov. 01, 2024, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/info15110697.

E. Jo, D. A. Epstein, H. Jung, and Y. H. Kim, “Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention,” in Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, Apr. 2023.

doi: 10.1145/3544548.3581503.

S. Kumar, “Smarter Governance: How Business AI Is Shaping The Future Of Public Services,” 2023.

A. Nikiforova, A. P. Rodriguez Müller, L. Tangi, and J. Martin-Bosch, “Proactive Public Services in the Age of Artificial Intelligence: Towards Post-Bureaucratic Governance,” in Lecture Notes in Computer Science, Springer Science and Business Media Deutschland GmbH, 2026, pp. 395–413. doi: 10.1007/978-3-032-01589-1_25.

T. Yigitcanlar, R. Y. M. Li, T. Inkinen, and A. Paz, “Public Perceptions on Application Areas and Adoption Challenges of AI in Urban Services,” Emerging Science Journal, vol. 6, no. 6, pp. 1199–1236, Dec. 2022, doi: 10.28991/ESJ-2022-06-06-01.

A. A. Chinchamalatpure and S. B. Khandare, “AI-Driven Digital Transformation in Local Governance: Towards Sustainable and Paperless Operations,” 2025.

M. Das, S. Kamalanathan, and P. Alphonse, “A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset,” 2020.

L. Heryawan, D. Novitaningrum, K. R. Nastiti, and S. N. Mahmudah, “Medical Record Document Search with TF-IDF and Vector Space Model (VSM),” vol. 14, no. 3, 2024.

D. Soyusiawaty, D. Hilmawan, and R. Wolley, “Hybrid Spelling Correction and Query Expansion for Relevance Document Searching,” 2021. [Online]. Available: www.ijacsa.thesai.org

S. Dai et al., “AI-based NLP section discusses the application and effect of words models and TF based NLP section discusses the application and effect of words models and TF-IDF in NLP tasks based NLP section discusses the application and effect of bag-of,” 2024. [Online]. Available: http://creativecommons.org/liJournalofArtificialIntelligenceGeneralScienceJAIGS

https://ojs.boulibrary.com/index.php/JAIGS//ojs.boulibrary.com/index.php/JAIGShttp://

creativecommons.org/licenses/by/4.0

F. V. Krasnov, “Embedding-based retrieval: measures of threshold recall and precision to evaluate product search,” Business Informatics, vol. 18, no. 2, pp. 22–34, 2024,

doi: 10.17323/2587-814X.2024.2.22.34.

F. Diaz and A. Ferraro, “Offline Retrieval Evaluation Without Evaluation Metrics,” in SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, Inc, Jul. 2022, pp. 599–609. doi: 10.1145/3477495.3532033.

P. Palai, K. Agrawal, D. P. Mishra, and S. R. Salkuti, “Text grouping: a comprehensive guide,” IAES International Journal of Artificial Intelligence, vol. 12, no. 3, pp. 1476–1483, Sep. 2023, doi: 10.11591/ijai.v12.i3.pp1476-1483.

E. Ben George, G. Jeba Rosline, N. Balasupramanian, and N. R. Wilfred Blessing, “Efficient Document Retrieval System using Locality Sensitive Hashing Nearest Neighbor Algorithm and Weighted Jaccard Distance for Retrieving Closest Personalities,” Jurnal Kejuruteraan, vol. 36, no. 4, pp. 1535–1543, Jul. 2024, doi: 10.17576/jkukm-2024-36(4)-19.

J. Tussupov et al., “Analysis of Short Texts Using Intelligent Clustering Methods,” Algorithms, vol. 18, no. 5, May 2025, doi: 10.3390/a18050289.

F. Lechtenberg, J. Farreres, A. L. Galvan-Cara, A. Somoza-Tornos, A. Espuña, and M. Graells, “Information retrieval from scientific abstract and citation databases: A query-by-documents approach based on Monte-Carlo sampling,” Expert Syst. Appl., vol. 199, Aug. 2022,

doi: 10.1016/j.eswa.2022.116967.

Downloads

Citation Tools

How to Cite
Shalehah, N., & Efgivia, M. (2026). Development and Query Analysis of an LLM-Powered Police Chatbot with Expert Accuracy Evaluation. International Journal Of Mathematics And Computer Research, 14(07), 6643-6652. https://doi.org/10.47191/ijmcr/v14i7.03