Taxonomy of Prompt Injection Attacks and Analysis of Defense Mechanisms in Large Language Model-Based Chatbots
Article Sidebar
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Main Article Content
Abstract
Large Language Models have become critical infrastructure across healthcare, finance, and software engineering, yet their instruction-following design creates a structural class of security vulnerabilities known as prompt injection. This paper presents a comprehensive taxonomy of prompt injection attacks targeting LLM-based chatbots, organizes the threat landscape across seven categories covering both direct and indirect injection vectors, and evaluates defense mechanisms at three operational layers. The theoretical framework is validated through an empirical benchmark (PI-Bench) testing 210 prompt injection payloads across seven attack categories against five open-weight LLM models in a controlled local environment. Empirical findings demonstrate attack success rates ranging from 10.5% (LLaMA 3.1 8B) to 83.8% (Mistral 7B), confirming that alignment training quality is the primary determinant of injection resistance rather than model size. Across all models, Jailbreak/Roleplay attacks proved most effective at 34.7% aggregate ASR, while Gradient-Based attacks were least effective at 22.7%. The literature review documents attack success rates reaching 87.2% against state-of-the-art proprietary models and an 86.1% success rate for indirect injection against production applications. No single defensive measure provides adequate coverage, and a defense-in-depth architecture is necessary for robust protection.
Article Details
References
Chen, Y., Li, H., Zheng, Z., Song, Y., Wu, D., & Hooi, B. (2024). Defense against prompt injection attack by leveraging attack techniques. arXiv preprint arXiv:2401.17983. DOI: https://doi.org/10.18653/v1/2025.acl-long.897
Chowdhury, A. G., Islam, M. M., Kumar, V., Shezan, F. H., Jain, V., & Chadha, A. (2024). Breaking down the defenses: A comparative survey of attacks on large language models. arXiv:2403.04786.
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023). MASTERKEY: Automated jailbreaking of large language model chatbots. arXiv:2307.08715. DOI: https://doi.org/10.14722/ndss.2024.24188
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., Bensalem, S., & Huang, X. (2024). Safeguarding large language models: A survey. arXiv:2406.02622. DOI: https://doi.org/10.1007/s10462-025-11389-2
Dong, Y., et al. (2025). Attacks, defenses and evaluations for LLM conversation safety: A survey. Proceedings of AAAI 2025.
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. Proceedings of AISec, 79–90. DOI: https://doi.org/10.1145/3605764.3623985
Jones, N., Whaiduzzaman, M., Jan, T., Adel, A., Alazab, A., & Alkreisat, A. (2025). A CIA triad-based taxonomy of prompt attacks on large language models. arXiv preprint. DOI: https://doi.org/10.3390/fi17030113
Liu, X., Xu, N., Chen, M., & Xiao, C. (2024). AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. Proceedings of ICLR 2024. arXiv:2310.04451.
Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., Zhang, L. Y., & Liu, Y. (2023). Prompt injection attack against LLM-integrated applications. arXiv:2306.05499.
Pathade, C. (2025). Red teaming the mind of the machine: A systematic evaluation of prompt injection and jailbreak vulnerabilities in LLMs. arXiv:2505.04806.
Perez, F., & Ribeiro, I. (2022). Ignore previous prompt: Attack techniques for language models. arXiv:2211.09527.
Phute, M., Helbling, A., Hull, M., Peng, S., Szyller, S., Cornelius, C., & Chau, D. H. (2023). LLM self defense: By self examination, LLMs know they are being tricked. arXiv:2308.07308.
Rossi, S., Marianne Michela, A., Mukhamala, R. R., & Thatcher, J. B. (2024). An early categorization of prompt injection attacks on large language models. arXiv:2402.00898.
Schulhoff, S., Pinto, J., Khan, A., Bouchard, L.-F., Si, C., Anati, S., Tagliabue, V., Kost, A. L., Carnahan, C., & Boyd-Graber, J. (2023). Ignore this title and HackAPrompt: Exposing systemic vulnerabilities of LLMs through a global scale prompt hacking competition. arXiv:2311.16119. DOI: https://doi.org/10.18653/v1/2023.emnlp-main.302
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023). "Do Anything Now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv:2308.03825. DOI: https://doi.org/10.1145/3658644.3670388
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
Wei, A., Haghtalab, N., & Steinhardt, J. (2023). Jailbroken: How does LLM safety training fail? arXiv:2307.02483. DOI: https://doi.org/10.52202/075280-3508
Yi, S., Liu, Y., Sun, Z., Cong, T., He, X., Song, J., Xu, K., & Li, Q. (2024). Jailbreak attacks and defenses against large language models: A survey. arXiv:2407.04295.
Zhang, W., Kong, X., & Dewitt, C. (2024). A study on prompt injection attack against LLM-integrated mobile robotic systems. arXiv:2408.03515. DOI: https://doi.org/10.1109/ISSREW63542.2024.00103
Zhao, Y., Wang, M., Feng, S., Yang, X., Wang, D., & Zhang, Y. (2024). Defending large language models against jailbreak attacks via in-decoding safety-awareness probing. arXiv:2601.10543. DOI: https://doi.org/10.18653/v1/2024.findings-emnlp.293