Large-scale language model inference services are rapidly developing in online question-answering, intelligent agents and enterprise knowledge retrieval. Inference requests have many differences in the size of the input and output, when they are sent, and how much performance is expected. The above three strategies (traditional round-robin, least-connection, and single-instance batch processing) cannot achieve both high throughput and low tail latency simultaneously. Therefore, a prediction-aware load-balancing method for mixed long- and short-request loads has been developed, and in order to achieve this, request cost prediction, instance pressure normalisation, stage-aware routing, batch adaptive adjustment and service-level-effective throughput evaluation have been integrated into a unified framework. Based on real-world data from publicly available English literature over the past three years, two statistical charts and three comparison tables are generated to summarize the size of throughput, service capacity, tail latency and resource savings in different systems. Based on the above analysis, load balancing should not only consider the number of requests but also determine dynamically how much these requests will be coupled at the pre-filling, decoding, and key-value cache usage stages. The new way is convenient for individuals to learn and scalable enough for a big-scale inference node.

Yiling Yuan. High Performance Load Balancing Method for Distributed Inference of Large-Scale Language Models. International Journal of Big Data Intelligent Technology (2026), Vol. 7, Issue 2: 51-60. https://doi.org/10.38007/IJBDIT.2026.070206.

[1] Kwon W., Li Z., Zhuang S., Sheng Y., Zheng L., Yu CH, Gonzalez JE, Zhang H., Stoica I. Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles, 2023: 611–626.

[2] Patel P., Choukse E., Zhang C., Shah A., Goiri Í., Maleki S., Bianchini R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture, 2024: 118–132.

[3] Guo, X. (2025, March). Research on Blockchain-Based Financial AI Algorithm Integration Methods and Systems. In 2025 IEEE International Conference on Electronics, Energy Systems and Power Engineering (EESPE) (pp. 816-821). IEEE.

[4] Yuan, Y. (2026). Research on Memory Management and Dynamic Reasoning Methods for Intelligent Agents Based on Large Language Models.

[5] Guo, X. (2025, April). Research on Financial Trading Algorithms Based on Deep Reinforcement Learning. In 2025 IEEE 3rd International Conference on Control, Electronics and Computer Technology (ICCECT) (pp. 1711-1715). IEEE.

[6] Zhang, Z. (2026). Research on the Optimization of Payment Risk Dynamic Monitoring Models Driven by High-dimensional Behavioral Features. Advances in Computer and Communication, 7(3).

[7] Xin, H. (2026). Research on Distributed Data Cleaning and Standardized Mapping for Heterogeneous Logistics Data. Advances in Computer and Communication, 7(2).

[8] Chen, J. (2026). Research on Traffic Peak Shaving and Data Consistency Guarantee Mechanism of E-commerce Flash Sale System Under Event-driven Architecture. Engineering Advances, 6(2).

[9] Zhang, Y. (2026). Autonomous Software Release Management and Cross-service Consistency Control Mechanisms Based on Large Language Models. Advances in Computer and Communication, 7(3).

[10] Su, J. (2026). Research on Machine Learning-based Performance Prediction and High-reliability Software Evolution Mechanisms for Android Communication Systems.

[11] Wang, Z. (2026). Valuation Enhancement Pathways for Resource Stocks Driven by the Growth of Energy Storage Demand. Engineering Advances, 6(3).

[12] Ma, X. (2026). Research on Elastic Scaling and Resource Scheduling Strategies for Distributed Systems Facing Traffic Fluctuations.

[13] Zhang, J. (2026). Research on Value Stratification and Precision Marketing of Retail Customers in Banks Based on Clustering Algorithm.

[14] Li, J. (2026). Research on the Path and Efficiency of Empowering Accurate Advertising Delivery with Data Infrastructure.

[15] Zhang, Y. (2026). A Framework for LLM-Based Semantic Configuration Understanding and Automated Change Generation in Microservice Orchestration.

[16] Zhang, Y. (2026). Research on LLM-Driven Intelligent Architecture Design and Autonomy Mechanism Construction for Cloud-Native Control Planes.

[17] Wang, N. (2026). Construction and Application of Interpretable Enterprise Performance Prediction Model Based on Multi-Source Operational Data.

[18] Liu, X. (2026). Research on the Application of Artificial Intelligence in Consumer Privacy Data Protection. Procedia Computer Science, 279, 956-965.

[19] Liu, X. (2026). Construction of Consumer Data Privacy Protection System Based On Blockchain Technology. Procedia Computer Science, 282, 2082-2091.

[20] Zhang, Y. (2026). Exploration of Enterprise Big Data Microservice Architecture Based on Domain-Driven Design (DDD). Procedia Computer Science, 282, 1994-2003.

[21] Yang, Y. (2026). The Integration and Application of Ai Technology in Marketing Data Analysis and Business Decision-Making.

[22] Qian, X. (2026). Market Data–Driven Demand Forecasting and Inventory Coordination Mechanisms in Retail Supply Chains.

[23] Chi, C. (2026). Quantitative Evaluation of Tranche-Level Returns and Loss Distributions of Structured Credit Assets under Stress Scenarios.

[24] Wang, Z. (2026). Research on Data Analysis and Quality Prediction of Manufacturing Processes Based on Improved Deep Learning Algorithms. Procedia Computer Science, 281, 1328-1337.

[25] Wang, N. (2026). Research on Integrated Analysis Method of Sales and Operations Data for Management Decision Support.

[26] Han, X. (2026). Research on Adaptive Optimization Methods for Multi-Objective Manufacturing Process Parameters in Complex Assembly Processes. Procedia Computer Science, 279, 366-374.