Welcome to Scholar Publishing Group

International Journal of Multimedia Computing, 2026, 7(2); doi: 10.38007/IJMC.2026.070206.

Research on Layered Idempotent Processing Mechanism for Kafka Event-Driven Payment Transactions with High-Concurrency Retry and Fault Replay

Author(s)

Kaiyu Liao

Corresponding Author:
Kaiyu Liao
Affiliation(s)

School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, 97331, Oregon, USA

Abstract

To solve the problem of duplicate deductions caused by network timeouts and producer retries in the Kafka event-driven payment chain, this paper proposes a layered idempotent processing mechanism that covers the access, messaging, consumption, state and reconciliation layers. The method establishes a stable identity using business idempo-tent keys and request digests, reduces the scope of duplication in logs via Kafka producer idempotency and transactions, ensures that external side effects occur only once through database unique constraints, conditional state transitions, and transactional outbox, and manages uncertain states by observable reconciliation closed loops. Using the Reserve Bank of India's 2024 payment statistics, this paper shows a high-concurrency environment and performs five repeated failure replay experiments in an SQLite WAL persistent environment. Based on the results, in a scenario with a 10% replay ratio and mid-restart, pure local caching still produces 50% duplicate side effects; database unique constraints and the hybrid scheme both reduce this to 0%, and the hybrid scheme achieves an average throughput of 1651.0 events/s. This study shows that Kafka's exactly-once semantics can-not substitute for business database idempotency, and financial-grade reliability should be achieved via cross-layer invariants.

Keywords

Kafka; event-driven architecture; payment transactions; idempotency handling; transactional outbox; fault replay

Cite This Paper

Kaiyu Liao, Research on Layered Idempotent Processing Mechanism for Kafka Event-Driven Payment Transactions with High-Concurrency Retry and Fault Replay. International Journal of Multimedia Computing (2026), Vol. 7, Issue 2: 48-57. https://doi.org/10.38007/IJMC.2026.070206.

References

[1] Fragkoulis M, Carbone P, Kalavri V, Katsifodimos A. A survey on the evolution of stream pro-cessing systems. The VLDB Journal, 2024, 33: 507–541.

[2] Motheram A, Buteau S. Supply side drivers of digital payment adoption in India. Discover Sus-tainability, 2026, 7: Article 560. https://link.springer.com/article/10.1007/s43621-026-02901-x

[3] Li, Z. (2026). Design of Collaborative Architecture for Android Mobile Smart Systems Based on Edge Computing.

[4] Lin, Q. (2026). Research on Multi-source Demand Perception and Inventory-Capacity Co-optimization of Sales and Operation Planning in Discrete Manufacturing Enterprises.

[5] Hu, J. (2026). Research on Fault Tolerant Mechanism of Payment Order State Machine Based on Raft Consensus Algorithm.

[6] Li, S. (2026). Research on Identification of Cost Drivers and Profitability Analysis from the Perspective of Large Enterprise P&L Management.

[7] Gao, C. (2026). Project Management Path and Key Technologies for Energy Efficiency Opti-mization in AI Data Centers.

[8] Liu, M. (2026). Evaluation of the Effectiveness of Financial Risk Management Tools Under the New Regulatory Environment.

[9] Song, W. (2026). Research on Load Balancing Strategy for Cloud Platforms Based on Docker Containers.

[10] Zhang, L. (2026). Research on the Construction of International Tax Compliance Systems and Tax Risk Control Strategies for Multinational Enterprises.

[11] Xu, Z. (2026). Server Energy Consumption Monitoring and Adaptive Control in Large-Scale Data Center.

[12] Sun, X. (2026). Design of Infrared Thermal Imaging Fault Detection and Early Warning Sys-tem for Data Center Equipment.

[13] Shi, C. (2026). Research on Cloud Native Lightweight Container Layout Scheme for Edge Computing Scenarios.

[14] Xin, H. (2026). Freight Operation Automation and State-Machine Processing Mechanism Based on a Distributed Workflow Engine.

[15] Sun, L. (2026). Optimization of Resource Scheduling for Multi Tenant Advertising Systems.

[16] Sun, L. (2026, August). Optimization and Practice of High-Performance Advertising Data Compression Based on LZ4 and ZSTD Algorithm Fusion. In 2026 IEEE 5th World Conference on Applied Intelligence and Computing (AIC) (pp. 326-332). IEEE.

[17] Yuan, Y. (2026). High Performance Load Balancing Method for Distributed Inference of Large-Scale Language Models.

[18] Sun, L. (2026, July). Low-Latency Optimized Architecture for Distributed Stream Processing Systems for RealTime Bidding Advertising. In 2026 5th International Conference on Distribut-ed Computing and Electrical Circuits and Electronics (ICDCECE) (pp. 1-7). IEEE.

[19] Zhang, Y. (2026). A Framework for Semantic Alignment and Privacy Compliance of Large-Scale Language Models for Cross-Border Data Governance. Procedia Computer Science, 285, 1039-1047.

[20] Ma, W. (2026). Research on Reliability Enhancement of Cloud Data Platform Driven by Adap-tive Sharding and Fault Tolerance Algorithms. Procedia Computer Science, 285, 1020-1029.

[21] Wang, N. (2026, May). Research and Development of an Enterprise Operation Load Forecast-ing Model Optimization and Resource Dynamic Allocation System based on A Temporal Atten-tion Mechanism. In 2026 4th International Conference on Integrated Circuits and Communi-cation Systems (ICICACS) (pp. 1-6). IEEE.

[22] Zhu, P. (2024, December). Deep Exploration of Enhanced Integrated Model Based on EM Al-gorithm in Biostatistical Analysis and Disease Association Research. In International Confer-ence on Big Data Analytics for Cyber-Physical System in Smart City (pp. 101-108). Singapore: Springer Nature Singapore.

[23] Su, J. (2026). Android Communication Reliability Enhancement Architecture Based on Multi Link Diversity Transmission.

[24] Su, J. (2026, July). Research on Machine Learning-based Abnormal Traffic Detection and An-ti-Interference Methods for Android End-to-End Encrypted Transmission in High-Reliability Application Scenarios. In 2026 5th International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE) (pp. 1-6). IEEE.

[25] Zhang, J. (2026). Research on Financial Data Governance Framework and Data Quality Evaluation in the Big Data Environment.[26]

[26] Qian, X. (2026, July). Application of the Combination Model of Prophet and LSTM in Multi-Level Inventory Demand Prediction for Retail Supply Chain of CPG-Consumer Packaged Foods. In 2026 5th International Conference on Distributed Computing and Electrical Cir-cuits and Electronics (ICDCECE) (pp. 1-7). IEEE.

[27] Jin, C. (2026). Application of Data Mining Technology in Predicting User Churn and Reten-tion Strategies in E-Commerce.

[28] Liu, S. (2026). Automatic Generation Method and Implementation of Web Front-End Compo-nents Based on Generative AI.

[29] Zhang, Z. (2026). User Behavior Device Reliability Joint Modeling and Intelligent Risk Con-trol Practice In Consumer Technology.