Welcome to Scholar Publishing Group

International Journal of Big Data Intelligent Technology, 2026, 7(3); doi: 10.38007/IJBDIT.2026.070305.

Data Warehouse Building Based on ETL Processes and Applications for Visual Analysis

Author(s)

Yun Lou

Corresponding Author:
Yun Lou
Affiliation(s)

Michigan State University, East Lansing, MI, 48824, US

Abstract

Addressing the issues of coarse-grained refresh scope and difficulty in consistently translating logical pruning into operational benefits in continuously updated data warehouses, this study constructs a star-structured data warehouse for three types of visual analytics workloads: time trends, regional revenue, and payment structure. A selective refresh method driven by a two-layer field-partition dependency is proposed. This method comprehensively considers change operations, changed fields, and pre- and post-change groupings to determine the necessary refresh keys for the three types of aggregations on the partition incremental baseline. This study conducts deterministic change replay based on NYC TLC Yellow Taxi open data and evaluates the refresh method from four aspects: refresh scope, runtime, stage cost, and analytical consistency. Results show that in low-coverage local corrections, fine-grained dependencies can eliminate irrelevant maintenance; however, as the impact scope expands, scan pruning and write granularity weaken wall-clock benefits. Under high-coverage updates, partition-level incremental updates have completed major pruning. This study demonstrates that the granularity of data warehouse maintenance should be selected in conjunction with the change structure and downstream analytical workloads.

Keywords

Data warehouse; incremental ETL; selective refresh; impact coverage; visual analytics

Cite This Paper

Yun Lou. Persistent Memory Mechanisms and Storage Architecture Design for Agent-based Information Systems. International Journal of Big Data Intelligent Technology (2026), Vol. 7, Issue 3: 42-52. https://doi.org/10.38007/IJBDIT.2026.070305.

References

[1] Walha A, Ghozzi F, Gargouri F. Data integration from traditional to big data: main features and comparisons of ETL approaches[J]. The Journal of Supercomputing, 2024, 80(19): 26687-26725. DOI: 10.1007/s11227-024-06413-1.

[2] Khan B, Jan S, Khan W, Chughtai M I. An overview of ETL techniques, tools, processes and evaluations in data warehousing[J]. Journal on Big Data, 2024, 6(1): 1-20. DOI: 10.32604/jbd.2023.046223.

[3] Setiawan D, Isa S M. Improving Data Warehouse ETL Efficiency with Partitioning and Parallelism in Financial Services[J]. Jurnal Sains dan Teknologi, 2025, 14(3): 590-599. DOI: 10.23887/jst-undiksha.v14i3.103391.

[4] Zhang, Z. (2026). Research on Service-oriented Packaging of AI Systems and Data-driven Decision Support for Real-world Business Processes. Engineering Advances, 6(3).

[5] Zhang, Y. (2025, January). Research on the Application and Optimization of Multi Dimensional Data Model Based on Kylin in Enterprise Technology Management. In International Conference on Innovative Computing (pp. 234-241). Singapore: Springer Nature Singapore.

[6] Fei, W., Xue, W., & Wang, B. (2025). Research on the innovative path of rural education teaching methods from the perspective of PBE and PBL integration. Int. Acad. J. Humanit. Soc. Sci.

[7] Zhang, Z. (2026). User Behavior Anomaly Identification Integrating Telemetry Data and Its Application in Consumer Finance Risk Control.

[8] Zhang, Y. (2026). Exploration of Enterprise Big Data Microservice Architecture Based on Domain-Driven Design (DDD). Procedia Computer Science, 282, 1994-2003.

[9] Cui, N. (2024, December). Research on Adaptive Urban Traffic Green Wave Control Algorithm and Safety Based on Reinforcement Learning DQN-COP. In International Conference on Big Data Analytics for Cyber-Physical System in Smart City (pp. 379-387). Singapore: Springer Nature Singapore.

[10] Liu, S. (2026). Automatic Generation Method and Implementation of Web Front-End Components Based on Generative AI.

[11] Wendan Xue, Wen Fei. An Analytical Study on the Differences and Determinants of Basic Education Resource Allocation in Rural China and the United States. International Journal of Educational Research and Development, 1(3), 13-17.

[12] Jingzhi Yin. (2026). Constructing a CMBS Default Risk Sentiment Index Using Financial Text Embeddings and Evaluating Its Predictive Effectiveness. Engineering Advances, 6(1), 50-54.

[13] Wang, Z. (2026). Research on Operation Data Analysis and Efficiency Improvement of the New Energy Industry Based on Machine Learning. Procedia Computer Science, 285, 605-612.

[14] Zhang, Z. (2026). User Behavior Device Reliability Joint Modeling and Intelligent Risk Control Practice In Consumer Technology.

[15] Wang, B. (2025, November). Research on Performance Improvement of Personalized Recommendation System for E-commerce Platform Based on Big Data and Collaborative Filtering Algorithm. In International Conference on Computational Technologies for Research in Data Analytics (pp. 36-46). Cham: Springer Nature Switzerland.

[16] Liu, S. (2026). Design and Implementation of Retrieval Enhancement Generation Method for Enterprise Knowledge Base.

[17] Fei, W., & Qian, J. (2026). Application and Exploration of Reverse Instructional Design in the Teaching of the Primary School Physical Education" Walking and Playing" Unit. Journal of Sociology and Education, 2(2).

[18] Boli Wang, Wen Fei. Innovations and practices in the classroom for rural aesthetic education in the perspective of school integration. International Journal of Educational Research and Development, 3(1), 36-41.

[19] Hou, Y. (2026, March). Adaptive Upgrading of Computing Power Platforms for Large Model Workloads. In International Conference on Computing and Communication Systems for Industrial Applications (pp. 429-437). Cham: Springer Nature Switzerland.

[20] Xin, H. (2026). Research on Distributed Data Pipelines and Event-Driven Architecture for Freight Visibility.

[21] Jin, C. (2026). Application of Data Mining Technology in Predicting User Churn and Retention Strategies in E-Commerce.

[22] Wen Fei, Jianrong Qian. Research on Value Guidance and Practical Pathways for Constructing the Professional Culture of Teachers. International Journal of Educational Research and Development, 2(4), 27-30.

[23] Qian, X. (2026, July). Application of the Combination Model of Prophet and LSTM in Multi-Level Inventory Demand Prediction for Retail Supply Chain of CPG-Consumer Packaged Foods. In 2026 5th International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE) (pp. 1-7). IEEE.

[24] Jin, C. (2026). Construction and Empirical Study of Credit Risk Assessment Index System for Small and Micro Enterprises in Commercial Banks.