Institutional Repository of Computer Application Research Laboratory
| Study on tiered storage algorithm based on heat correlation of astronomical data | |
Ye, Xin-Chen1,2,3 ; Zhang, Hai-Long1,2,3,4; Wang, Jie1,3; Zhang, Ya-Zhou1,2; Du, Xu1,2; Wu, Han1,2
| |
| 2024-03-14 | |
| Source Publication | FRONTIERS IN ASTRONOMY AND SPACE SCIENCES
 |
| ISSN | 2296-987X |
| Volume | 11Pages:1371249 |
| Contribution Rank | 1 |
| Abstract | With the surge in astronomical data volume, modern astronomical research faces significant challenges in data storage, processing, and access. The I/O bottleneck issue in astronomical data processing is particularly prominent, limiting the efficiency of data processing. To address this issue, this paper proposes a tiered storage algorithm based on the access characteristics of astronomical data. The C4.5 decision tree algorithm is employed as the foundation to implement an astronomical data access correlation algorithm. Additionally, a data copy migration strategy is designed based on tiered storage technology to achieve efficient data access. Preprocessing tests were conducted on 418GB NSRT (Nanshan Radio Telescope) formaldehyde spectral line data, showcasing that tiered storage can potentially reduce data processing time by up to 38.15%. Similarly, utilizing 802.2 GB data from FAST (Five-hundred-meter Aperture Spherical radio Telescope) observations for pulsar search data processing tests, the tiered storage approach demonstrated a maximum reduction of 29.00% in data processing time. In concurrent testing of data processing workflows, the proposed astronomical data heat correlation algorithm in this paper achieved an average reduction of 17.78% in data processing time compared to centralized storage. Furthermore, in comparison to traditional heat algorithms, it reduced data processing time by 5.15%. The effectiveness of the proposed algorithm is positively correlated with the associativity between the algorithm and the processed data. The tiered storage algorithm based on the characteristics of astronomical data proposed in this paper is poised to provide algorithmic references for large-scale data processing in the field of astronomy in the future. |
| Keyword | tiered strorage astronomical data processing load prediction decision tree high performance computing |
| DOI | 10.3389/fspas.2024.1371249 |
| Indexed By | SCI |
| Language | 英语 |
| Funding Project | National Key R&D Program of China[2021YFC2203502] ; National Key R&D Program of China[2022YFF0711502] ; National Natural Science Foundation of China (NSFC)[12173077] ; National Natural Science Foundation of China (NSFC)[12003062] ; Tianshan Innovation Team Plan of Xinjiang Uygur Autonomous Region[2022D14020] ; Tianshan Talent Project of Xinjiang Uygur Autonomous Region[2022TSYCCX0095] ; Scientific Instrument Developing Project of the Chinese Academy of Sciences[PTYQ2022YZZD01] ; China National Astronomical Data Center (NADC) ; Operation, Maintenance and Upgrading Fund for Astronomical Telescopes and Facility Instruments ; Ministry of Finance of China ; Natural Science Foundation of Xinjiang Uygur Autonomous Region[2022D01A360] |
| WOS Research Area | Astronomy & Astrophysics |
| WOS Subject | Astronomy & Astrophysics |
| WOS ID | WOS:001191944200001 |
| Publisher | FRONTIERS MEDIA SA |
| Funding Organization | National Key R&D Program of China ; National Natural Science Foundation of China (NSFC) ; Tianshan Innovation Team Plan of Xinjiang Uygur Autonomous Region ; Tianshan Talent Project of Xinjiang Uygur Autonomous Region ; Scientific Instrument Developing Project of the Chinese Academy of Sciences ; China National Astronomical Data Center (NADC) ; Operation, Maintenance and Upgrading Fund for Astronomical Telescopes and Facility Instruments ; Ministry of Finance of China ; Natural Science Foundation of Xinjiang Uygur Autonomous Region |
| Citation statistics | |
| Document Type | 期刊论文 |
| Identifier | http://ir.xao.ac.cn/handle/45760611-7/5959 |
| Collection | 计算机技术应用研究室 科研仪器设备产出_利用南山26米射电望远镜（NSRT）观测数据的文章 科研仪器设备产出_我台利用FAST观测数据文章 |
| Corresponding Author | Zhang, Hai-Long |
| Affiliation | 1.Chinese Acad Sci, Xinjiang Astron Observ, Urumqi, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Natl Astron Data Ctr, Beijing, Peoples R China 4.Chinese Acad Sci, Key Lab Radio Astron, Nanjing, Peoples R China |
| First Author Affilication | Xinjiang Astronomical Observatory, Chinese Academy of Sciences |
| Corresponding Author Affilication | Xinjiang Astronomical Observatory, Chinese Academy of Sciences |
| Recommended Citation GB/T 7714 | Ye, Xin-Chen,Zhang, Hai-Long,Wang, Jie,et al. Study on tiered storage algorithm based on heat correlation of astronomical data[J]. FRONTIERS IN ASTRONOMY AND SPACE SCIENCES,2024,11:1371249. |
| APA | Ye, Xin-Chen,Zhang, Hai-Long,Wang, Jie,Zhang, Ya-Zhou,Du, Xu,&Wu, Han.(2024).Study on tiered storage algorithm based on heat correlation of astronomical data.FRONTIERS IN ASTRONOMY AND SPACE SCIENCES,11,1371249. |
| MLA | Ye, Xin-Chen,et al."Study on tiered storage algorithm based on heat correlation of astronomical data".FRONTIERS IN ASTRONOMY AND SPACE SCIENCES 11(2024):1371249. |
| Files in This Item: | ||||||
| File Name/Size | DocType | Version | Access | License | ||
| Ye-2024-Study on tie（28131KB） | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | View Application Full Text | |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment