Portal de Periódicos da CAPES

Massive data transfer between supercomputers and grid clusters using SCP controlled by FTS

2011; IOP Publishing; Volume: 331; Issue: 4 Linguagem: Inglês

10.1088/1742-6596/331/4/042041

1742-6596

P. Tallada-Crespí, Gerard Bernabeu, Marc Caubet, M. Delfino, Francisco Javier Jiménez Martínez, C. Neissner, Maria A. Rodriguez, N. Tonello,

Distributed systems and fault tolerance

Data transfers between Supercomputing centers and Grid sites may be complicated since, in general, Supercomputing centers are not completely incorporated in the Grid environment. In particular, the Port d'Informació Científica (PIC) had to transfer data of cosmological simulations from the Barcelona Supercomputing Center (BSC) to Grid storage resources in order to post-process those data at PIC. The data connection is limited to the Secure Shell (SSH) protocol using the scp command. Besides a limitof bandwidth, we had to deal with a lack of automation and reliability of the transfers. PIC managed to control those scp transfers through gLite's File Transfer Service (FTS). Therefore, the discs containing the data at the BSC are mounted via SSH to a local Storage Resource Management (SRM) server at PIC. Once established the access to the data through the SRM server, the data can be transfered to any other SRM server, including the storage system at PIC, using grid protocols. Although we cannot improve the bandwidth, we reach a level of automation and reliability comparable to transfers from the Tier-0 to Tier-1 sites of the Large Hadron Collider (LHC) experiments.