s3fs 설정중 fuse-2.8.4.tar.gz 다운로드경로 (s3fs 설치)

CN AWS 를 이용하면서, 많은 부분에 있어서 문제를 확인하였습니다.

우선적으로 현시간적으로는 CN AWS 는 s3fs 가 지원이 되지않습니다. ㅠ_ㅠ

현재 fluentd(비둘기) 를 통해서 EC2 와 S3를 연결하는 작업을 하고있으나, 현재 진행중입니다.

완료되면 매뉴얼 작성하여 공유해 드리겠습니다.

아래의 URL 은 Fuse 2.8.4 버전 다운로드 경로 입니다.

저도 잊지않기위해서 기재해 놓았습니다. 필요하신분은 아래의 경로로 서버및 자체다운로드 이후에 설치 하시기 바랍니다.

http://pkgs.fedoraproject.org/lookaside/extras/fuse/fuse-2.8.4.tar.gz/a26ec54e410bb826a387947b10c0fd2c/fuse-2.8.4.tar.gz

설치 방법도 같이 기재해 드려야할것 같아서 괜찮은 사이트에서 가져온 내용 입니다.

Mount an Amazon S3 Bucket to a Local Linux File System

by Jeff Hunter, Sr. Database Administrator

Contents

• Introduction
• Requirements
• Mount S3 Bucket
• s3fs Limitations
• About the Author

Introduction

This guide describes how to mount an Amazon S3 bucket as a virtual drive to a local file system on Linux by using s3fs and FUSE.

The example provided in this guide will mount an S3 bucket named idevelopment-software to /mnt/s3/idevelopment-software on an EC2 instance running CentOS 6.3 (x86_64).

Requirements

Perform the following prerequisites on the target machine as root.

1. Install the following prerequisite packages.

   # yum -y install gcc # yum -y install libstdc++-devel # yum -y install gcc-c++ # yum -y install curl-devel # yum -y install libxml2-devel # yum -y install openssl-devel # yum -y install mailcap # yum -y install make # yum -y install svn

2. Download, compile, and install the FUSE (Filesystem in Userspace) module.

   Note: s3fs requires FUSE version 2.8.4 when running CentOS 5. Using a version of FUSE higher than version 2.8.4 on CentOS 5 will result in the following error when attempting to mount the S3 bucket:

   # s3fs idevelopment-software /mnt/s3/idevelopment-software /bin/mount: unrecognized option `--no-canonicalize'

   CentOS 6 works with the latest FUSE version which at the time of this writing is release 2.9.1.

   CentOS 6

   # yum remove fuse fuse* fuse-devel # cd /usr/local/src # wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.9.1/fuse-2.9.1.tar.gz # tar -xzvf fuse-2.9.1.tar.gz # cd fuse-2.9.1 # ./configure --prefix=/usr # make # make install # ldconfig # export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ # pkg-config --modversion fuse 2.9.1 # ldconfig -v | grep fuse libfuse.so.2 -> libfuse.so.2.9.1

   CentOS 5

   # yum remove fuse fuse* fuse-devel # cd /usr/local/src # wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz # tar -xzvf fuse-2.8.4.tar.gz # cd fuse-2.8.4 # ./configure --prefix=/usr # make # make install # ldconfig # export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ # pkg-config --modversion fuse 2.8.4 # ldconfig -v | grep fuse libfuse.so.2 -> libfuse.so.2.8.4

   Manually load the FUSE module for the first time.

   # modprobe fuse

3. Download, compile, and install the latest version of s3fs.

   # cd /usr/local/src # svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs # cd s3fs # autoreconf --install # ./configure --prefix=/usr # make # make install # ldconfig

   Verify the s3fs installation.

   # s3fs --version Amazon Simple Storage Service File System 1.61 Copyright (C) 2010 Randy Rizun License GPL2: GNU GPL version 2 This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

4. Create an s3fs password file for storing your AWS Access Key ID and Secret Access Key.

   The default location for the s3fs password file can be created:

   • using a .passwd-s3fs file in the users home directory (i.e. ~/.passwd-s3fs)
   • using the system-wide /etc/passwd-s3fs file

   The s3fs password file has the following format. Use this format if you have only one set of credentials. Specify your AWS Access Key ID and Secret Access Key separated by a colon with no spaces between.

   AccessKeyId:SecretAccessKey

   If have more than one set of credentials, then you can have default credentials as specified above, but the format includes the bucket name and the default S3 credentials, all separated by a colon with no spaces between.

   my-s3-bucket:AccessKeyId:SecretAccessKey

   where my-s3-bucket is the S3 bucket name to mount, AccessKeyId is the Access Key ID, and SecretAccessKey is the Secret Access Key.

   The AWS Access Key ID and Secret Access Key serve the purpose of ID and Password to access Amazon S3. Navigate to Security Credentials, click on the Access Keys tab under Access Credentials to create or view your Access Key ID and Secret Access Key.

   Note: The AWS Access Key ID and Secret Access Key displayed below are not my actual S3 security credentials. It is nothing more than a set of random characters and is only being shown to display example output. It is not valid. However, if you have a lot of time on your hands, give it a try!

   # touch /root/.passwd-s3fs # chmod 600 /root/.passwd-s3fs # vi /root/.passwd-s3fs idevelopment-software:B9W1IF9E1H0CQ17V1LQW:U1DfhLaQ1bhQ1B7/DcNai2C9qmaI8wpqDJnn3Qas

Mount S3 Bucket

Create a mount point directory for the S3 bucket and use the s3fs command to mount it.

1. Create a directory for mounting the S3 bucket.

   # mkdir -p /mnt/s3/idevelopment-software

2. Mount the S3 bucket using the correct permissions and any other options.

   # s3fs -o allow_other -o use_cache=/tmp idevelopment-software /mnt/s3/idevelopment-software

   • The -o allow_other option allows all other users on the server read / write access to the mounted directory. Obviously, this can be a huge security hole.
   • The -o use_cache=/tmp option specifies the location where to store locally cached folders and files. This enables local file caching which minimizes downloads.
     Note: make certain you have plenty of space dedicated to the cache directory before considering this option. Depending on the size of the files being copied, this directory can fill up very quickly!
   • The -o default_acl=public-read-write option will set anything you write to the bucket as publicly viewable and writable (by default, it's set to private).
   • The -o default_acl=public-read option will set anything you write to the bucket as publicly viewable (by default, it's set to private).

3. Verify the s3fs mounted file system.

   # grep s3fs /etc/mtab s3fs /mnt/s3/idevelopment-software fuse.s3fs rw,nosuid,nodev,allow_other 0 0 # df -Th /mnt/s3/idevelopment-software Filesystem Type Size Used Avail Use% Mounted on s3fs fuse.s3fs 256T 0 256T 0% /mnt/s3/idevelopment-software

4. Manually unmount the virtual drive using the umount command:

   # umount /mnt/s3/idevelopment-software

5. Automatically mount the S3 bucket when the server boots by adding an entry to /etc/fstab using the following syntax:

   s3fs#bucketname /mnt/mount_folder fuse option1,option2 0 0

   For example:

   s3fs#idevelopment-software /mnt/s3/idevelopment-software fuse allow_other,use_cache=/tmp 0 0

s3fs Limitations

• UID/GID Support

  At this time, there is no full UID/GID support. All files will be owned by root. Additionally, if you allow others to access the bucket (using the -o allow_other option), others can also remove files.

• Excessive Time-outs

  Currently s3fs can hang the CPU if you have lots of time-outs. This is not a fault of s3fs but rather libcurl. This happens when you try to copy thousands of files in 1 session, it doesn't happen when you upload hundreds of files or less.

• Moving Large Files

  Moving, renaming, or erasing files may take considerable time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the -o use_cache=<directory> mount option.

• File Size

  S3FS has a file size limit of 64GB for the current version (limited by s3fs, not Amazon).

• Directory Support

  Prior to configuring s3fs on my test system, I used the AWS Management Console to create several directories and to upload some files to the target S3 bucket I intended to mount (idevelopment-software). After mounting the bucket to the local file system, I was unable to see any of the directories. I could see the files I uploaded to the root directory of the bucket, but was unable to see any of the directories. I went back to the AWS Management Console and even to S3Fox and verified that the directories did indeed exist.

  It turns out that S3 does not have a native concept of a folder (i.e. directory). It is up to each S3 client tool (AWS Console, S3Fox, s3cmd, s3fs) to implement their own strategy for handling folders. Without a common specification in place for storing folders, certain S3 client tools will build directory structures that are not compatible with one another. In my example, the folders I created in the AWS Management Console were not compatible with s3fs.

  Unfortunately, the only solution for now is to adopt a single S3 tool exclusively to modify the contents of a bucket. For the S3 buckets I intend to mount, I will only be using s3fs against the contents of those buckets.

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.

그밖에 참조 URL : (아래와 같습니다. 한글 설명은 호스트웨 / wp.* / egloos 보시면 됩니다.

http://system-admins.ru/montiruem-amazon-s3-kak-fajlovuyu-sistemu/

http://www.idevelopment.info/data/AWS/AWS_Tips/AWS_Management/AWS_20.shtml

http://faq.hostway.co.kr/AWS_FAQ/7243

https://wp.yoonc.co.kr/2012/07/ec2%EC%97%90-s3%EB%A5%BC-mount-%ED%95%98%EB%8A%94%EB%B2%95on-ubuntu/

https://github.com/s3fs-fuse/s3fs-fuse/issues/105

http://rricketts.com/how-to-mount-an-amazon-s3-bucket-on-centosrhel-using-s3fs/

http://egloos.zum.com/hanvic/v/4051529

https://www.noodles.net.nz/2013/02/27/s3fsfuse-on-centosrhel/