├── Dockerfile ├── README.md ├── etc ├── sftp.d │ └── gcs-mounts.sh └── sftp │ └── users.conf ├── secrets └── gcloud-key.json └── sftp.yaml /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM atmoz/sftp 2 | MAINTAINER Michael Ghen 3 | 4 | # Install FUSE so we can mount GCS buckets 5 | # Ref: https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md 6 | RUN apt-get update 7 | RUN apt-get install -y curl lsb gnupg wget 8 | RUN echo "deb http://packages.cloud.google.com/apt gcsfuse-stretch main" | tee /etc/apt/sources.list.d/gcsfuse.list 9 | RUN cat /etc/apt/sources.list.d/gcsfuse.list 10 | RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - 11 | RUN apt-get update 12 | RUN apt-get install -y gcsfuse 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GCS SFTP Server 2 | SFTP Server designed to store data in Google Cloud Storage (GCS) Buckets 3 | 4 | This is based upon [atmoz/sftp](https://github.com/atmoz/sftp) project. 5 | 6 | # Dockerfile 7 | We need to setup an image (based on atomz/sftp) so that we can mount to Google Cloud Storage. That means just installing [gcsfuse](https://github.com/GoogleCloudPlatform/gcsfuse/tree/master/docs). 8 | 9 | Find and build your own image using the `Dockerfile` provided. 10 | 11 | 12 | # Mounting Buckets 13 | We use gcsfuse `--uid`, `--gid`, and `--only-dir` arguments to mount each SFTP users home directory to a single bucket. Inside the bucket, we create a directory for each user manually. (Not sure if using `--only-dir` will work unless the directory already exists) 14 | 15 | Sample Bucket Directory Structure: 16 | ``` 17 | bucket-name 18 | - /user1 19 | - /user2 20 | ``` 21 | 22 | The mounting is done in `etc/sftp.d/mount_user_directories.sh`. When deploying to Kubernetes, this script gets executed as a `postStart` command. 23 | 24 | ## Access Control for GCS Bucket 25 | We just need to ensure your GKE cluster is created with the OAuth scope https://www.googleapis.com/auth/devstorage.read_write, and everything else will be handled automatically. Alternatively, we can mount a file in Service Account JSON key. 26 | 27 | # Setup Instructions 28 | ## Dependancies 29 | For testing, you will need to have Minikube and Docker installed. 30 | 31 | For deployment, you will need to have the gcloud SDK. 32 | 33 | ## Configuration 34 | You can configure SFTP user accounts by adjusting what's in `etc/sftp/users.conf` and `etc/sftp.d/mount_user_directories.sh`. 35 | 36 | When adding a new user, add a new line into `etc/sftp/users.conf`: 37 | ``` 38 | username:password:uid:gid:directory 39 | ``` 40 | Where `uid` is a number (e.g. 1003) and `gid` is a number (e.g. 1003). 41 | And then add a new line into `etc/sftp.d/mount_user_directories.sh` to monunt their `directory` to a GCS bucket: 42 | ``` 43 | runuser -l partner1 -c \ 44 | 'export GOOGLE_APPLICATION_CREDENTIALS=/credentials/gcloud-key.json && \ 45 | gcsfuse -o nonempty --only-dir username bucket /home/username/ftp' 46 | ``` 47 | This command will mount the bucket as the given user. It also does some environment variable trickery. 48 | 49 | :warning: User passwords are committed to this repo as a demo. Not the best to commit them in practice. 50 | 51 | ## Production Deployment 52 | To deploy to GKE follow these steps: 53 | 54 | ### To Do 55 | - [ ] Push docker image to dockerhub 56 | - [ ] Document production deployment instructions 57 | 58 | ## Development Setup for Testing 59 | Follow these steps to run this locally with `minikube`. 60 | 61 | ### 1. Start minikube: 62 | ``` 63 | minikube start 64 | ``` 65 | 66 | ### 2. Tell minikube to use local docker images: 67 | ``` 68 | eval $(minikube docker-env) 69 | ``` 70 | 71 | ### 3. Build a local image from the `Dockerfile`: 72 | ``` 73 | docker build --rm -t mikeghen/kube-sftp . 74 | ``` 75 | 76 | ### 4. Setup Secrets and Config Mappings 77 | You'll need to adjust files in `etc` so that it reflects the SFTP users you're planning to use. You'll also need a Service Account as well. 78 | 79 | Then, you can run these commands to put these files on the cluster as secrets: 80 | ``` 81 | kubectl create secret generic users --from-file=users.conf=./etc/sftp/users.conf 82 | kubectl create secret generic sftp-gcloud-key --from-file=gcloud-key.json=./secrets/gcloud-key.json 83 | kubectl create configmap gcs-mounts --from-file=gcs-mounts.sh=./etc/sftp.d/gcs-mounts.sh 84 | ``` 85 | * **users** - Code for maintaining users credentials for SFTP access 86 | * **sftp-cloud-key** - JSON Key for GCS Service Account 87 | * **gcs-mounts** - Code for mounting GCS bucket 88 | 89 | ### 5. Deploy the SFTP server to Kubernetes: 90 | ``` 91 | kubectl apply -f sftp.yaml 92 | ``` 93 | ### 6. Get the test IP and port: 94 | ``` 95 | minikube service sftp --url 96 | ``` 97 | This will give you the IP and NodePort port. 98 | 99 | :information_source: We use NodePort 30022 for SFTP. 100 | 101 | ### 7. Confirm you can SFTP using the usernames and password you setup in `etc/sftp*` with `sftp` utility: 102 | ``` 103 | $ sftp -P 30022 username@192.168.99.100 104 | username@192.168.99.100's password: 105 | sftp> pwd 106 | /directory 107 | ``` 108 | -------------------------------------------------------------------------------- /etc/sftp.d/gcs-mounts.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # File mounted as: /etc/sftp.d/mount_user_directories.sh 3 | 4 | runuser -l partner1 -c \ 5 | 'export GOOGLE_APPLICATION_CREDENTIALS=/credentials/gcloud-key.json && \ 6 | gcsfuse -o nonempty --only-dir user1 bucket /home/user1/ftp' 7 | 8 | runuser -l partner2 -c \ 9 | 'export GOOGLE_APPLICATION_CREDENTIALS=/credentials/gcloud-key.json && \ 10 | gcsfuse -o nonempty --only-dir user2 bucket /home/user2/ftp' 11 | -------------------------------------------------------------------------------- /etc/sftp/users.conf: -------------------------------------------------------------------------------- 1 | user1:password:1003:1003:ftp 2 | user2:password:1004:1003:ftp 3 | -------------------------------------------------------------------------------- /secrets/gcloud-key.json: -------------------------------------------------------------------------------- 1 | # Key goes here 2 | -------------------------------------------------------------------------------- /sftp.yaml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | name: sftp 5 | labels: 6 | environment: production 7 | spec: 8 | type: NodePort 9 | ports: 10 | - name: ssh 11 | port: 22 12 | targetPort: 22 13 | nodePort: 30022 14 | selector: 15 | app: sftp 16 | --- 17 | apiVersion: apps/v1beta1 18 | kind: Deployment 19 | metadata: 20 | name: sftp-deployment 21 | spec: 22 | replicas: 1 23 | template: 24 | metadata: 25 | labels: 26 | app: sftp 27 | spec: 28 | volumes: 29 | - name: users 30 | secret: 31 | secretName: users 32 | - name: sftp-gcloud-key 33 | secret: 34 | secretName: sftp-gcloud-key 35 | - name: gcs-mounts 36 | configMap: 37 | name: gcs-mounts 38 | defaultMode: 0744 39 | containers: 40 | - name: sftp 41 | image: mikeghen/kube-sftp 42 | imagePullPolicy: Never 43 | ports: 44 | - containerPort: 22 45 | volumeMounts: 46 | - mountPath: /etc/sftp 47 | name: users 48 | - mountPath: /credentials 49 | name: sftp-gcloud-key 50 | - mountPath: /etc/sftp.d/ 51 | name: gcs-mounts 52 | securityContext: 53 | privileged: true 54 | capabilities: 55 | add: 56 | - SYS_ADMIN 57 | --------------------------------------------------------------------------------