Uploading to Google Cloud Buckets

Quick route for linux command line

(An alternative to Google's documentation)

The following were useful resources for putting this together:

• https://stackoverflow.com/questions/48860586/how-to-upload-and-save-large-data-to-google-colaboratory-from-local-drive

• https://cloud.google.com/storage/docs/gsutil_install

  • https://stackoverflow.com/questions/47527868/importerror-cannot-import-name-opentype-on-new-installation

• https://console.cloud.google.com/cloud-resource-manager

Installing the Cloud tools in a virtualenv

Sadly, Google's tools appear to be Python 2.x (and discussions indicate that Google considers updating to Python 3.x as a low priority). Thus, rather than 'dirty' Fedora's native 2.7 installation. it makes sense to create a virtualenv to contain the functionality cleanly. As user you just need :

virtualenv --system-site-packages gcloud-env
. gcloud-env/bin/activate
pip install gsutil
pip install --upgrade google-auth-oauthlib

Cbeck that the installation has worked :

gsutil version -l

Authenticate/configure Google Cloud account

This will ask you to authenticate against your Google Cloud account (and save the token, and other settings, in ~/.boto):

gsutil config

After going to the web authentication page to get the required code (which needs you to identify which Google account is linked to your cloud stuff), go to the project link suggested to get the list of project ids, and select the one required.

Look at the buckets available

gsutil ls

Actually upload to the bucket

Looking at the gsutil Google documentation :

cd XYZ/
gsutil help options
gsutil -m cp -r FOLDER-TO-UPLOAD gs://MY-BUCKET-NAME/
# ...
# [35.0k/35.0k files][ 14.3 GiB/ 14.3 GiB] 100% Done  12.0 MiB/s ETA 00:00:00
# Operation completed over 35.0k objects/14.3 GiB.

All done.