Integrating Github with Google Colab
Reading, executing, saving Python script from Google Colab
Various Deep Neural Network algorithms require a large amount of data and because of high complexity, they are resource-intensive. So when performed on the local system they are very slow and takes a lot of time. By parallelizing these models can be trained faster. We achieve this by using a GPU. But for many bottlenecks is access to such computational resources and in case we have easy access to them with various cloud services often they are priced high. Google has a perfect solution for anyone who wants to use such advanced hardware which is free to use.
Often it is easy to get started using Colab at https://colab.research.google.com which is fine when we are building and Adhoc standalone python program. Otherwise, we need to work a little extra. Here are some of the way to get most from the same by integrating it with Github & Google Drive
Firstly we have to get scripts ready. For scripting, you can use an IDE such as PyCharm, Visual Studio Code, or Jupyter Notebook. After the scripts are ready, we can push the scripts to the GitHub repository.
Working with Notebooks Directly from GitHub
Google Colab allows to load a publicly accessible notebook directly from Github Repo with below steps:
1. Navigate to https://colab.research.google.com/github/.
2. Provide Github link at “Enter a GitHub URL or search by organization or user” option.
3. Select the repo under the “Repository” option followed by the GitHub target git branch from listed under “Branch”.
4. Colab will now list all notebook files on the repo & these can be selected to open on Colab.
5. Further, we can execute modify the notebook on Colab. In case we want to save the modified notebook we can select File->Save a Copy In Github.
6. With resulting prompts by authenticating, selecting the branch & with commit message Colab will update the modified copy at Github.
Although this out of box solution from Google is simple but it may not be always useful as incase we need custom python imports from our repository files
Working with Python Source file at GitHub via Google Drive
Here we can use Google Drive to download the Github source code and work on them. In a brief we need to open a notebook would mount Google drive, clone from Github, run python file with local import & drive available data, and modify & save back. Detailed steps as below.
It’s better to add Google Drive mounting & GitHub notebook to be part of the script files on GitHub. So we open a notebook directly from Colab and make changes and run our target script.
1. Mounting of Drive can be done as below. Running this code with follow present a prompt. By following the link to allow google access to drive and other needed service & with entering the verification code on the notebook prompt our drive would be ready to be used from Colab.
import os
from google.colab import drive
ROOT = “/content/drive”
print(f’root folder: {ROOT}’)
drive.mount(ROOT)
Drive mount status can be verified by checking file explorer with Google Colab as seen left.
2. Provide information about the repo to be cloned as below.
workspace_rel_path = ‘My Drive/github’
git_repo = “colab_tutorial”
git_user_name = “Name”
git_token = “GIT HUB ACCESS TOKEN” #Change it to actual token found at Github->Settings->Developer settings->Generate new token
git_email = “Git VisibleEmail@gmail.com”
git_name = “Git Visible Name”workspace_abs_path = os.path.join(ROOT, workspace_rel_path)
print(f”workspace_abs_path: {workspace_abs_path}”)
if not os.path.exists(workspace_abs_path):
os.makedirs(workspace_abs_path)
pass
git_repo_url = f”@github.com/{git_user_name}/{git_repo}.git”>https://{git_token}@github.com/{git_user_name}/{git_repo}.git"
print(f”git_repo_url: {git_repo_url}”)% cd “{workspace_abs_path}”
! git clone “{git_repo_url}”
3. We can invoke a python program with local python import from the notebook cell by invoking.
! python file_name.py
4. Similarly, we can invoke a python program with reference to the data folder inside Drive.
! python file_name.py file_path
5. In case we need to update or modify any file it can be opened from the file explorer section inside Colab. And once changes are finalized it cab be add, commit and push to remote Github repo.
In case you wish to walk through the above steps then Complete walkthrough codebase is available at https://github.com/srikant86panda/colab_tutorial
Happy training Colab! Let me know your feedback in the comments.
