How to Execute Terminal Commands in Jupyter Notebook

As a data scientist or software engineer, you’re probably familiar with Jupyter Notebook, an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. Jupyter Notebook is widely used in data science and software development because it makes it easy to explore data, prototype code, and share results with others.

But what if you need to execute terminal commands in Jupyter Notebook? For example, what if you want to install a package using pip, clone a git repository, or run a shell script? In this blog post, I’ll show you how to execute terminal commands in Jupyter Notebook using magic commands, subprocess module, and bash kernel.

Table of Contents

1. Using Magic Commands
2. Using Subprocess Module
3. Using Bash Kernel
4. Conclusion

Using Magic Commands

Jupyter Notebook has a set of magic commands that allow you to perform various tasks, including executing terminal commands. To execute a terminal command in Jupyter Notebook using magic commands, you need to prefix the command with an exclamation mark (!). For example, to list the contents of the current directory, you can use the ls command as follows:

!ls

To install a package using pip, you can use the !pip command followed by the package name. For example, to install the numpy package, you can use the following command:

!pip install numpy

To clone a git repository, you can use the !git clone command followed by the repository URL. For example, to clone the tensorflow/models repository, you can use the following command:

!git clone https://github.com/tensorflow/models.git

Using magic commands is a simple and convenient way to execute terminal commands in Jupyter Notebook. However, it has some limitations. For example, you can’t capture the output of the command or interact with the command in real-time. To overcome these limitations, you can use the subprocess module.

Using Subprocess Module

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. You can use the subprocess module to execute terminal commands in Jupyter Notebook and capture their output. Here’s an example:

import subprocess

result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
print(result.stdout.decode())

In this example, we use the subprocess.run function to execute the ls -l command and capture its output. The stdout=subprocess.PIPE argument tells subprocess to capture the standard output of the command, and the result.stdout.decode() method converts the byte string output to a string.

You can also use the subprocess module to execute long-running commands or commands that require user input. For example, you can execute a shell script using the following code:

import subprocess

result = subprocess.run(['./myscript.sh'], input=b'yes\n', encoding='utf-8', stdout=subprocess.PIPE)
print(result.stdout)

In this example, we use the subprocess.run function to execute the ./myscript.sh script and provide user input (yes\n). The encoding='utf-8' argument tells subprocess to use UTF-8 encoding for the input and output, and the result.stdout attribute contains the output of the script.

Using the subprocess module gives you more control over the execution of terminal commands in Jupyter Notebook. However, it requires more code and can be more complex than using magic commands. If you’re comfortable with shell scripting, you can also use the bash kernel.

Using Bash Kernel

The bash kernel allows you to execute shell commands directly in Jupyter Notebook cells. To use the bash kernel, you need to install it first using the following command:

!pip install bash_kernel

Then, you need to register the bash kernel using the following command:

!python -m bash_kernel.install

After installing and registering the bash kernel, you can create a new notebook and select the bash kernel from the kernel dropdown menu. Then, you can execute shell commands directly in the notebook cells, like this:

ls -l

The bash kernel provides a familiar shell environment and allows you to execute complex shell scripts, use shell variables, and pipe commands together. However, it requires a separate installation and may not be available on all systems.

Conclusion

Executing terminal commands in Jupyter Notebook is a useful skill for data scientists and software engineers. You can use magic commands, subprocess module, or bash kernel to execute terminal commands in Jupyter Notebook. Magic commands are simple and convenient, but have some limitations. Subprocess module gives you more control and flexibility, but requires more code. Bash kernel provides a familiar shell environment, but requires a separate installation. Choose the method that suits your needs and skills, and enjoy the power of Jupyter Notebook!

SHARE:

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Request a demo today to learn more.