In another post recapping Minus80 (v1.0.0), I talk about incorporating persistence to Minus80 datasets using the
unfreeze commands. These tools ensure that datasets can be traced back to specific tags and facilitate a way to pivot and compare results between different experimental runs or points in time.
For example, when you are ready to publish your data you can
freeze it, assigning it a distinguishable tag, such that later access guarantees that you are working with the exact set of files you published with. In the meantime, you are free to continue updating and analyzing the data without worrying about losing or inadvertently changing anything related to when it was published.
This is great were it not for one thing. Your dataset, along with all the frozen snapshots, are local to the computer you are working on. Until now! Version
v1.1.0 of Minus80 adds several new commands that allow you to move tagged datasets to and from the cloud.
Four simple commands have been added to support cloud powered datasets.
cloud login- access your minus80 cloud account
cloud list- list minus80 datasets available in the cloud
cloud push- move a local, tagged dataset to the cloud
cloud pull- move a tagged, cloud dataset to your local machine
Logging into your minus80 cloud account
For the time being,
minus80 cloud accounts are still only being created for beta-testing users. If you'd like to participate in beta testing
minus80 cloud functionality, contact me at
minus80 CLI, Logging is is pretty straight forward:
Additionally, simple account management features are included. Forget your password? No problem. Reset your password by using a secure link sent to your account email address.
Seeing what's up there
The list command shows you what datasets are available in the cloud.
cloud list command mimics that same interface as the local
minus80 list command, along with the
name search and filter (not shown here). Since we don't have any datasets in the cloud, we get the message
Nothing here yet!. To get some data up into the cloud, let's check out the
Pushing a dataset to the cloud
Assume we have a tagged local
Project.foobar:v1 that we want to push to the cloud. We can see our local dataset using the
minus80 list --tags command, which shows the frozen tag name, the checksum, and the timestamp. Then, using the same tag (
v1), we can send the dataset (
Project.foobar) to the cloud using the push command. Putting it all together, using a similar syntax as the
freeze command, we have:
minus80 cloud push Project.foobar:v1.
Behind the scenes,
minus80 communicates with the cloud server and caluclates which files need to be uploaded. Files are uploaded in parallel, up to 5 at a time. Once the files are server-side, checksums are calucalted and a status is returned, in this case
If we re-try out
cloud list command, we can see our dataset in the cloud!
Pulling a dataset from the cloud
Pulling a dataset is just as straightforward. Since the tagged dataset we just pushed exists locally (we cannot pull a tag that exists locally), assume it was either deleted with the
minus80 remove command or the following demo is happening on a different computer.
When we list the local data sets, we can see that a
foobar has been created for us along with the tag we requested.
This was a difficult post to write. So much went into trying to make the
minus80 cloud push/pull API as simple as possible, it's hard to not dive deep into the implementation details. Luckily for you, I have several smaller blog posts planned in the future focusing on some fun, nitty gritty details that peel back whats going on behind the scenes.
Until then, stay tuned on more updates and make sure to check out out GitHub page for more incremental updates and connect on Twitter.
- This work was funded by a Mozilla Science Mini Grant
- Cover Photo by eberhard grossgasteiger on Unsplash