Literate Computing for Reproducible Infrastructure

What is LC4RI?

It is as essential as to share and communicate about infrastructure design and elaborated workflows with participants as to actually automate complex operations. Literate Computing for Reproducible Infrastructure is an approach both to describe automated operations as live code and to share predicted and reproducible outcomes among technical and non-technical alike in the form of narrative stories. We utilize Jupyter Notebook for sharing the reproducible experience. Operational engineering and DevOps should be one of the distinctive application areas for Jupyter.

“Automated Operation” [機械化] ≠ “Automation” [自働化]

We want to accomplish traceability and reproducibility in engineering operations. For those primary purposes, we utilize computational narrative tools, i.e., Jupyter Notebook. Every operation is described with no doubt and can be automated. Automated operation is always along with humans in the loop and bound up with a situation. It is something different from distilled automation, which tends to result in an anesthetizing effect (Nicholas InCarr. The Glass Cage) . Automated operation is a partnership between humans and machines and augments our ability to learn and expertise.

Collaboration and Communication

For reproducibility and resilience for long-term sustainability, it is crucial not only to share knowledge but also to share reproducible experience participating in both tech and non-tech alike. Narrative stories allow collaborative communication between experts and novices to accumulate infrastructure knowledge and operational experience within an operation team. Moreover, it is efficient to share an understanding of how infrastructure is usable and really works between tech ops and non-tech users. Narrative stories also help communicate with users about how services are delivered and customized, considering reproducibility.

Literate Computing tools

Literate Computing Tools are our enhancements to Jupyter for achieving the following goals:

• Giving awareness where you are; the cells are colored depending on their statuses. You can see how an operation is in progress and whether it has been succeeded. Light Cyan, Linen, and Pink represent “running”, “finished successfully” and “finished with errors” respectively.
• Preventing miss-operation; once a cell has been executed, it “freezes” against unintended execution. Frozen cells will not be executed nor edited until those will be unfrozen.
• A good operational outlook; as an enhancement for collapsible headings, collapsed code cells underneath are represented as bricks. The number of bricks represents operational complexities. Plus, you can run through whole bricks (collapsed code cells) with one click as a routine procedure. Color changes of bricks also represent the progress.
• Assure traceability; infrastructure operation can generate massive output lines. LC_wrapper kernel summarizes output lines, and simultaneously all original output lines are saved into an individual file with a timestamp at each execution. You can investigate the total output and compare it with previous results for reviewing.

Try the Demo

You can start the Notebook server on port 8888 with the following command.

docker run -it --rm -p 8888:8888 niicloudoperation/notebook:latest

Star

You can login the Notebook server with the authentication token in the startup message.

Jupyter Extensions

Other Extensions

• multi_outputs Star
• nblineage Star
• Jupyter-LC_index Star
• i18n_cells Star

...

...

Notebooks for Reproducible Infrastructure

• Literate-computing-Basics Star
• Literate-computing-Hadoop Star
• Literate-computing-Elasticsearch Star

The following video presents early practices back to late 2015 and 2016Q1. Since then, we have been developed Literate Computing Tools as enhancement of Jupyter.