How to Layout and Manage Your Machine Learning Project

Project layout is critical for machine learning projects just as it is for software development projects. I think of it like language. A project layout organizes thoughts and gives you context for ideas just like knowing the names for things gives you the basis for thinking.

In this post I want to highlight some considerations in the layout and management of your machine learning project. This is very much related to the goals of project and science reproducibility. There is no “best” way, you will want to select and adopt the practices that best meet your predilections and project requirements.

Workflow Motivating Questions

Jeromy Anglim gave a presentation at the Melbourne R Users group in 2010 on the state of project layout for R. The video is a bit shaky but provides a good discussion on the topic.

I really like the motivation questions from Jeromy’s presentation:

  • Divide a project into files and folders?
  • Incorporate R analyses into a report?
  • Convert default R output into publication quality tables, figures, and text?
  • Build the final product?
  • Sequence the analyses?
  • Divide code into functions?

You can checkout the summary of the presentation on Jeromy’s blog, the PDF presentation slides and the YouTube video of the presentation.

Goals for Project Workflow

David Smith provides a summary of what he believes are the goals of a good project workflow in the post titled A workflow for R. I think these are excellent and should be kept in mind when designing your own project layout.

  • Transparency: Logical and clear layout for the project making it intuitive for the reader.
  • Maintainability: Easy to modify the project with standard names for files and directories.
  • Modularity: Discrete tasks separated into separate scripts with a single responsibility.
  • Portability: Easy to move a project to another system (relative paths and known dependencies)
  • Reproducibility: Easily run and create the same artefacts by you in the future or another person.
  • Efficiency: Less thought on meta project details like the tools and more on the problems you are solving.

ProjectTemplate

John Myles White has an R project called ProjectTemplate that aims to automatically create a well defined layout for a statistical analysis project. It provides conventions and utilities for automatically loading and munging data.

The logo for ProjectTemplate, a project for laying out your R statistical analysis project.
The project layout is larger than I would prefer, but provides insight into a highly-structured way for organizing your project.

  • cache: Preprocessed datasets that don’t need to be re-generated every time you perform an analysis.
  • config: Configuration settings for the project
  • data: Raw data files.
  • munge: Preprocessing data munging code, the outputs of which are put in cache.
  • src: Statistical analysis scripts.
  • diagnostics: Scripts to diagnose data sets for corruption or outliers.
  • doc: Documentation written about the analysis.
  • graphs: Graphs created from analysis.
  • lib: Helper library functions but not the core statistical analysis.
  • logs: Output of scripts and any automatic logging.
  • profiling: Scripts to benchmark the timing of your code.
  • reports: Output reports and content that might go into reports such as tables.
  • tests: Unit tests and regression suite for your code.
  • README: Notes that orient any newcomers to the project.
  • TODO: list of future improvements and bug fixes you plan to make.

You can learn more on the ProjectTemplate homepage, the blog post on John’s website the GitHub page for development and the CRAN page for distribution.

Data Management

Software Carpentry provides a short presentation titled Data Management. The approach to data management is inspired by an article by William Stafford Noble titled A Quick Guide to Organizing Computational Biology Projects.

The presentation describes problems with maintaining multiple versions of data on disk or in version control. It comments that the main requirement in data archiving and proposes an approach of dated directory names and data file metadata files that are themselves managed in version control. It’s an interesting approach.

You can review the video and slides for the presentation here.

Best Practices

There is a lot of discussion of best practices for project layout and code organization for data analysis projects on question and answer sites. For example, some popular examples include:

A good example is the question How to efficiently manage a statistical analysis project? which was turned into a community wiki describing the best practices. In summary, these practices are divided into the following sections:

  • Data management: Use a directory structure, never modify raw data directly, check data consistency, use GNU make.
  • Coding: Organize code into functional units, document everything, custom functions in a dedicated file.
  • Analysis: Record your random seeds, separate parameters into config files, use multivariate plots
  • Versioning: use version control, backup everything, use an issue tracker.
  • Editing/Reporting: Combine code and reporting and use formal report generators.

More Practices

Each project I try to refine my project layout. It’s hard because the projects vary with data and aims as do the languages and tools. I’ve tried all compiled code and all scripting language versions. Some good tips I’ve discovered include:

  • Stick to a POSIX filesystem layout (var, etc, bin, lib, so on).
  • Put all commands in scripts.
  • Call all scripts from GNU make targets.
  • Have make targets that create environment and download public datasets.
  • Create recipes and let the infrastructure check and create any missing output products each run.

This last point is a game changer. It allows you to pipeline your workflow and define recipes with wild abandon for tasks like data analysis, preprocessing, model configuration, feature selection, etc. The framework knows how to execute recipes and creates results for you to review. I’ve talked about this approach before.

How do you layout and organize your machine learning project? Leave a comment.

10 Responses to How to Layout and Manage Your Machine Learning Project

  1. Avatar
    Raj April 7, 2017 at 4:17 pm #

    Hi,
    i am allocated for the machine learning project in my organization. But i have no idea about machine learning. The main goal of this project is, we need to generate the reports related to my organization like, present and future company standards, employees recruitment status etc..
    He has given me the different technologies to study like, machine learning, big data, NLP, data visualization….
    Now i need a clarity on how these all technologies integrated. and what is the flow. I mean which technology comes first and second and so on..
    I hope i will get reply either here or to my mail id.

    Thank you

  2. Avatar
    Jesús Martínez April 4, 2018 at 12:26 am #

    Thanks for sharing! Although I’m still developing a framework or methodology to tackle new machine learning projects, one practice that has really helped me is to keep daily journals of the activities I’ve performed, the things I tried and the results I got. I found myself in the past repeating the same steps over and over, which, of course, was a huge waste of valuable time.

  3. Avatar
    Butul May 31, 2018 at 8:47 am #

    Thanks Jason for sharing.

    May I ask you to elaborate on this point “Have make targets that create environment and download public datasets”

    • Avatar
      Jason Brownlee May 31, 2018 at 2:12 pm #

      I mean “make” targets (in a makefile) that will download/install required libraries and other targets that download the datasets.

      Does that help?

  4. Avatar
    suy December 27, 2018 at 9:16 pm #

    Thanks for sharing!!
    I’m looking for some help. If we use convolutional neural networks for feature extraction, use decision trees for classification, and if this project shows a gradual decline in prediction at a later stage and in application, how should we maintain the project? I hope I can get a response. Thank you very much.

  5. Avatar
    Shaily March 17, 2019 at 12:34 am #

    Hello
    Thanks for sharing valuable knowledge .
    I’m working on a ML project (basically my final year major project )of Gender recognition through faces , that are captures by webcam. I am done with the coding part ,but I’m confused that how to represent it. Whether to go for webpage or webapp or anything that you suggests.

    • Avatar
      Jason Brownlee March 17, 2019 at 6:22 am #

      Perhaps the specifics of the presentation don’t matter?

Leave a Reply