With more and more industries outside of the technology sector relying on artificial intelligence (AI) and machine learning (ML) systems in business operations, there is increased focus on the often treacherous path from development to production. A gap exists between the ML models developed by data scientists and the methods currently used to deploy their models into production. 

Though the development of ML systems is, in a number of ways, fundamentally different from deployment and continuous delivery of traditional software-based services, many organizations still adhere to the standard software engineering principles of DevOps. The limitations of DevOps processes in the development of ML systems have led to the rise of a new discipline — MLOps. Though both share the same goals, DevOps is solely driven by code iteration. The exploratory nature of MLOps – with its focus on reproducibility, observability, and scalability – creates challenges that are beyond DevOps practices.

Current MLOps solutions often require data scientists to change their development practices or adopt new ones. Organizations using these solutions usually experience operational friction as the ML models are deployed into production. 

Data scientists shoulder the burden existing technology solutions cannot address.

The readily available solutions create this friction because they are designed to support either development or production. The result is data scientists acting as liaisons between the two environments and undertaking more of the data and software engineering burden. 

Data science development is characterized by nimble, iterative experimentation and exploratory data analysis. The stream-of-consciousness workloads used by data scientists are often at odds with best software engineering practices required for productionizing end results. To bridge the gap, data scientists often find themselves learning new engineering frameworks and undertaking the manual and time-consuming processes themselves.

This practice slows workflows for both environments. Now an increasing number of organizations seek a better solution that will bridge the gap between data science development and production.

There is a better way. 

A frictionless path from development to production is achievable. It requires rethinking traditional approaches to better integrate data science development and production, shifting the software burden away from data scientists and automating the process from development to production. To materialize this path organizations need to focus on four key design principles:

• Instrument existing development environments. Production systems have very particular deployment methods that MLOps typically have to build from scratch when it is time to go to production. However, in most cases the blueprint for that production was a part of development. If that blueprint could be automatically captured and shared with production systems, it would eliminate the need for data scientists to author one. 
• Extract pipeline semantics. The process of translating development workflow into production pipelines is time consuming and error prone. Since data science workflows comprise well known libraries, it is easy to extract the execution semantics and use them to fully automate the translation between development and production.
• Capture the complete development process. Data scientists quickly iterate through many experiments during development, but manually logging their workflow often leads to an incomplete capture of development history. It is important to capture the history so that any part of development can be recovered and examined during the process for productionization. 
• Interoperate with existing tools. End-to-end production systems can have a ripple effect throughout the technology stack, eliminating the option for MLOps teams to choose best-in-class point solutions. Organizations should be able to integrate with existing solutions when it makes sense. This can eliminate adoption barriers and enable sharing of tools across both development and production to reduce duplicate work.

Following these principles, it is possible to supercharge your data science workflow and automate the transition from development to production. That’s why we built LineaPy: to create a frictionless path for taking ML models from development to production.

Why LineaPy

LineaPy automatically captures models developed by data scientists and deploys them to production using industry-leading MLOps platforms. LineaPy traces a Python program execution and creates an intermediate graph representation that captures a program’s semantic meaning. LineaPy can then transform these graphs into different MLOps toolchain outputs, which can automatically run in production.

LineaPy bridges the gap between development and production by taking the ML models developed by data scientists and automatically integrating them with industry-leading MLOps platforms and tools. LineaPy was built by taking these four design principles and translating them into specific actions.

• Instrument existing development environments. This is achieved by importing the LineaPy library into existing Python programs.
• Extract pipeline semantics by tracing program execution and creating an intermediate representation.
• Capture the complete development process using LineaArtifacts that extract a clean version of a program and its execution context.
• Interoperate with existing MLOps tools with the help of applications built on LineaArtifact.

Backed by a decade of research and industry expertise tackling hyperscale data challenges, LineaPy can help organizations drive innovation, improve the customer experience, and increase revenue. LineaPy is an open source initiative that will super charge your data science workflows. Using just two lines of code, capture, analyze, and transform Python code to extract production data pipelines within minutes.