How SMEs Can Easily Overcome Their Common Machine Learning Challenges. Not just SMEs can benefit from a ML orchestration platforms that automates the infrastructure, data, modeling, and deployment puzzles. A practical guide to Machine Learning.
Machine learning (ML) is often said to be implemented in all businesses within the next decade. It’s already proven itself in both complex industries such as pharmaceutical research and development and more commonplace uses such as providing insight into customer shopping habits.
But ML has lagged in one area: Small and Medium-sized Enterprises (SMEs). Fewer SMEs have decided to adopt ML relative to their “big” company counterparts. In the US alone, SMEs make up 99.9% of all businesses. Yet as of 2020, only about 20% of these organizations had in some way adopted ML into production. However, there is hope, as the number of businesses trying to implement ML is on the rise.
What are the general barriers to ML?
In this Machine Learning practical guide, we fist look at the general barriers to ML. ML and Artificial Intelligence (AI) technologies have evolved incredibly since what is considered their first real practical use in 1951. Yet general adoption was initially incredibly slow, only to pick up some steam in the last few years, as businesses considered the genuine benefits it brings.
Results from a recent survey show that AI/ML adoption is maturing from prototype to production. Yet obstacles remain [1]. The reasons for these obstacles vary as the industry changes to address them. Multiple yearly surveys try to isolate the most significant issues and suggest how to adapt to these problems.
Cited again and again is the need for skilled data-scientists to create models. In the last couple of years, the number of data scientists has increased dramatically [2] but so too has the number of companies using ML. As a result, the skills gap persists. As long as it does, so will the slow rate of adoption.
The time it takes to put a model into production, is another metric (and problem) becoming more widely used to garner how far along an implementation of AI or ML is. Over half of the companies polled in a survey done in 2020 reported the time it took was over 90 days. 8% of the companies spent more than one year to build and put a model into production. Going further on this statistic, the time it took to deploy a model alone made up a significant portion of the data scientist’s time: As much as 25% of the entire time spent on a production-ready model was spent on infrastructure tasks and not the model itself [3].
How SMEs share similar problems
Although the most significant issues encountered by large companies are the same for SMEs, the SMEs have a harder time dealing with them, as we will see in this Machine Learning practical guide.
Whenever it comes to the skills gap, SMEs fare worse than the more prominent companies. The latter will typically have a larger budget and, therefore, can offer more for an ML job.
SMEs located in geographical regions where the pool of data engineers and data scientists is smaller, have a hard time finding the skills they need. Bigger companies often solve this issue by opening offices in areas with larger concentrations of data engineers and data scientists. This leaves less qualified workers for the smaller companies, which inevitably means hiring people with less experience. This usually results in longer times to produce the same models, as less experience can result in learning new techniques and practice “on the job”.
In addition, a lack of experience will often lead to constraints on what models can be put into production. It is simply not viable to expect a year’s worth of experience to happen immediately, no matter how good the documentation is. Such lack of experience when developing a model for your business may result in sub-optimal predictions and model deterioration over time. In the worst cases, it may dissuade the smaller companies from continuing with their ML projects as the time, costs, and manpower increase and fail to match the often-pledged profits.
Machine learning practical guide for companies of all sizes?
Let’s look at the next problem companies run into, in this Machine Learning practical guide. This problem is shared by companies of all sizes. It’s about successfully identifying and retrieving the needed data to use in your model. This is one of the most intriguing issues, as you need at least a basic understanding of Machine Learning before you can definitively say that you can go ahead with the data you currently have. Unfortunately, this isn’t always the case. As a result, four problems can be encountered:
- There isn’t enough data,
- The data is of poor quality,
- The data is unrepresentative of the use case and/or includes irrelevant features.
- The most relevant data cannot be retrieved for security, compliance or technical reasons
This can lead to disenfranchisement with ML as poor models fail to live up to expectations. Bear in mind that while data-related problems are universal, this pitfall is more dangerous to SMEs where time and money are more precious and a failed project more harmful.
What are the difficulties of model deployment?
Another problem covered by this Machine Learning practical guide is the deployment of the model. For big business, deployment often means hiring, in addition to experienced Artificial Intelligence or Machine Learning practitioners, project managers, data engineers, and IT experts who are able to implement and deploy the AI and ML model. As larger companies decide to implement their AI/ML models in the cloud, we often find that they quickly hire the AWS®, Microsoft® Azure, or GCS experts they need to get the job done.
For inexperienced (and even some experienced) data-scientists, the difficulties of moving a model from deployment to production may come as an unwelcome surprise. These difficulties include unstable models, model dependencies, reliable pipelines, dynamic learning, unreliable data, cloud infrastructures, and security management. It becomes apparent to the business that it doesn’t have any of the necessary skills in-house, often late in the process. This frequently leads them to draw the unfortunate conclusion that AI/ML is not for them.
Machine learning practical guide: the next steps?
Fortunately for SMEs, there are AI platforms and MLOps platforms out there specifically designed to overcome these issues. The general recommendation this Machine Learning practical guide covers is the use of a ML platform. There are a lot of AI platforms and MLOps platforms to choose from. Some offer a lot more features than others. SMEs should look for a AI platforms or MLOps platform with DataOps capabilities, that can orchestrate a complete Artificial Intelligence and Machine Learning project from end-to-end: These automate a lot of the infrastructure, data store, modeling, and deployment puzzles.
Using a well-thought-out platform does not require advanced AI or ML knowledge. The better ones offer many models to choose from and guide you through the model set-up, training, scoring, and deployment. A multi-cloud DataOps/MLOps platform will allow SMEs to easily deploy their models on any cloud (public or private) without requiring specialized cloud expertise. The more advanced platforms will enable the organization to build Web applications with dashboards and other appealing business graphics to highlight the found insights.
These new generation platforms promise to bridge the skills gap between SMEs and larger businesses and avoid the problems faced late in AI/ML projects. SMEs should seriously consider them. They can not only help companies avoid devastating delayed or even derailed ML attempts but even considerably speed up AI/ML projects. By directly addressing these common ML problems, MLOps platforms with DataOps capabilities can help ML become more widespread and more affordable.
[1] (copy/paste the following link) https://www.oreilly.com/radar/ai-adoption-in-the-enterprise-2020/
[2] (copy/paste the following link) https://info.algorithmia.com/hubfs/2019/Whitepapers/The-State-of-Enterprise-ML-2020/Algorithmia_2020_State_of_Enterprise_ML.pdf
[3] https://medium.com/@ODSC/machine-learning-challenges-you-might-not-see-coming-9e3ed893491f
For more articles on data, analytics, machine learning, and data science, follow me Paul Sinaï on Towards Data Science.
Get started with ForePaaS for FREE!
Discover how to make your journey towards successful ML/ Analytics – Painless
The image used in this post is a royalty free image from Unsplash.
