STAGES IN ML DEVELOPMENT
This stage involves gathering data in a scalable manner and in a usable format by the machine learning models. In the data preparation phase we need to have data quality control measures, visualisations, hypothesis testing and use unsupervised techniques for exploratory data analysis
This is sometimes the hardest part in the entire flow. At this stage based on our data and overall business objectives we need to understand the most appropriate algorithm from a multitude of algorithms found in literature, design networks, validate assumptions of the model as well as verify the conceptual validity of using a model
Even if the training model is good we need to be certain as to whether the model will perform as expected in a data on which the model has not been trained. Hence we test the model on evaluation data and compare performance with the training model. A large deterioration of performance indicates overfitting or issue with sampling and we need to go back to stage 1.
Once we complete the evaluation phase we now come to the parameter tuning stage which is a method to improve our model performance by modulating the parameters (statistical/mathematical) of the model for example epochs or learning rates. Initial conditions might also play a pivotal role here. A deep understanding of statistical concepts as well as a grasp on the data and business requirements is need in this stage to create significant improvements