01 - Fashion Classification

Introduction to Deep Learning:
This lesson introduces the idea that deep learning, a subfield of machine learning, excels at tasks like image recognition. Convolutional Neural Networks (CNNs) are at the core of these advancements.

Deep Learning Explanation:

• What is a neural network? A system inspired by the human brain, composed of interconnected layers of neurons.
• Why deep learning? It enables capturing complex representations in data, such as textures or shapes in images.

Key Content:

• How a CNN analyzes an image pixel by pixel.
• Visualization of convolutional filters and activations in the layers.

01b - Saturn Cloud

Deep Learning in a Cloud Environment:
Deep learning often requires substantial computational resources (GPUs, TPUs). Saturn Cloud simplifies access to these resources by enabling cloud environments tailored for machine learning workflows.

Deep Learning Explanation:

• Importance of GPUs: They accelerate matrix calculations crucial for neural networks.
• Training large-scale models using distributed computing on the cloud.

Key Content:

• Running a deep learning model on Saturn Cloud.
• Optimizing costs for cloud-based ML projects.

02 - TensorFlow and Keras

Introduction to Deep Learning with TensorFlow and Keras:
TensorFlow and Keras are fundamental tools for building deep learning models. This lesson explores their role in creating neural networks.

Deep Learning Explanation:

• Difference between Machine Learning and Deep Learning: Traditional ML uses manually engineered features, whereas deep learning learns directly from raw data.
• Core Layers: Dense, Activation, Dropout, and Convolution.

Key Content:

• Building a simple multi-layer neural network.
• Using optimizers like Adam and SGD to train models effectively.

03 - Pretrained Models

Leveraging Knowledge with Deep Learning:
Pretrained models like ResNet or Inception capture general knowledge about images, obtained through training on massive datasets.

Deep Learning Explanation:

• Feature extraction: Early layers of pretrained models often capture generic patterns.
• Fine-tuning: Adjusting the final layers for specific tasks, e.g., classifying clothes instead of animals.

Key Content:

• Practical application of ResNet on a custom dataset.
• Comparison between full training and fine-tuning approaches.

04 - Convolutional Networks (ConvNets)

Deep Learning Fundamentals for Images:
ConvNets use specific layers to extract visual features such as edges, textures, and shapes.

Deep Learning Explanation:

• Convolutional layers: Apply filters to extract patterns.
• Pooling layers: Reduce data dimensions to retain essential features.
• Visualizing feature maps from the convolutional layers.

Key Content:

• Implementing a ConvNet from scratch.
• Optimizing parameters to enhance performance.

 

 05- Transfer Learning

Boosting Results with Deep Learning:
Transfer learning uses pretrained networks to avoid training from scratch, saving time and improving performance.

Deep Learning Explanation:

• Why does it work? Deep learning models capture universal patterns applicable to many tasks.
• When to use it? When you have limited data or tasks similar to those solved by existing models.

Key Content:

• Reusing weights from a model like MobileNet to classify new objects.
• Tuning hyperparameters for optimal results.

06 - Learning Rate

The Role of Learning Rate in Deep Learning:
The learning rate is a critical hyperparameter that controls the speed of model convergence during backpropagation.

Deep Learning Explanation:

• Too fast: The model may miss global minima.
• Too slow: Training may become excessively long or stall.
• Advanced techniques: annealing (progressive reduction), Cyclical Learning Rate.

Key Content:

• Visualizing the impact of the learning rate on loss curves.
• Implementing learning rate schedules in training.

07 - Checkpointing

Saving and Reusing Progress in Deep Learning:
Checkpointing is crucial when training complex models for long periods or testing modifications without starting from scratch.

Deep Learning Explanation:

• Deep learning models often require days (or weeks) of training due to their complexity.
• Checkpoints save progress to resume training or evaluate intermediate performance.

Key Content:

• Configuring checkpoints in TensorFlow/Keras.
• Saving weights at different training stages for reuse.

08 - Adding More Layers

Building Deeper Models in Deep Learning:
Adding more layers increases a model’s learning capacity but also makes training more challenging.

Deep Learning Explanation:

• Depth vs. width: When to increase depth (more layers) or width (more neurons per layer).
• Risks: overfitting, exploding/vanishing gradients.

Key Content:

• Creating a deep model with additional layers.
• Techniques for stabilizing training (batch normalization, weight initialization).

09 - Dropout

Reducing Overfitting with Dropout:
Dropout is a regularization technique used in deep learning to randomly deactivate neurons during training.

Deep Learning Explanation:

• Why? It forces the model to avoid relying on specific neurons, encouraging generalization.
• Integration into complex architectures like ResNet or LSTMs.

Key Content:

• Using dropout across different types of layers.
• Analyzing its impact on performance using TensorBoard.

10 - Data Augmentation

Generating More Data for Deep Learning:
Deep learning models need vast amounts of data to generalize well. Data augmentation artificially enriches datasets through transformations.

Deep Learning Explanation:

• Why is it helpful? It makes models robust to variations in data.
• Examples: Rotations, scaling, color adjustments, and blurring.

Key Content:

• Implementing data augmentation pipelines.
• Assessing the impact on model performance.

11 - Large Models

Managing Challenges of Large Models in Deep Learning:
Architectures like GPT, BERT, or Vision Transformers (ViT) push the boundaries of deep learning capabilities.

Deep Learning Explanation:

• Why are they large? These models learn billions of parameters to capture intricate relationships.
• Challenges: Data requirements, memory usage, and training time.

Key Content:

• Strategies to train large models (model distillation, sparsity techniques).
• Practical applications (translation, text summarization, image  generation).

 

Project Presentation: Cardiovascular Risk Prediction Using Machine Learning

Context and Motivation

Cardiovascular diseases (CVD) are among the leading causes of death worldwide. By analyzing common risk factors such as high cholesterol, chest pain, and age-related risks, this project aims to identify individuals at risk of developing CVD at an early stage. Early detection could significantly reduce the number of deaths attributed to these diseases.

The primary objective is to design a machine learning model capable of classifying patients based on their cardiovascular risk by leveraging existing medical data. Additionally, the model aims to provide insights into the correlations and relative importance of various risk factors to assist researchers.

 

Technical Approach and Implementation

Dataset

The project utilizes the Kaggle Heart Failure Prediction Dataset, which offers comprehensive medical data relevant to CVD prediction.

Exploratory Data Analysis (EDA)

Detailed data exploration is performed using the part_1_preprocessing.ipynb notebook. This process includes data cleaning, feature engineering, and visualization to understand distributions, correlations, and anomalies.

Model Training

1. Models Tested: The evaluated models include Logistic Regression, DecisionTreeClassifier, RandomForestClassifier, and XGBClassifier.
2. Evaluation Metrics: Performance was assessed using confusion matrices and AUC scores.
3. Final Model: After fine-tuning hyperparameters, the chosen model is an XGBClassifier, which delivered optimal accuracy.

Deployment

The project implements multiple deployment options:

1. Flask Application: Offers predictions through an API and can be containerized using Docker for increased portability.
2. Streamlit Application: Provides an intuitive user interface for local testing or online access via Streamlit Cloud.
3. Docker Containerization: Simplifies the management and deployment of the model across various environments.

Software Infrastructure

• Virtual Environment: Managed with Pipenv, ensuring complete reproducibility. The Pipfile specifies all necessary dependencies (numpy, scikit-learn, xgboost, etc.).
• Training Script: Data preprocessing, model training, and fine-tuning are consolidated into the training.py script, enabling automated execution.

Cloud Deployment

The Streamlit app is hosted on Streamlit Cloud and accessible via this URL: Streamlit App.

 

Evaluation Criteria

The project is assessed based on several dimensions:

• Problem Description: Clear understanding of clinical and technological challenges.
• Exploratory Data Analysis (EDA): Relevance and depth of the analyses performed.
• Model Training and Selection: Justification of methodological choices and thoroughness of evaluations.
• Deployment: Ability to deliver a robust, accessible solution locally and via the cloud.
• Reproducibility: Clear documentation enabling re-execution of the entire pipeline.
• Dependency Management: Optimal use of Pipenv to ensure a stable environment.
• Containerization: Use of Docker to facilitate flexible integration.

---

Conclusion

 

This project demonstrates a comprehensive and rigorous approach to analyzing and predicting cardiovascular risks using machine learning. By combining in-depth analysis, high-performing models, and versatile deployment solutions, it addresses practical challenges in medical research while offering an application that healthcare professionals can readily use.

Summary: Estimation of Obesity Levels Based on Eating Habits and Physical Condition

Introduction

Obesity is a critical global health issue linked to poor eating habits and insufficient physical activity. It can lead to severe and even fatal consequences. This project aims to estimate obesity levels using a dataset of individuals from Colombia, Peru, and Mexico, offering insights into their health based on their eating habits and physical condition.

The dataset contains 17 attributes and 2,111 records, categorized into the following classes:

1. Insufficient Weight
2. Normal Weight
3. Overweight Level I
4. Overweight Level II
5. Obesity Type I
6. Obesity Type II
7. Obesity Type III

Seventy-seven percent of the data was synthetically generated using the SMOTE technique in Weka, while 23% was collected from users via a web platform.

 

Data Analysis and Model Training

Data analysis and model training were performed in Jupyter Notebook, and a streamlined pipeline was developed to preprocess the dataset and train the models. The process resulted in a trained model and a serialized pipeline exported as obesity-levels-model.bin.

 

Local Deployment

1. Using Flask:

   • Deploy the model using the script predict.py.
   • Set up the environment with Pipenv (pipenv install), activate it, and run the Flask application.

   cd scripts
   python predict.py
   

   You can test the model locally using the provided test.py script.

2. Using Waitress:

   • Serve the Flask app using the Waitress WSGI server:

     waitress-serve --listen=0.0.0.0:9696 predict:app
     

3. Docker Deployment:

   • Build the container:

     docker build -t estimation-obesity-levels .
     

   • Run the container:

     docker run -p 9696:9696 -it estimation-obesity-levels:latest

Remote Deployment

Using AWS Elastic Beanstalk:

• Deploy the application using Docker with AWS Elastic Beanstalk.
• Follow steps to initialize and configure the EB CLI:
  • Specify region, application name, environment name, and load balancer type.
  • Use AWS free-tier compatible instance types such as t2.micro.

Once deployed, the application URL is available in the AWS Elastic Beanstalk environment section. Update the test.py script with the new URL to test the application remotely.

 

Testing

Testing scripts (test.py) are provided to evaluate the model locally or remotely. Update the url variable to reflect the deployment location (local or AWS).

 

Key Features

• Dataset: Analysis of obesity levels with data from Latin America.
• Model Pipeline: Preprocessing, training, and deployment encapsulated in a single pipeline.
• Local and Remote Deployment: Flask and Docker-based local setup, with AWS Elastic Beanstalk for scalable cloud deployment.
• Testing Framework: Comprehensive scripts for validating predictions.

 

This project demonstrates the practical application of machine learning for public health, offering scalable solutions to assess obesity risks effectively.

Summary: SuperStore Sales Analysis and Prediction

This project uses data science techniques to analyze and predict SuperStore's sales performance, identifying actionable insights and optimization strategies to enhance profitability and revenue.

 

Objectives

1. Predict: Forecast sales revenue and profit margins using machine learning models.
2. Classify: Determine whether orders are profitable or non-profitable.
3. Optimize: Provide data-driven recommendations to improve business performance.

Key Analyses and Models

1. Sales Prediction:
   • Utilized linear regression to estimate revenue trends and seasonal patterns.
2. Profitability Classification:
   • Employed logistic regression to categorize orders based on profitability with a 97.25% accuracy and a near-perfect AUC score (0.997 ± 0.001).
   • Fine-tuned the model’s regularization strength (C), achieving optimal performance at C = 0.5.

Findings

• Discounting Practices: Discounts significantly impact sales volume and profitability.
• Geographical Insights: Regions and customer segments show clear differences in performance, influencing sales strategies.
• Model Efficacy: Logistic regression effectively identifies profitable orders with robust cross-validation results.

Recommendations

1. Discount Strategies: Tailor discounts based on their effect on profitability across regions and customer segments.
2. Customer Targeting: Prioritize high-performing regions and customer categories.
3. Inventory Management: Adjust stock levels to meet regional demand trends and product performance.

 

This project combines robust machine learning techniques and actionable insights, offering a comprehensive approach to enhancing SuperStore's business performance.

Machine Learning - Serverless Hosting

 

Machine learning models can be deployed using serverless computing services. When hosting these models, considerations need to be made regarding the size of the packages and security configurations.

 

Serverless Hosting

 

Cloud providers such as Azure Functions, Google Cloud Functions (GCP), and AWS Lambda offer serverless computing services that allow you to execute code without managing or provisioning infrastructure. These services enable the hosting of machine learning models as serverless functions, where you can run inference in response to events without worrying about the underlying server management. Below is a technical breakdown of deploying machine learning models using these serverless platforms:

 

Azure Functions

 

Azure Functions is part of Microsoft's serverless computing offering on Azure. Here's how to deploy a machine learning model using Azure Functions:

 

1. Select a Runtime: Azure Functions supports several runtimes, including .NET, Node.js, Python, and Java. Choose the one that matches your model’s requirements. For instance, if your model uses Python-based libraries like TensorFlow or PyTorch, select Python as the runtime.

2. Create a Function: You can create a new function using the Azure Portal, Visual Studio, or Azure CLI. Define triggers such as HTTP requests, Timer events, or Queue events, depending on your use case.

3. Dependencies and Environment: Install the necessary dependencies for your machine learning model. For Python-based models, you might use the pip package manager to install packages like TensorFlow, Scikit-learn, or PyTorch.

4. Code Integration: Integrate the code that loads and runs your machine learning model. You can use popular ML libraries, such as TensorFlow, PyTorch, or Scikit-learn. This code would typically load the model, perform inference, and return results based on the input data.

5. Deployment: Once your function is ready, deploy it using the Azure tools, such as the Azure Portal, Visual Studio, or Azure CLI.

 

AWS Lambda

 

AWS Lambda is a core part of AWS's serverless architecture. Here's how to host machine learning models on Lambda:

 

1. Choose a Runtime: AWS Lambda supports multiple runtimes, including Node.js, Python, Java, and more. Choose a runtime compatible with your model. If you're using TensorFlow or Keras models, Python is the most common choice.

2. Create a Lambda Function: Create a Lambda function using the AWS Management Console or AWS CLI. You can configure triggers, such as API Gateway, S3 events, or CloudWatch events.

3. Dependencies and Environment: Package any required dependencies and libraries into your Lambda function’s deployment package. If you are using Python, ensure that your deployment package includes the relevant .whl files or dependencies that AWS Lambda doesn’t natively support.

4. Code Integration: Write the code that loads the machine learning model and executes inference using libraries compatible with Lambda, such as TensorFlow Lite or PyTorch.

5. Deployment: Upload your deployment package to AWS Lambda via the AWS Management Console, AWS CLI, or tools like AWS SAM (Serverless Application Model).

 

Google Cloud Functions

 

Google Cloud Functions is another serverless option provided by Google Cloud Platform. Here's how you can deploy machine learning models:

 

1. Choose a Runtime: Google Cloud Functions supports runtimes like Node.js, Python, Go, and others. Select the one that works best with your model.

2. Create a Function: Create a Cloud Function using the Google Cloud Console, gcloud CLI, or other deployment methods. Configure triggers such as HTTP requests, Cloud Storage events, or Pub/Sub messages.

3. Dependencies and Environment: Install dependencies required for your model. For example, for Python-based models, use pip to install any necessary libraries.

4. Code Integration: Integrate the necessary code to load the model and perform inference. Popular machine learning libraries like TensorFlow, Keras, and Scikit-learn are widely used for model execution.

5. Deployment: Deploy the function to Google Cloud Functions using the Cloud Console or gcloud CLI.

 

Considerations for Serverless Hosting

 

• Cold Starts: Both Azure Functions, AWS Lambda, and Google Cloud Functions may experience "cold starts." A cold start happens when the function is invoked for the first time after being idle, leading to higher latency during that initial execution. This is a crucial consideration for real-time applications where low latency is essential.

• Resource Limits: Each serverless platform has resource limitations, such as maximum execution time, memory, and payload size. Ensure that your model's size and inference requirements are within the platform's constraints.

• Integration with Other Services: You can integrate serverless functions with other cloud services, such as Azure Blob Storage, AWS S3, Google Cloud Storage, or BigQuery, to store and manage data, log events, or trigger functions based on specific criteria.

 

TensorFlow Lite for Edge Devices

 

TensorFlow Lite is a lightweight version of TensorFlow specifically designed for mobile devices and edge devices, offering optimized inference for resource-constrained environments. It's ideal for deploying machine learning models on devices with limited computational power and memory, such as smartphones, IoT devices, and embedded systems.

 

TensorFlow Lite Overview

 

1. Model Optimization: TensorFlow Lite provides tools for converting and optimizing TensorFlow models to run efficiently on mobile and embedded devices. Optimization techniques like quantization reduce model size by lowering the precision of model weights and activations, thus reducing memory usage and improving inference speed.

2. Hardware Accelerators: TensorFlow Lite supports a range of hardware accelerators, including GPU and TPU on supported devices, enhancing inference performance.

3. Interpreter: TensorFlow Lite uses a lightweight interpreter that can execute models on edge devices, with support for both CPU and specialized accelerators like TPUs.

 

Converting a TensorFlow Model to TensorFlow Lite

 

To convert a TensorFlow model (e.g., a SavedModel or Keras model) to TensorFlow Lite, follow these steps:

 

1. Install TensorFlow and TensorFlow Lite Converter: Ensure you have TensorFlow installed and also install the TensorFlow Lite Converter.

   pip install tensorflow
   pip install tflite-model-maker
   

2. Convert the Model: You can use the TensorFlow Lite Converter to convert your trained model to the .tflite format.

   import tensorflow as tf
   from tensorflow.keras.models import load_model
   
   model = load_model("path/to/model.h5")
   converter = tf.lite.TFLiteConverter.from_keras_model(model)
   tflite_model = converter.convert()
   
   with open("model.tflite", "wb") as f:
       f.write(tflite_model)
   

3. Optimization (Optional): Quantization or other optimization techniques can be applied to further reduce the model size for constrained devices, ensuring it fits within the resource limits.

4. Deploy to Serverless Platform (e.g., Azure Functions): After converting to .tflite, you can deploy the model to serverless platforms like Azure Functions by integrating the model loading and inference code. Ensure that the function is optimized to handle potential cold starts and resource limits.

 

Image Classification using TensorFlow Lite

 

In this section, we demonstrate how to download, preprocess, and classify images using a TensorFlow Lite model. The image will be processed through a series of functions:

 

1. Download Image: This function fetches an image from the provided URL and converts it into a PIL Image object.

   from urllib import request
   from PIL import Image
   
   def download_image(url):
       with request.urlopen(url) as resp:
           buffer = resp.read()
       return Image.open(BytesIO(buffer))
   

2. Prepare Image: This function resizes the image to the target size and ensures it’s in RGB format.

   def prepare_image(img, target_size):
       if img.mode != 'RGB':
           img = img.convert('RGB')
       return img.resize(target_size, Image.NEAREST)
   

3. Preprocess Image: The function converts the image to a NumPy array, normalizes the pixel values to the range [0, 1], and ensures the data type is float32.

   def preprocess_image(img):
       img = np.array(img).astype('float32')
       return img / 255.0
   

4. Run Inference with TensorFlow Lite Model: Use the TensorFlow Lite model for inference:

   def load_lite_model(path):
       interpreter = tf.lite.Interpreter(model_path=path)
       interpreter.allocate_tensors()
       input_details = interpreter.get_input_details()
       output_details = interpreter.get_output_details()
       return interpreter, input_details, output_details
   
   interpreter, input_details, output_details = load_lite_model('model.tflite')
   interpreter.set_tensor(input_details[0]['index'], [img_normalized])
   interpreter.invoke()
   
   output_data = interpreter.get_tensor(output_details[0]['index'])
   print(f"Classification Result: {round(output_data[0][0], 3)}")
   

 

By following these steps, you can deploy a TensorFlow Lite model in a serverless environment, preprocess the image data, and run inference in real-time on edge devices with minimal latency.

 

Deployment with Docker

 

To run the image classification model within a Docker container, you can set up a Dockerfile that includes all necessary dependencies and your application code. The steps include creating the environment, installing dependencies, and running the application inside the container. You can use the following Dockerfile for deployment:

 

FROM agrigorev/zoomcamp-bees-wasps:v2

WORKDIR .

COPY main.py .
COPY img_ai ./img_ai/

COPY Pipfile Pipfile.lock ./

RUN pip install numpy
RUN pip install Pillow
RUN pip install https://github.com/alexeygrigorev/tflite-aws-lambda/raw/main/tflite/tflite_runtime-2.14.0-cp310-cp310-linux_x86_64.whl

CMD ["main.main"]

 

Once the Docker image is built, deploy it as a containerized function on your serverless platform. The Docker container will ensure a consistent runtime environment and provide scalability for inference tasks.

 

In this section, we dive deep into the integration of Kubernetes with TensorFlow Serving, a high-performance framework designed to deploy and serve machine learning models. We'll explore how to leverage Kubernetes' power to scale machine learning services, optimize resource usage, and ensure reliability in production environments. The key areas covered include the architecture of scalable ML systems, deployment workflows, GPU optimization, and scaling strategies using Kubernetes' built-in features.

---

• TensorFlow Serving: Optimized for serving TensorFlow models written in C++, focusing on minimal latency and high throughput for inference tasks.
• Kubernetes: Orchestrating Docker containers, scaling workloads, and managing the lifecycle of applications efficiently across a cluster.
• Scalable Architectures: Using Kubernetes to ensure that services, such as image preprocessing and model inference, can scale independently depending on the resource requirements.
• Cloud Deployment with EKS: Transitioning from local development environments to cloud-based Kubernetes clusters with AWS's Elastic Kubernetes Service.

• Learn Kubernetes basics: Pods, Deployments, Services, and StatefulSets.
• Deploy a TensorFlow Serving model: Use Kubernetes to manage and scale ML models.
• Optimizing resources: How to leverage both CPU and GPU resources effectively.
• Cloud-native deployment: Moving from a local setup to the cloud with services like AWS EKS for easy scaling and resource management.

---

The gateway service is responsible for the initial image processing. This includes downloading images, resizing, and converting them into numpy arrays—tasks that are relatively lightweight and do not require GPU acceleration. The gateway can be containerized in Docker and deployed in Kubernetes Pods for isolation and scalability.

• Why CPU? The pre-processing workload is computationally trivial compared to model inference, so it can be handled by CPU nodes to save GPU resources.
• Kubernetes Pods: Each service (e.g., the gateway) is encapsulated in a pod. A pod may contain multiple containers if needed, but for simplicity, each service can be housed in its own pod.

This Dockerfile sets up a simple Python-based service to handle image preprocessing.

Once the images are preprocessed, they are sent to the TensorFlow Serving model for inference. TensorFlow Serving is highly optimized for inference tasks and is written in C++, ensuring performance is maximized for real-time predictions. Since inference tasks are computationally expensive, the model service is configured to run on GPU-enabled nodes.

• Why GPU? The core operations of deep learning models (e.g., matrix multiplications) benefit greatly from GPU acceleration, providing significant speed-ups for complex computations.
• gRPC Protocol: The communication between services (e.g., the gateway and the model service) is handled by gRPC, a high-performance binary protocol that allows for low-latency communication.

Kubernetes excels at managing scalable applications, which is crucial for deploying machine learning models in production environments. We'll discuss both horizontal scaling (adding more replicas) and vertical scaling (increasing resource allocation for specific workloads) and how these can be managed with Kubernetes.

• Horizontal Pod Autoscaling (HPA): Automatically scale the number of Pods based on CPU or memory utilization, ensuring that the system can handle more load when required.
• Vertical Scaling: For GPU-heavy tasks, Kubernetes can be configured to run workloads on GPU-optimized nodes to handle the intensive computations.

This configuration automatically scales the number of TensorFlow Serving pods between 2 and 10 based on CPU usage.

---

Once the local Kubernetes setup is functioning well, the next step is to transition to the cloud. AWS Elastic Kubernetes Service (EKS) is a fully managed service that simplifies the deployment, scaling, and management of Kubernetes clusters in the cloud. EKS integrates seamlessly with other AWS services like Elastic Load Balancing, IAM for security, and CloudWatch for monitoring.

To deploy your machine learning services in the cloud, you can use the eksctl CLI to create and manage your Kubernetes clusters on AWS.

---

In production systems, you often need to persist data between container restarts. Kubernetes supports Persistent Volumes (PV), which can be backed by various storage systems, including network-attached storage (NAS) or cloud-based services like AWS EBS.

This allows for managing data that should persist even when a container is terminated or restarted, such as storing models or datasets.

Kubernetes ConfigMaps and Secrets are used to store configuration data (non-sensitive) and sensitive information (like API keys, passwords), respectively. This allows for flexible management of environment variables, model parameters, and other configurations.

ConfigMaps are ideal for storing general application settings, whereas Secrets should be used to store sensitive data securely, with encryption at rest.

---

• Kubernetes simplifies the management of machine learning models in production by automating scaling, resource management, and container orchestration.
• TensorFlow Serving provides an optimized, high-performance solution for deploying machine learning models.
• Cloud deployment using EKS offers easy scalability, security, and managed infrastructure.
• Proper resource management—using both GPU for model inference and CPU for lighter workloads—ensures efficiency and cost-effectiveness.
• Horizontal scaling via Kubernetes allows services to handle varying workloads seamlessly.

---

• 📖 Kubernetes Documentation
• 📘 TensorFlow Serving Documentation
• 🎥 ML Zoomcamp Course Videos

---

1. Deploy a TensorFlow model on Kubernetes: Set up both the gateway and model services as Docker containers and deploy them using Kubernetes.
2. Configure Horizontal Scaling: Implement Horizontal Pod Autoscaling based on CPU usage to manage varying traffic loads.
3. Create Persistent Storage: Set up a Persistent Volume for storing model data.
4. Set up EKS: Deploy your Kubernetes configuration to AWS EKS for cloud-based management.

💡 Tip: Kubernetes is a powerful tool, but mastering it for machine learning requires understanding both infrastructure management and the needs of your specific workloads (e.g., GPUs for inference tasks, CPUs for lighter workloads).

 🌟 Skin Lesion Analyzer  

Detection and classification of skin lesions using ResNet50, MLOps, and Flask.

 

🗂️ Table of Contents

1. ✨ Introduction]

2. 🔍 Data Exploration (EDA)]

3. 🛠️ Data Preparation]

4. 🏋️‍♂️ Model Training

5. ✅ Testing and Evaluation

6. 🚀 Deployment with Streamlit

    - 📦 Creating a Docker Image

    - 🌐 Server Deployment

7. 📈 Monitoring and MLOps

    - 🔄 Building an MLOps Pipeline

8. 📂 Data 

9. 🙌 Contributors

 

✨ 1. Introduction

The Skin Lesion Analyzer is an application that uses artificial intelligence to detect and classify skin lesions from images.  

This approach automates and improves the accuracy of dermatological diagnostics through deep learning algorithms.  

 

Theoretical Context:

Image classification models rely on convolutional neural networks (CNNs), such as ResNet50. These models can detect complex patterns and differentiate lesion classes based on visual features.

 

🎯 Main Objectives:  

- Provide a fast and accurate model for analyzing skin lesions.  

- Deploy a user-friendly and accessible application.  

- Integrate a continuous training and deployment pipeline to ensure consistent performance improvements.

 

---

 

## 🔍 2. Data Exploration (EDA)  

Data Exploration is an essential step in data-driven projects. It helps understand the structure of the data and detect any anomalies or trends.Here are the different labels for this dataset.

 

Class indices and their corresponding labels:

 

Index 0: Actinic keratosis.

 

Index 1: Basal cell carcinoma.

 

Index 2: Dermatofibroma.

 

Index 3: Melanoma.

 

Index 4: Nevus.

 

Index 5: Pigmented benign keratosis.

 

Index 6: Squamous cell carcinoma.

 

Index 7: Vascular lesion.

 

 

Key Steps:

1. Analyze the distribution of classes (types of lesions).  

2. Study metadata, such as age or anatomical location.  

3. Visualize correlations between different attributes.  

 

Theoretical Context:

The quality and diversity of data directly influence model performance. An imbalanced class distribution may require techniques like sampling or weighting to ensure reliable predictions.  

 

 🛠️ 3. Data Preparation  

Data preparation is crucial to ensure uniform and relevant input to the model.  

 

 Key Steps:

1. Data Cleaning: Remove missing or anomalous values in the metadata.  

2. Image Preprocessing: Resize images to 224x224 pixels to ensure compatibility with ResNet50.  

3. Data Augmentation: Generate variations of images (rotation, zoom, flipping) to enrich the dataset and prevent overfitting.  

4. Normalization: Adjust pixel values between 0 and 1 to stabilize training.  

 

Theoretical Context:

Neural networks are sensitive to data scales. Normalization accelerates convergence and improves model robustness.

 

📷 Normalize images:  

🏋️‍♂️ 4. Model Training

The model used, ResNet50, is a convolutional neural network pretrained on ImageNet. It is fine-tuned to classify skin lesions into 7 categories.  

 

Training Steps:

1. Load cleaned and normalized data.  

2. Perform data augmentation to improve robustness.  

3. Train by adjusting the model's final layers.  

 

 Theoretical Context:

- Transfer Learning: Using a pretrained model reduces data requirements and training time.  

- Fine-Tuning: Adapting a general-purpose model (like ResNet50) to a specific task.  

 

📈 Visual Indicators:  

 

 

✅ 5. Testing and Evaluation  

Once trained, the model's performance is evaluated on an independent test set.

- Evaluation : Evaluation of Models on test set:

 

 Metrics Used:

1. Accuracy: Proportion of correct predictions.  

2. Recall: Model's ability to correctly identify positive cases.  

3. F1-score: Harmonic mean of precision and recall, useful for imbalanced data.  

4. Confusion Matrix: Visualizes classification errors.  

 

 Theoretical Context:

Metrics like F1-score are particularly useful in medical contexts where classification errors have critical implications.  

 

 Test whit new image and Grad-CAM :

Here is the result of our model on a new image, along with the explainability analysis using the Grad-CAM method. This technique allows us to visualize which areas of the image influenced the model's decision. By using Grad-CAM, we generate a heatmap highlighting the important regions of the image that the model focused on for making its prediction. This approach provides a better understanding of the model's decision-making process, making its predictions more transparent and interpretable, which is essential for ensuring the reliability of the model, especially in sensitive applications.  

 

 

 

 🚀 6. Deployment with Streamlit  

Once validated, the model is integrated into a Streamlit application for interactive use.  

 

📦 Creating a Docker Image

Docker is used to ensure application portability. A Docker image contains all necessary dependencies.  

 

 

 

 🌐 Server Deployment  

The Docker image is deployed on a server or cloud platform (AWS, Azure, Google Cloud).  

 

 

 

 📈 7. Monitoring and MLOps  

Monitoring and MLOps practices ensure the model's maintenance and continuous improvement after deployment.  

 

Monitoring with Prometheus and Grafana:

- Collect performance metrics (response time, error rate).  

- Real-time visualization on dashboards.  

 

Building an MLOps Pipeline:

1. CI/CD: Continuous integration and deployment with GitHub Actions.  

2. Kubernetes: Container orchestration for scalable deployment.  

 

 📂 8. Data  

The dataset HAM10000 used in this project is a collection of 10,015 images of skin lesions, with detailed medical annotations.  

## 🙌 **9. Contributors**  

This project was created by:  

-[M. Domche]  

For any questions or suggestions,

✨ Thank you for your interest in this project!