Quantify commits

Quantified Commit Activity Over 14 Days

Developer	Avatar	Branches	PRs	Commits	Files	Changes
Tianyu Yu		1	0/0/0	2	27	2901
YuzaChongyi		1	3/3/0	3	6	40
qianyu chen		1	0/1/0	1	9	37
Hongji Zhu		1	0/0/0	5	3	35
Cui Junbo		1	0/0/0	2	2	14
uuhc		1	1/1/0	1	1	3
Yuan Yao		1	0/0/0	1	1	2

_{PRs: created by that dev and opened/merged/closed-unmerged during the period}

Report On: Fetch commits

Project Overview

The MiniCPM-V project, managed by the organization OpenBMB, aims to develop a series of multimodal large language models (MLLMs) that can be efficiently deployed on mobile devices. The latest model in this series, MiniCPM-Llama3-V 2.5, boasts performance comparable to GPT-4V and supports over 30 languages. The project is actively maintained with frequent updates and improvements, indicating a robust development trajectory. The repository has garnered significant interest with 2,198 stars and 152 forks, reflecting its relevance and utility in the AI community.

Team Members and Recent Activities

Hongji Zhu (iceflame89)

1. 1 day ago - Merge pull request #108 from uuhc/uuhc

   • Files: requirements.txt (+3, -0)
   • Details: Updated requirements.
   • Collaborated with: uuhc

2. 1 day ago - Update README_en.md

   • Files: README_en.md (+2, -2)
   • Details: Minor updates to the English README.

3. 1 day ago - Update README.md

   • Files: README.md (+2, -2)
   • Details: Minor updates to the main README.

4. 2 days ago - fix web_demo_2.5 for int4

   • Files: web_demo_2.5.py (+8, -3)
   • Details: Fixed issues related to int4 in the web demo script.

5. 2 days ago - Update README_en.md

   • Files: README_en.md (+4, -4)
   • Details: Updated the English README for clarity.

6. 2 days ago - Update README.md

   • Files: README.md (+4, -4)
   • Details: Updated the main README for clarity.

YuzaChongyi

1. 1 day ago - Merge pull request #112 from YuzaChongyi/main

   • Files: finetune/readme.md (+3, -0)
   • Details: Updated finetune readme.
   • Collaborated with: cjm

2. 1 day ago - update finetune readme

   • Files: finetune/readme.md (+3, -0)
   • Details: Minor updates to the finetune readme.

3. 1 day ago - Merge pull request #111 from YuzaChongyi/main

   • Files: finetune/finetune_ds.sh (+1, -0), finetune/trainer.py (+1, -2)
   • Details: Fixed issues in finetuning scripts.
   • Collaborated with: cjm

4. 1 day ago - fix finetune

   • Files: finetune/finetune_ds.sh (+1, -0), finetune/trainer.py (+1, -2)
   • Details: Minor fixes in finetuning scripts.

5. 2 days ago - Merge pull request #92 from YuzaChongyi/main

   • Files: README.md (+8, -8), README_en.md (+8, -9), assets/airplane.jpeg (added)
   • Details: Updated demo-related content.
   • Collaborated with: cjm

6. 2 days ago – update demo – Files: README.md (+8,-8), README_en.md (+8,-9), assets/airplane.jpeg (added) – Details: Updated demo-related content.

uuhc

1. 1 day ago – feat: update requirements – Files: requirements.txt (+3,-0) – Details: Updated requirements.

Yuan Yao (yaoyuanTHU)

1. 2 days ago – Update README.md – Files: README.md (+1,-1) – Details: Minor update to main README.

Cui Junbo (Cuiunbo)

1. 2 days ago – update table – Files: README.md (+2,-2), README_en.md (+2,-2) – Details: Updated tables in READMEs.

2. 2 days ago – update readme – Files: README.md (+2,-2), README_en.md (+1,-1) – Details: Minor updates to READMEs.

Tianyu Yu (yiranyyu)

1. 2 days ago – update readme – Files: README.md (+1,-1) – Details: Minor update to main README.

2. 2 days ago – Update to MiniCPM-Llama3-V 2.5 – Files: Multiple files updated and added including images and markdown files. – Details: Major update introducing MiniCPM-Llama3-V 2.5.

qianyu chen (qyc-98)

1. 14 days ago – update finetuning code – Files: Multiple files related to finetuning scripts. – Details: Added and updated finetuning scripts.

Patterns and Conclusions

The recent activities indicate a highly collaborative environment with multiple team members frequently updating documentation and refining features related to model deployment and fine-tuning capabilities:

• Frequent updates to documentation (README.md, README_en.md) suggest an emphasis on clear communication and user guidance.
• Several commits focused on fine-tuning scripts (finetune/readme.md, finetune/finetune_ds.sh, finetune/trainer.py) highlight ongoing efforts to enhance model training processes.
• Updates related to demo scripts (web_demo_2.5.py) indicate active work on making the models more accessible for testing and demonstration purposes.
• The introduction of MiniCPM-Llama3-V 2.5 marks a significant milestone in the project's development trajectory.

Overall, the team appears well-coordinated with a clear focus on both improving model performance and ensuring ease of use for end-users through comprehensive documentation and support for various deployment scenarios.

Report On: Fetch issues

GitHub Issues Analysis

Recent Activity Analysis

Recent GitHub issue activity for the OpenBMB/MiniCPM-V project has been notably high, with a significant number of issues created and updated within the last few days.

Several issues exhibit notable anomalies or complications. For instance, #121 highlights a significant performance discrepancy between two model versions, which could indicate underlying inefficiencies or bugs in the int4 model. Issue #120 reports conflicting dependencies in the requirements.txt file, which could hinder new users from setting up the project. Additionally, issue #116 reports a critical runtime error due to shape mismatches in tensors, which could be a blocker for users trying to run the model.

Themes among the issues include performance concerns (e.g., #121), dependency conflicts (e.g., #120), and runtime errors (e.g., #116). There are also multiple requests for documentation and technical reports (e.g., #122), indicating a demand for more comprehensive project documentation.

Issue Details

Most Recently Created Issues

1. Issue #122: tech report or docs

   • Priority: Not specified
   • Status: Open
   • Created: 0 days ago
   • Updated: N/A

2. Issue #121: int4和bffloat16推理时间问题（着急）

   • Priority: High (indicated by "着急")
   • Status: Open
   • Created: 0 days ago
   • Updated: N/A

3. Issue #120: package versions have conflicting dependencies.

   • Priority: High (installation blocker)
   • Status: Open
   • Created: 0 days ago
   • Updated: N/A

4. Issue #119: 是否有，仿OpenAI API风格的demo运行文件

   • Priority: Not specified
   • Status: Open
   • Created: 0 days ago
   • Updated: N/A

5. Issue #118: 请问能不能训练特定场景，然后转成ONNX

   • Priority: Not specified
   • Status: Open
   • Created: 0 days ago
   • Updated: N/A

Most Recently Updated Issues

1. Issue #115: M1,16G带不动如何卸载？

   • Priority: Not specified
   • Status: Closed
   • Created: 0 days ago
   • Updated: 0 days ago

2. Issue #112: update finetune readme

   • Priority: Not specified
   • Status: Closed
   • Created: 1 day ago
   • Updated: 1 day ago

3. Issue #111: fix finetune

   • Priority: Not specified
   • Status: Closed
   • Created: 1 day ago
   • Updated: 1 day ago

4. Issue #108: feat: update requirements

   • Priority: Not specified
   • Status: Closed
   • Created: 1 day ago
   • Updated: 1 day ago

5. Issue #106: 支持ollama部署，thx

   • Priority: Not specified
   • Status: Closed
   • Created: 1 day ago
   • Updated: 1 day ago

Report On: Fetch pull requests

Analysis of Pull Requests for OpenBMB/MiniCPM-V

Open Pull Requests

PR #61: Implement chatbot functionality using Streamlit

• State: Open
• Created: 34 days ago, edited 1 day ago
• Description: This PR introduces a chatbot built with Streamlit, integrating the MiniCPM-V-2.0 model. It features a user-friendly interface, real-time interaction, and customizable parameters.
• Comments:
  • JamePeng: Update MiniCPM-Llama3-V-2_5 streamlit demo
  • Hongji Zhu (iceflame89): Thanks for your contribution，we will review this PR soon.
• Commits:
  • Initial implementation of chatbot functionality.
  • Recent updates to align with the main branch and enhance the demo.
• Files Added:
  • web_demo_streamlit-2_5.py (+98 lines)
  • web_demo_streamlit.py (+99 lines)

Notable Points:

• The PR has been open for over a month, indicating it may need more attention or review resources.
• Recent edits suggest active development and improvements are ongoing.

PR #36: [Draft] Add minicpmv finetune script

• State: Open
• Created: 42 days ago
• Description: This draft PR adds fine-tuning code for MiniCPM-V, tested on an 8xA100 GPU environment. It currently supports single image processing.
• Commits:
  • Initial commit adding fine-tuning scripts and configurations.
• Files Added:
  • ds_config_zero2.json (+52 lines)
  • finetune_ds.sh (+48 lines)
  • finetune_minicpmv.py (+477 lines)
  • minicpmv/model/configuration_minicpm.py (+216 lines)
  • minicpmv/model/modeling_minicpm.py (+1454 lines)
  • minicpmv/model/modeling_minicpmv.py (+422 lines)
  • minicpmv/model/resampler.py (+164 lines)

Notable Points:

• The draft status and lack of recent updates might indicate it is still under heavy development or awaiting further testing/feedback.

Recently Closed Pull Requests

PR #112: update finetune readme

• State: Closed
• Created: 1 day ago, closed 1 day ago
• Description: Minor update to the finetune README to include special instructions for Llama3's chat template during training and inference.
• Commits:
  • Single commit updating the README.
• Files Changed:
  • finetune/readme.md (+3 lines)

PR #111: fix finetune

• State: Closed
• Created: 1 day ago, closed 1 day ago
• Description: Fixes in the fine-tuning script and trainer to address issues with model max length and loss computation.
• Commits:
  • Single commit fixing the fine-tuning script and trainer.
• Files Changed:
  • finetune/finetune_ds.sh (+1 line)
  • finetune/trainer.py (+1 line, -2 lines)

PR #108: feat: update requirements

• State: Closed
• Created: 1 day ago, closed 1 day ago
• Description: Updates to the requirements file to include new dependencies like socksio and gradio.
• Commits:
  • Single commit updating requirements.txt.
• Files Changed:
  • requirements.txt (+3 lines)

PR #92: update demo

• State: Closed
• Created: 2 days ago, closed 2 days ago
• Description: Updates to the demo files including README changes and adding an image asset.
• Commits:
  • Single commit updating demo files.
• Files Changed:
  • README.md (+8 lines, -8 lines)
  • README_en.md (+8 lines, -9 lines)
  • assets/airplane.jpeg (added)

Notable Points from Closed PRs

Quick Turnaround on Recent Fixes and Updates

Several recent PRs (#112, #111, #108) were created and closed within a day. This indicates a responsive maintenance process for minor fixes and updates.

Significant Changes in Fine-Tuning Scripts

PRs like #81 introduced substantial additions to the fine-tuning scripts using Huggingface Trainer. These changes are crucial for users looking to customize their models.

Documentation Improvements

PRs such as #71 and #49 focused on improving documentation by adding descriptions and correcting typos. These enhancements are essential for maintaining clear communication with users.

Conclusion

The project shows active development with frequent updates and quick resolutions for minor issues. However, some open PRs like #61 and #36 have been pending for over a month, suggesting they may need more attention or resources for review. The recent focus on fine-tuning capabilities and documentation improvements reflects ongoing efforts to enhance usability and functionality.

Report On: Fetch Files For Assessment

Source Code Assessment

File: chat.py

• Purpose: Main code for interacting with the model, including chat functionalities.
• Structure & Quality:
  • Imports: Organized and relevant imports. Uses standard libraries and third-party libraries like torch, transformers, and PIL.
  • Functions:
  • init_omni_lmm: Initializes the OmniLMM model, tokenizer, and image processor. Handles device setup and model loading.
  • expand_question_into_multimodal: Converts questions into a multimodal format by embedding image tokens.
  • wrap_question_for_omni_lmm: Prepares the input data for the model by tokenizing and formatting the conversation.
  • Classes:
  • OmniLMM12B, OmniLMM3B, MiniCPMV2_5: Wrapper classes for different model versions, encapsulating methods to handle chat interactions.
  • OmniLMMChat: Main class to select the appropriate model version based on the provided path.
  • Code Quality:
  • Clear separation of concerns with well-defined functions and classes.
  • Adequate error handling (e.g., image decoding).
  • Good use of assertions to ensure assumptions about configurations are met.
  • Consistent use of logging and print statements for debugging.

File: finetune/finetune.py

• Purpose: Handles fine-tuning of the model, crucial for adapting it to specific tasks or datasets.
• Structure & Quality:
  • Imports: Relevant imports from standard libraries, torch, transformers, and custom modules.
  • Data Classes:
  • ModelArguments, DataArguments, TrainingArguments: Define configurations for model paths, data paths, and training parameters.
  • Functions:
  • rank0_print: Utility function to print messages only from rank 0 in distributed settings.
  • make_supervised_data_module: Prepares datasets and collators for supervised fine-tuning.
  • get_parameter_number: Calculates the number of trainable and total parameters in the model.
  • train: Main function to handle training logic, including parsing arguments, setting up the model, tokenizer, and data modules, and initiating training using a custom trainer (CPMTrainer).
  • Code Quality:
  • Well-organized with clear separation between configuration definitions and functional logic.
  • Proper handling of distributed training setups (e.g., DeepSpeed).
  • Use of data classes for clean argument management.

File: web_demo_2.5.py

• Purpose: Code for running a web-based demonstration of the model using Gradio, useful for understanding deployment and user interaction.
• Structure & Quality:
  • Imports: Includes necessary libraries like Gradio, PIL, transformers, etc.
  • Functions:
  • create_component: Utility to create Gradio components dynamically based on provided parameters.
  • chat, upload_img, respond, regenerate_button_clicked: Functions to handle various interactions within the Gradio interface (e.g., uploading images, responding to user queries).
  • Main Script:
  • Sets up an argument parser to choose between CUDA and MPS devices.
  • Loads the model and tokenizer based on the provided path.
  • Defines Gradio interface components (e.g., sliders, buttons) and their interactions.
  • Code Quality:
  • Clear separation between UI component creation and interaction logic.
  • Adequate error handling within interaction functions (e.g., invalid image uploads).
  • Well-documented usage instructions as comments at the top.

File: requirements.txt

• Purpose: Lists dependencies required to run the project, essential for setting up the environment.
• Structure & Quality:
  • Lists a comprehensive set of dependencies with specific versions ensuring reproducibility.
  • Includes common libraries (torch, transformers) as well as specialized ones (gradio, timm).

File: finetune/dataset.py

• Purpose: Handles dataset-related operations for fine-tuning, important for understanding how data is processed and used.
• Structure & Quality:
  • Classes:
  • SupervisedDataset: Custom dataset class for supervised fine-tuning. Implements necessary methods like __len__ and __getitem__.
  • Functions:
  • Various utility functions (data_collator, conversation_to_ids, etc.) to preprocess data, convert conversations into token IDs, slice images, etc.
  • Code Quality:
  • Comprehensive preprocessing logic covering multiple scenarios (e.g., image slicing).
  • Clear documentation within functions explaining their purpose and usage.

File: omnilmm/model/omnilmm.py

• Purpose: Core model architecture and implementation details.
• Structure & Quality:
  • Classes & Functions:
  • Defines custom configurations (OmniLMMConfig), vision modules (create_vision_module), main model class (OmniLMMModel), and causal LM class (OmniLMMForCausalLM).
  • Implements methods to initialize vision modules, get vision embeddings, forward pass logic, etc.
  • Code Quality:
  • Detailed implementation covering various aspects of multimodal learning (e.g., vision embedding integration).
  • Proper use of inheritance to extend base classes from transformers library.

Summary

The source code is well-structured with clear separation of concerns across different files. Each file serves a distinct purpose ranging from model interaction (chat.py), fine-tuning (finetune/finetune.py), web demo setup (web_demo_2.5.py), dependency management (requirements.txt), dataset handling (finetune/dataset.py), to core model architecture (omnilmm/model/omnilmm.py). The code quality is high with adequate error handling, clear documentation, and proper use of modern Python features like data classes.