Quantify commits

Quantified Commit Activity Over 14 Days

Developer	Avatar	Branches	PRs	Commits	Files	Changes
hoshi-hiyouga		1	0/0/0	64	69	2365
Ledzy		1	1/1/0	3	10	227
Marco		1	3/1/2	2	10	111
marko1616		1	1/1/0	7	2	83
codingma		1	2/1/1	2	1	51
Lao		1	2/1/1	2	2	19
Erich Schubert		1	1/1/0	1	1	2
liuzc		1	0/0/0	1	1	2
None (linpan)		0	1/0/1	0	0	0
Yingbei Tong (tybalex)		0	1/0/1	0	0	0
Xu Song (xu-song)		0	1/0/1	0	0	0
None (Katehuuh)		0	1/0/1	0	0	0
None (liu-zichen)		0	1/1/0	0	0	0
None (BUAADreamer)		0	1/0/0	0	0	0
Saken Tsukenofu (sakentsunofu)		0	1/0/0	0	0	0
kevinpro (Ricardokevins)		0	1/0/1	0	0	0
Louis Brulé Naudet (louisbrulenaudet)		0	0/0/1	0	0	0

_{PRs: created by that dev and opened/merged/closed-unmerged during the period}

Report On: Fetch issues

Based on the provided information and the context of the issues, here is a detailed analysis of the open issues for the software project "LLaMA-Factory":

Notable Problems:

1. Issue #3398: The issue reports an out-of-memory error during model evaluation. This indicates that the evaluation process is consuming more memory than available, which could be due to large batch sizes or model sizes. It's critical to address this to prevent crashes and ensure smooth evaluations.

2. Issue #3397 and #3396: These issues involve problems with model loading after fine-tuning, which could indicate inconsistencies in how models are saved or restored, especially concerning different fine-tuning methods like full-parameter fine-tuning and LoRA.

3. Issue #3395: The user is experiencing difficulties understanding how to properly train with DPO (Dynamic Prompt Optimization). The lack of clarity in the README about merging LoRA weights and inference procedures suggests that documentation may need to be improved for better user guidance.

4. Issue #3394: This issue is about adding support for multimodal LLM finetuning, which is currently a work in progress. Multimodal capabilities are significant as they expand the use cases of LLMs beyond text-based tasks.

5. Issue #3393: The user is asking for examples related to longlora and streaming llm support, indicating a need for more comprehensive examples or documentation for these features.

6. Issue #3392: The user requests support for num_beams>1 during SFT prediction, which implies a need for enhanced generation capabilities during inference.

7. Issue #3391: There seems to be an issue with identity adaptation not reflecting in model responses, suggesting potential problems with adapter training or merging.

8. Issue #3389: The user is asking about implementing a custom tool for outputting weather information, indicating a desire for extending functionality through custom tool integration.

9. Issue #3388 and #3387: These issues report errors related to file handling and parameter passing during inference, which could affect usability and robustness.

10. Issue #3386: A user reports increased memory consumption when training llama3 compared to qwen14b, suggesting possible inefficiencies or bugs in memory management for different models.

11. Issue #3385: An export error during llamafactory finetuning indicates potential issues with serialization or tensor sharing that need to be addressed.

12. Issue #3384: A user reports that training gets stuck after a few iterations, which could indicate deadlocks or inefficiencies in the training loop.

13. Issue #3382 and #3381: Users are reporting errors when using Llama3-8B-Chinese-Chat, pointing towards potential compatibility issues or bugs with specific models.

14. Issue #3374: A user suggests that shuffle should be written earlier in code, indicating potential improvements in data processing logic.

15. Issue #3373: A user reports that using lora sft sometimes results in non-stopping model outputs during beam search inference, suggesting possible issues with stopping criteria or beam search implementation.

16. Issue #3371: A user has initiated a Russian README file but it's still under development, indicating ongoing efforts to make documentation more accessible to non-English speakers.

17. Issue #3370: This issue discusses potential problems with model format and extended training length when merging LoRA adapters, suggesting complexities in managing training states across different configurations.

18. Issue #3361: A user reports an inability to perform inference with LLaMA3-8B, pointing towards possible issues with model compatibility or setup procedures.

19. Issue #3359: A user experiences access denial when saving files during training, which could be related to system permissions or resource locks on the storage medium.

20. Issue #3353: A report of loss being zero during lora sft training indicates either a bug in loss computation or an issue with the training setup that needs investigation.

21. Issue #3347: Users report errors related to data type mismatches when using vllm inference with bf16 precision, suggesting potential issues with datatype handling during model execution.

22. Issue #3344: Users encounter CUDA errors when using dbrx-base and dbrx-instruct models for inference, indicating potential compatibility issues with specific hardware or software configurations.

Uncertainties and TODOs:

• Several users have reported issues related to specific models (e.g., Llama3-8B-Chinese-Chat) that need further investigation to determine whether the problems are due to user configuration errors or inherent bugs within the software.
• The documentation may need updates to clarify processes such as DPO training and inference.
• There are ongoing efforts to add support for multimodal LLM finetuning (e.g., LLaVA and Instruct-BLIP), which are currently listed as TODO items.
• Some users have requested additional examples or detailed steps for features like longlora and streaming llm support.
• There is a need for better error handling and messaging when users encounter issues such as CUDA errors or file access denials during training.

Anomalies:

• Issue #3370 discusses problems related to merging LoRA weights but also includes comments about unrelated topics such as web UI usage.
• Issue #3361 combines discussions about inference issues with LLaMA3-8B alongside general questions about other unrelated topics.
• Issue #3359 mixes discussions about file access denials during training with comments about transferring files between Linux and Windows systems.

General Context and Trends:

• There is a trend of users encountering compatibility issues when using specific models (e.g., Llama3) or finetuning methods (e.g., LoRA).
• Users frequently request clarification on how to use certain features or resolve specific errors they encounter.
• There are ongoing efforts to improve documentation and add support for new features such as multimodal LLM finetuning.
• Some users are contributing by initiating translations of README files into other languages like Russian.

Recently Closed Issues:

The recently closed issues indicate active maintenance and responsiveness from maintainers in addressing reported bugs and feature requests. For instance:

• Issue #3338 was closed after providing instructions on how to normalize port numbers in URL parsing.
• Issue #3337 was closed by adding support for Llama3 models.
• Issue #3336 was closed by fixing import errors related to gradio dependency.

Report On: Fetch pull requests

Analysis of Pull Requests for the hiyouga/LLaMA-Factory Repository

Open Pull Requests

PR #3394: Add Multimodal LLM Finetuning

• Status: Open and marked as work in progress.
• Notable Issues: The PR lacks commit timestamps, which makes it difficult to track the progress and recency of changes. It is also missing completion of key tasks such as adding LLaVA and Instruct-BLIP support.
• Files Changed: Multiple files have been added or modified, indicating a significant update to the project.
• Action Required: Monitor progress and await completion of TODO items before considering a merge.

PR #3371: Russian Readme file init

• Status: Open, created 1 day ago.
• Notable Issues: None observed. The PR adds a Russian translation of the README file, which can help widen the project's audience.
• Files Changed: A single file (README_ru.md) has been added.
• Action Required: Review for accuracy in translation and consider merging to support Russian-speaking users.

PR #3103: [FEATURE: ADD LISA ALGORITHM]

• Status: Open, created 21 days ago.
• Notable Issues: Several comments indicate issues when combining LISA with multiple GPUs, Zero3, and gradient checkpointing. There are also comments about high VRAM consumption and memory spikes during fine-tuning.
• Files Changed: Four files have been added or modified, suggesting a moderate update focused on adding the LISA algorithm.
• Action Required: Address reported issues and test thoroughly before merging.

PR #3004: Supports custom data set sampling quantity

• Status: Open, created 27 days ago.
• Notable Issues: None observed. The PR aims to support custom dataset sample quantities, which could be beneficial for training with self-awareness data.
• Files Changed: Four files have been modified, indicating changes related to dataset handling.
• Action Required: Review changes for potential impacts on existing dataset handling and consider merging if beneficial.

PR #2845: Add dependency environment check and add script to quick start the web UI

• Status: Open, created 40 days ago.
• Notable Issues: None observed. The PR adds an environment check script and a quick-start script for the web UI.
• Files Changed: Two new files have been added, indicating utility enhancements for the project.
• Action Required: Review scripts for functionality and merge if they provide value to users.

PR #1624: Support separate value model

• Status: Open, created 151 days ago.
• Notable Issues: None observed. The PR seems to add support for a separate value model based on an issue link provided in the description.
• Files Changed: Three files have been modified, suggesting changes related to fine-tuning arguments and PPO training workflow.
• Action Required: Review the necessity of this feature and its impact on the current workflow before considering a merge.

PR #844: add Fine-tuning method: AdaLoRA

• Status: Open, created 228 days ago.
• Notable Issues: Comments suggest that AdaLoRA may perform better in some tasks but not significantly better than normal LoRA. This indicates that while it may be an enhancement, it's not a critical update.
• Files Changed: Two files have been modified, indicating changes related to fine-tuning arguments and tuner core adapter logic.
• Action Required: Evaluate the performance benefits of AdaLoRA compared to existing methods before deciding on a merge.

Closed Pull Requests

PR #3383: Update requirements.txt

Closed without being merged due to being a duplicate of another pull request (#2445). This indicates good housekeeping by avoiding redundant updates.

PR #3357: Dev

Closed after several updates by the contributor. It appears that this was more of a development branch rather than a specific feature or bug fix.

PR #3338: Adding Mixture of Depth

Merged successfully after review. This indicates that new features are being added to the project after thorough review.

PR #3332: Fix --max_samples error in streaming mode

Closed after addressing an error related to --max_samples in streaming mode. This indicates responsiveness to fixing bugs that affect functionality.

PR #3291: support for previewing custom dataset in directory format

Merged successfully after review. This indicates enhancements to dataset handling within the project.

PR #3288: fix MoD related stuff

Closed without being merged due to unresolved issues with MoD (Mixture of Depths) implementation. This indicates ongoing challenges with integrating new algorithms.

PR #3287: [Feature] Add BAdam algorithm

Merged successfully after review and necessary changes. This indicates that performance-enhancing algorithms are being considered for inclusion in the project.

PR #3276: fix: turn on output_router_logits of mixtral

Merged successfully after review. This indicates attention to detail in ensuring model configurations are correct.

PR #3275: Update dataset the-stack-v2

Closed without being merged due to direct loading issues with the dataset. This indicates attentiveness to dataset usability within the project.

PR #3267: add eval example for rouge and bleu

Closed without being merged after contributor acknowledgment. It appears there was no need for these changes at this time.

PR #3263: Adding support for Mixture of Depths Training

Closed without being merged due to unresolved issues with training using MoD. This suggests that complex features may require additional development time before they are ready for integration.

PR #3261: Added specimens for single-card full parameter prediction

Merged successfully after review. This suggests improvements in prediction capabilities for single-card setups.

PR #3256: Update README.md

Closed without being merged as it appears there was no significant change required or it was addressed elsewhere.

PR #3254: Add template&support for c4ai-command-r/plus (tested)

Merged successfully after review and necessary changes. This suggests that templates supporting specific models or use cases are being added to enhance usability.

Summary

The analysis shows active development within the hiyouga/LLaMA-Factory repository with several open pull requests focused on adding new features such as multimodal fine-tuning (#3394), new algorithms like LISA (#3103), and utility scripts (#2845). Closed pull requests indicate responsiveness to community contributions, with several merges introducing new algorithms (e.g., BAdam in #3287) and enhancements (e.g., custom dataset previewing in #3291).

Key Observations: 1. Some pull requests remain open for extended periods (e.g., AdaLoRA in #844), suggesting either complexity in integration or lower priority compared to other updates. 2. Closed pull requests without merges often lack necessary information or face unresolved issues (e.g., MoD-related problems in #3263). 3. Merged pull requests show careful consideration by maintainers through reviews and discussions before integration into the main branch (e.g., Mixture of Depth addition in #3338).

Recommendations: 1. Prioritize resolving open pull requests with notable issues such as high VRAM consumption during fine-tuning (#3103). 2. Consider closing long-standing open pull requests if they are no longer relevant or if alternative solutions have been implemented (e.g., AdaLoRA in #844). 3. Continue monitoring newly opened pull requests closely (e.g., Russian Readme file init in #3371) for timely integration into the project after thorough review.

Report On: Fetch commits

LLaMA Factory Project Report

Project Overview

The LLaMA Factory is a software project that provides a unified framework for the efficient fine-tuning of over 100 different large language models (LLMs). It is hosted on GitHub under the repository hiyouga/LLaMA-Factory and is licensed under the Apache License 2.0. The project was created on May 28, 2023, and has seen active development since then. As of the last update, the repository has amassed a significant following with 17,803 stars and 2,149 forks.

The LLaMA Factory offers a wide range of features for model training and fine-tuning, including support for various models such as LLaMA, Mistral, Mixtral-MoE, Qwen, Yi, Gemma, Baichuan, ChatGLM, Phi, and others. It integrates methods like pre-training, supervised fine-tuning, reward modeling, PPO, DPO, and ORPO. The project also provides scalable resources for different precision levels during training and advanced algorithms like GaLore and BAdam.

The project's README file provides comprehensive documentation on getting started with the software, including links to Colab notebooks for easy access to GPU resources for training. It also includes benchmarks showcasing the performance improvements offered by the LLaMA Factory's tuning methods compared to other approaches.

Development Team Activity

Recent Commits (Reverse Chronological Order)

• hoshi-hiyouga:

  • Updated examples and README files.
  • Added support for various features and fixed issues related to model training and inference.
  • Released version updates (v0.6.3).
  • Addressed GitHub issues and pull requests.

• mlinmg:

  • Contributed to fixing issues with dataset downloads.

• Ledzy:

  • Added the BAdam algorithm feature.

• codemayq:

  • Improved Docker support for the project.

• marko1616:

  • Fixed issues related to model saving during full parameter training.

• liuzc:

  • Fixed an issue with Mixtral output router logits.

Pull Requests (PRs)

• mlinmg: Opened PRs related to dataset download configurations; one was merged.
• Ledzy: Opened a PR adding the BAdam algorithm which was merged.
• codemayq: Opened PRs improving Docker support; one was merged.
• marko1616: Opened a PR fixing model saving which was merged.
• Other contributors opened or commented on PRs related to specific features or fixes.

Patterns and Conclusions

The development team is actively maintaining the project with frequent commits addressing new features, bug fixes, performance improvements, and user-reported issues. The contributions are mainly from a core group of developers with occasional contributions from the community.

The team seems responsive to community feedback as evidenced by their interaction with issues and pull requests. There is a focus on ensuring compatibility across different environments (e.g., Docker support) and improving user experience (e.g., updating examples and documentation).

Given the high level of activity and responsiveness of the development team, it can be concluded that the LLaMA Factory project is in a healthy state with ongoing efforts to enhance its capabilities and usability.

Note: The detailed commit messages have been omitted for brevity but can be provided upon request.

Report On: Fetch Files For Assessment

The provided source code files from the LLaMA-Factory repository are part of a larger framework designed for efficient fine-tuning of large language models (LLMs). Here's an analysis of each file based on its structure, quality, and potential areas for improvement:

1. src/llmtuner/extras/constants.py

Purpose: This file contains constants and configurations for different models, providing insights into the model's architecture and settings.

Analysis:

• Pros: Centralizing constants in one file simplifies maintenance and updates. It also helps in managing multiple model configurations efficiently.
• Cons: The file can grow significantly as more models or configurations are added, making it harder to manage. Using a more dynamic configuration management system or splitting constants by model type could improve scalability.

2. src/llmtuner/data/template.py

Purpose: Manages templates for data formatting, crucial for understanding how data is processed and used within the project.

Analysis:

• Pros: Using templates for data formatting ensures consistency across different parts of the application and simplifies changes to data presentation logic.
• Cons: Hardcoding templates can reduce flexibility. It might be beneficial to allow runtime modifications of templates or load them from an external source to enhance flexibility.

3. src/llmtuner/train/utils.py

Purpose: Includes utility functions for training, revealing important mechanisms or optimizations used in the training process.

Analysis:

• Pros: Utility functions help encapsulate logic making the main training code cleaner and more maintainable. Functions like creating custom optimizers or schedulers are essential for flexible training regimes.
• Cons: The file seems to mix different levels of abstraction (e.g., optimizer creation and API response formatting). Splitting these into more focused modules could improve clarity and reusability.

4. src/llmtuner/model/loader.py

Purpose: Responsible for loading models, providing details on how different models are integrated and managed within the framework.

Analysis:

• Pros: Centralizing model loading logic in one place simplifies the management of different model architectures and their specific configurations.
• Cons: The function load_model appears to be doing too much, handling multiple conditional paths for different configurations. Refactoring to separate these concerns into smaller, more focused functions would enhance readability and maintainability.

5. src/llmtuner/api/app.py

Purpose: Defines the API structure and endpoints, useful for understanding how external interactions are handled.

Analysis:

• Pros: The API endpoints are clearly defined, and use of FastAPI provides automatic swagger documentation which is excellent for API usability.
• Cons: The error handling could be more comprehensive to cover more edge cases, ensuring that the API is robust against various types of input errors or failures.

General Observations

• Code Quality: The code is generally well-structured with clear separation of concerns in most parts. However, some files could benefit from further modularization.
• Documentation: There is a lack of inline comments explaining complex logic or decisions which could make the codebase more accessible to new developers or contributors.
• Testing: There is no direct evidence from the provided snippets about testing practices. Unit tests, integration tests, and other automated tests are crucial for maintaining large codebases especially dealing with complex models like LLMs.

Recommendations

1. Refactor Large Functions: Break down large functions into smaller ones that handle a single responsibility.
2. Improve Configuration Management: Consider using a more dynamic approach for managing configurations especially when dealing with multiple models.
3. Enhance Documentation: Add more inline comments and update documentation to reflect changes in logic or architecture.
4. Strengthen Testing Practices: Establish a robust testing framework that covers unit tests, integration tests, and performance tests to ensure reliability and performance.

Overall, the project demonstrates good software engineering practices but could benefit from some refinements to handle complexity as it scales.