Quantify commits

Quantified Commit Activity Over 14 Days

Developer	Avatar	Branches	PRs	Commits	Files	Changes
Nalaso		1	2/2/0	2	46	1334
ayush rajgor		1	1/1/0	15	37	893
Ahmet0691		1	0/1/0	1	2	19
Wickes Wong		1	1/1/0	1	4	17
Younes DARRASSI		1	2/1/0	1	1	12
FroDK		1	1/1/0	1	1	3
Henri Jamet (hjamet)		0	1/0/0	0	0	0
Ihab P (ITP2023)		0	1/0/0	0	0	0
Dobromir Popov (d-popov)		0	0/0/1	0	0	0
Cheifsam (hassamc)		0	1/0/0	0	0	0
Ikko Eltociear Ashimine (eltociear)		0	1/0/0	0	0	0
Suvodeep Chowdhury (ImSuvodeep)		0	0/0/2	0	0	0
Rawknee.69 (Rawknee-69)		0	4/0/3	0	0	0
ViscousGuy (ViscousGuy)		0	0/0/1	0	0	0
AMJID DADER (amjiddader)		0	1/0/0	0	0	0
Jerome Sonnet (letzdoo) (letzdoo-js)		0	1/0/1	0	0	0
Dmitry Vasilev (scythargon)		0	1/0/1	0	0	0
Kesavan (Nallandavar)		0	1/0/1	0	0	0
Robert Paulson (PlanetMacro)		0	0/0/1	0	0	0
None (prathyushnallamothu)		0	0/0/1	0	0	0

_{PRs: created by that dev and opened/merged/closed-unmerged during the period}

Report On: Fetch issues

Analysis of Open GitHub Issues for stitionai/devika

Notable Open Issues

Issue #512: SOCKET inference error due to incorrect API key

• Problem: Incorrect API key provided for OpenAI, causing a SOCKET inference error.
• Notability: This is a critical issue as it prevents any interaction with the OpenAI platform, which is essential for the project's functionality.
• TODO: Validate the API key and ensure it's correctly set up in the configuration.

Issue #511: Devika encounters an error and stops

• Problem: An exception in thread Thread-30 causes Devika to stop functioning.
• Notability: This issue is notable due to its impact on the stability of Devika. It seems to be related to thread management and execution flow within the application.
• TODO: Investigate the stack trace provided and identify the root cause of the exception.

Issue #510: Add hide API keys flag

• Problem: Request to add an option to hide the settings/API keys section from the UI.
• Notability: This is a security-related request that would help prevent accidental exposure of sensitive API keys.
• TODO: Implement a feature to toggle visibility of API keys in the UI.

Issue #507: Feature request for model retry

• Problem: The agent sometimes stops and doesn't retry the prompt.
• Notability: This feature request is notable as it addresses user experience and reliability of Devika when handling tasks.
• TODO: Consider adding a retry mechanism in the UI for failed tasks.

Issue #506: Feature request for Claude 3 tool use

• Problem: Suggestion to implement Claude 3's tool use (function calling) within Devika.
• Notability: This feature could significantly enhance Devika's functionality by enabling more autonomous operations based on prompt goals.
• TODO: Assess the feasibility of integrating Claude 3 function calling into Devika.

Issue #503: Experiment with advanced planning agents

• Problem: Proposal to experiment with more long-term stable planning agents, referencing specific research papers.
• Notability: This issue is notable as it suggests ways to improve the planning capabilities of agents within Devika, potentially leading to more sophisticated behavior.
• TODO: Review the suggested approaches and consider their integration into Devika's planning agents.

Issue #502: Inference shouldn't have a timeout

• Problem: Current timeout settings are causing issues, especially with larger local models on OLLAMA.
• Notability: This issue is significant because it affects the usability of Devika with certain setups, rendering it ineffective for some users.
• TODO: Reevaluate and adjust timeout settings for inference processes.

Issue #501: Learning resources for understanding Devika's codebase

• Problem: A user has created a playlist dedicated to learning about Devika's backend operations.
• Notability: This issue is notable as it provides valuable resources for new contributors or users who want to understand how Devika works behind the scenes.
• TODO: Possibly integrate these learning resources into official documentation or contributor guides.

Issue #500: Devika does not search in Google

• Problem: User reports that Devika fails to search on Google despite all APIs being registered.
• Notability: This issue is significant as it affects one of the core functionalities of Devika - using search engines to gather information for tasks.
• TODO: Investigate why searches are failing and resolve any issues with search engine integration.

Issue #499: Failed to connect to server

• Problem: Users are unable to connect to the server through the UI, receiving "failed to connect server" messages.
• Notability: This issue is critical as it prevents users from interacting with Devika entirely.
• TODO: Troubleshoot connection issues and ensure that server communication is functioning correctly.

Closed Issues

The closed issues provide context on recent trends and fixes. Notably, issues related to Docker (#492), API key input (#498), and CORS blocking requests (#423) have been resolved. These resolutions indicate active maintenance and responsiveness to community feedback. However, they do not directly impact the current state of open issues.

Summary

The open issues highlight several areas requiring attention:

1. Security concerns regarding API key exposure (#510).
2. Reliability and error handling improvements (#511, #507).
3. Feature enhancements for better functionality (#506).
4. Development support and learning resources (#501).
5. Core functionality issues with search engines and server connectivity (#500, #499).

The project maintainers should prioritize resolving these issues, particularly those affecting core functionalities like search engine integration and server connectivity, as they directly impact user experience. Additionally, implementing security features like hiding API keys and enhancing agent reliability will contribute positively to user trust and overall project robustness.

Report On: Fetch pull requests

Analysis of Pull Requests for stitionai/devika

Open Pull Requests

Notable Issues

• PR #491: This PR seems to have a significant amount of discussion and edits, indicating a complex change. There are concerns about the necessity of certain features, the understanding of how offline models work, and the diversity settings. The conversation indicates that there might be some misunderstandings about the capabilities of GPT-4 and other models. It's important to resolve these issues before considering merging.

• PR #485: This PR is related to updating the knowledge base using SentenceTransformer and FAISS. It appears to be an enhancement to improve search capabilities within the knowledge base.

• PR #480: This PR addresses a typo in the Netlify deployment script. Although it seems like a simple fix, it's notable that this PR was mentioned in a comment on another PR (#483), indicating that it might have been closed without being merged due to an oversight.

• PR #475: The changes in this PR aim to improve error handling and JSON parsing in devika.py and runner.py. These improvements could be significant for the robustness of the software.

• PR #469: This PR introduces regex parsing for LLM responses and a new parent class for agents, which could significantly simplify code and improve response handling from LLMs.

Recently Closed Pull Requests

Notable Resolutions

• PR #508: Closed without being merged. It was a new addition with GitHub Actions configuration, but it seems like it was closed immediately after creation, possibly due to being created by mistake or not being ready for review.

• PR #490 & PR #483: Both were closed without being merged. They appear to be duplicates or related to PR #491, which is still open and under discussion.

• PR #457: Merged recently. It addressed a Python version requirement issue, which is important for ensuring compatibility with the codebase.

• PR #452: Merged recently. It addressed security concerns by removing API keys from logs and fixed an issue with link handling in the UI.

• PR #442: Merged recently. It added support for custom OpenAI API endpoints, which can be significant for users who need to configure custom endpoints for their OpenAI services.

General Observations

• There are several PRs that address minor fixes or enhancements, such as documentation updates (e.g., PR #505) and typo corrections (e.g., PR #465).

• Some PRs seem to have been closed without being merged due to duplication or being superseded by other PRs (e.g., PR #437).

• There is active discussion and review on many PRs, indicating a healthy collaborative environment among contributors.

Recommendations

1. For PRs with extensive discussions like PR #491, it's crucial to ensure that all concerns are addressed before merging. A thorough review by maintainers or domain experts is recommended.

2. For simple fixes that have been closed without merging (like PR #480), verify if the changes were indeed incorporated into the codebase through other means or if they need to be reopened and reviewed again.

3. Closed PRs that have been merged should be checked for any follow-up actions required post-merge, such as updating documentation or notifying users about new features or fixes.

4. Open PRs with enhancements like PR #469 should be prioritized for review as they can bring significant improvements to the project's functionality.

5. Monitor for duplicate or redundant PRs and encourage contributors to check existing PRs before opening new ones to avoid confusion and streamline the review process.

6. Ensure that all merged changes are properly documented and communicated to users, especially when they involve changes in requirements or introduce new features.

Report On: Fetch commits

Devika Project Analysis Report

Project Overview

Devika is an Agentic AI Software Engineer designed to understand high-level human instructions, decompose them into actionable steps, conduct relevant research, and write code to fulfill specified objectives. It is an open-source project maintained by the organization stitionai and serves as a competitive alternative to Devin by Cognition AI. The project is in its early experimental stages with many features yet to be implemented or currently broken. Devika supports various large language models and offers advanced AI planning, reasoning capabilities, web browsing for information gathering, and code writing in multiple languages. It is licensed under the MIT License and has garnered significant interest with 16,593 stars on GitHub.

Development Team and Recent Activities

Ayush Rajgor (ARajgor)

• Recent Commits: 15 commits with 893 changes across 37 files.
• Files Worked On: devika.py, src/agents/*.py, src/llm/llm.py, src/state.py, ui/*, among others.
• Collaborations: Co-authored commits with Андрей Васильев and FroDK.
• Patterns & Conclusions: ARajgor has been actively working on improving error handling, socket management, agent state management, and UI enhancements. They are also responsible for updating dependencies and fixing issues related to configuration file key checking.

Nalaso (nalaso)

• Recent Commits: 2 commits with 1334 changes across 46 files.
• Files Worked On: Various UI components (ui/src/lib/components/*.svelte), ui/package.json, ui/bun.lockb, among others.
• Collaborations: Co-authored commits with Kartik, ARajgor, and Ayush Rajgor.
• Patterns & Conclusions: Nalaso's recent work revolves around a major overhaul of the UI components and configuration files. They have been involved in cleaning up the UI codebase and adding new UI components.

Younes DARRASSI (darrassi1)

• Recent Commits: 1 commit with 12 changes in ui/src/lib/components/MessageContainer.svelte.
• Patterns & Conclusions: Darrassi1's recent activity indicates a focus on UI improvements, specifically enhancing the message container component.

FroDK

• Recent Commits: 1 commit with 3 changes in src/config.py.
• Collaborations: Co-authored a commit with Андрей Васильев.
• Patterns & Conclusions: FroDK's recent work includes fixing an unsupported operation issue within the configuration file.

Ahmet0691

• Recent Commits: 1 commit with 19 changes across 2 files (src/agents/feature/feature.py and src/agents/patcher/patcher.py).
• Patterns & Conclusions: Ahmet0691 has contributed to solving problems caused by version incompatibility in patching functionality.

Wickes Wong (wickes1)

• Recent Commits: 1 commit with 17 changes across 4 files including README.md, sample.config.toml, src/config.py, and src/llm/openai_client.py.
• Patterns & Conclusions: Wickes Wong has added custom API endpoints for OpenAI to the project configuration, indicating work on extensibility for different AI model integrations.

Patterns and Conclusions

The development team's recent activities show a concerted effort towards enhancing Devika's stability, user interface, and configurability. There is a strong emphasis on improving error handling mechanisms and refining the user experience through UI updates. The team is also focused on expanding support for various language models and ensuring compatibility across different versions of dependencies. Collaboration among team members is evident from co-authored commits, suggesting a cooperative development environment.

Note: The above analysis is based on the provided repository information up to April 2024. It does not include any activities that may have occurred outside of the GitHub repository or after the knowledge cutoff date.

Report On: Fetch Files For Assessment

Analysis of Source Code Files

1. src/agents/runner/runner.py

• Purpose: Manages the execution of commands in a project environment, including handling retries and errors.
• Structure: The class Runner is well-structured with methods for rendering templates, validating responses, running commands, and handling retries. It uses Jinja2 for templating which is a good choice for dynamic content generation.
• Quality:
  • Positive: Use of subprocess for command execution is appropriate. The retry mechanism is a thoughtful addition for handling command failures.
  • Improvement Areas: Error handling could be more robust. The method validate_response and validate_rerunner_response could be refactored to reduce duplication. The use of print statements for debugging should be replaced with a logging framework for better production-level diagnostics.

2. src/config.py

• Purpose: Manages configuration settings for the application, ensuring that all necessary configurations are loaded and available.
• Structure: Singleton pattern is used to ensure only one instance of the configuration is active using the _instance variable, which is a classic approach in configuration management.
• Quality:
  • Positive: Checks and loads missing keys from a sample configuration which ensures the application's robustness against incomplete configuration files.
  • Improvement Areas: The class could benefit from more comments explaining the purpose of each method, especially around the dynamic setting and updating of configuration values.

3. ui/src/lib/components/ControlPanel.svelte

• Purpose: Provides a UI component for managing projects, models, and search engines within the Devika application.
• Structure: The Svelte component is well-organized with reactive statements and modular functions handling UI interactions like selecting projects or models.
• Quality:
  • Positive: Good use of Svelte's reactivity model to update UI elements based on user interactions. Componentization of UI elements like buttons and dropdowns enhances readability.
  • Improvement Areas: Inline styles could be extracted to separate CSS files or blocks for better maintainability. Some functions are quite large and could be broken down further.

4. src/llm/llm.py

• Purpose: Manages interactions with different language models and handles inference requests.
• Structure: The class LLM encapsulates model interactions and token management effectively, providing a centralized point for model management.
• Quality:
  • Positive: The separation of model configurations into a dictionary is clean, allowing easy additions or modifications of models. Usage of concurrent futures for handling model inference asynchronously is a strong implementation detail.
  • Improvement Areas: Exception handling could be more granular to differentiate between different types of failures. The method inference is quite long and could benefit from being split into smaller sub-methods.

5. src/browser/search.py

• Purpose: Handles web search functionality using different search engines like Bing and Google.
• Structure: Classes for each search engine (BingSearch, GoogleSearch) encapsulate specific API interactions cleanly.
• Quality:
  • Positive: Each class provides a clear interface for performing searches and extracting results. Use of requests library for HTTP calls is appropriate.
  • Improvement Areas: Error handling could be improved by not just returning errors but also managing them appropriately (e.g., retry mechanisms). Some commented-out code should be removed if it's not intended to be used.

General Observations

The codebase demonstrates a good understanding of object-oriented principles and software design patterns. However, there are areas where error handling and logging could be improved for better maintainability and reliability. Additionally, some parts of the codebase show signs of potential refactorings to improve modularity and readability.