Khoj is an open-source, self-hostable AI application designed to serve as a "second brain" for users, enabling interaction with local and online language models. It supports various interfaces and offers features like semantic search and image generation. The project is actively maintained under the GNU Affero General Public License v3.0, with significant community interest reflected in its repository statistics. Currently, the project is focused on enhancing user experience, expanding capabilities, and addressing self-hosting challenges.
Self-Hosting Challenges: Users face difficulties with Docker and Kubernetes configurations.
Compatibility Issues: Installation problems on macOS and Windows are prevalent.
Feature Expansion: Efforts to integrate more LLMs and improve document summarization.
Security Enhancements: Recent PRs focus on increasing security measures.
Active Development: Frequent updates and enhancements indicate a rapid development cycle.
Recent Activity
Team Members
Debanjum
Saba Imran
Recent Activities (Reverse Chronological Order)
Debanjum
Released multiple versions (1.28.0 to 1.28.3) with improvements and bug fixes.
Enhanced web app agent management features.
Implemented prompt tracing for debugging.
Improved UI elements in chat interfaces.
Saba Imran
Released versions (1.27.0 to 1.28.3) focusing on stability.
Improved subscription management logic.
Enhanced document conversion functionalities in chats.
Developed advanced reasoning capabilities in research mode.
Patterns and Themes
Active enhancement of user experience and performance evaluation.
Strong emphasis on advanced reasoning features and research mode improvements.
Frequent updates indicate a rapid development cycle.
Risks
Self-Hosting Issues: Persistent configuration errors with Docker/Kubernetes could hinder user adoption.
Compatibility Problems: Recurring installation issues on macOS/Windows may limit accessibility.
Telemetry Concerns: Default-enabled telemetry without clear consent could raise privacy issues.
Of Note
Community Engagement: High level of community interest as indicated by stars and forks, suggesting robust user engagement.
Security Focus: Recent PRs emphasize security improvements, showcasing a proactive approach to potential vulnerabilities.
Rapid Development Cycle: Frequent updates and active contributions reflect a dynamic and evolving project trajectory.
Quantified Reports
Quantify issues
Recent GitHub Issues Activity
Timespan
Opened
Closed
Comments
Labeled
Milestones
7 Days
3
1
8
0
1
30 Days
9
8
18
0
1
90 Days
29
40
72
0
1
1 Year
150
136
501
28
1
All Time
444
399
-
-
-
Like all software activity quantification, these numbers are imperfect but sometimes useful. Comments, Labels, and Milestones refer to those issues opened in the timespan in question.
Rate pull requests
4/5
The pull request addresses a critical security concern by preventing CRUD operations on entries, files, and conversations in the database when the user is null. This change enhances the application's defense-in-depth strategy, which is significant for maintaining data integrity and security. The PR includes comprehensive changes across multiple files and introduces a new function to validate user presence, ensuring robustness. However, while the changes are thorough, they primarily focus on one aspect of security without broader application improvements or additional features, which limits it from being rated as exemplary.
[+] Read More
4/5
The pull request introduces a significant feature by enabling file attachments in chat messages, leveraging larger context windows in models. It includes comprehensive changes across multiple files and methods, ensuring the attached files are processed correctly through the backend. The PR also addresses potential UI improvements and ensures backward compatibility by adjusting the document processing flow. However, it breaks some prior behaviors, requiring users to manually upload documents for embedding, which might inconvenience some users. Overall, the PR is well-executed but could benefit from more detailed documentation on the new workflow for users.
[+] Read More
4/5
The pull request introduces a significant feature by allowing user files to be analyzed in the code sandbox, which enhances the functionality of the application. It also simplifies the logic by running one program at a time, improving maintainability. The changes are well-documented and include useful examples in the prompt updates. However, while the changes are beneficial, they are not groundbreaking or exceptionally innovative, which is why it doesn't merit a perfect score.
PRs: created by that dev and opened/merged/closed-unmerged during the period
Quantify risks
Project Risk Ratings
Risk
Level (1-5)
Rationale
Delivery
3
The project faces moderate delivery risks due to several factors. The backlog of 45 open issues, including critical bugs like the online search authentication error (#962) and Docker API key configuration issue (#961), could impact delivery timelines if not addressed promptly. Self-hosting challenges, particularly with Docker and Kubernetes configurations, pose significant risks as they can delay deployment and affect user satisfaction. Additionally, compatibility issues across different operating systems exacerbate these challenges. While there is strong engagement with community feedback and continuous improvement efforts, the recurrence of similar issues indicates possible gaps in testing or documentation processes.
Velocity
3
The project's velocity shows both strengths and weaknesses. High developer activity, especially from contributors like Sabaimran and Debanjum, suggests a strong development momentum. However, the sheer volume of changes raises concerns about code quality and technical debt if not adequately reviewed and tested. The balance between opened and merged pull requests indicates a healthy review process, but the presence of unmerged PRs might require attention to avoid potential bottlenecks. Additionally, the need for further validation and testing of complex features could slow down integration, impacting overall velocity.
Dependency
4
Dependency risks are relatively high due to several factors. The project's reliance on external libraries and models, such as those used for file handling in the code sandbox (PR #960) and speech-to-text capabilities, introduces potential risks if these dependencies are not robustly implemented or become deprecated. Deployment challenges on platforms like Docker and Windows further highlight dependency risks that need to be mitigated to ensure smooth operations across different environments. The integration with external APIs (e.g., OpenAI, Anthropic) also poses risks if there are changes in these APIs that could affect system functionality.
Team
3
The team shows strong engagement with community feedback and continuous improvement efforts. However, there are potential risks related to team dynamics and resource allocation. The minimal contribution from some team members, like MaxData, could impact team dynamics if not aligned with project needs. Additionally, the false reporting of issues and pull requests suggests potential gaps in communication or documentation within the team, which could impact the team's ability to address real problems effectively.
Code Quality
3
Code quality is generally good but faces some risks due to the high volume of changes being made rapidly. The use of modern frameworks like FastAPI for backend services and React for front-end components indicates a focus on maintaining code quality. However, the complexity of asynchronous operations and extensive use of external libraries introduces potential challenges in error handling and debugging. The lack of detailed documentation for some changes could lead to overlooked errors or insufficient test coverage.
Technical Debt
4
Technical debt is a significant concern due to the complexity of recent changes and rapid iteration cycles. The introduction of new features and enhancements across multiple files increases the risk of accumulating technical debt if not managed carefully. The complexity of managing multiple states and asynchronous operations in components like 'chatMessage.tsx' could contribute to technical debt if not documented and maintained properly. Additionally, recurring issues suggest possible gaps in testing or documentation processes that could exacerbate technical debt over time.
Test Coverage
3
Test coverage is moderate but could be improved to ensure comprehensive validation of new features and prevent regressions. While there are efforts to maintain test coverage through updated tests in pull requests, the sheer volume of changes without detailed documentation or comprehensive testing could lead to overlooked errors or insufficient coverage. The need for further validation and testing of complex features highlights a risk in ensuring adequate test coverage.
Error Handling
3
Error handling is generally well-implemented but faces some risks due to the complexity of recent changes. The use of HTTP exceptions in API endpoints indicates a focus on error handling, but the lack of detailed logging for certain operations might hinder troubleshooting efforts. The complexity of asynchronous operations and threading increases the risk of concurrency-related bugs that could complicate error handling. Overall, while error handling practices are in place, there is room for improvement to ensure robust error management across all components.
Detailed Reports
Report On: Fetch issues
Recent Activity Analysis
Recent GitHub issue activity for the Khoj project shows a mix of bug reports, feature requests, and user inquiries. There is a notable focus on fixing bugs related to self-hosting, Docker configurations, and compatibility issues with various platforms and dependencies. Additionally, there are ongoing discussions about enhancing user experience and expanding the project's capabilities.
Anomalies and Themes
Self-Hosting Challenges: Several issues highlight difficulties users face when setting up Khoj in self-hosted environments, particularly with Docker and Kubernetes. Common problems include configuration errors, dependency conflicts, and network access issues.
Compatibility and Installation: Users report challenges with installing Khoj on different operating systems, especially macOS and Windows. Issues like missing dependencies (libGL.so.1), Python version incompatibilities, and installation errors are recurring themes.
Offline Model Issues: There are multiple reports of problems with offline models, such as Llama V2. Users encounter errors related to model loading, GPU utilization, and memory allocation.
User Interface Improvements: Requests for UI enhancements are frequent, including better error messages, improved chat interfaces, and support for additional file types (e.g., ODT files).
Telemetry Concerns: Some users express concerns about telemetry being enabled by default without clear documentation or consent prompts.
Feature Expansion: There is interest in expanding Khoj's capabilities, such as integrating more LLMs (e.g., Gemini), supporting additional languages (i18n), and improving document summarization features.
Issue Details
Most Recently Created Issues
#962: [FIX] Online Search Broken - Created 1 day ago by jakobd04
Priority: High
Status: Open
Description: Authentication error due to missing bearer token during online search.
#961: [FIX] Resend API key not found - Created 1 day ago by Quentin D.
Priority: Medium
Status: Open
Description: Docker setup issue related to missing Resend API key configuration.
Most Recently Updated Issues
#841: [IDEA] using function calling to integrate more tools - Edited 0 days ago
Priority: Low
Status: Open
Description: Suggestion to enhance agent capabilities by integrating additional tools via function calling.
Description: Resolved issue related to model configuration on a self-hosted Windows system.
Overall, the Khoj project is actively addressing user-reported issues while exploring new features to enhance its functionality as a versatile AI tool for personal knowledge management.
Report On: Fetch pull requests
Analysis of Pull Requests for Khoj Project
Open Pull Requests
PR #960: Enable Passing External Documents for Analysis in Code Sandbox
Overview: This PR introduces the ability to pass user files into the code sandbox for analysis, updates prompts with more complex code examples, and simplifies the logic for running programs.
Notable Aspects: This feature could significantly enhance user interaction by allowing more dynamic and complex code analysis. The simplification of logic to run one program at a time could improve system stability.
Potential Issues: None apparent from the description, but thorough testing will be necessary to ensure that the new input handling does not introduce security vulnerabilities.
PR #958: Do not CRUD on entries, files & conversations in DB for null user
Overview: Aims to increase security by preventing database operations when the user is unset.
Notable Aspects: This change enhances security by reducing potential attack vectors. It is labeled as a maintenance update, indicating its importance in fortifying the system.
Potential Issues: Careful review is needed to ensure that legitimate operations are not inadvertently blocked.
PR #957: Support including file attachments in the chat message
Overview: Allows full texts of certain files to be included in messages when a file filter is set, leveraging larger context windows of models.
Notable Aspects: This feature could greatly improve the context and relevance of responses by integrating more comprehensive data directly into conversations.
Potential Issues: The change breaks prior behaviors related to document processing and embedding generation, which may require users to adjust their workflows.
Notable Issue: This PR was closed without merging and appears to have been an accidental submission or script error. It involved significant deletions across various workflow and documentation files. The comment from sabaimran suggests it was unintended.
PR #955: Add Script to Evaluate Khoj on Google's FRAMES benchmark
Overview: Introduces an evaluation script using Google's FRAMES benchmark to measure performance shifts.
Significance: Provides a standardized method for assessing multi-step retrieval and reasoning capabilities, which is crucial for ongoing development and optimization.
PR #954: Research Mode [Part 2]: Improve Prompts, Edit Chat Messages. Set LLM Seed for Reproducibility
Overview: Enhances research mode with improved prompts, chat message editing capabilities, and reproducible LLM outputs through seed setting.
Significance: These improvements are likely to enhance user experience by providing more consistent outputs and better debugging tools.
PR #953 & #947: Edit Chat Messages from Web App. Set LLM Seed for Reproducibility / Separate notes, online context from user message sent to chat models
Not Merged: Both these PRs were closed without merging. They aimed at improving chat message handling and context separation but were likely superseded by other changes or merged through different branches.
General Observations
Security Enhancements: Several PRs focus on improving security and stability (e.g., #958), reflecting an ongoing effort to fortify the platform against potential vulnerabilities.
Feature Expansion: The project is actively expanding its capabilities with features like enhanced document handling (#960) and improved research mode (#954).
Community Engagement: The presence of multiple contributors and active discussions around PRs indicate a healthy community engagement.
Rapid Development Cycle: With numerous closed PRs in recent weeks, it's clear that Khoj is under active development with frequent updates and enhancements.
Unmerged PRs: Some PRs were closed without merging (#956, #953, #947), which might indicate either redundant submissions or issues identified during review that necessitated alternative solutions.
Overall, Khoj's development activity reflects its commitment to enhancing functionality while maintaining security and usability. The project continues to evolve rapidly with contributions from various developers, indicating robust community involvement.
Structure and Dependencies: The file is well-structured, adhering to the TOML format, and clearly delineates sections for build-system requirements, project metadata, dependencies, and optional dependencies.
Dependencies Management: The dependencies are specified with version constraints that ensure compatibility while allowing for minor updates. This is crucial for maintaining stability across environments.
Optional Dependencies: The inclusion of optional dependencies for production and development environments is a good practice, allowing for environment-specific installations.
Tool Configurations: Tools like mypy, black, and isort are configured, indicating a focus on code quality and consistency.
Scripts: The presence of scripts under [project.scripts] suggests automation capabilities within the project.
Purpose and Functionality: This script evaluates the Khoj system using the Google FRAMES benchmark. It loads datasets, sends prompts to the Khoj API, and evaluates responses using Google's Gemini model.
Code Quality: The code is modular with functions dedicated to specific tasks such as loading datasets, getting agent responses, and evaluating responses. This enhances readability and maintainability.
Logging: Comprehensive logging is implemented, which aids in debugging and monitoring the evaluation process.
Error Handling: There is basic error handling in place with try-except blocks, though it could be more granular to handle specific exceptions.
Concurrency: The use of concurrent.futures.ThreadPoolExecutor for batch processing indicates an effort to optimize performance by parallelizing API calls.
Functionality: This module handles conversations with Anthropic models. It includes functions for extracting questions and sending messages to models.
Modularity: Functions are well-defined with clear responsibilities, such as extract_questions_anthropic and converse_anthropic.
Integration with External Services: The module integrates with Anthropic's API using helper functions like anthropic_completion_with_backoff, which likely implements retry logic for robustness.
Logging and Debugging: Logging is used effectively to track the flow of data and operations within the functions.
Similarity to Anthropic Chat Module: This file mirrors the structure of the Anthropic chat module but is tailored for Google's Gemini models. It suggests a consistent design pattern across different model integrations.
Functionality: Similar functions are present for extracting questions and conversing with models, indicating a standardized approach to handling different AI models.
Code Reuse: There seems to be potential for refactoring common functionalities between this module and the Anthropic chat module into shared utilities.
Complexity and Size: At 1199 lines, this file is quite large, which can make it difficult to navigate. Consider breaking it down into smaller modules or classes.
API Endpoints: Implements numerous endpoints related to chat functionality, including conversation management, feedback submission, and text-to-speech conversion.
Security Considerations: Uses decorators like @requires(["authenticated"]) to enforce authentication on sensitive endpoints.
Error Handling and Logging: Error handling is present but could be improved by using more specific exceptions. Logging is used throughout to track operations.
UI Component Structure: This React component handles user input for chat interactions. It includes features like file uploads, voice recording, and command suggestions.
State Management: Utilizes hooks like useState and useEffect effectively for managing component state.
User Experience Enhancements: Features like drag-and-drop file uploads and voice transcription indicate a focus on improving user interaction.
Code Organization: The component is relatively large; consider breaking down into smaller sub-components if possible.
Database Interaction Layer: This module acts as an adapter layer for database operations related to users, subscriptions, agents, etc.
Asynchronous Operations: Makes extensive use of asynchronous Django ORM methods (afirst, aupdate_or_create) which is suitable for I/O-bound operations in web applications.
Complexity Management: Contains numerous utility functions; consider organizing related functions into separate modules or classes to improve maintainability.
Documentation Quality: Provides clear instructions on setting up code execution capabilities within Khoj using Cohere's Terrarium.
Clarity and Conciseness: The documentation is concise yet informative, offering both setup instructions and verification steps.
Overall, the source code exhibits good practices in terms of modularity, logging, error handling, and integration with external services. However, there are opportunities for improvement in terms of code organization (especially in larger files) and more granular error handling.
Report On: Fetch commits
Development Team and Recent Activity
Team Members
Debanjum
Saba Imran
Recent Activities
Debanjum
Commits: 63 commits with 1858 changes across 60 files in 3 branches.
Activities:
Added a script to evaluate Khoj using Google's FRAMES benchmark for multi-step retrieval and reasoning.
Improved user interface elements, such as aligning chat input buttons and enhancing chat message handling.
Worked on research mode enhancements, including setting LLM generation seeds for reproducibility and improving prompt structures.
Implemented prompt tracing for debugging Khoj's train of thought using Git.
Released multiple versions of Khoj (1.28.0 to 1.28.3) with various improvements and bug fixes.
Enhanced the web app's agent management features, allowing users to sort agents by most recent use.
Saba Imran
Commits: 66 commits with 2434 changes across 56 files in 2 branches.
Activities:
Worked on handling file retrieval when agents are not set and deprecated certain user model configurations.
Released new versions of Khoj (1.27.0 to 1.28.3) focusing on stability and feature enhancements.
Improved subscription management logic and integrated new functionalities for document conversion and processing within chats.
Collaborated on the development of advanced reasoning capabilities in research mode, including running code in a sandbox environment.
Enhanced the web app's ability to handle multiple images in chats and improved UI components for better user experience.
Patterns, Themes, and Conclusions
The development team is actively enhancing Khoj's capabilities, focusing on improving user experience, performance evaluation, and debugging tools.
There is a strong emphasis on integrating advanced reasoning features and improving the research mode for more complex queries.
Both team members are involved in releasing frequent updates, indicating a rapid development cycle aimed at refining existing features and introducing new ones.
Collaboration between Debanjum and Saba Imran is evident in shared tasks such as version releases and feature implementations, highlighting a cohesive team effort towards project goals.
Summary
Quantified Reports
Detailed Reports.png?size=50)