‹ Reports
The Dispatch

GitHub Repo Analysis: mendableai/firecrawl

Executive Summary

The mendableai/firecrawl project is an advanced tool designed to convert websites into LLM-ready markdown or structured data using sophisticated scraping and crawling techniques. Managed by Mendable AI, the project has garnered significant attention with over 22,373 stars on GitHub, reflecting its popularity and utility. The project is in a robust state with active development and a trajectory focused on enhancing web data extraction capabilities for AI applications.

Recent Activity

Team Members and Activities

  1. Gergő Móricz (mogery)

    • Focused on sitemap functionality, logging enhancements, and fixing crawling issues.
    • Collaborated with Nicolas on merges and fixes.

  2. Nicolas (nickscamara)

    • Engaged in formatting, bug fixes, and feature developments like extract billing.
    • Worked closely with Gergő Móricz, Rafael Miller, and Eric Ciarla.

  3. Rafael Miller (rafaelsideguide)

    • Worked on extraction options and schema handling improvements.
    • Collaborated with Nicolas on feature enhancements.

  4. Eric Ciarla (ericciarla)

    • Added a new web crawler example script.

  5. Thomas Kosmas (tomkosm)

    • Enhanced extraction service with caching capabilities.

  6. Dependabot[bot]

    • Automated dependency updates across branches.

Recent Commits and PRs

Risks

Of Note

  1. Integration Complexity: Ongoing efforts to support Azure OpenAI endpoints (#1081) reflect the project's adaptability but also highlight integration challenges.
  2. Proxy Support Limitations: Lack of proxy settings in the fetch engine (#1035) could hinder users dealing with restricted network environments.
  3. Advanced Feature Demand: User interest in robust proxy support and authentication handling indicates a growing demand for more sophisticated scraping capabilities.

Quantified Reports

Quantify issues



Recent GitHub Issues Activity

Timespan Opened Closed Comments Labeled Milestones
7 Days 4 1 2 2 1
30 Days 17 7 31 5 1
90 Days 72 41 155 14 1
All Time 432 322 - - -

Like all software activity quantification, these numbers are imperfect but sometimes useful. Comments, Labels, and Milestones refer to those issues opened in the timespan in question.

Rate pull requests



2/5
The pull request introduces a minor change by adding an environment variable to the docker-compose file. While it addresses a specific issue (#930), the change is minimal, involving only a single line addition. The impact of this change is limited, and it does not demonstrate significant complexity or improvement to warrant a higher rating. The PR lacks substantial documentation or explanation beyond the comment, making it relatively insignificant in scope.
[+] Read More
3/5
The pull request addresses a specific bug by fixing the WebSocket URL creation in the CrawlWatcher class, which is a necessary and functional improvement. The use of regex to replace 'http' with 'ws' is a straightforward solution, but it lacks complexity and does not handle edge cases such as URLs not starting with 'http'. The change is minor, affecting only a few lines of code, and while it improves functionality, it doesn't introduce significant new features or optimizations. Overall, it's an average PR that fixes a bug but doesn't go beyond the basic requirements.
[+] Read More
3/5
The pull request updates development dependencies for the test suite, including @types/jest, artillery, and typescript. These are minor version updates that typically include bug fixes and performance improvements. While keeping dependencies up-to-date is good practice, this PR does not introduce any significant new features or improvements to the codebase itself. The changes are routine and necessary for maintenance but do not warrant a higher rating due to their lack of impact on the functionality or performance of the application.
[+] Read More
3/5
This pull request updates several dependencies in the project, which is a routine maintenance task. While updating dependencies is important for security and performance improvements, this PR does not introduce any significant new features or bug fixes. The changes are straightforward and do not require complex code modifications, thus it can be considered average. It fulfills its purpose but lacks any remarkable or exceptional aspects.
[+] Read More
3/5
This pull request updates 12 development dependencies in the /apps/api directory, which is a routine maintenance task. While keeping dependencies up-to-date is important for security and compatibility, the changes are not particularly significant or complex. The updates do not introduce new features or major improvements to the codebase. Therefore, this PR is considered average, as it is a necessary but unremarkable update.
[+] Read More
3/5
This pull request primarily involves updating 47 dependencies in the /apps/api project, which is a routine maintenance task. While it is important to keep dependencies up-to-date for security and compatibility reasons, the PR does not introduce any significant new features or improvements to the codebase. The updates are mostly minor or patch-level changes, with some major version updates that require careful testing. Overall, this PR is average in significance and complexity, hence an average rating of 3 is appropriate.
[+] Read More
3/5
This pull request is a routine dependency update managed by Dependabot, which bumps the versions of FastAPI and Playwright. While keeping dependencies up-to-date is important for security and compatibility, this PR does not introduce any new features or significant changes to the codebase. The updates are minor version bumps, indicating backward-compatible improvements and bug fixes. There are no apparent issues or conflicts, but the PR lacks any substantial impact beyond maintenance. Thus, it is an average and unremarkable update.
[+] Read More
3/5
The pull request introduces changes to improve the handling of scrape options and threshold options, which are relevant for the functionality of the project. The code modifications involve refactoring and enhancing existing functions, such as adjusting the calculation of total wait times and improving type definitions. While these changes are beneficial, they are not groundbreaking or exceptionally innovative. The PR is well-structured but lacks significant impact or complexity that would warrant a higher rating. Overall, it is a solid contribution with moderate improvements.
[+] Read More
4/5
The pull request introduces a new feature by adding support for Qdrant vector index, which is a significant enhancement to the project. It employs an inheritance-based approach to support multiple vector DB providers, indicating a well-thought-out design. The implementation involves substantial code additions and modifications, including the creation of new files and refactoring existing ones, which reflects thoroughness and attention to detail. However, while the PR is quite good and impactful, it lacks detailed documentation or examples of usage that could help other developers understand and utilize the new feature more effectively. This minor shortcoming prevents it from being rated as exemplary.
[+] Read More
4/5
The pull request introduces caching capabilities to the extraction service, which is a significant enhancement for testing purposes. The changes are well-structured and comprehensive, with a substantial addition of 188 lines and modification of 84 lines across three files. The implementation includes new options for cache management and error handling, improving the service's efficiency and reliability. However, the PR could benefit from additional documentation or comments explaining the new caching logic for maintainability and ease of understanding. Overall, it's a solid improvement but lacks some clarity in code comments.
[+] Read More

Quantify commits



Quantified Commit Activity Over 14 Days

Developer Avatar Branches PRs Commits Files Changes
Nicolas 3 5/5/0 85 93 55956
None (dependabot[bot]) 5 13/0/8 5 5 10601
Rafael Miller (rafaelsideguide) 3 1/0/1 17 24 2749
Gergő Móricz 2 0/1/0 53 47 1867
Thomas Kosmas (tomkosm) 1 1/0/0 1 3 272
Eric Ciarla 1 0/0/0 3 2 181
CyberKn1ght (aanokh) 0 1/0/0 0 0 0
Ademílson Tonato (ftonato) 0 0/0/1 0 0 0
aee (aeelbeyoglu) 0 1/0/1 0 0 0
Hercules Smith (ProfHercules) 0 1/0/0 0 0 0

PRs: created by that dev and opened/merged/closed-unmerged during the period

Quantify risks



Project Risk Ratings

Risk Level (1-5) Rationale
Delivery 4 The project faces significant delivery risks due to a backlog of unresolved issues and critical open issues like #1082 and #1071. The volume of changes in pull requests, such as PR #1083, and the lack of comprehensive testing and documentation further exacerbate these risks. Additionally, dependency management challenges, particularly with self-hosting configurations, could impact delivery timelines.
Velocity 4 The project's velocity is at risk due to the accumulation of unresolved issues and the high volume of open pull requests. While there is active development, as seen in PR #1083 and recent commits by key contributors, the uneven distribution of workload among team members could lead to bottlenecks. The reliance on a few developers for major contributions also poses a risk to sustained velocity.
Dependency 3 Dependency risks are moderate, with active management through dependabot updates (e.g., PRs #1079, #1078). However, the integration challenges posed by numerous dependency changes could impact stability if not thoroughly tested. Issues like #1082 highlight potential risks from external system dependencies.
Team 3 The team faces potential risks related to uneven workload distribution and possible burnout among key contributors like Nicolas and Gergő Móricz. The need for improved documentation and communication, as indicated by user feedback, suggests challenges in team collaboration and onboarding processes.
Code Quality 4 Code quality is at risk due to the concentration of contributions among a few developers and the lack of comprehensive testing details in pull requests like PR #1083. Frequent changes to critical files without thorough reviews could lead to technical debt. The presence of unresolved bugs (#1071) further indicates potential quality issues.
Technical Debt 4 Technical debt is accumulating due to unresolved issues and frequent modifications to core files like 'extraction-service.ts'. The lack of detailed documentation and test coverage in significant feature additions (e.g., PR #1047) contributes to this risk. Ongoing development without addressing these gaps could exacerbate technical debt.
Test Coverage 4 Test coverage is insufficient, as indicated by the lack of explicit testing details in major pull requests like PR #1083. While some unit tests exist (e.g., 'mix-schemas.test.ts'), they may not cover all edge cases or integration scenarios, increasing the risk of undetected bugs.
Error Handling 3 Error handling mechanisms are present but may not be comprehensive enough to catch all potential failures, especially given the reliance on asynchronous operations and external API calls in critical components like 'extraction-service.ts'. The ongoing improvements in logging and error handling are positive but require further enhancement.

Detailed Reports

Report On: Fetch issues



Recent Activity Analysis

Recent activity in the mendableai/firecrawl GitHub repository shows a diverse range of issues being reported and addressed. The project has an active community with numerous open issues, indicating ongoing development and user engagement. Key themes include self-hosting challenges, feature requests for enhanced scraping capabilities, and bug reports related to specific website interactions.

Notable anomalies include recurring issues with self-hosted deployments, particularly around Redis configuration and resource management. There are also several reports of unexpected behavior when dealing with dynamic content or specific web technologies like Cloudflare protection. Additionally, users have expressed interest in more robust proxy support and better handling of authentication-required pages.

A significant portion of the issues revolves around enhancing the tool's ability to handle complex web environments, such as those requiring JavaScript execution or dealing with nested sitemaps. There's also a clear demand for improved documentation and examples to aid new users in setting up and utilizing Firecrawl effectively.

Issue Details

Most Recently Created Issues

  • #1082: "[Self-Host] Working with Dify" - Created 3 days ago; Status: Open; Priority: High
  • #1081: "[Feat] Support for Azure OpenAI endpoints" - Created 3 days ago; Status: Open; Priority: Medium
  • #1071: "[Bug] Metadata fields being returned as lists" - Created 6 days ago; Status: Open; Priority: High

Most Recently Updated Issues

  • #1064: "[Feat] Issue to retrieve similar link URLs given a URL." - Updated 9 days ago; Status: Open; Priority: Medium
  • #1052: "[Self-Host] The model gpt-4o does not exist or you do not have access to it." - Updated 11 days ago; Status: Open; Priority: Low

Critical Issues

  • #1046: "[Bug] Dependency Resolution Issues with Bazel (Cannot find module 'ws' from)" - Edited 14 days ago; Status: Open; Priority: High
  • #1035: "[Self-Host] fetch engine does not support proxy settings" - Edited 2 days ago; Status: Open; Priority: High

These issues highlight ongoing challenges in both the hosted and self-hosted environments, particularly around integration with other tools and services, as well as handling complex web structures. The project's maintainers are actively engaging with the community to address these concerns, indicating a responsive development process.

Report On: Fetch pull requests



Analysis of Pull Requests for mendableai/firecrawl

Open Pull Requests

Recent and Notable Open PRs

  1. #1083: Rafa/extract options

    • State: Open
    • Created: 2 days ago
    • Summary: This PR involves extracting scrape and threshold options, which could enhance the flexibility of data extraction processes. It has seen multiple updates and merges from the main branch, indicating active development.
    • Files Changed: Significant changes across multiple files, with a focus on types and reranker functionalities.
    • Concerns: None apparent; however, continuous updates suggest ongoing testing and refinement.

  2. #1079: apps/test-suite(deps-dev): bump the dev-deps group in /apps/test-suite with 3 updates

    • State: Open
    • Created: 3 days ago
    • Summary: This is a dependabot PR to update development dependencies like @types/jest, artillery, and typescript. Keeping dependencies up-to-date is crucial for security and performance.
    • Concerns: Dependabot PRs are generally safe but should be tested thoroughly to ensure compatibility.

  3. #1078: apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 7 updates

    • State: Open
    • Created: 3 days ago
    • Summary: Another dependabot PR focusing on production dependencies, including updates to packages like @anthropic-ai/sdk and openai.
    • Concerns: Similar to #1079, this requires thorough testing to prevent breaking changes in production environments.

  4. #1077: apps/api(deps-dev): bump the dev-deps group in /apps/api with 12 updates

    • State: Open
    • Created: 3 days ago
    • Summary: Updates a wide range of development dependencies for the API component, which is critical for maintaining a robust development environment.
    • Concerns: Ensure that all updates are compatible with existing codebase functionalities.

  5. #1076: apps/api(deps): bump the prod-deps group in /apps/api with 47 updates

    • State: Open
    • Created: 3 days ago
    • Summary: A major update to production dependencies, which includes significant libraries such as @supabase/supabase-js and mongoose.
    • Concerns: Given the number of updates, comprehensive testing is essential to avoid any disruptions in production.

Other Open PRs

  • There are several other open PRs focusing on feature enhancements (e.g., #1047 for Qdrant vector index support) and bug fixes (e.g., #1053 for fixing WebSocket URL issues). These indicate an active effort to improve functionality and address existing issues.

Closed Pull Requests

Recently Closed Notable PRs

  1. #1073: (feat/index) Index/Insertion queue

    • State: Closed (Merged)
    • Summary: Introduced a new queue system for batch insertion operations, which can significantly improve database interaction efficiency.
    • Impact: Enhances scalability and performance for large-scale data operations.

  2. #1072: (feat/formats) Extract format renamed to json format

    • State: Closed (Merged)
    • Summary: Renamed extract formats to JSON, aligning naming conventions with industry standards.
    • Impact: Improves clarity and consistency in API usage.

  3. #1068: (feat/extract) - LLMs usage analysis + billing

    • State: Closed (Merged)
    • Summary: Added usage analysis and billing features for LLM extractions, which is crucial for managing resources in cloud offerings.
    • Impact: Provides better cost management and transparency for users leveraging LLM capabilities.

General Observations

  • The project is actively maintained with frequent updates to both features and dependencies.
  • There is a strong emphasis on keeping dependencies current, which is vital for security and performance.
  • Feature enhancements are focused on improving scalability, performance, and usability, reflecting the project's commitment to meeting user needs effectively.
  • The community engagement through contributions suggests a healthy open-source project with collaborative development efforts.

Recommendations

  • For open dependabot PRs (#1079, #1078, #1077), ensure rigorous testing before merging to avoid any unforeseen issues due to dependency conflicts.
  • Monitor recently merged feature enhancements (#1073, #1072) for any post-deployment issues that may arise due to integration complexities.
  • Continue encouraging community contributions while ensuring that all changes align with the project's long-term vision and quality standards.

Report On: Fetch Files For Assessment



Analysis of Source Code Files

1. extraction-service.ts

  • Purpose and Functionality: This file is central to the extraction functionality of the project. It handles the extraction of data from URLs using schemas and prompts, leveraging OpenAI's API for classification and data generation.
  • Structure and Organization: The file is well-organized with clear separation of concerns. Functions like analyzeSchemaAndPrompt and performExtraction are logically structured, each handling specific tasks in the extraction process.
  • Code Quality: The use of TypeScript interfaces such as ExtractServiceOptions and ExtractResult enhances type safety. However, the file is quite lengthy (834 lines), which could be broken down into smaller modules for better maintainability.
  • Error Handling: There is comprehensive error handling, especially in asynchronous operations, which is crucial for a service dealing with external APIs.
  • Logging: Extensive logging is implemented using a logger, which aids in debugging and monitoring.
  • Dependencies: The file imports several modules from within the project, indicating a high level of integration with other components.

2. crawler.ts

  • Purpose and Functionality: This file implements a web crawler that can navigate through websites, respecting robots.txt rules, and extract links for further processing.
  • Structure and Organization: The class-based structure (WebCrawler) encapsulates crawling logic effectively. Methods like filterLinks, getRobotsTxt, and extractLinksFromHTML are well-defined.
  • Code Quality: The code is clean with descriptive method names. However, the constructor has many parameters, which could be refactored into a configuration object for clarity.
  • Error Handling: There is robust error handling, especially when dealing with network requests using Axios.
  • Logging: Logging is thorough, providing insights into the crawling process and any issues encountered.
  • Scalability: The design supports scalability with features like sitemap fetching and link filtering based on depth and patterns.

3. map.ts

  • Purpose and Functionality: This controller handles mapping URLs to extract links from websites. It integrates with Redis for caching results and managing crawl jobs.
  • Structure and Organization: Functions are well-organized, with getMapResults being the core function handling URL mapping logic.
  • Code Quality: The use of TypeScript enhances type safety. However, some functions are quite long and could benefit from further decomposition into smaller helper functions.
  • Error Handling: Error handling is present but could be more granular in certain areas to provide specific feedback on failures.
  • Logging and Monitoring: Logging is implemented to track the mapping process and any errors that occur.

4. mix-schema-objs.ts

  • Purpose and Functionality: This helper file merges schema objects by combining single-answer results with multi-entity results based on a given schema.
  • Structure and Organization: The file is concise and focused on its task. The recursive function mergeResults effectively handles nested structures within schemas.
  • Code Quality: The code is clean and easy to understand. TypeScript's type system could be leveraged more to define input types explicitly.
  • Logging: Minimal logging is present, which might be sufficient given the simplicity of the task.

5. index.ts (Scraping Engines)

  • Purpose and Functionality: This file manages different scraping engines, determining which engine to use based on feature flags and environmental configurations.
  • Structure and Organization: The use of enums for engines (Engine) and feature flags (FeatureFlag) provides clarity. Functions like buildFallbackList are well-designed to handle engine selection logic.
  • Code Quality: The code is modular with clear separation between engine definitions, handlers, and options. Environment variables are used effectively to toggle features.
  • Scalability: The design allows for easy addition of new scraping engines or features by updating respective lists or handlers.

General Observations

  1. Consistency in Style: Across all files, there is consistent use of TypeScript features such as interfaces, enums, and async/await for asynchronous operations.

  2. Modularity: While some files are lengthy (e.g., extraction-service.ts), they generally follow good modular design principles. Further decomposition could improve readability.

  3. Error Handling & Logging: Comprehensive error handling combined with detailed logging provides robustness against runtime errors and aids in troubleshooting.

  4. Integration & Dependencies: There is strong integration between different parts of the codebase, indicating a well-thought-out architecture that supports complex functionalities like web crawling and data extraction.

  5. Potential Improvements:

    • Break down large functions into smaller units for better readability and maintainability.
    • Enhance type safety by defining more explicit types where possible.
    • Consider using configuration objects instead of long parameter lists in constructors or functions.

Overall, the codebase demonstrates a high level of sophistication suitable for a project focused on web scraping and data extraction at scale.

Report On: Fetch commits



Development Team and Recent Activity

Team Members and Their Activities

Gergő Móricz (mogery)

  • Recent Work: Focused on improving the sitemap functionality, logging enhancements, and fixing various issues related to crawling and scraping. Made significant changes to the extraction service and crawler components.
  • Collaboration: Worked alongside Nicolas on several merges and fixes.
  • In Progress: Continues to refine the sitemap features and address crawl-related bugs.

Nicolas (nickscamara)

  • Recent Work: Engaged in extensive formatting, bug fixes, and improvements across multiple files. Contributed to the development of new features like extract billing and JSON format updates. Made numerous updates to SDKs and test files.
  • Collaboration: Frequent collaboration with Gergő Móricz, Rafael Miller, and Eric Ciarla on merges and feature implementations.
  • In Progress: Ongoing work on SDK improvements and extract-related features.

Rafael Miller (rafaelsideguide)

  • Recent Work: Worked on extraction options, reranker improvements, and merging branches for feature integration. Addressed issues related to schema handling in extraction processes.
  • Collaboration: Collaborated with Nicolas on several branches for feature enhancements.

Eric Ciarla (ericciarla)

  • Recent Work: Added a new web crawler example script. Minor contributions compared to other team members.

Thomas Kosmas (tomkosm)

  • Recent Work: Enhanced the extraction service with caching capabilities. Limited recent activity but contributed significantly to specific features.

Dependabot[bot]

  • Recent Work: Automated dependency updates across various branches, ensuring that development dependencies are up-to-date.

Patterns, Themes, and Conclusions

  1. Active Development: The project is under active development with frequent commits from key contributors like Gergő Móricz and Nicolas. There is a strong focus on improving existing functionalities such as sitemap handling, logging, and extraction services.

  2. Collaborative Efforts: Team members often collaborate on merges and feature implementations, indicating a cohesive development process. This is evident in the frequent merges between branches managed by different developers.

  3. Feature Enhancements: Recent activities highlight ongoing efforts to enhance the project's capabilities, particularly in extraction services, logging improvements, and sitemap functionality.

  4. Dependency Management: Regular updates by Dependabot indicate a proactive approach to maintaining dependency health within the project.

  5. Diverse Contributions: While some team members focus heavily on core functionalities (e.g., Gergő Móricz), others contribute through examples or specific enhancements (e.g., Eric Ciarla).

Overall, the Firecrawl project demonstrates a dynamic development environment with continuous improvements and active collaboration among team members.