Report On: Fetch commits

Development Team and Recent Activity

Team Members

• dependabot[bot]

• Luca Steeb (steebchen)

• Alexander Belanger (abelanger5)

• Gabe Ruttner (grutt)

Recent Activity

dependabot[bot]

• 0 days ago: Bumped google.golang.org/api from 0.185.0 to 0.186.0.
• 2 days ago: Bumped github.com/go-co-op/gocron/v2 from 2.6.0 to 2.7.0.
• 5 days ago: Bumped github.com/go-co-op/gocron/v2 from 2.5.0 to 2.6.0.

Luca Steeb (steebchen)

• 0 days ago:
  • Deprecated On in favor of RegisterWorkflow.
  • Added serverless support via webhook workers.
• 1 day ago:
  • Refactored random string generation.
  • Added String variants using base64 for encryption.
  • Updated TypeScript SDK docs.
  • Applied style changes to navbar links.

Alexander Belanger (abelanger5)

• 1 day ago:
  • Corrected lengths on random.Generate.
  • Made RabbitMQ connection optional and disabled for token generation.
  • Overhauled TypeScript SDK docs.
• 2 days ago:
  • Ported dynamic limits feature from a feature branch.

Gabe Ruttner (grutt)

• 1 day ago:
  • Implemented billing features, including Lago integration and plan management.
• 2 days ago:
  • Ported dynamic limits feature from a feature branch.

Patterns, Themes, and Conclusions

1. Dependency Updates: Dependabot continues to manage dependency updates, ensuring the project remains secure and up-to-date with the latest versions of external libraries.
2. Feature Enhancements: Significant new features were added, such as serverless support via webhook workers and comprehensive billing functionalities.
3. Refactoring and Optimization: Several refactoring efforts were made to improve code quality and maintainability, such as the deprecation of older methods in favor of new ones and the refactor of random string generation.
4. Documentation Improvements: Both the Python SDK and TypeScript SDK documentation saw significant updates, indicating a focus on improving developer experience and usability.
5. Collaborative Development: Multiple team members collaborated on various features and fixes, demonstrating effective teamwork and coordination.

Analysis of Progress Since Last Report

Since the last report, there has been significant activity across various aspects of the project:

• Major features like webhook workers and billing functionalities have been added.
• Numerous bug fixes and optimizations have been implemented to enhance system stability and performance.
• Documentation has been extensively updated to support new features and improve clarity.

Overall, the development team has made substantial progress in enhancing the platform's capabilities while maintaining a strong focus on code quality and user experience.

Report On: Fetch issues

Analysis of Progress Since Last Report

Since the previous analysis 7 days ago, there has been notable activity in the hatchet-dev/hatchet repository. Here is a detailed breakdown of the changes and their implications:

New Issues

1. Issue #642 - Improve step error message on workflow scheduling timeout

   • Created: 0 days ago by None (pveierland)
   • Significance: This issue suggests enhancing the error message displayed when a workflow scheduling times out. Improving this message will provide users with more informative feedback, aiding in troubleshooting and understanding system behavior.

2. Issue #640 - Refactor: make engine runnable with config instead of loader

   • Created: 0 days ago by None (abelanger5)
   • Significance: This refactor aims to allow the engine to run with a configuration object instead of a loader object. This change could simplify configuration management and improve code maintainability.

3. Issue #639 - feat: check security service

   • Created: 1 day ago by Gabe Ruttner (grutt)
   • Significance: This feature introduces a security check for users running vulnerable instances. It enhances security by alerting users about potential vulnerabilities in their running versions.

4. Issue #637 - Go SDK: prefix action ids with workflow name

   • Created: 1 day ago by Luca Steeb (steebchen)
   • Significance: Prefixing action IDs with the workflow name can help in better identification and organization of actions within workflows, improving clarity and debugging.

5. Issue #636 - Go SDK: SetTimeout should accept time.Duration

   • Created: 1 day ago by Luca Steeb (steebchen)
   • Significance: Allowing SetTimeout to accept time.Duration will provide more flexibility and precision in setting timeouts, enhancing the SDK's usability.

6. Issue #629 - User guide documentation updates

   • Created: 3 days ago by Akshat Sharma (akshat2602), edited 0 days ago
   • Significance: Updates to the user guide documentation, including adding examples and removing outdated information, are crucial for improving user onboarding and experience.

Closed Issues

1. Issue #641 - chore(deps): bump google.golang.org/api from 0.185.0 to 0.186.0

   • Created and Closed: 0 days ago by None (dependabot[bot])
   • Significance: Regular dependency updates ensure that the project remains secure and up-to-date with the latest features and fixes from dependencies.

2. Issue #638 - fix: correct lengths on random.Generate

   • Created and Closed: 1 day ago by None (abelanger5)
   • Significance: Fixing the length of random generators for generating a cookie hash and block key ensures correct functionality and security.

3. Issue #635 - fix: make rabbitmq connection optional and disable for token generation

   • Created and Closed: 1 day ago by None (abelanger5)
   • Significance: Making RabbitMQ connection optional for API token generation simplifies setup and reduces unnecessary dependencies.

4. Issue #634, #633, #632, #631, #630, and others closed within the last few days.

   • These issues include various enhancements, refactors, new features, and fixes that contribute to overall system stability and functionality.

Observations

• The repository continues to see active development with new features, bug fixes, and documentation improvements.
• There is a focus on enhancing user experience through better error messages (#642) and documentation updates (#629).
• Security remains a priority with the introduction of a security check feature (#639).

Summary

The recent activity in the hatchet-dev/hatchet repository includes critical updates that enhance functionality, security, and user experience. The development team remains proactive in addressing issues promptly, which is a positive indicator for the project's health.

Detailed Breakdown of New Issues

• #642, #640, and others highlight ongoing efforts to improve code quality, user experience, and security.
• Documentation updates (#629) are particularly important for user onboarding and reducing friction.

Detailed Breakdown of Closed Issues

• The closure of issues like #641 through #630 reflects continuous efforts to keep dependencies current while also adding new features and fixing critical bugs.

Conclusion

The hatchet-dev/hatchet repository has seen significant progress over the past 7 days with numerous bug fixes, feature enhancements, and dependency updates. The development team remains responsive and proactive in addressing issues, which bodes well for the project's future stability and usability.

This report captures only activity since the last analysis 7 days ago, focusing on new developments while providing context from previously closed issues where relevant.

Report On: Fetch PR 640 For Assessment

PR #640

Summary of Changes

This pull request refactors the hatchet-engine to be runnable with a configuration object (server.ServerConfig) instead of a loader object (loader.ConfigLoader). The changes include:

1. Introduction of RunWithConfig Function: A new function RunWithConfig is introduced to handle the engine's execution using the server.ServerConfig object.
2. Refactoring Teardown Structure: The Teardown struct fields are made public (Name and Fn) to facilitate their use outside the package.
3. Code Reorganization: The main logic for running the engine is moved into the RunWithConfig function, which returns a list of teardown functions.
4. Error Handling Adjustments: Error handling is adjusted to return both the teardown list and errors from RunWithConfig.
5. Removal of Redundant Code: The original Run function is simplified to call RunWithConfig and handle the returned teardown functions.

Detailed Code Analysis

Changes in cmd/hatchet-engine/engine/run.go

• Imports:

  • Added import for server.ServerConfig.

• Teardown Struct:

  • Fields name and fn are renamed to Name and Fn, respectively, making them public.

• Function Run:

  • Simplified to load server config, call RunWithConfig, and handle teardown.

• New Function RunWithConfig:

  • Handles initialization of various services (health, ticker, events controller, jobs controller, etc.) using the provided server configuration.
  • Collects teardown functions for each service.
  • Returns the list of teardown functions and any errors encountered.

Code Quality Assessment

1. Clarity and Readability:

   • The refactoring improves code readability by separating concerns. The main function (Run) is now more concise, focusing on high-level logic.
   • The introduction of RunWithConfig encapsulates detailed initialization logic, making it easier to understand and maintain.

2. Error Handling:

   • Error handling is consistent and clear. Each step checks for errors and returns them appropriately, ensuring that issues are caught early.

3. Modularity:

   • The changes enhance modularity by decoupling the engine's initialization from its configuration loading mechanism.
   • Public fields in the Teardown struct allow for better integration with other parts of the codebase.

4. Maintainability:

   • By moving complex logic into a dedicated function (RunWithConfig), future modifications can be localized, reducing the risk of introducing bugs.
   • The use of descriptive names for teardown functions improves maintainability.

5. Performance Considerations:

   • There are no significant performance implications introduced by this refactor. The primary focus is on improving code structure without altering runtime behavior.

6. Testing and Validation:

   • While the diff does not include tests, it would be prudent to ensure that existing tests cover these changes adequately.
   • Additional tests could be added to validate the new behavior introduced by RunWithConfig.

Conclusion

The refactor in PR #640 is well-executed, improving code clarity, modularity, and maintainability without altering core functionality or performance characteristics. It adheres to good coding practices by encapsulating complex logic into a dedicated function and maintaining consistent error handling throughout. Further testing should be conducted to ensure comprehensive coverage of the new implementation.

Report On: Fetch pull requests

Analysis of Progress Since Last Report

Since the previous analysis conducted 7 days ago, there has been significant activity in the repository with various pull requests being opened and closed. Here's a detailed report on the changes:

Notable Problems with Open PRs:

1. PR #640: refactor: make engine runnable with config instead of loader

   • State: Open
   • Created: 0 days ago
   • Description: Allows engine to be run with a config object instead of a loader object.
   • Comments: Vercel bot provided deployment updates.
   • Commits: 2 commits focusing on making the hatchet-engine runnable programmatically.
   • Files Changed: cmd/hatchet-engine/engine/run.go (+87, -71)

2. PR #639: feat: check security service

   • State: Open
   • Created: 1 day ago
   • Description: Alerts users running vulnerable instances by checking the security service on API start.
   • Comments: Vercel bot provided deployment updates.
   • Commits: 4 commits focusing on checking the security service and propagating version information.
   • Files Changed: Multiple files including cmd/hatchet-admin/cli/token.go, cmd/hatchet-api/api/run.go, and others.

3. PR #629: User guide documentation updates

   • State: Open (Draft)
   • Created: 3 days ago, edited 0 days ago
   • Description: Added examples to the user guide and removed redundant information.
   • Comments: Issues with linting and formatting were discussed and addressed.
   • Commits: 8 commits focusing on adding examples and fixing linting issues.
   • Files Changed: Multiple documentation files including frontend/docs/pages/home/features/cancellation.mdx, frontend/docs/pages/home/features/child-workflows.mdx, and others.

4. PR #541: Feat retry delay

   • State: Open (Draft)
   • Created: 27 days ago
   • Description: Introduces retry delay parameters to step definitions.
   • Comments: Vercel bot provided deployment updates.
   • Commits: 2 commits focusing on the implementation of retry delays.
   • Files Changed: Multiple files including api-contracts/workflows/workflows.proto, internal/repository/prisma/dbsqlc/models.go, and others.

5. PR #501: docs: format go snippet

   • State: Open
   • Created: 42 days ago, edited 41 days ago
   • Description: Formats Go code snippets in the documentation.
   • Comments: Issues with the linter that need to be resolved before merging.
   • Commits: 2 commits focusing on documentation formatting.
   • Files Changed: frontend/docs/pages/home/features/on-failure-step.mdx (+22, -22)

6. PR #493: docs: Docker compose update

   • State: Open
   • Created: 43 days ago
   • Description: Updates Docker compose documentation to fix connection issues.
   • Comments: Vercel bot provided deployment updates.
   • Commits: 1 commit updating self-hosted page documentation.
   • Files Changed: frontend/docs/pages/self-hosting/docker-compose.mdx (+3, -3)

Recently Closed/Merged PRs of Interest:

1. PR #641: chore(deps): bump google.golang.org/api from 0.185.0 to 0.186.0 -State Closed -Created/Closed0 days ago -Description Updates Golang API dependency to version 0.186.0.

2. PR#638fixcorrectlengthsonrandomGenerate -Stateclosed -Created/Closed1dayago -Description**Fixeslengthofrandomgeneratorsforgeneratingacookiehashandblockkey

3.PR#635fixmakerabbitmqconnectionoptionalanddisablefortokengeneration -Stateclosed -Created/Closed1dayago -Description**Addsoptiontooverrideconfigvaluesintheconfigloaderanddisablestherabbitmqconnectionfromtheapi-tokencommand

4.PR#634feat(encryption)addStringvariantsusingbase64 -Stateclosed -Created/Closed2daysagoedited1dayagoclosed1dayagoMergedbyLucaSteebsteebchen1dayagoAddsstringmethodswhichusebase64internallytoeasilyreadandwritestringsfrom/tothedatabase

5.PR#633refactor(random)refactorrandomstringgeneration -Stateclosed -Created/Closed2daysagoedited1dayagoclosed1dayagoMergedbyLucaSteebsteebchen1dayagoRefactorrandomstringgenerationincludingmovingmethodtonewrandompackageandaddingtests

6.PR#632Featdynamiclimits -Stateclosed -Created/Closed2daysagoclosed2daysagoMergedbyGabeRuttnergrutt2daysagoExposesmapforcustomlimits

7.PR#631feat(frontend)cloudapiintegration -Stateclosed -Created/Closed2daysagoedited1dayagoclosed1dayagoDraftNotmergedAddsadditionalAPIclienttothefrontendfortheHatchetCloudAPI

8.PR#630chore(deps)bumpgithubcomgo-co-opgocronv2from260to270 -Stateclosed -Created/Closed2daysagoclosed2daysagoMergedbyNonegithub-actionsbot2daysagoUpdatesGoCrondependencytoversion270

9.PR#628chore(deps)bumpgithubcomgo-co-opgocronv2from250to260 -Stateclosed -Created/Closed5daysagoclosed5daysagoMergedbyNonegithub-actionsbot5daysagoUpdatesGoCrondependencytoversion260

10.PR#627fix(worker)deprecateOninfavorofRegisterWorkflow -Stateclosed -Created/Closed6daysagoedited0daysagoclosed0daysagoMergedbyLucaSteebsteebchen0daysagoDeprecatesOnmethodinfavorofRegisterWorkflow

11.PR#626docs(typescript)fiximports -Stateclosed -Created/Closed6daysagoclosed6daysagoMergedbyLucaSteebsteebchen6daysagoFixesinvalidimportsinTypescriptSDKdocumentation

12.PR#625docs(ts)adaptsomeinvalidimports -Stateclosed -Created/Closed6daysagoclosed6daysagonotmergedFixesinvalidimportsinTypescriptSDKdocumentation

13.PR#624featbilling -Stateclosed -Created/Closed6daysagoedited1dayagoclosed1dayagoMergedbyNoneabelanger51dayagoAddsnewbillingfeaturewithvariousimprovementsandfixes

14.PR#623style(navbar)usecursorpointerfornavbarlinks -Stateclosed -Created/Closed7daysagoedited1dayagoclosed1dayagoMergedbyNoneabelanger51dayagoUsesapointercursorfornavitemsinthenavigationbar

15.PR#622fixclosecreateeventdialogonsuccessCreateeventdialogshouldclosewheneventissuccessfullycreated

16.PR#621docstypescriptsdkdocsoverhaulOverhaulstheTypescriptSDKdocssothey'resimilartothePythonSDKdocsTake note of current TODOs

17.PR#620chore(migrate)improvemigratetaskImprovementsinmigratetaskincludingusingthealreadyprovidedmigrationnameintheprismamigrationandnotattemptingtocreateanewmigrationingenerate-sqlc

18.PR#619docsremoverequestaccesslinksRemovesrequestaccesslinksinvariousdocumentationpages

19.PR#618wwmergeMergesmainintowebhook-workersbranchwithvariousfixesandimprovements

20.PR#617featadduri-basedfilteringStoresfiltersinthesearchparamsso reloads/backbuttons will maintain filtered state

Report On: Fetch Files For Assessment

Source Code Assessment

File: go.mod

Structure and Quality

• Module Declaration: The file correctly declares the module name as github.com/hatchet-dev/hatchet.
• Go Version: Specifies Go 1.21, ensuring compatibility with the latest Go features and improvements.
• Dependencies:
  • Direct Dependencies: Lists essential libraries such as github.com/Masterminds/semver, github.com/fatih/color, github.com/google/go-github, etc. These are well-known and widely used packages in the Go ecosystem.
  • Indirect Dependencies: Includes a comprehensive list of indirect dependencies, ensuring that all transitive dependencies are accounted for.
  • Versioning: Uses semantic versioning for dependencies, which is a best practice for maintaining compatibility and stability.
• Comments: The file lacks comments or documentation, which could help in understanding the purpose of specific dependencies.

Recommendations

• Documentation: Adding comments to explain why certain dependencies are included would be beneficial.
• Dependency Management: Regularly review and update dependencies to ensure security and compatibility with the latest versions.

File: pkg/worker/service.go

Structure and Quality

• Package Declaration: Correctly declares the package as worker.
• Imports: Imports necessary packages such as fmt and github.com/hatchet-dev/hatchet/pkg/client/types.
• Service Struct:
  • Defines a Service struct with fields for Name, middlewares, and Worker.
  • Provides methods to use middlewares, register workflows, and actions.
• Deprecation Notice:
  • Clearly marks the On method as deprecated in favor of RegisterWorkflow.
  • Provides a backward-compatible implementation by calling On within RegisterWorkflow.
• Error Handling: Proper error handling is implemented throughout the methods.
• Code Quality:
  • The code is clean, well-organized, and follows Go conventions.
  • Uses meaningful variable names and maintains consistency in function signatures.

Recommendations

• Documentation: Adding comments to describe the purpose of each method would improve readability.
• Deprecation Handling: Ensure that deprecated methods are removed in future major releases to maintain code cleanliness.

File: api-contracts/openapi/components/schemas/webhook_worker.yaml

Structure and Quality

• Schema Definition:
  • Defines multiple schemas related to webhook workers such as WebhookWorker, WebhookWorkerCreated, WebhookWorkerCreateRequest, etc.
  • Uses $ref to reference common components like metadata, promoting reusability.
• Properties:
  • Specifies properties with appropriate types (e.g., string) and descriptions.
  • Enforces required properties using the required keyword.
• Validation:
  • Implements validation constraints like minLength for the secret key.

Recommendations

• Consistency: Ensure consistent naming conventions for properties across different schemas.
• Documentation: Add more detailed descriptions for each property to improve clarity.

File: api/v1/server/handlers/webhook-worker/create.go

Structure and Quality

• Package Declaration: Correctly declares the package as webhookworker.
• Imports: Imports necessary packages such as echo, gen, transformers, etc.
• Function Implementation:
  • Implements the WebhookCreate function to handle webhook creation requests.
  • Uses context to retrieve tenant information and generate secrets if not provided.
  • Encrypts the secret before storing it using the configured encryption service.
  • Handles errors appropriately and returns a response using transformers.

Recommendations

• Error Handling: Consider adding more granular error messages to aid debugging.
• Documentation: Add comments to describe the flow of operations within the function.

File: frontend/app/src/pages/main/tenant-settings/webhooks/index.tsx

Structure and Quality

• Imports: Imports necessary React components, hooks, and API utilities.
• Component Implementation:
  • Implements a functional component to manage webhook workers within tenant settings.
  • Uses React hooks (useState, useQuery, useMutation) for state management and API interactions.
  • Provides UI elements like buttons, dialogs, dropdown menus, etc., for user interactions.
• Error Handling:
  • Implements basic error handling using conditional rendering for loading states and errors.

Recommendations

• Code Splitting: Consider splitting large components into smaller sub-components for better maintainability.
• Styling Consistency: Ensure consistent styling across different UI elements.

File: pkg/encryption/cloudkms.go

Structure and Quality

• Package Declaration: Correctly declares the package as encryption.
• Imports: Imports necessary packages such as Tink crypto libraries, Google Cloud KMS client, etc.
• Encryption Service Implementation:
  • Implements a cloud KMS-backed encryption service with methods for encryption/decryption of data and strings.
  • Provides functions to generate JWT keysets from Cloud KMS.

Recommendations

• Error Handling: Ensure all potential errors are handled gracefully with meaningful messages.
• Documentation: Add comments to describe each function's purpose and usage.

File: pkg/random/random.go

Structure and Quality

• Package Declaration: Correctly declares the package as random.
• Imports: Imports necessary packages such as crypto/rand and math/big.
• Random String Generation Implementation:
  • Implements functions to generate random strings of specified length using secure random number generation.

Recommendations

• Extensibility: Consider adding more utility functions for different types of random data generation if needed in the future.

File: frontend/docs/pages/self-hosting/configuration-options.mdx

Structure and Quality

• Markdown Documentation:
  • Provides comprehensive documentation on configuration options for self-hosting Hatchet server and engine.
  • Groups environment variables based on their configuration sections (e.g., Runtime Configuration, Services Configuration).

Recommendations

• Clarity and Detail:
  • Ensure all configuration options are up-to-date with the latest codebase changes.
  • Add examples where applicable to illustrate how to set specific environment variables.

Aggregate for risks

Notable Risks

Critical bug causing double execution of steps when restarting the engine

Severity: High (3/3)

Rationale

This bug can lead to significant workflow reliability issues, causing unexpected behavior and potentially duplicating actions that should only occur once.

• Evidence: Issue #552 highlights a critical bug where restarting the engine causes the same step to execute twice.
• Reasoning: This issue directly impacts the core functionality of the system, leading to potential data corruption, unintended side effects, and unreliable operation of workflows.

Next Steps

• Prioritize fixing this bug immediately.
• Implement a robust testing mechanism to ensure that restarting the engine does not cause duplicate executions in the future.

Prolonged disagreement or argumentative engagement among team members

Severity: Medium (2/3)

Rationale

Disagreements can indicate deeper issues within the team that may affect productivity and project direction.

• Evidence: PR discussions showing considerable disagreement about code and architectural issues (#524).
• Reasoning: While healthy debate is part of development, prolonged disagreements can slow down progress and lead to fragmented solutions.

Next Steps

• Escalate the discussion to a tech lead or technical executive for resolution.
• Facilitate a meeting to align on architectural decisions and ensure all team members are on the same page.

Multiple rewrites of the same source code files in a short period

Severity: Medium (2/3)

Rationale

Frequent changes to the same files can indicate instability or unclear requirements, which may introduce bugs or inconsistencies.

• Evidence: Multiple commits by Gabe Ruttner and Alexander Belanger on features like worker semaphore v2 and email alert groups (#540, #547).
• Reasoning: While these changes are aimed at improving functionality, frequent rewrites can lead to integration issues and potential bugs if not managed carefully.

Next Steps

• Conduct a thorough review of recent changes to ensure stability.
• Establish clearer requirements and design specifications before implementing further changes.

Ambiguous specifications or direction for important functionality

Severity: Medium (2/3)

Rationale

Ambiguity in specifications can lead to misaligned implementations and wasted effort.

• Evidence: Issue #541 requests retry delay parameters but lacks detailed defining criteria.
• Reasoning: Without clear specifications, developers may implement features that do not meet user needs or project goals, leading to rework and delays.

Next Steps

• Clarify and document detailed specifications for high-priority features.
• Ensure all stakeholders review and agree on these specifications before development begins.

Non-critical PRs left open for several days without any updates

Severity: Low (1/3)

Rationale

While not urgent, leaving PRs open without updates can indicate potential bottlenecks in the review process.

• Evidence: PR #501 has been open for 21 days with unresolved linter issues.
• Reasoning: Delays in merging non-critical PRs can slow down overall development velocity and introduce merge conflicts over time.

Next Steps

• Assign reviewers promptly and set clear deadlines for reviewing non-critical PRs.
• Encourage regular updates on open PRs to keep them moving towards closure.