Report On: Fetch commits

Project Overview

The PostHog development team is working on an open-source analytics platform that provides product analytics, session recording, feature flagging, and A/B testing capabilities. It is designed for self-hosting, allowing users to maintain control over their data while leveraging the platform's extensive analytics and optimization tools. The project is hosted on GitHub and continues to see active development with a balanced focus on both front-end improvements and back-end stability.

Recent Development Activities

Team Contributions (Reverse Chronological Order)

Thomas Obermüller (thmsobrmlr)

• 0 days ago: Added toTypeName function to HogQL (posthog/hogql/functions/mapping.py).
• 0 days ago: Supported non-string funnel breakdowns in HogQL (posthog/hogql_queries/insights/funnels/utils.py).
• 1 day ago: Fixed trends actors include recordings in HogQL (posthog/hogql_queries/insights/trends/trends_actors_query_builder.py).
• 1 day ago: Fixed explore tab for trends queries in HogQL (frontend/src/scenes/trends/persons-modal/persons-modal-utils.tsx).

Sandy Spicer (aspicer)

• 0 days ago: Implemented active CPU time feature (frontend/src/queries/schema.json, posthog/clickhouse/client/execute_async.py).
• 1 day ago: Added means to retention chart (frontend/src/scenes/retention/RetentionTable.tsx).
• 1 day ago: Removed 10,000 cohort limit when loading from actors (posthog/models/cohort/util.py).

David Newell (daibhin)

• 0 days ago: Added CNAME in row feature (frontend/src/scenes/settings/project/Proxy.tsx).
• 1 day ago: Fixed proxy UI issues (frontend/src/scenes/settings/project/Proxy.tsx).
• 1 day ago: Ignored external style on canvas image (frontend/src/scenes/session-recordings/player/rrweb/canvas/canvas-plugin.ts).

Tim Glaser (timgl)

• 0 days ago: Made small loading bar improvements (frontend/src/scenes/insights/EmptyStates/EmptyStates.tsx).

Frank Hamand (frankh)

• 0 days ago: Deleted proxy records immediately if in waiting or erroring states (posthog/api/proxy_record.py).
• 0 days ago: Fixed validation job not checking target_cname (posthog/tasks/validate_proxy_domains.py).
• 1 day ago: Fixed target_cname in proxy UI (frontend/src/scenes/settings/project/Proxy.tsx).

Paul D'Ambra (pauldambra)

• 1 day ago: Showed response status more effectively (frontend/src/scenes/session-recordings/player/inspector/performance-event-utils.ts).
• 1 day ago: Put session summaries back (ee/session_recordings/session_summary/summarize_session.py).
• 1 day ago: Recorded heatmap as dropped only when there is heatmap data (plugin-server/src/worker/ingestion/event-pipeline/extractHeatmapDataStep.ts).

Michael Matloka (Twixes)

• 1 day ago: Fixed query support for InsightCachingState mechanism (posthog/api/test/test_insight.py, posthog/caching/calculate_results.py).

Peter Abordan (Pety99)

• 1 day ago: Made links clickable in table (frontend/src/scenes/insights/views/InsightsTable/columns/BreakdownColumn.tsx).

Ben White (benjackwhite)

• 2 days ago: Added more options for hedgehog mode (frontend/src/lib/components/HedgehogBuddy/HedgehogOptions.tsx).

Robbie Coomber (robbie-c)

• 2 days ago: Fixed referring domain with sessions in web analytics (posthog/hogql_queries/web_analytics/stats_table.py).

Bianca Yang (xrdt)

• 2 days ago: Fixed dict accessor for subscription status on addons (ee/billing/billing_manager.py, ee/billing/billing_types.py, ee/billing/test/test_billing_manager.py).

Phani Raj (Phanatic)

• 3 days ago: Randomized questions & choices in surveys (frontend/src/scenes/surveys/SurveyEditQuestionRow.tsx, posthog/api/test/test_survey.py).

Feedanal

• 3 days ago: Fixed object storage endpoints for hobby deployment (docker-compose.hobby.yml).

Juraj Majerik (jurajmajerik)

• 6 days ago: Brought MDE setting back for experiments (frontend/src/scenes/experiments/ExperimentView/DataCollection.tsx, frontend/src/scenes/experiments/ExperimentView/DataCollectionCalculator.tsx).

Tiina Turban (tiina303)

• 6 days ago: Allowed sorting plugin logs and made pipeline UI visual improvements (frontend/__snapshots__/scenes-app-pipeline--pipeline-node-edit-configuration-stateless-plugin--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-edit-configuration-stateless-plugin--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-destination--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-destination--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-sequence-timer--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-sequence-timer--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-transformation--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-node-new-transformation--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-nodes-management-page--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-nodes-management-page--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-overview-page--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-overview-page--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-site-apps-page--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-site-apps-page--light.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-transformations-page-empty--dark.png, frontend/__snapshots__/scenes-app-pipeline--pipeline-transformations-page-empty--light.png, frontend/src/scenes/pipeline/Destinations.tsx, frontend/src/scenes/pipeline/PipelineNodeNew.tsx).

Conclusions

The PostHog development team has been actively working on various features and fixes across the platform. Recent activities show a strong focus on enhancing the user interface, improving backend functionalities, and adding new features such as active CPU time tracking, dynamic CNAME targets, and randomizing survey questions. The team has also been addressing bugs and optimizing existing functionalities to ensure a robust and user-friendly experience. The ongoing efforts reflect a proactive approach to maintaining and improving the platform's capabilities.

Report On: Fetch issues

Recent Activity Analysis

Overview

Since the last report 6 days ago, there has been a significant amount of activity in the repository. Numerous new issues have been created, and several issues have been closed. This indicates ongoing development and maintenance efforts.

Notable New Issues

1. Issue #22348: Fixes a blocking issue in bin/e2e-test-runner due to malformed SQL in 0061_add_ttl_to_heatmaps migration when settings.TEST is true.
2. Issue #22347: Ensures consistent use of person_distinct_id_override instead of person_override, which may change data for certain groups but is necessary for correctness.
3. Issue #22346: Adds a "no data yet" empty state for insights that haven't loaded data yet, with TODOs for adding tests and applying this to all visualization types.
4. Issue #22345: Prevents full dashboard refresh from being canceled on query timeout by handling subsequent requests more gracefully.
5. Issue #22344: Requests bringing back the "Time" field in HogQL for better user experience.

Recently Closed Issues

1. Issue #22343: Fixed a failing Cypress test by accounting for variable label outputs ("2 months ago" or "1 month ago").
2. Issue #22340: Addressed the issue where dashboard refreshes were canceled on query timeout.
3. Issue #22331: Added active CPU time to loading bars for better performance feedback.
4. Issue #22330: Supported non-string funnel breakdowns in HogQL to prevent errors related to type mismatches.
5. Issue #22329: Improved table loading states and fixed deletion issues in the proxy UI.

General Trends

Backend Improvements

• Several fixes and enhancements have been made to backend functionalities, particularly around query handling (#22348, #22347).
• There is a focus on improving error handling and logging to ensure smoother operations (#22250, #22268).

Frontend Enhancements

• Multiple UI/UX improvements have been introduced, such as better loading states (#22346), improved dashboard refresh logic (#22345), and enhanced insight displays (#22293).
• New features like dynamic CNAME targets for proxies (#22315) and additional customization options for Hedgehog mode (#22266) have been added.

Bug Fixes

• Numerous bugs have been addressed, including issues with session replay not capturing certain network statuses in Safari (#22309), and fixing malformed SQL queries (#22348).

Performance and Usability

• Efforts to improve performance are evident with changes like concurrent loading of insights on dashboards (#22197) and reducing excessive logging that could lead to disk space issues (#22250).

Conclusion

The recent activity in the repository reflects a robust development cycle focused on both expanding features and maintaining system integrity. The team's responsiveness to issues, both in terms of introducing enhancements and resolving bugs, indicates a strong commitment to user satisfaction and continuous improvement. The introduction of new features alongside critical bug fixes suggests a balanced approach to development, ensuring both innovation and stability.

Overall, the project appears to be progressing well with significant improvements made over the past week.

Report On: Fetch PR 22340 For Assessment

Report On: Fetch pull requests

Analysis of Progress Since Last Report

Summary

Since the last analysis 6 days ago, there has been significant activity in the PostHog/posthog repository. Numerous pull requests (PRs) have been opened and closed, with various enhancements, bug fixes, and new features being introduced. Below is a detailed report of the changes:

Open Pull Requests

1. PR #22340: A fix to ensure that full dashboard refreshes are not canceled on query timeout.
2. PR #22339: Introduction of a feature to enable session replay during product analytics onboarding.
3. PR #22338: Fix to avoid conditional calls to useActions & useValues.
4. PR #22337: Deprecation of most frontend references to available_features.
5. PR #22336: Addition of the "Hog" tab for insights and under /debug to run Hog programs.
6. PR #22335: Sorting functionality added to the simple key-value list.
7. PR #22334: Hedgehog Mode 2.0 with additional customization options.
8. PR #22333: Update to sensitive settings token age from 1 hour to 6 hours.
9. PR #22332: Allowing upcoming/past ranges in LemonCalendarSelect component.
10. PR #22326: Fix for displaying errored funnel insights correctly on dashboards.

Notable Closed Pull Requests

1. PR #22331: Addition of active CPU time to loading bars.
2. PR #22330: Support for non-string funnel breakdowns in HogQL.
3. PR #22329: Enhancements to proxy records UI including expanded table rows and better table loading state.
4. PR #22328: Implementation of Zenhog tickets UI with backend support.
5. PR #22327: Small improvements to loading bars for smoother number appearance.
6. PR #22325: Immediate deletion of proxy records in waiting or erroring states.
7. PR #22324: Fix for validation job not checking target_cname.
8. PR #22323: Fix for target_cname in proxy UI.
9. PR #22322: Addition of toTypeName function to HogQL for debugging purposes.
10. PR #22319: Fix for CodeSnippet content which cannot be an element.

Significant Issues and Resolutions

• The closure of PRs such as #22079 without merging indicates potential reevaluation or alternative solutions being implemented, particularly concerning large payload handling in webhooks.
• Several PRs focused on improving system stability and performance, such as PRs related to dashboard refreshes (#22340) and funnel insights (#22326).
• The introduction of new features like Hedgehog Mode 2.0 (#22334) and Zenhog tickets UI (#22328) shows ongoing efforts to enhance user experience and functionality.

Observations

• There has been a notable focus on refining existing functionalities, improving error handling, and enhancing user experience across various components of PostHog.
• The project continues to evolve with significant contributions aimed at both expanding capabilities and ensuring reliability.

Overall, the recent activity indicates an active development cycle aimed at both expanding capabilities and ensuring the reliability of the platform.

This level of activity suggests that the PostHog team is actively working on both new features and improvements to existing functionalities, ensuring that the platform remains robust and user-friendly.

Report On: Fetch Files For Assessment

Source Code Assessment

1. File: posthog/hogql/functions/mapping.py

Reason for Review: This file was recently updated to add a new function 'toTypeName' to hogql. Understanding this function could provide insights into recent feature additions.

Analysis:

• Structure and Organization: The file is quite long (814 lines), which suggests it might be handling multiple responsibilities. It would be beneficial to break down the file into smaller, more manageable modules if feasible.
• Functionality: The addition of the toTypeName function indicates an enhancement to the type handling capabilities of hogql. This function likely maps various data types to their corresponding type names.
• Code Quality: Without the specific lines of code, it's challenging to comment on the exact implementation. However, given the nature of the update, it would be crucial to ensure that the function is well-documented and includes error handling for unsupported types.
• Testing: Ensure that there are unit tests covering various scenarios for this new function, including edge cases.

2. File: posthog/hogql_queries/insights/funnels/utils.py

Reason for Review: This file was updated to support non-string funnel breakdowns. It is essential for understanding how the system handles different data types in funnel analyses.

Analysis:

• Structure and Organization: The file is concise (92 lines) and appears well-organized with clear separation of utility functions.
• Functionality:
  • get_funnel_order_class: Determines the class to use based on the funnel order type.
  • get_funnel_actor_class: Determines the class for funnel actors based on visualization type and order.
  • funnel_window_interval_unit_to_sql: Converts funnel window interval units to SQL-compatible strings.
  • get_breakdown_expr: Constructs expressions for breakdowns, supporting both single and multiple breakdowns, with optional URL normalization.
• Code Quality: The code is clean and uses type hints, which enhances readability and maintainability. The use of ast and parse_expr suggests dynamic query generation, which is powerful but should be carefully managed to avoid injection vulnerabilities.
• Testing: Ensure comprehensive tests cover all possible types of breakdowns, including edge cases like empty strings or unsupported types.

3. File: frontend/src/scenes/settings/project/Proxy.tsx

Reason for Review: This file has multiple updates related to proxy settings and UI improvements. It is crucial for understanding recent changes in proxy management.

Analysis:

• Structure and Organization: The file is moderately sized (191 lines) and appears well-organized with clear separation of components.
• Functionality:
  • The main component (Proxy) handles displaying a table of proxy records and a form for adding new domains.
  • Conditional rendering based on whether the environment is cloud or development (isCloudOrDev).
  • Use of LemonTable, LemonButton, and other UI components from the PostHog design system.
  • Handling of different statuses (valid, erroring, waiting) with appropriate UI feedback (e.g., spinners, tooltips).
• Code Quality: The code is clean and leverages modern React practices such as hooks (useValues, useActions). The use of TypeScript enhances type safety.
• Testing: Ensure that there are tests covering user interactions (e.g., adding a domain, deleting a record) and edge cases (e.g., invalid domain input).

4. File: posthog/clickhouse/client/execute_async.py

Reason for Review: This file was updated for active CPU time tracking, which is important for performance monitoring and optimization.

Analysis:

• Structure and Organization: The file is moderately large (309 lines) but appears well-organized with clear separation of concerns.
• Functionality:
  • Manages asynchronous query execution with Redis for status tracking.
  • Includes metrics collection (QUERY_WAIT_TIME, QUERY_PROCESS_TIME) using Prometheus.
  • Enhanced functionality for tracking active CPU time during query execution (OSCPUVirtualTimeMicroseconds).
• Code Quality: The code is robust with good use of exception handling (QueryNotFoundError, QueryRetrievalError). The use of structured logging (structlog) enhances observability.
• Testing: Ensure that there are tests covering various scenarios, including successful query execution, errors during execution, and progress tracking.

5. File: posthog/api/proxy_record.py

Reason for Review: This file has been frequently updated with fixes and features related to proxy records. It is significant for understanding backend changes in proxy management.

Analysis:

• Structure and Organization: The file is concise (67 lines) and well-organized with clear separation between serializer and viewset definitions.
• Functionality:
  • Defines a serializer (ProxyRecordSerializer) for proxy records with appropriate read-only fields.
  • Implements a viewset (ProxyRecordViewset) with custom list, create, and destroy methods.
  • Custom logic in the create method to generate target CNAME using a hash function (generate_target_cname).
• Code Quality: The code is clean and leverages Django REST framework best practices. The use of custom permissions (OrganizationAdminWritePermissions) ensures proper access control.
• Testing: Ensure that there are tests covering CRUD operations on proxy records, including edge cases like attempting to delete a record in an invalid state.

6. File: plugin-server/src/worker/ingestion/event-pipeline/extractHeatmapDataStep.ts

Reason for Review: This file was updated to fix heatmap data extraction issues. It is important for understanding how heatmap data is processed and validated.

Analysis:

• Structure and Organization: The file is moderately sized (173 lines) and appears well-organized with clear separation of functions.
• Functionality:
  • Main function (extractHeatmapDataStep) processes heatmap data from events and produces Kafka messages.
  • Utility functions (replacePathInUrl, isValidString, isValidNumber) enhance readability by encapsulating common checks.
  • Handles different types of heatmap data (e.g., scroll depth) with appropriate validation and scaling (SCALE_FACTOR).
• Code Quality: The code is clean with good use of TypeScript types. Error handling includes capturing ingestion warnings via Kafka messages (captureIngestionWarning).
• Testing: Ensure that there are tests covering various scenarios, including valid heatmap data extraction, invalid data handling, and edge cases like missing properties.

Overall, the reviewed files exhibit good structure, organization, and code quality. Testing coverage should be comprehensive to ensure robustness across different scenarios.

Aggregate for risks

Notable Risks

1. Complete Lack of Test Coverage for New Functionality in PRs

• Risk Severity: High (3/3)
  • Rationale: The absence of test coverage for new functionalities can lead to undetected bugs, regressions, and overall instability in the codebase. This is a critical issue that needs immediate attention to ensure the reliability and robustness of the platform.
• Detail: Recent updates, such as the addition of the toTypeName function in posthog/hogql/functions/mapping.py, do not mention any corresponding unit tests. Similarly, changes in posthog/hogql_queries/insights/funnels/utils.py and other files lack explicit references to new or updated tests.
  • Supporting Examples:
  • posthog/hogql/functions/mapping.py: Addition of toTypeName function without mention of tests.
  • posthog/hogql_queries/insights/funnels/utils.py: Updates to support non-string funnel breakdowns without mention of tests.
• Next Steps:
  • Implement comprehensive unit tests for all new functionalities.
  • Ensure that edge cases are covered in the tests.
  • Review and update existing tests to cover recent changes.

2. Multiple Rewrites of Source Code Files in a Short Period

• Risk Severity: Medium (2/3)
  • Rationale: Frequent rewrites of the same source code files can introduce instability and increase the likelihood of bugs. It may also indicate unclear requirements or design issues that need to be addressed.
• Detail: Several files have been updated multiple times within a short period, indicating potential instability or ongoing issues.
  • Supporting Examples:
  • frontend/src/scenes/settings/project/Proxy.tsx: Multiple updates related to proxy settings and UI improvements within a few days.
  • posthog/api/proxy_record.py: Frequent updates with fixes and features related to proxy records.
• Next Steps:
  • Conduct a thorough review of the affected files to identify and address underlying issues.
  • Ensure clear requirements and design specifications are in place before making further changes.
  • Implement code reviews and pair programming sessions to improve code stability.

3. Prolonged, Argumentative Engagement Among Team Members

• Risk Severity: Medium (2/3)
  • Rationale: Prolonged and argumentative discussions among team members can lead to delays in development, reduced team morale, and potential misalignment on project goals.
• Detail: There have been instances of prolonged discussions on certain issues and pull requests, indicating potential disagreements or lack of clarity on requirements.
  • Supporting Examples:
  • Issue #22348: Discussions around fixing a blocking issue in bin/e2e-test-runner due to malformed SQL migration.
  • PR #22340: Discussions on ensuring full dashboard refreshes are not canceled on query timeout.
• Next Steps:
  • Facilitate regular team meetings to ensure alignment on project goals and requirements.
  • Encourage constructive feedback and collaborative problem-solving approaches.
  • Implement conflict resolution strategies to address disagreements promptly.

4. Ambiguous Specifications for Important Functionality

• Risk Severity: Medium (2/3)
  • Rationale: Ambiguous specifications can lead to misinterpretations, incorrect implementations, and increased rework. It is crucial to have clear and detailed specifications for critical functionalities.
• Detail: Some issues and pull requests lack clear defining criteria, leading to potential misunderstandings during implementation.
  • Supporting Examples:
  • Issue #22346: Adds a "no data yet" empty state for insights with TODOs for adding tests and applying this to all visualization types, but lacks detailed criteria for implementation.
  • PR #22339: Introduction of session replay during product analytics onboarding without clear specifications on expected behavior and edge cases.
• Next Steps:
  • Ensure that all issues and pull requests have clear, detailed specifications before starting implementation.
  • Use templates or checklists to standardize the documentation of requirements.
  • Involve stakeholders early in the process to validate requirements and expectations.

5. Moderate Code Quality Issues

• Risk Severity: Low (1/3)
  • Rationale: Moderate code quality issues can introduce bugs but aren't certain to do so. Addressing these issues can improve maintainability and reduce technical debt over time.
• Detail: Some files exhibit moderate code quality issues such as long file lengths, lack of modularization, or insufficient error handling.
  • Supporting Examples:
  • posthog/hogql/functions/mapping.py: The file is quite long (814 lines), suggesting it might be handling multiple responsibilities that could be broken down into smaller modules.
  • posthog/clickhouse/client/execute_async.py: While robust, further improvements in error handling could enhance reliability.
• Next Steps:
  • Refactor large files into smaller, more manageable modules where feasible.
  • Enhance error handling mechanisms across the codebase.
  • Conduct regular code reviews focusing on improving code quality.

6. Non-Critical PRs Left Open for Several Days

• Risk Severity: Low (1/3)
  • Rationale: Non-critical PRs left open for several days can indicate potential bottlenecks in the review process but do not pose an immediate risk to project success.
• Detail: Some non-critical PRs have been left open for several days without resolution, which could delay feature releases or bug fixes.
  • Supporting Examples:
  • PR #22336: Addition of the "Hog" tab for insights has been open for several days without merging.
  • PR #22335: Sorting functionality added to the simple key-value list has been open for several days without merging.
• Next Steps:
  • Streamline the review process to ensure timely resolution of PRs.
  • Prioritize PR reviews based on their impact on project timelines and goals.