Compare commits
3 Commits
abb7f743b8
...
b24ab838f8
| Author | SHA1 | Date | |
|---|---|---|---|
 jared
|
b24ab838f8 | ||
|
jared
|
cf7c66b99a | ||
|
jared
|
04b56ffacd |
+206
-59
@@ -8,86 +8,233 @@ This document tracks identified bugs, edge cases, and architectural discrepancie
|
||||
|
||||
## 🚩 Critical & UI Bugs
|
||||
|
||||
### 1. Avatar Decoration Displacement in Profile
|
||||
### 1. No Camera Focus During Screenshare
|
||||
- **File:** `cinny/src/app/features/call/CallControls.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Automatic screenshare spotlighting forces primary display override, preventing users from manually focusing on camera feeds.
|
||||
- **Root Cause:** Current spotlighting logic prioritizes active screenshare streams over manual participant selections, effectively ignoring or overriding user-initiated focus states.
|
||||
- **Proposed Fix:** Introduce a manual 'Focus' state that takes precedence over automatic screenshare spotlighting, implemented via a toggle/click UI on participant tiles. Update the video renderer to respect this manual override.
|
||||
|
||||
**File:** `src/app/components/user-profile/UserHero.tsx`
|
||||
**Status:** **OPEN**
|
||||
### 2. Chat Background Animation Flickering
|
||||
- **File:** `cinny/src/app/features/lotus/chatBackground.ts`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Animated background properties cause visible flickering on message text and the composer area, particularly on browsers/GPUs susceptible to repaint-induced artifacts.
|
||||
- **Root Cause:** Animation triggers excessive repaints or layout recalculations on descendant elements, likely due to animating non-GPU accelerated properties on parent containers without proper rendering context isolation.
|
||||
- **Proposed Fix:** Promote background container to a compositor layer using `will-change: transform`, strictly limit animations to `transform` and `opacity` properties, and utilize `contain: paint;` to isolate the background rendering context.
|
||||
|
||||
- **Issue:** Avatar decorations appear displaced left of the avatar when viewing the profile modal.
|
||||
- **Root Cause:** The `AvatarPresence` badge sticking out to the right shifts the center of the `inline-flex` container. The decoration centers on the container, not the avatar.
|
||||
- **Recommended Fix:** Wrap only the `Avatar` component with `AvatarDecoration`.
|
||||
### 3. Avatar Decorations in Element Call
|
||||
- **File:** `cinny/src/app/components/avatar-decoration/AvatarDecoration.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Avatar decorations are failing to render within the call/room interface member lists.
|
||||
- **Root Cause:** Likely a mismatch between the expected `member` object structure required by the `AvatarDecoration` component and the data actually provided by the call/room UI components. Matrix event data for decorations might not be propagating correctly to these UI member objects.
|
||||
- **Proposed Fix:** Analyze the data propagation chain from Matrix events to the member object in `cinny/src/app/components/call` and `room`, ensuring that decoration-related properties are correctly mapped and passed to the `AvatarDecoration` component.
|
||||
|
||||
### 2. Inconsistent Settings Dropdown Styling
|
||||
### 4. DM and Group Message Calls
|
||||
- **File:** `cinny/src/app/components/CallEmbedProvider.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Incoming call ringtone is hardcoded, lacks volume control, and is suppressed if the user is already in an active call.
|
||||
- **Root Cause:** Ringing logic is tightly coupled to `RTCNotification` events in `CallEmbedProvider.tsx`, using a hardcoded audio file path. It lacks an abstraction for sound management or user-configurable settings for ringtones/volumes.
|
||||
- **Proposed Fix:** Migrate sound asset management to a dedicated audio service. Implement user-configurable settings for ringtone and notification volume. Update the `IncomingCallListener` to support ringing even during active calls (if appropriate) by enhancing event handling.
|
||||
|
||||
**Files:** `Profile.tsx`, `SystemNotification.tsx`
|
||||
**Status:** **OPEN**
|
||||
### 5. Seasonal Themes and Chat Backgrounds Design
|
||||
- **File:** `cinny/src/app/hooks/useTheme.ts`, `cinny/src/app/features/lotus/chatBackground.ts`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Basic CSS or random moving lines are insufficient for high-fidelity wallpaper/theming. They lack professional design theory, coherence, and aesthetic depth.
|
||||
- **Root Cause:** Current implementation relies on basic CSS, lacks advanced design theory, and does not leverage modern, performant CSS wallpaper techniques.
|
||||
- **Proposed Fix (Extreme Depth Redesign):**
|
||||
- **Research-Backed Implementation:** Implement advanced design techniques (layered `oklch` gradients, `backdrop-filter` for refractive "liquid glass" effects, GPU-accelerated `transform` animations) to create living, breathing backgrounds.
|
||||
- **Performance Optimization:** Ensure all animations strictly use compositor-thread properties (`transform`, `opacity`) and apply `contain: paint` / `will-change: transform` to prevent layout thrashing/flickering.
|
||||
- **Design Resources (Examples/Inspiration):**
|
||||
- [Uiverse.io Patterns](https://uiverse.io/patterns)
|
||||
- [MagicPattern CSS Backgrounds](https://www.magicpattern.design/tools/css-backgrounds)
|
||||
- [Prismic Blog: CSS Background Effects](https://prismic.io/blog/css-background-effects)
|
||||
- [CSS-Pattern.com](https://css-pattern.com) (Pure CSS pattern library)
|
||||
- [BGJar](https://bgjar.com) (Performance-focused generators)
|
||||
- **Goal:** Treat each theme/background as a week-long development sprint to ensure professional polish, WCAG AA contrast compliance for overlaying UI, and seamless integration with the Lotus TDS.
|
||||
|
||||
- **Issue:** Dropdowns for Status Expiry and Notification Sounds use raw HTML `<select>` elements.
|
||||
- **Recommended Fix:** Replace with the custom-styled `Menu` + `PopOut` pattern used in `General.tsx`.
|
||||
### 6. Exclusive Background vs. Seasonal Choice
|
||||
- **File:** `cinny/src/app/state/settings.ts`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Concurrent application of both Chat Backgrounds and Seasonal Themes causes visual clutter and high GPU usage.
|
||||
- **Root Cause:** These are currently handled as independent settings in the `settingsAtom` and applied simultaneously without mutual exclusion.
|
||||
- **Proposed Fix:** Introduce mutual exclusion in the settings application logic. Update the settings UI to present these as a single choice (e.g., a radio group or toggled selection) where activating one deactivates the other. Enforce this rule in `cinny/src/app/features/lotus/chatBackground.ts` and `cinny/src/app/components/seasonal/SeasonalEffect.tsx`.
|
||||
|
||||
### 3. Ringing Modal Fires in Voice Rooms
|
||||
### 7. Tiny Touch Targets in Composer Toolbar
|
||||
- **File:** `cinny/src/app/features/room/RoomInput.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Toolbar buttons have hit areas smaller than the WCAG-recommended 44x44px for touch, hindering mobile accessibility.
|
||||
- **Root Cause:** The `IconButton` component does not explicitly enforce a 44x44px touch target, and the default size is too small for mobile touch precision.
|
||||
- **Proposed Fix:** Apply CSS `min-width: 44px; min-height: 44px;` to all toolbar icons, potentially via a dedicated `MobileTouchTarget` style wrapper or by overriding the `IconButton` component's style for mobile viewports using media queries.
|
||||
|
||||
**File:** `src/app/components/CallEmbedProvider.tsx`
|
||||
**Status:** **OPEN**
|
||||
### 8. Horizontal Overflow in Room Settings
|
||||
- **File:** `cinny/src/app/features/room-settings/RoomSettings.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Wide tables and input elements in room settings cause horizontal overflow on mobile viewports.
|
||||
- **Root Cause:** A fixed-width constraint applied to the `PageNav` component in `cinny/src/app/components/page/style.css.ts` overrides responsive layout attempts on smaller screens.
|
||||
- **Proposed Fix:** Update the `PageNav` style definition to be responsive, setting its width to `100%` on mobile viewports using CSS media queries, and ensure all child containers in `RoomSettings` use `flex-wrap: wrap` or similar overflow-handling layouts.
|
||||
|
||||
- **Issue:** Joining a static voice room triggers the "Incoming Call" ringing.
|
||||
- **Recommended Fix:** Check `notification_type` in the Matrix RTC event. Only 'ring' should trigger the modal.
|
||||
### 9. Modal Float-Style Responsiveness
|
||||
- **File:** `cinny/src/app/components/modal/Modal.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Modals appear as floating boxes on mobile, creating navigation and readability challenges.
|
||||
- **Root Cause:** Missing viewport-dependent styling in the base modal component.
|
||||
- **Proposed Fix:** Implement media-query-based responsiveness in the base `Modal` wrapper. For viewports below a certain threshold, override floating styles with full-screen layout: `width: 100vw; height: 100vh; border-radius: 0;`.
|
||||
|
||||
### 4. No Camera Focus During Screenshare
|
||||
### 10. Composer Keyboard Obscurity
|
||||
- **File:** `cinny/src/app/features/room/RoomInput.tsx`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** The chat composer is often partially or fully obscured by the virtual keyboard on mobile.
|
||||
- **Root Cause:** Use of `vh` units for container height does not adapt when the virtual keyboard appears, pushing the `RoomInput` container off-screen.
|
||||
- **Proposed Fix:** Adopt modern viewport units (`100svh`) for main layout containers to ensure the height adapts dynamically. Optionally, use `scrollIntoView` on the `RoomInput` container when it receives focus.
|
||||
|
||||
**File:** `src/app/features/call/CallControls.tsx`
|
||||
**Status:** **OPEN**
|
||||
|
||||
- **Issue:** When someone is screensharing and another participant turns on their camera, there is no way to switch the primary display to the camera or go fullscreen on it.
|
||||
- **Recommended Fix:** Implement a "Focus" toggle on participant tiles that overrides the automatic screenshare spotlight.
|
||||
|
||||
### 5. Chat Background Animation Flickering
|
||||
|
||||
**File:** `src/app/features/lotus/chatBackground.ts`
|
||||
**Status:** **OPEN**
|
||||
|
||||
- **Issue:** Some animated backgrounds (like Fireflies) cause flickering/flashing of the message text and composer area on certain browsers/GPUs.
|
||||
- **Recommended Fix:** Ensure animations are scoped strictly to background properties (`background-position`, `background-size`) and do not use properties like `filter` or `opacity` on the main container.
|
||||
|
||||
### 6. Avatar Decorations not displaying in Element Call UI
|
||||
|
||||
- **Issue:** When in a voice call or room with users it displays the avatar of the users and their mute status etc... but doesn't display their avatar decorations on their pictures.
|
||||
|
||||
### 7. DM and Group Message Calls
|
||||
|
||||
- **Issue:** Call ring is very loud and not customizable, also if the user is already in an active call/or voice room it won't ring at all until the user leaves.
|
||||
|
||||
### 8. Seasonal Themes and Chat Backgrounds need EXTREME design improvements.
|
||||
|
||||
- **Issue:** Basic css or random moving lines are not good artwork or design theory. Requires extensive research on css backgrounds wallpapers and app theming, these should be multi-day projects PER background and theme. As if a whole team spent a entire project sprint on a single one.
|
||||
### 11. Inline Jotai atom creation
|
||||
- **File:** `cinny/src/app/hooks/useSpaceHierarchy.ts`
|
||||
- **Status:** **OPEN**
|
||||
- **Issue:** Inline Jotai atom creation in a hook risks re-rendering components unnecessarily.
|
||||
- **Proposed Fix:** Lift atom definition out of the hook or utilize `useMemo` to ensure atom stability.
|
||||
|
||||
---
|
||||
|
||||
## 📱 PWA & Mobile Issues
|
||||
## 📦 Barrel File Audit
|
||||
|
||||
### 1. Exclusive Background vs. Seasonal Choice
|
||||
|
||||
**Status:** **OPEN**
|
||||
|
||||
- **Issue:** Users can have both a Chat Background and a Seasonal Theme active, causing visual clutter and excessive GPU usage on mobile.
|
||||
- **Recommended Fix:** Implement a "Choose One" toggle in Settings.
|
||||
| File Path | Note | Status |
|
||||
| :--- | :--- | :--- |
|
||||
| `cinny/src/app/plugins/call/index.ts` | Extensive `export *` usage | OPEN |
|
||||
| `cinny/src/app/plugins/text-area/index.ts` | Extensive `export *` usage | OPEN |
|
||||
| `cinny/src/app/components/message/index.ts` | Extensive `export *` usage | OPEN |
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Technical & Performance Refinements
|
||||
|
||||
### 1. Decrypted Media Memory Leak (Gallery & Lightbox)
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| State Sync | Fire-and-forget network call to set offline presence during `pagehide` event may not complete reliably, potentially causing UI drift in presence status. | `cinny/src/app/hooks/usePresenceUpdater.ts` | OPEN |
|
||||
| State Sync | Fire-and-forget network call `setPresence().catch(...)` suppresses errors, meaning the app may falsely assume presence update success. | `cinny/src/app/hooks/usePresenceUpdater.ts` | OPEN |
|
||||
| Memory Leak | Decrypted Media Memory Leak (Gallery & Lightbox) due to missing virtualization and blob revocation. | `cinny/src/app/features/room/MediaGallery.tsx` | OPEN |
|
||||
| Data Persistence | Scheduled Messages are ephemeral (lost on refresh) due to fragile `localStorage` parsing. | `cinny/src/app/state/scheduledMessages.ts` | OPEN |
|
||||
| Memory Leak | Potential memory leak due to uncleaned `handleMouseMove` listener in `usePan`. | `cinny/src/app/hooks/usePan.ts` | OPEN |
|
||||
| Asset Optimization | Large unoptimized media asset (213KB) found in `public/res`. | `public/res/Lotus.png` | OPEN |
|
||||
| Data Persistence | Non-atomic `localStorage` updates in session management can lead to inconsistent state. | `cinny/src/app/state/sessions.ts` | OPEN |
|
||||
| Data Persistence | Lack of cross-tab synchronization for `localStorage` updates in session management risks race conditions. | `cinny/src/app/state/sessions.ts` | OPEN |
|
||||
| Network Resilience | `uploadContent` lacks retry logic, failing immediately upon network error. | `cinny/src/app/utils/matrix.ts` | OPEN |
|
||||
| Network Resilience | `rateLimitedActions` uses basic retry logic without exponential backoff, which may exacerbate 429 issues. | `cinny/src/app/utils/matrix.ts` | OPEN |
|
||||
| Matrix Event Robustness | `useMatrixEventRenderer` handles unknown events gracefully by returning `null`, which may hide potentially important unrendered data. | `cinny/src/app/hooks/useMatrixEventRenderer.ts` | OPEN |
|
||||
| Data Contract | `MatrixError` instantiation with `UploadResponse` might be brittle. | `cinny/src/app/utils/matrix.ts` | OPEN |
|
||||
| Type Safety | `addRoomIdToMDirect` uses `as any` cast for `AccountDataEvent.Direct`, bypassing type contract validation. | `cinny/src/app/utils/matrix.ts` | OPEN |
|
||||
| Robustness | `rateLimitedActions` relies on `MatrixError.httpStatus` which might not exist on all error variants. | `cinny/src/app/utils/matrix.ts` | OPEN |
|
||||
| Type Contract | Custom types in `cinny/src/types/matrix` mirror SDK types instead of using them, risking drift and contract mismatches. | `cinny/src/types/matrix/` | OPEN |
|
||||
|
||||
**File:** `src/app/features/room/MediaGallery.tsx`
|
||||
**Status:** **OPEN**
|
||||
## 🏗️ Architectural & Hygiene Audit
|
||||
|
||||
- **Issue:** Every image in the gallery history is decrypted and converted to a Blob URL simultaneously.
|
||||
- **Recommended Fix:** Implement virtualization for the gallery grid.
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Hygiene | No stale development notes or TypeScript strictness issues found | N/A | OPEN |
|
||||
|
||||
### 2. Scheduled Messages are Ephemeral
|
||||
---
|
||||
|
||||
**File:** `src/app/state/scheduledMessages.ts`
|
||||
**Status:** **OPEN**
|
||||
## 🏗️ TDS Compliance & Styling Issues
|
||||
|
||||
- **Issue:** Refreshing the page clears the "Scheduled" tray, making it impossible for users to see or cancel messages they have already scheduled.
|
||||
- **Recommended Fix:** Persist the scheduled message metadata in `localStorage`.
|
||||
| Issue Description | File Path |
|
||||
| :--- | :--- |
|
||||
| Hardcoded inline style `cursor: 'pointer'` | `cinny/src/app/plugins/react-custom-html-parser.tsx` |
|
||||
| Hardcoded color `#00D4FF`, `#FFB300` | `cinny/src/app/components/event-readers/EventReaders.tsx` |
|
||||
| Hardcoded color `#EE1D52`, `#9146ff`, `#ff4500`, `#cb3837`, `#f48024` | `cinny/src/app/components/url-preview/UrlPreviewCard.tsx` |
|
||||
| Massive number of hardcoded `backgroundColor` values | `cinny/src/app/features/lotus/chatBackground.ts` |
|
||||
| Hardcoded colors `#00FF88`, `#FF6B00` | `cinny/src/app/features/call/CallControls.tsx` |
|
||||
| Hardcoded fallback hexes in toast colors | `cinny/src/app/features/toast/LotusToastContainer.tsx` |
|
||||
|
||||
---
|
||||
|
||||
## 🌐 Localization, Accessibility & Performance
|
||||
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Localization | Hardcoded UI string: "Chat Room" | `src/app/components/create-room/CreateRoomTypeSelector.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Messages, photos, and videos." | `src/app/components/create-room/CreateRoomTypeSelector.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Voice Room" | `src/app/components/create-room/CreateRoomTypeSelector.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Live audio and video conversations." | `src/app/components/create-room/CreateRoomTypeSelector.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Download" | `src/app/components/image-viewer/ImageViewer.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Open Location" | `src/app/components/message/MsgTypeRenderers.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Thread" | `src/app/components/message/Reply.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "View" | `src/app/components/message/content/ImageContent.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Spoiler" | `src/app/components/message/content/ImageContent.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Retry" | `src/app/components/message/content/ImageContent.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Close" | `src/app/components/DeviceVerification.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Accept" | `src/app/components/DeviceVerification.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "They Match" | `src/app/components/DeviceVerification.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Okay" | `src/app/components/DeviceVerification.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Join Server" | `src/app/components/url-preview/UrlPreviewCard.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Invite" | `src/app/components/invite-user-prompt/InviteUserPrompt.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Files" | `src/app/components/upload-board/UploadBoard.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Send" | `src/app/components/upload-board/UploadBoard.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Upload Failed" | `src/app/components/upload-board/UploadBoard.tsx` | OPEN |
|
||||
| Localization | Hardcoded UI string: "Password" | `src/app/components/uia-stages/PasswordStage.tsx` | OPEN |
|
||||
| Bundle Size | Large unoptimized media asset (213KB) | `public/res/Lotus.png` | OPEN |
|
||||
| Matrix Logic | Inefficient repeated `mx.getRoom()` calls in component render loops | `src/app/features/lobby/Lobby.tsx` | OPEN |
|
||||
| Matrix Logic | Inefficient repeated `mx.getRoom()` calls in component render loops | `src/app/components/emoji-board/EmojiBoard.tsx` | OPEN |
|
||||
| Performance | Numerous event handlers (e.g., handleUserClick, handleReplyClick) lack `useCallback`, leading to unnecessary re-renders of message components. | `cinny/src/app/features/room/RoomTimeline.tsx` | OPEN |
|
||||
| Performance | The `submit` function and file handling callbacks (e.g., handleSendUpload) are re-created on every render, causing re-renders of the editor and toolbar components. | `cinny/src/app/features/room/RoomInput.tsx` | OPEN |
|
||||
| Accessibility | `button` for edit history lacks `aria-label` | `cinny/src/app/components/message/content/FallbackContent.tsx` | OPEN |
|
||||
| Accessibility | `button` for reaction lacks `aria-label` | `cinny/src/app/components/message/Reaction.tsx` | OPEN |
|
||||
| Accessibility | `button` for ThreadIndicator lacks `aria-label` | `cinny/src/app/components/message/Reply.tsx` | OPEN |
|
||||
| Accessibility | `button` for ReplyLayout lacks `aria-label` | `cinny/src/app/components/message/Reply.tsx` | OPEN |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Infrastructure, DevEx & Type Safety
|
||||
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Dependencies | `lodash` pinned to non-existent version `4.18.1` | `cinny/package.json` | OPEN |
|
||||
| Dependencies | Various pinned versions of `@atlaskit`, `matrix-js-sdk` | `cinny/package.json` | OPEN |
|
||||
| Dependencies | `matrix-js-sdk` pinned to Release Candidate (`41.6.0-rc.0`) | `cinny/package.json` | OPEN |
|
||||
| Dependencies | Unstable/experimental versions for build tools (`vite` 8.0.14, `typescript` 6.0.3, `eslint` 9.39.4) | `cinny/package.json` | OPEN |
|
||||
| CI/CD | `package-manager-cache` set to `false` | `cinny/.github/workflows/build-pull-request.yml` | OPEN |
|
||||
| CI/CD | Inefficient sequential execution in deployment | `cinny/.github/workflows/prod-deploy.yml` | OPEN |
|
||||
| CI/CD | Aggressive 1-minute timeout for Netlify deploy | `cinny/.github/workflows/prod-deploy.yml` | OPEN |
|
||||
| DevEx | Stale upstream bug tracker link/donations/CLA | `cinny/CONTRIBUTING.md` | OPEN |
|
||||
| DevEx | Alignment issue between README and CONTRIBUTING | `cinny/README.md` | OPEN |
|
||||
| Testing | No evident automated testing configuration/files | `cinny/src/` | OPEN |
|
||||
| Type Safety | Extensive use of `as any` type assertions | `cinny/src/` | OPEN |
|
||||
| Security | Hardcoded public CDN URL; consider moving to environment variable | /root/code/cinny/scripts/syncDecorations.mjs | OPEN |
|
||||
| Architecture | Modifying node_modules directly is brittle; use patch-package instead | /root/code/cinny/scripts/patch-folds.mjs | OPEN |
|
||||
| Robustness | Missing security headers (HSTS, CSP, etc.) and inefficient asset serving using rewrites instead of try_files | /root/code/cinny/contrib/nginx/cinny.domain.tld.conf | OPEN |
|
||||
| Robustness | Incomplete documentation/placeholder path in Caddyfile | /root/code/cinny/contrib/caddy/caddyfile | OPEN |
|
||||
| Matrix SDK | Inefficient listener management (`setMaxListeners: 150`) and incomplete SDK state transition handling. | `src/client/initMatrix.ts` | OPEN |
|
||||
| PWA Robustness | Service worker lacks caching strategy for application assets, resulting in no offline capability. | `cinny/src/sw.ts` | OPEN |
|
||||
| PWA Integrity | `manifest: false` in `vite.config.js` might prevent correct PWA installation if not handled externally. | `cinny/vite.config.js` | OPEN |
|
||||
| PII Leakage | Potential PII exposure via console.error (parameter e likely contains event data). | `cinny/src/app/plugins/call/CallEmbed.ts` | OPEN |
|
||||
| PII Leakage | Potential PII exposure via console.warn (parameter imgError/videoError/thumbError object). | `cinny/src/app/features/room/msgContent.ts` | OPEN |
|
||||
| PII Leakage | Potential PII exposure via console.error (parameter e likely contains event data). | `cinny/src/app/features/room/RoomInput.tsx` | OPEN |
|
||||
|
||||
## 🏗️ Architectural & Resilience Audit
|
||||
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Element Call Integration | Lacks robust iframe failure monitoring beyond initial 'preparing' event; can result in a permanently hung 'Loading...' state with no user-visible error or recovery path. | `src/app/plugins/call/CallEmbed.ts` | OPEN |
|
||||
| Component Resilience | `RoomTimeline` has no `ErrorBoundary` wrapper — a single malformed event crashing the renderer takes down the entire timeline with no fallback UI. | `src/app/features/room/RoomTimeline.tsx` | OPEN |
|
||||
| Component Resilience | `RoomInput` has no `ErrorBoundary` wrapper — a crash in the composer leaves users unable to send messages. | `src/app/features/room/RoomInput.tsx` | OPEN |
|
||||
| Fallback Logic | No explicit empty/error fallback for Matrix SDK data calls in `RoomTimeline`; relies purely on SDK internal error propagation, meaning silent failures show a blank timeline. | `src/app/features/room/RoomTimeline.tsx` | OPEN |
|
||||
| Dependency | Potential for complex dependency chains due to deep nesting in `src/app/features/` and `src/app/hooks/`. | `src/app/` | OPEN |
|
||||
| Hydration/Race Condition | The SyncState listener registered by useSyncState may miss the initial 'PREPARED' event if the client initializes synchronously from IndexedDB before the effect runs, leading to an infinite loading state. | `cinny/src/app/pages/client/ClientRoot.tsx` | OPEN |
|
||||
| Structure | High number of small, highly coupled utility hooks (`src/app/hooks/`) may obscure dependency graphs. | `src/app/hooks/` | OPEN |
|
||||
| Dead Code | Potential for unused CSS modules or UI components in `src/app/features/`. | `src/app/` | OPEN |
|
||||
| Security | Sensitive session data (access tokens, device ID) stored in `localStorage` is vulnerable to XSS. | `src/app/state/sessions.ts` | OPEN |
|
||||
| Privacy | Sensitive user status messages and expiry timestamps are persisted in `localStorage`. | `src/app/features/settings/account/Profile.tsx` | OPEN |
|
||||
| Privacy | Unsent composer drafts stored in `localStorage` without encryption could leak info on shared devices. | `src/app/features/room/RoomInput.tsx` | OPEN |
|
||||
| Persistence | Scheduled messages relying on fragile `localStorage` parsing are prone to data loss on session expiry or error. | `src/app/state/scheduledMessages.ts` | OPEN |
|
||||
| Bundle Bloat | Inefficient `lodash` import; risks including entire library instead of necessary utilities. | `cinny/package.json` | OPEN |
|
||||
| Bundle Bloat | Large `matrix-js-sdk` (RC version) dependency; high potential for tree-shaking overhead. | `cinny/package.json` | OPEN |
|
||||
| Build-Time Overhead | `lotusDenoise` plugin performs heavy, sequential `fs` operations during `closeBundle`, significantly slowing build times. | `cinny/vite.config.js` | OPEN |
|
||||
| Build-Time Overhead | Complex manual `viteStaticCopy` configuration requiring multiple renames and path manipulations; risks redundant processing. | `cinny/vite.config.js` | OPEN |
|
||||
| Architectural Debt | Redundant style variant logic in `SpacingVariant` could be simplified. | `cinny/src/app/components/message/layout/layout.css.ts` | OPEN |
|
||||
| Overhead Analysis | Potential CSS bloat from `DropTarget` composition across multiple recipes (`SidebarItem`, `SidebarFolder`). | `cinny/src/app/components/sidebar/Sidebar.css.ts` | OPEN |
|
||||
|
||||
## 🏗️ Git Workflow & History Audit
|
||||
|
||||
| Category | Issue Description | File Path | Status |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Workflow | Monolithic "Fix all bugs" commits (e.g., `10f6544e`, `aa48c9ef`) make `git bisect` difficult. | Git History | OPEN |
|
||||
| Workflow | Inconsistent commit message prefixes (e.g., `fix`, `feat`, `docs`, `assets`). | Git History | OPEN |
|
||||
| Workflow | Use of `fix` or `feat` for large-scale changes affecting multiple disparate systems (e.g., `938ead79`). | Git History | OPEN |
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
# Engineering Review: Multi-Model ML Noise Suppression Upgrade (P5-30)
|
||||
|
||||
## Overview
|
||||
|
||||
This PR implements a robust, modular, and high-fidelity client-side audio processing pipeline for noise suppression (NS) within Lotus Chat. It addresses issues with static noise artifacts, suboptimal sample rate resampling, and the lack of transparency in the audio processing chain.
|
||||
|
||||
## 1. Architectural Changes
|
||||
|
||||
### 1.1 Audio Processing Pipeline (`lotus-denoise.js`)
|
||||
|
||||
- **Decoupled Initialization:** The shim now treats the audio chain as a configurable graph: `Source` → `Noise Gate` (optional) → `ML Model` → `LiveKit`.
|
||||
- **Series Processing:** We enabled the browser-native suppressor (Google NSNet2) to run in series with the ML model. The native engine handles stationary noise (fan hum) efficiently, while the ML model focuses on transient "life" noise (keyboard clicks, mouse taps).
|
||||
- **Hardware Fidelity:** Removed forced `48kHz` capture constraints in `getUserMedia`. This allows high-end audio interfaces (e.g., Rode/Scarlett at 48kHz) to pass raw audio without low-quality browser-level resampling, which was previously creating "static" artifacts.
|
||||
- **SIMD Optimization:** Added runtime `WebAssembly.validate` checks to detect SIMD support. The pipeline dynamically selects `rnnoise_simd.wasm` over standard WASM if supported, reducing CPU utilization.
|
||||
- **Failure Resilience:** Wrapped the entire graph initialization in `Promise.all` + `try/catch`. If any component (WASM loading, AudioWorklet initialization) fails, the shim sends a `postMessage` failure report and falls back to the raw microphone stream, ensuring calls never drop due to suppression errors.
|
||||
|
||||
### 1.2 Multi-Model Support
|
||||
|
||||
Added support for 4 distinct processing models:
|
||||
|
||||
1. **RNNoise (Mozilla):** Default lightweight hybrid model.
|
||||
2. **Speex (Legacy):** DSP-based fallback for extremely low-CPU requirements.
|
||||
3. **DTLN (Balanced):** Deep learning model (~15% CPU). Improved transient handling.
|
||||
4. **DeepFilterNet 3 (Pro):** Studio-grade Deep Learning (~25-50%+ CPU). Designed for high-fidelity noise removal.
|
||||
|
||||
## 2. Infrastructure & Build Integration (`vite.config.js`)
|
||||
|
||||
- **Automated Asset Pipeline:** Added rules to copy model assets (TFLite models, WASM runtimes) from `node_modules` into the `denoise/` directory during build.
|
||||
- **CI-Friendly:** The copy logic now includes `console.warn` fallbacks for missing assets to prevent build failures in environments where `npm install` hasn't yet finished, facilitating robust CI/CD integration.
|
||||
- **Self-Hosting:** All assets are explicitly served from the `/denoise/` path, ensuring full privacy and avoiding external CDN dependencies at runtime.
|
||||
|
||||
## 3. UI & UX Improvements
|
||||
|
||||
### 3.1 Settings & Tuning (`General.tsx`)
|
||||
|
||||
- **Capability Detection:** Created `lotusDenoiseUtils.ts` to verify support for `AudioContext` and `AudioWorklet`. The ML option is programmatically disabled in unsupported browsers (e.g., Safari/Mobile) with a clear requirement list.
|
||||
- **Comparison Chart:** Added a UI table listing `Model`, `CPU Usage`, `Quality`, and `Transient Handling` to allow users to make informed decisions based on their hardware.
|
||||
- **Live Tuning:** Added a `MicMeter` component using an `AnalyserNode` to provide real-time visual feedback, enabling users to calibrate the **Noise Gate Threshold** (-100dB to 0dB) precisely to their microphone's noise floor.
|
||||
|
||||
### 3.2 Error Reporting
|
||||
|
||||
- **Inter-Iframe Comms:** The shim now reports status and failures to the parent `LotusChat` host via `window.parent.postMessage`.
|
||||
- **System Toasts:** Added `LotusDenoiseFeature` in `ClientNonUIFeatures.tsx`. It listens for these events and triggers a non-intrusive system toast if the noise suppression falls back to raw mic, ensuring users know their microphone status.
|
||||
|
||||
## 4. Technical Debt & Safety
|
||||
|
||||
- **Settings Persistence:** Added strongly-typed settings fields for `callDenoiseModel`, `callDenoiseNativeNS`, `callDenoiseGate`, and `callDenoiseGateThreshold` to `settings.ts`.
|
||||
- **Clean Teardown:** Improved `cleanup()` logic in `lotus-denoise.js` to ensure the `AudioContext` and `MediaStreamTracks` are properly released, preventing potential memory leaks or microphone "hanging" after calls.
|
||||
|
||||
## Testing Instructions for Senior Engineer
|
||||
|
||||
1. **Calibration:** Go to Settings, enable ML NS, toggle on Noise Gate, and click "Test Microphone". Confirm the meter reflects real-time audio.
|
||||
2. **Validation:** Test "Series Suppression ON" vs "OFF" with a fan running in the background to confirm native NS is effectively handling the stationary noise.
|
||||
3. **Fallback Test:** Introduce a malformed model request (via devtools console) to verify the System Toast notification functions.
|
||||
+393
-47
@@ -5,6 +5,23 @@
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Infrastructure & Maintenance
|
||||
|
||||
- [ ] **Upgrade Synapse to v1.155.0**
|
||||
- **Context:** Synapse 1.155.0 is the last version supporting Debian 12 Bookworm.
|
||||
- **Reference:** https://github.com/element-hq/synapse/releases/tag/v1.155.0
|
||||
- **Plan:** Review release notes, backup database and media store on LXC 151, perform upgrade in a staging environment if possible, then production. Prepare for OS migration to Debian 13 afterward.
|
||||
|
||||
---
|
||||
|
||||
## 📱 Quick Feature Additions
|
||||
|
||||
- [ ] **Full-Screen Camera Broadcasts**
|
||||
- **Context:** Element Call currently supports full-screening screenshares. We need to parity this functionality for camera broadcasts.
|
||||
- **Goal:** Users should be able to toggle any camera feed to full-screen mode, similar to the existing screenshare full-screen implementation.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ TDS DESIGN LAW — READ BEFORE TOUCHING ANY UI
|
||||
|
||||
> **ALL Lotus Terminal Design System (TDS) styling — colors, animations, glows, borders, fonts, spacing — MUST come exclusively from `/root/code/web_template/base.css` CSS variables.**
|
||||
@@ -40,32 +57,32 @@ Status: `[ ]` pending · `[~]` in progress · `[x]` completed
|
||||
|
||||
### Confirmed facts
|
||||
|
||||
| Finding | Impact |
|
||||
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| Finding | Impact |
|
||||
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| **MSC flags ON:** `msc4140` · `msc3771` · `msc3440.stable` · `msc4133.stable` · `simplified_msc3575` | All safe to use now |
|
||||
| **MSC flags OFF:** `msc4306` (thread subscriptions) · `msc3882` · `msc3912` · `msc4155` | These features are BLOCKED |
|
||||
| **MSC3266** room summary: returns 404 | Room Preview feature BLOCKED |
|
||||
| **MSC3892** relation redaction: not in flags | Reaction Redaction feature BLOCKED |
|
||||
| **MSC4260** report user: server at v1.12, endpoint may not exist | Report User feature BLOCKED |
|
||||
| **MSC4151** report room: HTTP 405 on GET = endpoint exists (POST only) | Report Room live ✅ |
|
||||
| `folds AvatarImage` does NOT accept children | Add frame/overlay inside `UserAvatar.tsx` itself — optional `frameName` prop |
|
||||
| No in-app toast system exists (was) | Built `ToastProvider` + Jotai queue; at `App.tsx:65` |
|
||||
| `useUnverifiedDeviceCount()` hook exists | `src/app/hooks/useDeviceVerificationStatus.ts:65-106` |
|
||||
| Voice player: `AudioContent.tsx:44-223` | Playback rate on hidden `<audio>` at line 217 |
|
||||
| `CallControl.setMicrophone(bool)` at `CallControl.ts:206-212` | For AFK auto-mute |
|
||||
| `CallControl.toggleSound()` at `CallControl.ts:230-251` | Push-to-deafen — just wire a hotkey to this |
|
||||
| matrix-js-sdk has NO arbitrary profile field methods | Use `mx.http.authedRequest()` for MSC4133 |
|
||||
| Sanitizer (`sanitize.ts`) allows table, div, span, a, code, hr | LFG HTML card is safe locally; test on Element/FluffyChat |
|
||||
| Sanitizer STRIPS `<math>`/MathML tags | Math/LaTeX task must also modify sanitizer |
|
||||
| Service worker EXISTS at `src/sw.ts` | Quick-reply task: add `notificationclick` handler |
|
||||
| `knockSupported()` utility exists at `matrix.ts:376-391` | Knock UX: only need "Request to Join" in `RoomIntro.tsx` |
|
||||
| `KeywordMessages.tsx` already has custom keyword push rules | Full push rule editor: only non-keyword rule types need new UI |
|
||||
| `getMatrixToRoom()` in `matrix-to.ts` generates invite URLs | Invite link: just add QR code to room settings |
|
||||
| Cindy CANNOT inject audio into EC call stream | In-call soundboard must be redesigned as local-only |
|
||||
| Folds uses vanilla-extract in non-TDS, NOT CSS custom properties | Custom accent color: must create new vanilla-extract theme variant dynamically |
|
||||
| Theme presets need ~50 CSS custom properties each | Significant design work before coding |
|
||||
| `useCallSpeakers.ts` CSS MutationObserver polling | Visual speaking indicator: TDS ring animation on top of existing data |
|
||||
| MSC3489/3672 live location: BOTH false on server | Live Location BLOCKED |
|
||||
| **MSC3266** room summary: returns 404 | Room Preview feature BLOCKED |
|
||||
| **MSC3892** relation redaction: not in flags | Reaction Redaction feature BLOCKED |
|
||||
| **MSC4260** report user: server at v1.12, endpoint may not exist | Report User feature BLOCKED |
|
||||
| **MSC4151** report room: HTTP 405 on GET = endpoint exists (POST only) | Report Room live ✅ |
|
||||
| `folds AvatarImage` does NOT accept children | Add frame/overlay inside `UserAvatar.tsx` itself — optional `frameName` prop |
|
||||
| No in-app toast system exists (was) | Built `ToastProvider` + Jotai queue; at `App.tsx:65` |
|
||||
| `useUnverifiedDeviceCount()` hook exists | `src/app/hooks/useDeviceVerificationStatus.ts:65-106` |
|
||||
| Voice player: `AudioContent.tsx:44-223` | Playback rate on hidden `<audio>` at line 217 |
|
||||
| `CallControl.setMicrophone(bool)` at `CallControl.ts:206-212` | For AFK auto-mute |
|
||||
| `CallControl.toggleSound()` at `CallControl.ts:230-251` | Push-to-deafen — just wire a hotkey to this |
|
||||
| matrix-js-sdk has NO arbitrary profile field methods | Use `mx.http.authedRequest()` for MSC4133 |
|
||||
| Sanitizer (`sanitize.ts`) allows table, div, span, a, code, hr | LFG HTML card is safe locally; test on Element/FluffyChat |
|
||||
| Sanitizer STRIPS `<math>`/MathML tags | Math/LaTeX task must also modify sanitizer |
|
||||
| Service worker EXISTS at `src/sw.ts` | Quick-reply task: add `notificationclick` handler |
|
||||
| `knockSupported()` utility exists at `matrix.ts:376-391` | Knock UX: only need "Request to Join" in `RoomIntro.tsx` |
|
||||
| `KeywordMessages.tsx` already has custom keyword push rules | Full push rule editor: only non-keyword rule types need new UI |
|
||||
| `getMatrixToRoom()` in `matrix-to.ts` generates invite URLs | Invite link: just add QR code to room settings |
|
||||
| Cindy CANNOT inject audio into EC call stream | In-call soundboard must be redesigned as local-only |
|
||||
| Folds uses vanilla-extract in non-TDS, NOT CSS custom properties | Custom accent color: must create new vanilla-extract theme variant dynamically |
|
||||
| Theme presets need ~50 CSS custom properties each | Significant design work before coding |
|
||||
| `useCallSpeakers.ts` CSS MutationObserver polling | Visual speaking indicator: TDS ring animation on top of existing data |
|
||||
| MSC3489/3672 live location: BOTH false on server | Live Location BLOCKED |
|
||||
|
||||
---
|
||||
|
||||
@@ -114,38 +131,66 @@ Status: `[ ]` pending · `[~]` in progress · `[x]` completed
|
||||
- Reading messages in the timeline (screen reader announces new messages)
|
||||
- Composing and sending a reply
|
||||
- Opening and closing modals (focus trap, return focus)
|
||||
- ARIA labels on all icon-only buttons
|
||||
**Scope:** Do NOT attempt to make every corner of the app AA-compliant in one pass — focus on the golden path (open app → find room → read → reply → send).
|
||||
**[AUDIT REQUIRED]** — Run an automated audit first: `npx axe-core` or browser DevTools accessibility tree. Document every violation before writing a single line of code. Prioritize by severity (critical > serious > moderate).
|
||||
**Complexity:** Medium-High (audit is the main work).
|
||||
- ARIA labels on all icon-only buttons
|
||||
|
||||
**Scope:** Do NOT attempt to make every corner of the app AA-compliant in one pass — focus on the golden path (open app → find room → read → reply → send).
|
||||
**[AUDIT REQUIRED]** — Run an automated audit first: `npx axe-core` or browser DevTools accessibility tree. Document every violation before writing a single line of code. Prioritize by severity (critical > serious > moderate).
|
||||
|
||||
**Investigation Findings:**
|
||||
- **Root Cause:** Inconsistent focus management, missing `aria-live` regions for dynamic timeline updates, and sparse global keyboard shortcuts.
|
||||
- **Approach:** Standardize `focus-trap-react` usage (reference `RoomNavItem.tsx`). Add `aria-live` regions to the timeline. Expand `useKeyDown.ts` for section navigation shortcuts.
|
||||
- **Complexity:** Medium-High (audit is the main work).
|
||||
|
||||
---
|
||||
|
||||
### [ ] P3-8 · Thread Panel (full side drawer)
|
||||
|
||||
**⚠️ LARGEST FEATURE — requires its own planning session before implementation.**
|
||||
**What:** A right-side drawer for threaded conversations. Currently "Reply in Thread" exists but there is no panel to read or write thread replies.
|
||||
Features:
|
||||
**What:** A right-side drawer for threaded conversations. Currently "Reply in Thread" exists but there is no panel to read or write thread replies.
|
||||
|
||||
Features:
|
||||
- Click "Reply in Thread" → opens thread drawer on the right
|
||||
- Thread root event shown at the top of the panel
|
||||
- Full message rendering for all in-thread replies (reuse timeline components)
|
||||
- Reply input at the bottom (full composer with formatting, emoji, etc.)
|
||||
- Unread count badge on the thread button in the main timeline
|
||||
- Keyboard shortcut to close thread panel
|
||||
**Architecture:**
|
||||
- Keyboard shortcut to close thread panel
|
||||
|
||||
**Architecture:**
|
||||
- New Jotai atom: `activeThreadEventId: string | null`
|
||||
- New component: `src/app/features/room/thread/ThreadPanel.tsx`
|
||||
- Rendered alongside `RoomView` as a conditional right panel (mirror the members drawer pattern)
|
||||
- Filter events in timeline to `m.thread` relation for the active root event ID
|
||||
- Shares the same `mx` client and room reference as the main timeline
|
||||
**[AUDIT REQUIRED]** — Deeply audit how `m.thread` relation events are currently stored and retrieved in the matrix-js-sdk. Understand the thread aggregation API: `GET /rooms/{roomId}/relations/{eventId}/m.thread`. Check if `RoomTimeline.tsx` currently filters out thread replies from the main timeline (it should — confirm).
|
||||
**Complexity:** High.
|
||||
- Shares the same `mx` client and room reference as the main timeline
|
||||
|
||||
**[AUDIT REQUIRED]** — Deeply audit how `m.thread` relation events are currently stored and retrieved in the matrix-js-sdk. Understand the thread aggregation API: `GET /rooms/{roomId}/relations/{eventId}/m.thread`. Check if `RoomTimeline.tsx` currently filters out thread replies from the main timeline (it should — confirm).
|
||||
|
||||
**Investigation Findings:**
|
||||
- **Root Cause:** Current `m.thread` events are treated as standard `m.room.message` events and rendered in the main timeline.
|
||||
- **Approach:** Introduce new Jotai atom `activeThreadEventId`. Create `ThreadPanel.tsx`. Update `RoomTimeline.tsx` to filter out thread relations (`m.relates_to`). Implement aggregation fetch using `GET /rooms/{roomId}/relations/{eventId}/m.thread`. Use `thread.timelineSet` directly for the most accurate thread view.
|
||||
- **Complexity:** High.
|
||||
|
||||
---
|
||||
|
||||
## Priority 4 — Specialized, high complexity, or low priority
|
||||
|
||||
### [ ] P4-7 · Virtualized Infinite Scroll for Search Results
|
||||
|
||||
**What:** Replace the manual "load more" button with an automated, virtualized infinite scroll for search results.
|
||||
**Approach:** Utilize `@tanstack/react-virtual` in `MessageSearch.tsx` to handle the `nextToken` automatically as the user scrolls.
|
||||
|
||||
### [ ] P4-8 · Encrypted Message Search Indexing & Caching
|
||||
|
||||
**What:** Implement a persistent local cache for search results, optimized for encrypted rooms.
|
||||
**Approach:** Use `IndexedDB` to store search metadata (event IDs, timestamps) to prevent redundant server-side decryption/fetching.
|
||||
|
||||
### [ ] P4-9 · Advanced Search Filter UI
|
||||
|
||||
**What:** Introduce a more robust search filter UI in `SearchFilters.tsx`.
|
||||
**Approach:** Add UI components for easier filtering and a visual date-range picker that correctly maps to `fromTs` and `toTs`.
|
||||
|
||||
---
|
||||
|
||||
### [ ] P4-1 · Thread Notification Mode Per-Thread (MSC3771)
|
||||
|
||||
**Spec:** MSC3771 (stable). Depends on Thread Panel (#P3-8).
|
||||
@@ -169,7 +214,7 @@ Features:
|
||||
**Spec:** CS-API §11.5 (stable) — `formatted_body` can contain LaTeX.
|
||||
**What:** Render `$...$` or `$$...$$` LaTeX expressions in message bodies. Use KaTeX (lightweight, ~100KB, renders server-side-compatible CSS). Must gracefully fall back to raw LaTeX text if KaTeX fails.
|
||||
**Note:** This is LOW PRIORITY — only useful for academic/technical communities. Implement last.
|
||||
**[AUDIT REQUIRED]** — Confirm KaTeX bundle size impact on the Vite bundle. Check if matrix-js-sdk's HTML sanitizer strips LaTeX before it reaches the renderer. The formatted_body sanitization pipeline is the main risk here.
|
||||
**[AUDIT REQUIRED]** — Confirm KaTeX bundle size impact on the Vite bundle. Check if matrix-js-sdk's HTML sanitizer strips LaTeX before it reaches the renderer. The formatted_body sanitization pipeline is the main risk here. (Confirmed: sanitizer STRIPS `<math>` tags — must be patched alongside the renderer.)
|
||||
**Complexity:** Low-Medium.
|
||||
|
||||
---
|
||||
@@ -201,23 +246,24 @@ Features:
|
||||
|
||||
**What:** A hex/HSL color picker in Settings → Appearance. Chosen color replaces the primary accent throughout the UI: buttons, badges, active states, highlights, presence dot, links. Applied via a CSS custom property override injected into `<head>`.
|
||||
**IMPORTANT:** This feature is completely inactive when TDS is enabled — TDS has its own fixed palette. Add this setting under a "Non-TDS Themes" section that is hidden when TDS is active.
|
||||
**[AUDIT REQUIRED]** Identify all CSS custom properties that constitute the "accent color" in non-TDS mode. Map them to the folds/vanilla-extract token names.
|
||||
**[AUDIT REQUIRED]** Identify all CSS custom properties that constitute the "accent color" in non-TDS mode. Map them to the folds/vanilla-extract token names. (Confirmed: folds uses vanilla-extract, NOT CSS custom properties — must create a new vanilla-extract theme variant dynamically.)
|
||||
**Complexity:** Medium.
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-2 · Additional Color Theme Presets
|
||||
|
||||
**What:** 5 new one-click theme presets alongside TDS. Each must be a complete, polished system with proper contrast ratios (WCAG AA). All implemented as vanilla-extract themes matching the existing TDS pattern.
|
||||
Themes:
|
||||
**What:** 5 new one-click theme presets alongside TDS. Each must be a complete, polished system with proper contrast ratios (WCAG AA). All implemented as vanilla-extract themes matching the existing TDS pattern.
|
||||
|
||||
Themes:
|
||||
1. **Cyberpunk** — deep navy bg (`#0a0015`), electric purple (`#bf5fff`) + hot pink (`#ff2d9b`) accents, neon glow
|
||||
2. **Ocean** — deep sea blue bg (`#020b18`), teal (`#00c9b1`) + aqua (`#0096d6`) accents, soft feel
|
||||
3. **Blood Red** — near-black bg (`#0d0203`), deep crimson (`#7a0010`) + bright red (`#ff2233`) accents
|
||||
4. **Classic Matrix** — pure black bg (`#000000`), phosphor green (`#00ff41`) text + accents
|
||||
5. **Midnight** — dark charcoal (`#111827`), cool blue-grey (`#6b7ca8`) accents, clean minimal
|
||||
**[AUDIT REQUIRED]** Study `src/lotus-terminal.css.ts` for the full token list before designing themes. All tokens must be covered.
|
||||
**Complexity:** Medium (design effort is the main cost).
|
||||
5. **Midnight** — dark charcoal (`#111827`), cool blue-grey (`#6b7ca8`) accents, clean minimal
|
||||
|
||||
**[AUDIT REQUIRED]** Study `src/lotus-terminal.css.ts` for the full token list before designing themes. All tokens must be covered (~50 CSS custom properties each).
|
||||
**Complexity:** Medium (design effort is the main cost).
|
||||
|
||||
---
|
||||
|
||||
@@ -227,6 +273,18 @@ Themes:
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-5 · Intersection-Based Lazy Loading
|
||||
|
||||
**What:** Use `IntersectionObserver` to trigger media decryption and loading only when components approach the viewport.
|
||||
**Approach:** Reduce initial memory footprint and improve timeline load times by deferring decryption of images/videos until they are visible.
|
||||
|
||||
### [ ] P5-6 · Context-Aware Thumbnail Previews
|
||||
|
||||
**What:** Enhance thumbnail rendering in the timeline for consistent, polished aesthetics.
|
||||
**Approach:** Use CSS `object-fit: cover` with improved focal-point centering within `ThumbnailContent` to prevent media stretching or awkward aspect-ratio cropping.
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-15 · In-Call Soundboard
|
||||
|
||||
**What:** Grid of short audio clips playable into the call audio stream via Web Audio API (AudioBufferSourceNode → MediaStreamDestinationNode → mixed with mic). Built-in clips + user-uploadable custom clips (stored as mxc://). Accessible from call controls bar.
|
||||
@@ -238,7 +296,7 @@ Themes:
|
||||
### [ ] P5-20 · Quick Reply from Browser Notification
|
||||
|
||||
**What:** Inline reply field in browser notification toasts via Notification Actions API. Reply sends as threaded reply to the triggering message.
|
||||
**[AUDIT REQUIRED]** (1) Verify browser Notification Actions API support in target browsers. (2) This requires a Service Worker to handle the reply event — confirm if Lotus Chat has one or needs one.
|
||||
**[AUDIT REQUIRED]** (1) Verify browser Notification Actions API support in target browsers. (2) Confirmed: service worker EXISTS at `src/sw.ts` — add `notificationclick` handler there.
|
||||
**Complexity:** Medium-High.
|
||||
|
||||
---
|
||||
@@ -246,9 +304,15 @@ Themes:
|
||||
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
|
||||
|
||||
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
|
||||
**Shipped:** 3-tier setting (Off / Browser-native / ML beta) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (3-Tier)".
|
||||
**Shipped:** 3-tier setting (Off / Browser-native / ML) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (Advanced Multi-Tier)".
|
||||
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps.
|
||||
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC (AEC runs on the native track) — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta". Validate echo quality on real multi-party calls after deploy.
|
||||
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta".
|
||||
|
||||
**Model Roadmap (priority order):**
|
||||
- [ ] **Verify DTLN** (16 kHz narrowband fix) in a real call before investing further — wired but unverified.
|
||||
- [ ] **DeepFilterNet 3** — best self-hostable upgrade: Rust→WASM, CPU real-time, 48 kHz fullband. Effort: self-host `df_bg.wasm` + DFN3 ONNX model, wire a 48 kHz worklet.
|
||||
- [ ] **Desktop-only / HW-gated:** FRCRN or NVIDIA Maxine (RTX/Tensor only) — impossible in-browser; would run in Tauri Rust backend + bridge a virtual mic into the webview. Must detect capability and only offer on supported hardware; web falls back to RNNoise.
|
||||
- **Excluded:** Krisp (LiveKit Cloud only); FRCRN/Maxine on web (GPU/server-bound).
|
||||
|
||||
---
|
||||
|
||||
@@ -281,13 +345,102 @@ Themes:
|
||||
|
||||
### [ ] P5-40 · Desktop — Proactive Update Notifications (Tauri)
|
||||
|
||||
**What:** Automatically check for app updates on launch and periodically during long sessions. If an update is available, show an in-app toast or badge (e.g. on the Settings icon) to alert the user without requiring them to manual check in settings.
|
||||
**What:** Automatically check for app updates on launch and periodically during long sessions. If an update is available, show an in-app toast or badge (e.g., on the Settings icon) to alert the user without requiring a manual check in settings.
|
||||
**Mechanism:** Use the `useTauriUpdater` hook in a global component like `ClientNonUIFeatures.tsx`.
|
||||
**Note:** Ensure the check is throttled (e.g. once every 12 hours) to avoid redundant Tauri commands.
|
||||
**Note:** Ensure the check is throttled (e.g., once every 12 hours) to avoid redundant Tauri commands.
|
||||
**Complexity:** Low-Medium.
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-41 · Desktop — Native WinRT Toast Notifications
|
||||
|
||||
**What:** Replace emulated notifications with native WinRT Toast notifications.
|
||||
**Approach:** Implement native WinRT Toast integration using `windows-rs` to enable full Action Center integration, including native Quick Reply functionality.
|
||||
|
||||
### [ ] P5-42 · Desktop — Persistent Background Sync
|
||||
|
||||
**What:** Maintain light connection to homeserver when WebView2 is suspended.
|
||||
**Approach:** Implement a headless Rust sidecar to fetch unread counts/notifications while the webview is suspended to ensure instant notification delivery.
|
||||
|
||||
### [ ] P5-43 · Desktop — System Media Transport Controls (SMTC)
|
||||
|
||||
**What:** Integrate with Windows SMTC for volume flyout call/media control.
|
||||
**Approach:** Use Windows SMTC API to expose call status, mic mute/unmute, and media controls to the Windows volume flyout/media overlay.
|
||||
|
||||
### [ ] P5-44 · Desktop — Taskbar Thumbnail Toolbar
|
||||
|
||||
**What:** Add persistent call controls to the taskbar preview.
|
||||
**Approach:** Implement a COM thumbnail toolbar in the application preview window, featuring Mute/Deafen/End Call buttons.
|
||||
|
||||
### [ ] P5-46 · Desktop — System Power Management (Call Continuity)
|
||||
|
||||
**What:** Prevent system sleep/hibernate during active calls.
|
||||
**Approach:** Use Tauri/Rust `power-manager` or platform-specific APIs to block system power saving states while a voice/video session is active.
|
||||
|
||||
### [ ] P5-47 · Desktop — TDS-Styled Native Window Chrome
|
||||
|
||||
**What:** Replace system titlebar with custom Lotus TDS chrome.
|
||||
**Approach:** Configure Tauri window (`decorations: false`) and implement custom, TDS-token compliant titlebar controls (Close/Max/Min) for a cohesive UI.
|
||||
|
||||
### [ ] P5-48 · Desktop — Native File System Drag-and-Drop Improvements
|
||||
|
||||
**What:** Enhance drag-and-drop support for Windows.
|
||||
**Approach:** Improve handling for Windows file shortcuts, recursive folder uploads, and shell-integrated "Send To" context menu actions.
|
||||
|
||||
### [ ] P5-49 · Desktop — Network Awareness (NCSI Integration)
|
||||
|
||||
**What:** Proactively detect Windows network connectivity changes.
|
||||
**Approach:** Integrate with the Windows Network Connectivity Status Indicator (NCSI) API to improve offline mode transition latency and network recovery.
|
||||
|
||||
### [ ] P5-50 · Desktop — Windows Hardware-Accelerated Media Pipeline
|
||||
|
||||
**What:** Replace standard browser decoding with native Windows Media Foundation.
|
||||
**Approach:** Leverage DirectShow/Media Foundation to offload video/audio decoding from the CPU to the GPU, significantly reducing power consumption and latency during calls.
|
||||
|
||||
### [ ] P5-51 · Desktop — Federated "Identity Contexts" (Isolation Manager)
|
||||
|
||||
**What:** Compartmentalize sessions, local databases, and caches into isolated "Contexts."
|
||||
**Approach:** Implement a zero-leak boundary for personas (e.g., Work vs. Personal) by isolating `IndexedDB`, filesystem caches, and session persistence per context.
|
||||
**Priority:** Extreme Low (Multi-sprint/Architectural).
|
||||
|
||||
### [~] P5-52 · Desktop — Room-Level Sync Governor (Performance Control) [STILL_CONSIDERING]
|
||||
|
||||
**What:** Granular sync tuning for individual rooms.
|
||||
**Approach:** Allow per-room overrides for sync frequency and event type filtering (e.g., disable read receipts/typing in heavy rooms) to optimize performance. Implementation requires careful UX to prevent complexity fatigue.
|
||||
|
||||
### [ ] P5-53 · Desktop — Local-Only "Scripting" Plugin System (Tampermonkey-like)
|
||||
|
||||
**What:** A sandboxed environment for local execution of user scripts on Matrix events.
|
||||
**Approach:** Implement a WASM-based execution engine that allows users to write local-only, client-side scripts to interact with incoming Matrix events, trigger sounds/notifications, or inject custom UI elements based on event payload rules. Designed for privacy — all logic runs exclusively on the local machine.
|
||||
|
||||
### [ ] P5-55 · Desktop — Composer Toolbar Drag-and-Drop Reordering
|
||||
|
||||
**What:** Allow users to reorder toolbar icons via drag-and-drop.
|
||||
**Approach:** Extend the current settings-based toolbar toggle system to include a drag-and-drop UI mode in the composer settings, allowing users to personalize their icon order.
|
||||
|
||||
### [ ] P5-56 · Desktop — Windows "Focus Assist" (DND) Sync
|
||||
|
||||
**What:** Automatically toggle notification state based on Windows Focus Assist.
|
||||
**Approach:** Integrate with the Windows `NotificationCenter` / `Focus` state via Tauri/Rust to automatically enable/disable Lotus Chat's internal notification suppression mode when Windows Focus Assist is toggled.
|
||||
|
||||
### [ ] P5-57 · Desktop — Visual Draft Persistence Indicator
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Features to Add
|
||||
|
||||
- [ ] **Mobile Audit:** Comprehensive audit of all features in LOTUS_FEATURES.md for mobile PWA usability and layout responsiveness.
|
||||
- [ ] **Remind Me Later:** Slack-style reminders for messages (ping user at a set time, add to bookmarks).
|
||||
- **Root Cause:** Feature does not exist. Reminders require persistence in Matrix account data and a background worker to trigger notifications.
|
||||
- **Approach:** Expand `useReminders.ts` to include `removeReminder`. Add a "Remind me" menu item to `Message.tsx` that triggers a `DatePicker`/`TimePicker` modal (reusing logic from `JumpToTime.tsx`). Implement a `ReminderMonitor` in `ClientNonUIFeatures.tsx` that polls active reminders from `io.lotus.reminders` and pushes to `toastQueueAtom` in `state/toast.ts` when due.
|
||||
- **Complexity:** Medium.
|
||||
- [ ] **Mobile Bookmarks:** Fix visibility issue where bookmarks do not show up when selected via PWA on mobile.
|
||||
- **Root Cause:** `ClientLayout.tsx` explicitly restricts `BookmarksPanel` rendering to `ScreenSize.Desktop` (lines 51-56).
|
||||
- **Approach:** Update `ClientLayout.tsx` to remove the `ScreenSize.Desktop` restriction. Pass `isMobile={screenSize !== ScreenSize.Desktop}` to `BookmarksPanel`. `BookmarksPanel.tsx` already supports this prop (line 127) to enable full-screen absolute positioning and reactive layout.
|
||||
- **Complexity:** Low.
|
||||
|
||||
---
|
||||
|
||||
## Blocked Features
|
||||
|
||||
These features are confirmed desirable but cannot be built until the listed dependency is resolved.
|
||||
@@ -335,6 +488,199 @@ Research whether Matrix spec or MSC4133 (v1.16) defines a standard profile banne
|
||||
|
||||
---
|
||||
|
||||
## 📚 Implementation Reference
|
||||
|
||||
Exhaustive, low-level implementation details for backlog items. Follow these patterns to ensure code is "Lotus-perfect" (idiomatic, performant, and TDS-compliant).
|
||||
|
||||
### P3-8 · Thread Panel (Full Side Drawer)
|
||||
|
||||
**Architecture:** Mirror the `MembersDrawer` pattern but with a specialized timeline.
|
||||
|
||||
- **State (`src/app/state/room/thread.ts`):**
|
||||
```typescript
|
||||
export const activeThreadIdAtom = atom<string | null>(null);
|
||||
```
|
||||
- **Layout (`src/app/features/room/Room.tsx`):** Insert `ThreadPanel` conditionally alongside `RoomTimeline`:
|
||||
```tsx
|
||||
{activeThreadId && (
|
||||
<>
|
||||
<Line variant="Background" direction="Vertical" size="300" />
|
||||
<ThreadPanel roomId={roomId} threadId={activeThreadId} />
|
||||
</>
|
||||
)}
|
||||
```
|
||||
- **Component (`src/app/features/room/thread/ThreadPanel.tsx`):** Use `room.getThread(threadId)` from the SDK. Render a `Header` with a "Close" button that sets `activeThreadIdAtom` to `null`. Reuse `RoomTimeline` but pass a filtered `EventTimelineSet`. Use `thread.timelineSet` directly for the most accurate thread view.
|
||||
|
||||
---
|
||||
|
||||
### P4-4 · Math / LaTeX Rendering
|
||||
|
||||
**Mechanism:** KaTeX injection into the HTML parser.
|
||||
|
||||
- **Sanitizer (`src/app/utils/sanitize.ts`):** Allow KaTeX-specific tags and classes (e.g., `span`, `annotation`, `math`). Use a specialized allowed list for math blocks.
|
||||
- **Parser (`src/app/plugins/react-custom-html-parser.tsx`):** Detect `$ ... $` and `$$ ... $$` patterns in text nodes:
|
||||
```tsx
|
||||
if (node.type === 'text') {
|
||||
const parts = node.data.split(/(\$\$.*?\$\$|\$.*?\$)/g);
|
||||
return parts.map(p => {
|
||||
if (p.startsWith('$')) return <KaTeX math={p.replace(/\$/g, '')} />;
|
||||
return p;
|
||||
});
|
||||
}
|
||||
```
|
||||
- **CSS (`src/app/styles/CustomHtml.css.ts`):** Import `katex/dist/katex.min.css` only when a math block is rendered to save initial bundle size.
|
||||
|
||||
---
|
||||
|
||||
### P4-6 · OIDC / SSO Next-Gen Auth (MSC3861)
|
||||
|
||||
**Mechanism:** Matrix Authentication Service (MAS) Integration.
|
||||
|
||||
- **Architecture:** Shift from password-based `/login` to OAuth2 `authorization_code` flow.
|
||||
- **Key Files:** `src/app/pages/auth/Login.tsx` and `src/app/hooks/useAuth.ts`.
|
||||
- **Implementation:** Use `oidc-client-ts` or a similar lightweight OIDC library. Check for `m.authentication` in `/.well-known/matrix/client`. Redirect to the MAS authorization endpoint. Handle the callback in a new `OidcCallback` route and store the OIDC `refresh_token`.
|
||||
|
||||
---
|
||||
|
||||
### P5-1 · Custom Accent Color Picker (Non-TDS only)
|
||||
|
||||
**Mechanism:** Dynamic CSS variable injection.
|
||||
|
||||
- **Setting (`src/app/state/settings.ts`):** Add `customAccentColor: string` (hex).
|
||||
- **Manager (`src/app/pages/ThemeManager.tsx`):** Inside the `useEffect` that monitors theme changes:
|
||||
```typescript
|
||||
if (!lotusTerminal && customAccentColor) {
|
||||
document.documentElement.style.setProperty('--lt-accent-orange', customAccentColor);
|
||||
document.documentElement.style.setProperty('--lt-accent-orange-glow', `${customAccentColor}80`);
|
||||
}
|
||||
```
|
||||
- **UI (`src/app/features/settings/general/General.tsx`):** Use `<Input type="color">`. Hide this section if `lotusTerminal` is `true`.
|
||||
|
||||
---
|
||||
|
||||
### P5-15 · In-Call Soundboard
|
||||
|
||||
**Mechanism:** Local-to-Global Audio Bridge via Web Audio API.
|
||||
|
||||
- Create an `AudioContext` and a `MediaStreamDestinationNode`.
|
||||
- Create an `AudioBufferSourceNode` for each clip.
|
||||
- Route the mic `MediaStream` and the clip source to the destination node.
|
||||
- Pass the destination's `.stream` to the call bridge.
|
||||
|
||||
---
|
||||
|
||||
### P5-20 · Quick Reply from Browser Notification
|
||||
|
||||
**Mechanism:** Service Worker `notificationclick` Action.
|
||||
|
||||
```typescript
|
||||
// src/sw.ts
|
||||
self.addEventListener('notificationclick', (event) => {
|
||||
if (event.action === 'reply' && event.reply) {
|
||||
const { roomId, threadId } = event.notification.data;
|
||||
const session = sessions.get(event.clientId);
|
||||
fetch(`${session.baseUrl}/_matrix/client/v3/rooms/${roomId}/send/m.room.message`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${session.accessToken}` },
|
||||
body: JSON.stringify({
|
||||
msgtype: 'm.text',
|
||||
body: event.reply,
|
||||
'm.relates_to': threadId ? { rel_type: 'm.thread', event_id: threadId } : undefined,
|
||||
}),
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P5-30 · Advanced ML Noise Suppression — Model Roadmap
|
||||
|
||||
See shipped implementation in LOTUS_FEATURES.md → "Noise Suppression (Advanced Multi-Tier)".
|
||||
|
||||
**Models status:**
|
||||
- **RNNoise** (sapphi, 48 kHz) — ✅ working, default fallback. Keep — runs on any hardware.
|
||||
- **Speex** (sapphi, 48 kHz) — ✅ working, low value; candidate to drop.
|
||||
- **DTLN** (@workadventure, 16 kHz) — 🟡 wired; sample-rate fix applied (was robotic at 48 kHz). **TODO: verify in a real call.** Narrowband (16 kHz) = slightly telephone-y even when correct.
|
||||
|
||||
**Constraints:** client-side AudioWorklet, fully self-hosted, no GPU, self-hosted SFU (no LiveKit Cloud).
|
||||
|
||||
**Roadmap:**
|
||||
- [ ] Verify DTLN 16 kHz fix in a real call.
|
||||
- [ ] **DeepFilterNet 3** — best self-hostable upgrade: Rust→WASM, CPU real-time, 48 kHz fullband. Self-host `df_bg.wasm` + DFN3 ONNX model; wire a 48 kHz worklet. Audio quality unverifiable without a real-call test.
|
||||
- [ ] **Desktop-only / HW-gated:** FRCRN (Alibaba) or NVIDIA Maxine (RTX/Tensor only). Runs in Tauri Rust backend + bridges a virtual mic into the webview. Must detect capability; web + weak HW falls back to RNNoise/DTLN.
|
||||
|
||||
---
|
||||
|
||||
### P5-31 · Granular Voice & Screenshare Quality Controls
|
||||
|
||||
**Mechanism:** WebRTC Encoding Parameters + Backend Quality Guard.
|
||||
|
||||
- **State Event:** `io.lotus.room_quality` (state key `""`) containing:
|
||||
```json
|
||||
{ "audio_bitrate": 128000, "screen_max_res": "1080p", "screen_max_fps": 60 }
|
||||
```
|
||||
- **Screenshare:** In `src/app/plugins/call/CallControl.ts`, map the "Quality" setting to `getDisplayMedia` constraints.
|
||||
- **Audio Bitrate:** After the call joins, find the `RTCRtpSender` for the audio track:
|
||||
```typescript
|
||||
const sender = peerConnection.getSenders().find(s => s.track?.kind === 'audio');
|
||||
const params = sender.getParameters();
|
||||
params.encodings[0].maxBitrate = roomBitrate || 128000;
|
||||
await sender.setParameters(params);
|
||||
```
|
||||
- **Backend Sidecar:** Extend `voice-limit-guard.py` (LXC 151) to fetch `io.lotus.room_quality` and inject limits into the LiveKit JWT or return them as an authorized config packet.
|
||||
|
||||
---
|
||||
|
||||
### P5-40 · Desktop — Proactive Update Notifications (Tauri)
|
||||
|
||||
**Key Files:** `src/app/hooks/useTauriUpdater.ts`, `src/app/pages/client/ClientNonUIFeatures.tsx`, `src/app/features/toast/LotusToastContainer.tsx`.
|
||||
|
||||
1. Create a `TauriUpdateFeature` component. Use `useTauriUpdater()` to get the `check` function and `status`.
|
||||
2. In a `useEffect`, call `check()` on mount and then on a `setInterval` (every 12 hours).
|
||||
3. When status transitions to `{ state: 'available', version: '...' }`, fire a Lotus Toast: "Lotus Chat v[version] is available!" with an "Update" button that calls `install()`.
|
||||
4. Store `lastCheck` timestamp in `localStorage` to prevent redundant checks on refresh.
|
||||
|
||||
---
|
||||
|
||||
### Mobile Bookmarks Visibility Fix
|
||||
|
||||
**Issue:** `ClientLayout.tsx` explicitly restricts `BookmarksPanel` to `ScreenSize.Desktop` (lines 51-56).
|
||||
|
||||
```tsx
|
||||
// ClientLayout.tsx
|
||||
{bookmarksOpen && (
|
||||
<BookmarksPanel
|
||||
onClose={() => setBookmarksOpen(false)}
|
||||
isMobile={screenSize !== ScreenSize.Desktop}
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
`BookmarksPanel.tsx` already supports the `isMobile` prop (line 127) to enable full-screen absolute positioning. No other changes required.
|
||||
|
||||
---
|
||||
|
||||
### Remind Me Later (Slack-style)
|
||||
|
||||
**Mechanism:** Account Data + Timer/Service Worker.
|
||||
|
||||
- **Storage (`src/app/hooks/useReminders.ts`):** Store in account data `io.lotus.reminders` as `Array<{ id: string, roomId: string, eventId: string, timestamp: number }>`.
|
||||
- **Context Menu (`src/app/features/room/message/MessageContextMenu.tsx`):** Add "Remind me" option → opens date/time picker modal (reuse `JumpToTime.tsx` logic).
|
||||
- **Trigger (foreground):** `setTimeout` in a hook inside `ReminderMonitor` in `ClientNonUIFeatures.tsx` → pushes to `toastQueueAtom` in `state/toast.ts` when due.
|
||||
- **Trigger (background):** Use Service Worker — `setTimeout` in the main thread will not fire when the PWA is suspended.
|
||||
|
||||
---
|
||||
|
||||
### Mobile Usability Audit — Methodology
|
||||
|
||||
1. **Viewport & Touch:** All interactive elements must have at least `44px × 44px` touch targets. Audit for horizontal overflow (horizontal scrolling must be disabled).
|
||||
2. **Modal Responsiveness:** All modals (Settings, Profile, etc.) MUST cover the full screen on mobile, not float as overlays.
|
||||
3. **Sidebar / Panels:** On mobile, sidebar panels (Members, Bookmarks, Media) must become full-screen overlays (using a `Drawer` or `Modal` pattern) rather than side-by-side flexbox panels.
|
||||
4. **Input & Composer:** Ensure the composer doesn't get obscured by the mobile keyboard. Test focus trap and blur behaviors.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### ⚠️ TDS DESIGN LAW (repeated here for emphasis)
|
||||
|
||||
@@ -1,214 +0,0 @@
|
||||
# Lotus Chat — Technical Implementation Field Guide
|
||||
|
||||
**Date:** June 2026
|
||||
|
||||
This document provides exhaustive, low-level implementation details for the remaining items in `LOTUS_TODO.md`. Follow these patterns to ensure code is "Lotus-perfect" (idiomatic, performant, and TDS-compliant).
|
||||
|
||||
---
|
||||
|
||||
## 🧵 Priority 3 — Higher Complexity
|
||||
|
||||
### P3-8 · Thread Panel (Full Side Drawer)
|
||||
|
||||
**Architecture:** Mirror the `MembersDrawer` pattern but with a specialized timeline.
|
||||
|
||||
- **1. State (src/app/state/room/thread.ts):**
|
||||
```typescript
|
||||
export const activeThreadIdAtom = atom<string | null>(null);
|
||||
```
|
||||
- **2. Layout (src/app/features/room/Room.tsx):**
|
||||
Insert the `ThreadPanel` conditionally alongside the `RoomTimeline`.
|
||||
```tsx
|
||||
{
|
||||
activeThreadId && (
|
||||
<>
|
||||
<Line variant="Background" direction="Vertical" size="300" />
|
||||
<ThreadPanel roomId={roomId} threadId={activeThreadId} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
```
|
||||
- **3. Component (src/app/features/room/thread/ThreadPanel.tsx):**
|
||||
- Use `room.getThread(threadId)` from the SDK.
|
||||
- Render a `Header` with a "Close" button that sets `activeThreadIdAtom` to `null`.
|
||||
- Reuse `RoomTimeline` but pass a filtered `EventTimelineSet`.
|
||||
- **Pro Tip:** Use `thread.timelineSet` directly for the most accurate thread view.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Priority 4 — Specialized Features
|
||||
|
||||
### P4-4 · Math / LaTeX Rendering
|
||||
|
||||
**Mechanism:** KaTeX injection into the HTML parser.
|
||||
|
||||
- **1. Sanitizer (src/app/utils/sanitize.ts):**
|
||||
You must allow KaTeX-specific tags and classes (e.g., `span`, `annotation`, `math`). Use a specialized allowed list for math blocks.
|
||||
- **2. Parser (src/app/plugins/react-custom-html-parser.tsx):**
|
||||
Detect `$ ... $` and `$$ ... $$` patterns in text nodes.
|
||||
```tsx
|
||||
if (node.type === 'text') {
|
||||
const parts = node.data.split(/(\$\$.*?\$\$|\$.*?\$)/g);
|
||||
return parts.map((p) => {
|
||||
if (p.startsWith('$')) return <KaTeX math={p.replace(/\$/g, '')} />;
|
||||
return p;
|
||||
});
|
||||
}
|
||||
```
|
||||
- **3. CSS (src/app/styles/CustomHtml.css.ts):**
|
||||
Import `katex/dist/katex.min.css` only when a math block is rendered to save initial bundle size.
|
||||
|
||||
### P4-6 · OIDC / SSO Next-Gen Auth (MSC3861)
|
||||
|
||||
**Mechanism:** Matrix Authentication Service (MAS) Integration.
|
||||
|
||||
- **Architecture:** Shift from password-based `/login` to OAuth2 `authorization_code` flow.
|
||||
- **Key Files:** `src/app/pages/auth/Login.tsx` and `src/app/hooks/useAuth.ts`.
|
||||
- **Implementation:**
|
||||
1. Use `oidc-client-ts` or a similar lightweight OIDC library.
|
||||
2. Check for `m.authentication` in `/.well-known/matrix/client`.
|
||||
3. Redirect to the MAS authorization endpoint.
|
||||
4. Handle the callback in a new `OidcCallback` route and store the OIDC `refresh_token`.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Priority 5 — Gamer / Aesthetic / Customization
|
||||
|
||||
### P5-1 · Custom Accent Color Picker (Non-TDS only)
|
||||
|
||||
**Mechanism:** Dynamic CSS variable injection.
|
||||
|
||||
- **1. Setting (src/app/state/settings.ts):**
|
||||
Add `customAccentColor: string` (hex).
|
||||
- **2. Manager (src/app/pages/ThemeManager.tsx):**
|
||||
Inside the `useEffect` that monitors theme changes:
|
||||
```typescript
|
||||
if (!lotusTerminal && customAccentColor) {
|
||||
document.documentElement.style.setProperty('--lt-accent-orange', customAccentColor);
|
||||
// Also derive a 'glow' version (e.g. 50% opacity)
|
||||
document.documentElement.style.setProperty('--lt-accent-orange-glow', `${customAccentColor}80`);
|
||||
}
|
||||
```
|
||||
- **3. UI (src/app/features/settings/general/General.tsx):**
|
||||
Use a `<Input type="color">` component. Hide this section if `lotusTerminal` is `true`.
|
||||
|
||||
### P5-40 · Desktop — Proactive Update Notifications (Tauri)
|
||||
|
||||
**Mechanism:** Global Background Check via `useTauriUpdater`.
|
||||
|
||||
- **Objective:** Alert users to app updates without requiring a manual check in settings.
|
||||
- **Key Files:**
|
||||
- `src/app/hooks/useTauriUpdater.ts`: Logic source.
|
||||
- `src/app/pages/client/ClientNonUIFeatures.tsx`: Background mounting point.
|
||||
- `src/app/features/toast/LotusToastContainer.tsx`: UI for notification.
|
||||
- **Implementation:**
|
||||
1. Create a `TauriUpdateFeature` component.
|
||||
2. Use `useTauriUpdater()` to get the `check` function and `status`.
|
||||
3. In a `useEffect`, call `check()` on mount and then on a `setInterval` (e.g., every 12 hours).
|
||||
4. Watch the `status`. When it transitions to `{ state: 'available', version: '...' }`, trigger an in-app **Lotus Toast**.
|
||||
5. The toast should say "Lotus Chat v[version] is available!" with an "Update" button that calls the `install()` function from the hook.
|
||||
6. **Persistence:** Store the `lastCheck` timestamp in `localStorage` to ensure the background check doesn't fire redundant commands every time the user refreshes or re-opens the app.
|
||||
|
||||
---
|
||||
|
||||
## 🔊 Audio & Communications
|
||||
|
||||
### P5-15 · In-Call Soundboard
|
||||
|
||||
**Mechanism:** Local-to-Global Audio Bridge.
|
||||
|
||||
- **Architecture:** Use the `Web Audio API` to mix sounds into the `MediaStream` before it enters the Element Call widget.
|
||||
- **Implementation:**
|
||||
1. Create an `AudioContext`.
|
||||
2. Create a `MediaStreamDestinationNode`.
|
||||
3. Create an `AudioBufferSourceNode` for the clip.
|
||||
4. Route the mic `MediaStream` and the clip source to the destination.
|
||||
5. Pass the destination's `.stream` to the call bridge.
|
||||
|
||||
### P5-20 · Quick Reply from Browser Notification
|
||||
|
||||
**Mechanism:** Service Worker `notificationclick` Action.
|
||||
|
||||
- **1. Registration (src/sw.ts):**
|
||||
```typescript
|
||||
self.addEventListener('notificationclick', (event) => {
|
||||
if (event.action === 'reply' && event.reply) {
|
||||
const { roomId, threadId } = event.notification.data;
|
||||
const session = sessions.get(event.clientId); // Uses existing session mapping
|
||||
// Send via direct fetch to bypass SDK loading
|
||||
fetch(`${session.baseUrl}/_matrix/client/v3/rooms/${roomId}/send/m.room.message`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${session.accessToken}` },
|
||||
body: JSON.stringify({
|
||||
msgtype: 'm.text',
|
||||
body: event.reply,
|
||||
'm.relates_to': threadId ? { rel_type: 'm.thread', event_id: threadId } : undefined,
|
||||
}),
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔬 Extreme Complexity Projects
|
||||
|
||||
### P5-30 · Advanced ML Noise Suppression (Krisp-style)
|
||||
|
||||
**Mechanism:** RNNoise WASM + Web Audio Worklet Pipeline.
|
||||
|
||||
- **Objective:** Filter non-vocal noise from the microphone stream in real-time.
|
||||
- **Architecture:**
|
||||
1. **Engine:** Use `RNNoise` (Recurrent Neural Network for noise suppression). It is lightweight and highly effective for speech.
|
||||
2. **Pipeline:** `Mic Stream` -> `AudioWorkletNode` (Processing) -> `MediaStreamDestination` -> `Element Call`.
|
||||
- **Implementation Steps:**
|
||||
1. **WASM Wrapper:** Compile the `RNNoise` C library to WebAssembly. Use a library like `rnnoise-wasm` or `noise-suppression-js`.
|
||||
2. **Audio Worklet:** Create `src/app/utils/audio/RnnoiseWorklet.ts`. This must handle 480-sample chunks (10ms of audio at 48kHz), which is the standard frame size for RNNoise.
|
||||
3. **Client Integration:**
|
||||
- In `CallControl.ts`, intercept the `localStream`.
|
||||
- Pass the stream through the Worklet.
|
||||
- Crucially, you must ensure that the processed stream is used by the `RTCPeerConnection` within the Element Call iframe.
|
||||
|
||||
### P5-31 · Granular Voice & Screenshare Quality Controls
|
||||
|
||||
**Mechanism:** WebRTC Encoding Parameters + Backend Quality Guard.
|
||||
|
||||
- **Objective:** Per-room and per-user control over audio fidelity and screenshare smoothness.
|
||||
- **Architecture:**
|
||||
1. **State Event:** `io.lotus.room_quality` (state key `""`) containing:
|
||||
```json
|
||||
{
|
||||
"audio_bitrate": 128000,
|
||||
"screen_max_res": "1080p",
|
||||
"screen_max_fps": 60
|
||||
}
|
||||
```
|
||||
2. **Client-Side (RoomInput / CallControl):**
|
||||
- **Screenshare:** In `src/app/plugins/call/CallControl.ts`, when initiating screenshare, map the "Quality" setting to `getDisplayMedia` constraints:
|
||||
|
||||
```typescript
|
||||
const constraints = {
|
||||
video: {
|
||||
width: { ideal: 1920 }, // 1080p
|
||||
frameRate: { ideal: 60 },
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
- **Audio Bitrate:** After the call joins, find the `RTCRtpSender` for the audio track and update parameters:
|
||||
|
||||
```typescript
|
||||
const sender = peerConnection.getSenders().find((s) => s.track?.kind === 'audio');
|
||||
const params = sender.getParameters();
|
||||
params.encodings[0].maxBitrate = roomBitrate || 128000;
|
||||
await sender.setParameters(params);
|
||||
```
|
||||
|
||||
3. **Backend Sidecar (The "Quality Guard"):**
|
||||
- **Pattern:** Extend the `voice-limit-guard.py` (on LXC 151) to handle quality metadata.
|
||||
- **Mechanism:** When a user requests a LiveKit JWT to join a room, the Guard fetches the `io.lotus.room_quality` event for that room via the Synapse Admin API.
|
||||
- **Enforcement:** The Guard injects these limits into the LiveKit token claims (if supported) or simply returns them to the client as an authorized "config" packet that the client must respect.
|
||||
|
||||
- **Challenges:**
|
||||
- **LiveKit Compatibility:** Ensuring the SFU doesn't over-compress a high-bitrate stream from a "Pro" user.
|
||||
- **Network Stability:** High bitrates (512kbps audio + 60fps 1080p video) require significant upstream bandwidth. Implement a "Network Warning" UI if packets are dropped.
|
||||
Binary file not shown.
Binary file not shown.
+40
-3
@@ -38,9 +38,9 @@
|
||||
|
||||
var MODEL = params.get('lotusModel') || 'rnnoise';
|
||||
// DTLN (@workadventure) targets 16 kHz and does not resample internally, so
|
||||
// its whole graph runs in a 16 kHz context; RNNoise/Speex (sapphi) need
|
||||
// 48 kHz. The processed MediaStreamTrack is published to LiveKit either way
|
||||
// (WebRTC/Opus resamples as needed).
|
||||
// its whole graph runs in a 16 kHz context; RNNoise/Speex (sapphi) and
|
||||
// DeepFilterNet 3 are 48 kHz fullband. The processed MediaStreamTrack is
|
||||
// published to LiveKit either way (WebRTC/Opus resamples as needed).
|
||||
var SAMPLE_RATE = MODEL === 'dtln' ? 16000 : 48000;
|
||||
var USE_NATIVE_NS = params.get('lotusNativeNS') === 'true';
|
||||
var USE_GATE = params.get('lotusGate') === 'true';
|
||||
@@ -65,6 +65,15 @@
|
||||
// node, rather than addModule-ing a flat worklet ourselves.
|
||||
helper: 'workadventure/audio-worklet.js',
|
||||
},
|
||||
deepfilternet: {
|
||||
// deepfilternet3-noise-filter ships an ESM whose AudioWorklet processor +
|
||||
// wasm-bindgen glue are INLINED as a string (loaded via a Blob URL — no
|
||||
// CDN for the worklet). The only assets it fetches are its single-threaded
|
||||
// df_bg.wasm + ONNX model, which we vendor + self-host under
|
||||
// deepfilternet/v2/... We dynamic-import the ESM, build a DeepFilterNet3Core
|
||||
// pointed at the self-hosted base, and let it create the worklet node.
|
||||
esm: 'deepfilternet/index.esm.js',
|
||||
},
|
||||
gate: {
|
||||
name: '@sapphi-red/web-noise-suppressor/noise-gate',
|
||||
script: 'noiseGateWorklet.js',
|
||||
@@ -164,6 +173,34 @@
|
||||
return mod.createNoiseSuppressionAudioWorklet(ctx, { bypassUntilReady: true });
|
||||
});
|
||||
}
|
||||
if (MODEL === 'deepfilternet') {
|
||||
// Resolve an absolute self-hosted base so the package's cdnUrl override
|
||||
// fetches our vendored df_bg.wasm + ONNX model (never the upstream CDN).
|
||||
var dfnBase = new URL(ASSET_BASE + 'deepfilternet', window.location.href).href;
|
||||
return import(ASSET_BASE + PROCESSORS.deepfilternet.esm).then(function (mod) {
|
||||
var core = new mod.DeepFilterNet3Core({
|
||||
sampleRate: SAMPLE_RATE,
|
||||
noiseReductionLevel: 80,
|
||||
assetConfig: { cdnUrl: dfnBase },
|
||||
});
|
||||
// initialize() fetches + compiles the wasm and loads the model on the
|
||||
// main thread; the worklet node only exists once that resolves, so the
|
||||
// graph is connected with a ready model (no half-initialised passthrough).
|
||||
return core.initialize().then(function () {
|
||||
return core.createAudioWorkletNode(ctx).then(function (node) {
|
||||
return {
|
||||
node: node,
|
||||
ready: Promise.resolve(),
|
||||
dispose: function () {
|
||||
try {
|
||||
core.destroy();
|
||||
} catch (e) {}
|
||||
},
|
||||
};
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
var node = new AudioWorkletNode(ctx, PROCESSORS[MODEL].name, {
|
||||
channelCount: 1,
|
||||
numberOfInputs: 1,
|
||||
|
||||
Generated
+13
@@ -35,6 +35,7 @@
|
||||
"classnames": "2.5.1",
|
||||
"dateformat": "5.0.3",
|
||||
"dayjs": "1.11.20",
|
||||
"deepfilternet3-noise-filter": "1.2.1",
|
||||
"domhandler": "6.0.1",
|
||||
"dompurify": "3.4.5",
|
||||
"emojibase": "17.0.0",
|
||||
@@ -6399,6 +6400,18 @@
|
||||
"integrity": "sha512-Rn+RuwkmkDwCi2/oXOFS9Gsr5lJZu/yTGpK7wAaAIE75CC+LCGEZHpY6VQJa/RoJcrmaA/docWJZvYohlNkWPA==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/deepfilternet3-noise-filter": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/deepfilternet3-noise-filter/-/deepfilternet3-noise-filter-1.2.1.tgz",
|
||||
"integrity": "sha512-OAyrHTDlUHH+AhfpVNKYEOhVqb9cZpu0fdNThplA/tB/Ts4PF/UsI+abl2n1IbSxUkhiF0OqDejEhk1n42Oqpw==",
|
||||
"license": "(Apache-2.0 OR MIT)",
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"livekit-client": "^2.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/deepmerge": {
|
||||
"version": "4.3.1",
|
||||
"resolved": "https://registry.npmjs.org/deepmerge/-/deepmerge-4.3.1.tgz",
|
||||
|
||||
@@ -59,6 +59,7 @@
|
||||
"classnames": "2.5.1",
|
||||
"dateformat": "5.0.3",
|
||||
"dayjs": "1.11.20",
|
||||
"deepfilternet3-noise-filter": "1.2.1",
|
||||
"domhandler": "6.0.1",
|
||||
"dompurify": "3.4.5",
|
||||
"emojibase": "17.0.0",
|
||||
|
||||
@@ -71,8 +71,8 @@ export const EventReaders = as<'div', EventReadersProps>(
|
||||
style={
|
||||
lotusTerminal
|
||||
? {
|
||||
borderBottom: '1px solid rgba(0,212,255,0.30)',
|
||||
boxShadow: '0 2px 12px rgba(0,212,255,0.08)',
|
||||
borderBottom: '1px solid var(--lt-border-color)',
|
||||
boxShadow: 'var(--lt-box-glow-cyan)',
|
||||
}
|
||||
: undefined
|
||||
}
|
||||
@@ -83,8 +83,8 @@ export const EventReaders = as<'div', EventReadersProps>(
|
||||
style={
|
||||
lotusTerminal
|
||||
? {
|
||||
color: '#00D4FF',
|
||||
textShadow: '0 0 6px rgba(0,212,255,0.45)',
|
||||
color: 'var(--lt-accent-cyan)',
|
||||
textShadow: 'var(--lt-glow-cyan)',
|
||||
letterSpacing: '0.05em',
|
||||
}
|
||||
: undefined
|
||||
@@ -144,8 +144,8 @@ export const EventReaders = as<'div', EventReadersProps>(
|
||||
style={
|
||||
lotusTerminal
|
||||
? {
|
||||
color: '#FFB300',
|
||||
textShadow: '0 0 5px rgba(255,179,0,0.45)',
|
||||
color: 'var(--lt-accent-amber)',
|
||||
textShadow: 'var(--lt-glow-amber)',
|
||||
fontSize: '0.72rem',
|
||||
}
|
||||
: { opacity: 0.6 }
|
||||
|
||||
@@ -126,9 +126,10 @@ function BookmarkItem({ bookmark, onJump, onRemove }: BookmarkItemProps) {
|
||||
|
||||
type BookmarksPanelProps = {
|
||||
onClose: () => void;
|
||||
isMobile?: boolean;
|
||||
};
|
||||
|
||||
export function BookmarksPanel({ onClose }: BookmarksPanelProps) {
|
||||
export function BookmarksPanel({ onClose, isMobile }: BookmarksPanelProps) {
|
||||
const { bookmarks, removeBookmark } = useBookmarks();
|
||||
const { navigateRoom } = useRoomNavigate();
|
||||
const [filter, setFilter] = useState('');
|
||||
@@ -154,10 +155,14 @@ export function BookmarksPanel({ onClose }: BookmarksPanelProps) {
|
||||
<Box
|
||||
direction="Column"
|
||||
style={{
|
||||
width: '266px',
|
||||
width: isMobile ? '100%' : '266px',
|
||||
height: '100%',
|
||||
position: isMobile ? 'absolute' : 'static',
|
||||
top: isMobile ? 0 : 'auto',
|
||||
left: isMobile ? 0 : 'auto',
|
||||
zIndex: isMobile ? 100 : 'auto',
|
||||
flexShrink: 0,
|
||||
borderLeft: `1px solid ${color.Surface.ContainerLine}`,
|
||||
borderLeft: isMobile ? 'none' : `1px solid ${color.Surface.ContainerLine}`,
|
||||
background: color.Surface.Container,
|
||||
overflow: 'hidden',
|
||||
display: 'flex',
|
||||
|
||||
@@ -246,8 +246,8 @@ export function CallControls({ callEmbed }: CallControlsProps) {
|
||||
top: '-2.5rem',
|
||||
left: '50%',
|
||||
transform: 'translateX(-50%)',
|
||||
background: pttActive ? 'rgba(0,255,136,0.18)' : 'rgba(255,107,0,0.12)',
|
||||
border: `1px solid ${pttActive ? 'rgba(0,255,136,0.55)' : 'rgba(255,107,0,0.35)'}`,
|
||||
background: pttActive ? 'var(--lt-accent-green-dim)' : 'var(--lt-accent-orange-dim)',
|
||||
border: `1px solid ${pttActive ? 'var(--lt-accent-green-border)' : 'var(--lt-accent-orange-border)'}`,
|
||||
borderRadius: '99px',
|
||||
padding: '0.2rem 0.9rem',
|
||||
pointerEvents: 'none',
|
||||
@@ -257,7 +257,7 @@ export function CallControls({ callEmbed }: CallControlsProps) {
|
||||
<Text
|
||||
size="T200"
|
||||
style={{
|
||||
color: pttActive ? '#00FF88' : '#FF6B00',
|
||||
color: pttActive ? 'var(--lt-accent-green)' : 'var(--lt-accent-orange)',
|
||||
fontWeight: 700,
|
||||
letterSpacing: '0.08em',
|
||||
fontFamily: 'JetBrains Mono, monospace',
|
||||
|
||||
@@ -81,6 +81,7 @@ import { PowerIcon } from '../../../components/power';
|
||||
import colorMXID from '../../../../util/colorMXID';
|
||||
import { getPowerTagIconSrc } from '../../../hooks/useMemberPowerTag';
|
||||
import { ForwardMessageDialog } from './ForwardMessageDialog';
|
||||
import { RemindMeDialog } from './RemindMeDialog';
|
||||
import { useBookmarks } from '../../../hooks/useBookmarks';
|
||||
import { PresenceRingAvatar } from '../../../components/presence';
|
||||
import { AvatarDecoration } from '../../../components/avatar-decoration/AvatarDecoration';
|
||||
@@ -809,6 +810,7 @@ export const Message = React.memo(
|
||||
const [menuAnchor, setMenuAnchor] = useState<RectCords>();
|
||||
const [emojiBoardAnchor, setEmojiBoardAnchor] = useState<RectCords>();
|
||||
const [forwardOpen, setForwardOpen] = useState(false);
|
||||
const [remindOpen, setRemindOpen] = useState(false);
|
||||
const { addBookmark, removeBookmark, isBookmarked } = useBookmarks();
|
||||
|
||||
const senderDisplayName =
|
||||
@@ -1204,6 +1206,26 @@ export const Message = React.memo(
|
||||
</Text>
|
||||
</MenuItem>
|
||||
)}
|
||||
{!mEvent.isRedacted() && mEvent.getId() && (
|
||||
<MenuItem
|
||||
size="300"
|
||||
after={<Icon size="100" src={Icons.Clock} />}
|
||||
radii="300"
|
||||
onClick={() => {
|
||||
setRemindOpen(true);
|
||||
closeMenu();
|
||||
}}
|
||||
>
|
||||
<Text
|
||||
className={css.MessageMenuItemText}
|
||||
as="span"
|
||||
size="T300"
|
||||
truncate
|
||||
>
|
||||
Remind Me
|
||||
</Text>
|
||||
</MenuItem>
|
||||
)}
|
||||
{!isThreadedMessage && (
|
||||
<MenuItem
|
||||
size="300"
|
||||
@@ -1372,6 +1394,14 @@ export const Message = React.memo(
|
||||
{forwardOpen && (
|
||||
<ForwardMessageDialog mEvent={mEvent} onClose={() => setForwardOpen(false)} />
|
||||
)}
|
||||
{remindOpen && mEvent.getId() && (
|
||||
<RemindMeDialog
|
||||
roomId={room.roomId}
|
||||
eventId={mEvent.getId()!}
|
||||
previewText={(mEvent.getContent()?.body as string | undefined)?.slice(0, 120) ?? ''}
|
||||
onClose={() => setRemindOpen(false)}
|
||||
/>
|
||||
)}
|
||||
</MessageBase>
|
||||
);
|
||||
},
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
import React, { useMemo } from 'react';
|
||||
import FocusTrap from 'focus-trap-react';
|
||||
import {
|
||||
Box,
|
||||
Button,
|
||||
Header,
|
||||
Icon,
|
||||
IconButton,
|
||||
Icons,
|
||||
Overlay,
|
||||
OverlayBackdrop,
|
||||
OverlayCenter,
|
||||
Text,
|
||||
config,
|
||||
} from 'folds';
|
||||
import { stopPropagation } from '../../../utils/keyboard';
|
||||
import { useReminders } from '../../../hooks/useReminders';
|
||||
|
||||
type RemindMeDialogProps = {
|
||||
roomId: string;
|
||||
eventId: string;
|
||||
previewText: string;
|
||||
onClose: () => void;
|
||||
};
|
||||
|
||||
function getPresets(): Array<{ label: string; ms: number }> {
|
||||
const tomorrow = new Date();
|
||||
tomorrow.setDate(tomorrow.getDate() + 1);
|
||||
tomorrow.setHours(9, 0, 0, 0);
|
||||
const tomorrowMs = tomorrow.getTime() - Date.now();
|
||||
const timeLabel = tomorrow.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
|
||||
return [
|
||||
{ label: 'In 20 minutes', ms: 20 * 60_000 },
|
||||
{ label: 'In 1 hour', ms: 60 * 60_000 },
|
||||
{ label: 'In 3 hours', ms: 3 * 60 * 60_000 },
|
||||
{ label: `Tomorrow at ${timeLabel}`, ms: tomorrowMs },
|
||||
];
|
||||
}
|
||||
|
||||
export function RemindMeDialog({ roomId, eventId, previewText, onClose }: RemindMeDialogProps) {
|
||||
const { addReminder } = useReminders();
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
const presets = useMemo(() => getPresets(), []);
|
||||
|
||||
const handlePick = async (ms: number) => {
|
||||
await addReminder({
|
||||
roomId,
|
||||
eventId,
|
||||
timestamp: Date.now() + ms,
|
||||
message: previewText || 'Reminder',
|
||||
});
|
||||
onClose();
|
||||
};
|
||||
|
||||
return (
|
||||
<Overlay open backdrop={<OverlayBackdrop />}>
|
||||
<OverlayCenter>
|
||||
<FocusTrap
|
||||
focusTrapOptions={{
|
||||
initialFocus: false,
|
||||
returnFocusOnDeactivate: false,
|
||||
onDeactivate: onClose,
|
||||
clickOutsideDeactivates: true,
|
||||
escapeDeactivates: stopPropagation,
|
||||
}}
|
||||
>
|
||||
<Box
|
||||
direction="Column"
|
||||
style={{
|
||||
width: 300,
|
||||
borderRadius: 12,
|
||||
background: 'var(--bg-surface)',
|
||||
border: '1px solid var(--bg-surface-border)',
|
||||
overflow: 'hidden',
|
||||
}}
|
||||
>
|
||||
<Header variant="Surface" size="500">
|
||||
<Box grow="Yes" alignItems="Center" gap="200">
|
||||
<Icon size="100" src={Icons.Clock} />
|
||||
<Text size="H4">Remind Me</Text>
|
||||
</Box>
|
||||
<IconButton size="300" onClick={onClose} aria-label="Close">
|
||||
<Icon src={Icons.Cross} />
|
||||
</IconButton>
|
||||
</Header>
|
||||
{previewText && (
|
||||
<Box
|
||||
style={{
|
||||
padding: `${config.space.S100} ${config.space.S400}`,
|
||||
borderBottom: '1px solid var(--bg-surface-border)',
|
||||
}}
|
||||
>
|
||||
<Text size="T200" priority="300" truncate>
|
||||
{previewText}
|
||||
</Text>
|
||||
</Box>
|
||||
)}
|
||||
<Box
|
||||
direction="Column"
|
||||
gap="100"
|
||||
style={{ padding: `${config.space.S200} ${config.space.S200} ${config.space.S300}` }}
|
||||
>
|
||||
{presets.map((p) => (
|
||||
<Button
|
||||
key={p.label}
|
||||
variant="Secondary"
|
||||
fill="Soft"
|
||||
size="300"
|
||||
radii="300"
|
||||
style={{ justifyContent: 'flex-start' }}
|
||||
onClick={() => handlePick(p.ms)}
|
||||
>
|
||||
<Text size="B300">{p.label}</Text>
|
||||
</Button>
|
||||
))}
|
||||
</Box>
|
||||
</Box>
|
||||
</FocusTrap>
|
||||
</OverlayCenter>
|
||||
</Overlay>
|
||||
);
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
import React, { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { Box, Button, Text } from 'folds';
|
||||
import { DenoiseModelId } from '../../../state/settings';
|
||||
import { DENOISE_MODELS } from '../../../utils/lotusDenoiseUtils';
|
||||
import {
|
||||
DenoiseNode,
|
||||
buildGateNode,
|
||||
@@ -307,7 +308,7 @@ export function DenoiseTester({ model, useGate, gateThreshold, nativeNS }: Denoi
|
||||
[stopLive, stopPlayback],
|
||||
);
|
||||
|
||||
const modelLabel = model === 'rnnoise' ? 'RNNoise' : model === 'speex' ? 'Speex' : 'DTLN';
|
||||
const modelLabel = DENOISE_MODELS.find((m) => m.id === model)?.name ?? model;
|
||||
|
||||
return (
|
||||
<Box direction="Column" gap="400">
|
||||
@@ -367,6 +368,7 @@ export function DenoiseTester({ model, useGate, gateThreshold, nativeNS }: Denoi
|
||||
{ label: 'RNNoise', model: 'rnnoise' },
|
||||
{ label: 'Speex', model: 'speex' },
|
||||
{ label: 'DTLN', model: 'dtln' },
|
||||
{ label: 'DeepFilterNet', model: 'deepfilternet' },
|
||||
] as const
|
||||
).map((b) => (
|
||||
<Button
|
||||
|
||||
@@ -1240,6 +1240,7 @@ function Calls() {
|
||||
const deafenBind = useKeyBind(setDeafenKey);
|
||||
|
||||
const mlSupported = isMLDenoiseSupported();
|
||||
const selectedDenoiseModel = DENOISE_MODELS.find((m) => m.id === callDenoiseModel);
|
||||
|
||||
return (
|
||||
<Box direction="Column" gap="100">
|
||||
@@ -1258,46 +1259,9 @@ function Calls() {
|
||||
<Box direction="Column" gap="200">
|
||||
<Text>
|
||||
Filter background noise from your mic during calls. Browser-native uses the built-in
|
||||
WebRTC suppressor (Google NSNet2).
|
||||
WebRTC suppressor (Google NSNet2). ML runs a dedicated model for stronger removal.
|
||||
</Text>
|
||||
|
||||
<Box direction="Column" gap="100" style={{ overflowX: 'auto' }}>
|
||||
<Box
|
||||
direction="Row"
|
||||
gap="100"
|
||||
style={{ borderBottom: '1px solid var(--lt-border-color)', paddingBottom: '4px' }}
|
||||
>
|
||||
<Box style={{ width: '120px' }}>
|
||||
<Text size="T200">Model</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200">CPU</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200">Quality</Text>
|
||||
</Box>
|
||||
<Box grow="Yes">
|
||||
<Text size="T200">Transients</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
{DENOISE_MODELS.map((model) => (
|
||||
<Box key={model.id} direction="Row" gap="100">
|
||||
<Box style={{ width: '120px' }}>
|
||||
<Text size="T200">{model.name}</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200">{model.cpuUsage}</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200">{model.voiceQuality}</Text>
|
||||
</Box>
|
||||
<Box grow="Yes">
|
||||
<Text size="T200">{model.transients}</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
))}
|
||||
</Box>
|
||||
|
||||
{!mlSupported && (
|
||||
<Box direction="Column" gap="100">
|
||||
<Text size="T200" priority="400">
|
||||
@@ -1312,11 +1276,6 @@ function Calls() {
|
||||
</Box>
|
||||
</Box>
|
||||
)}
|
||||
{callNoiseSuppression === 'ml' && (
|
||||
<Text size="T200" priority="400">
|
||||
Note: Applying changes requires rejoining the call.
|
||||
</Text>
|
||||
)}
|
||||
</Box>
|
||||
}
|
||||
after={
|
||||
@@ -1339,72 +1298,174 @@ function Calls() {
|
||||
{callNoiseSuppression === 'ml' && (
|
||||
<Box
|
||||
direction="Column"
|
||||
gap="300"
|
||||
gap="400"
|
||||
style={{
|
||||
padding: '16px',
|
||||
marginTop: '8px',
|
||||
borderTop: '1px solid var(--lt-border-color)',
|
||||
background: 'rgba(0,0,0,0.1)',
|
||||
borderTop: '1px solid var(--border-color)',
|
||||
background: 'var(--bg-card)',
|
||||
}}
|
||||
>
|
||||
<SettingTile
|
||||
title="ML Model"
|
||||
description="Choose the machine learning model to use for noise removal."
|
||||
after={
|
||||
<SettingsSelect<DenoiseModelId>
|
||||
value={callDenoiseModel}
|
||||
onChange={setCallDenoiseModel}
|
||||
options={[
|
||||
{ value: 'rnnoise', label: 'RNNoise' },
|
||||
{ value: 'speex', label: 'Speex (Legacy)' },
|
||||
{ value: 'dtln', label: 'DTLN (beta)' },
|
||||
]}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
<SettingTile
|
||||
title="Series Suppression"
|
||||
description="Run the browser's native stationary noise filter before the ML model. Recommended for eliminating fan hum."
|
||||
after={
|
||||
<Switch
|
||||
variant="Primary"
|
||||
value={callDenoiseNativeNS}
|
||||
onChange={setCallDenoiseNativeNS}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
<SettingTile
|
||||
title="Noise Gate"
|
||||
description="Hard-cut audio when you aren't speaking to ensure absolute silence between sentences."
|
||||
after={
|
||||
<Switch variant="Primary" value={callDenoiseGate} onChange={setCallDenoiseGate} />
|
||||
}
|
||||
/>
|
||||
|
||||
{callDenoiseGate && (
|
||||
<Box direction="Column" gap="100">
|
||||
<Box direction="Row" justifyContent="SpaceBetween">
|
||||
<Text size="T200">Gate Threshold</Text>
|
||||
<Text size="T200">{callDenoiseGateThreshold} dB</Text>
|
||||
{/* ── Model selection ───────────────────────────────────────── */}
|
||||
<Box direction="Column" gap="200">
|
||||
<Text size="L400">Model</Text>
|
||||
<SettingTile
|
||||
title="ML Model"
|
||||
description="Choose the machine learning model used for noise removal. Heavier models clean more aggressively at a higher CPU cost."
|
||||
after={
|
||||
<SettingsSelect<DenoiseModelId>
|
||||
value={callDenoiseModel}
|
||||
onChange={setCallDenoiseModel}
|
||||
options={DENOISE_MODELS.map((m) => ({ value: m.id, label: m.name }))}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
{selectedDenoiseModel && (
|
||||
<Box
|
||||
direction="Column"
|
||||
gap="200"
|
||||
style={{
|
||||
padding: '12px',
|
||||
borderRadius: '8px',
|
||||
border: '1px solid var(--border-color)',
|
||||
background: 'var(--bg-input)',
|
||||
}}
|
||||
>
|
||||
<Text size="T300">{selectedDenoiseModel.name}</Text>
|
||||
<Text size="T200" priority="300">
|
||||
{selectedDenoiseModel.description}
|
||||
</Text>
|
||||
<Box direction="Row" gap="400" wrap="Wrap">
|
||||
{(
|
||||
[
|
||||
{ label: 'CPU', value: selectedDenoiseModel.cpuUsage },
|
||||
{ label: 'Voice quality', value: selectedDenoiseModel.voiceQuality },
|
||||
{ label: 'Transients', value: selectedDenoiseModel.transients },
|
||||
{ label: 'Download', value: selectedDenoiseModel.binarySize },
|
||||
] as const
|
||||
).map((stat) => (
|
||||
<Box key={stat.label} direction="Column" gap="100">
|
||||
<Text size="T200" priority="300">
|
||||
{stat.label}
|
||||
</Text>
|
||||
<Text size="T300">{stat.value}</Text>
|
||||
</Box>
|
||||
))}
|
||||
</Box>
|
||||
</Box>
|
||||
<input
|
||||
type="range"
|
||||
min="-100"
|
||||
max="0"
|
||||
step="1"
|
||||
value={callDenoiseGateThreshold}
|
||||
onChange={(e) => setCallDenoiseGateThreshold(parseInt(e.target.value, 10))}
|
||||
style={{ width: '100%', accentColor: 'var(--accent-orange)' }}
|
||||
/>
|
||||
</Box>
|
||||
)}
|
||||
)}
|
||||
|
||||
<details>
|
||||
<summary style={{ cursor: 'pointer' }}>
|
||||
<Text as="span" size="T200" priority="300">
|
||||
Compare all models
|
||||
</Text>
|
||||
</summary>
|
||||
<Box direction="Column" gap="100" style={{ overflowX: 'auto', marginTop: '8px' }}>
|
||||
<Box
|
||||
direction="Row"
|
||||
gap="100"
|
||||
style={{
|
||||
borderBottom: '1px solid var(--border-color)',
|
||||
paddingBottom: '4px',
|
||||
}}
|
||||
>
|
||||
<Box style={{ width: '160px' }}>
|
||||
<Text size="T200" priority="300">
|
||||
Model
|
||||
</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200" priority="300">
|
||||
CPU
|
||||
</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '90px' }}>
|
||||
<Text size="T200" priority="300">
|
||||
Quality
|
||||
</Text>
|
||||
</Box>
|
||||
<Box grow="Yes">
|
||||
<Text size="T200" priority="300">
|
||||
Transients
|
||||
</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
{DENOISE_MODELS.map((model) => (
|
||||
<Box key={model.id} direction="Row" gap="100">
|
||||
<Box style={{ width: '160px' }}>
|
||||
<Text size="T200">{model.name}</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '80px' }}>
|
||||
<Text size="T200">{model.cpuUsage}</Text>
|
||||
</Box>
|
||||
<Box style={{ width: '90px' }}>
|
||||
<Text size="T200">{model.voiceQuality}</Text>
|
||||
</Box>
|
||||
<Box grow="Yes">
|
||||
<Text size="T200">{model.transients}</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
))}
|
||||
</Box>
|
||||
</details>
|
||||
|
||||
<Text size="T200" priority="400">
|
||||
Note: Applying changes requires rejoining the call.
|
||||
</Text>
|
||||
</Box>
|
||||
|
||||
{/* ── Enhancement toggles ───────────────────────────────────── */}
|
||||
<Box
|
||||
direction="Column"
|
||||
gap="300"
|
||||
style={{ paddingTop: '12px', borderTop: '1px solid var(--border-color)' }}
|
||||
>
|
||||
<Text size="L400">Enhancements</Text>
|
||||
<SettingTile
|
||||
title="Series Suppression"
|
||||
description="Run the browser's native stationary noise filter before the ML model. Recommended for eliminating fan hum."
|
||||
after={
|
||||
<Switch
|
||||
variant="Primary"
|
||||
value={callDenoiseNativeNS}
|
||||
onChange={setCallDenoiseNativeNS}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
<SettingTile
|
||||
title="Noise Gate"
|
||||
description="Hard-cut audio when you aren't speaking to ensure absolute silence between sentences."
|
||||
after={
|
||||
<Switch variant="Primary" value={callDenoiseGate} onChange={setCallDenoiseGate} />
|
||||
}
|
||||
/>
|
||||
|
||||
{callDenoiseGate && (
|
||||
<Box direction="Column" gap="100">
|
||||
<Box direction="Row" justifyContent="SpaceBetween">
|
||||
<Text size="T200">Gate Threshold</Text>
|
||||
<Text size="T200">{callDenoiseGateThreshold} dB</Text>
|
||||
</Box>
|
||||
<input
|
||||
type="range"
|
||||
min="-100"
|
||||
max="0"
|
||||
step="1"
|
||||
value={callDenoiseGateThreshold}
|
||||
onChange={(e) => setCallDenoiseGateThreshold(parseInt(e.target.value, 10))}
|
||||
style={{ width: '100%', accentColor: 'var(--accent-orange)' }}
|
||||
/>
|
||||
</Box>
|
||||
)}
|
||||
</Box>
|
||||
|
||||
{/* ── Test & calibrate ──────────────────────────────────────── */}
|
||||
<Box
|
||||
direction="Column"
|
||||
gap="200"
|
||||
style={{ paddingTop: '8px', borderTop: '1px solid var(--border-color)' }}
|
||||
style={{ paddingTop: '12px', borderTop: '1px solid var(--border-color)' }}
|
||||
>
|
||||
<Text size="L400">Test & calibrate</Text>
|
||||
<Text size="T200" priority="300">
|
||||
|
||||
+31
-26
@@ -1,4 +1,4 @@
|
||||
import { MouseEventHandler, useEffect, useState } from 'react';
|
||||
import { MouseEventHandler, useEffect, useRef, useState } from 'react';
|
||||
|
||||
export type Pan = {
|
||||
translateX: number;
|
||||
@@ -16,36 +16,39 @@ export const usePan = (active: boolean) => {
|
||||
active ? 'grab' : 'initial',
|
||||
);
|
||||
|
||||
// Track the exact handler references that were passed to addEventListener so
|
||||
// we can remove them even if the component re-renders or unmounts mid-drag.
|
||||
const attachedRef = useRef<{ move: (e: MouseEvent) => void; up: (e: MouseEvent) => void } | null>(
|
||||
null,
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
setCursor(active ? 'grab' : 'initial');
|
||||
}, [active]);
|
||||
|
||||
const handleMouseMove = (evt: MouseEvent) => {
|
||||
evt.preventDefault();
|
||||
evt.stopPropagation();
|
||||
|
||||
setPan((p) => {
|
||||
const { translateX, translateY } = p;
|
||||
const mX = translateX + evt.movementX;
|
||||
const mY = translateY + evt.movementY;
|
||||
|
||||
return { translateX: mX, translateY: mY };
|
||||
});
|
||||
};
|
||||
|
||||
const handleMouseUp = (evt: MouseEvent) => {
|
||||
evt.preventDefault();
|
||||
setCursor('grab');
|
||||
|
||||
document.removeEventListener('mousemove', handleMouseMove);
|
||||
document.removeEventListener('mouseup', handleMouseUp);
|
||||
};
|
||||
|
||||
const handleMouseDown: MouseEventHandler<HTMLElement> = (evt) => {
|
||||
if (!active) return;
|
||||
evt.preventDefault();
|
||||
setCursor('grabbing');
|
||||
|
||||
const handleMouseMove = (e: MouseEvent) => {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
setPan((p) => ({
|
||||
translateX: p.translateX + e.movementX,
|
||||
translateY: p.translateY + e.movementY,
|
||||
}));
|
||||
};
|
||||
|
||||
const handleMouseUp = (e: MouseEvent) => {
|
||||
e.preventDefault();
|
||||
setCursor('grab');
|
||||
document.removeEventListener('mousemove', handleMouseMove);
|
||||
document.removeEventListener('mouseup', handleMouseUp);
|
||||
attachedRef.current = null;
|
||||
};
|
||||
|
||||
attachedRef.current = { move: handleMouseMove, up: handleMouseUp };
|
||||
document.addEventListener('mousemove', handleMouseMove);
|
||||
document.addEventListener('mouseup', handleMouseUp);
|
||||
};
|
||||
@@ -54,13 +57,15 @@ export const usePan = (active: boolean) => {
|
||||
if (!active) setPan(INITIAL_PAN);
|
||||
}, [active]);
|
||||
|
||||
// Clean up document listeners if component unmounts during an active drag
|
||||
// Remove listeners if the component unmounts while a drag is in progress.
|
||||
useEffect(
|
||||
() => () => {
|
||||
document.removeEventListener('mousemove', handleMouseMove);
|
||||
document.removeEventListener('mouseup', handleMouseUp);
|
||||
if (attachedRef.current) {
|
||||
document.removeEventListener('mousemove', attachedRef.current.move);
|
||||
document.removeEventListener('mouseup', attachedRef.current.up);
|
||||
attachedRef.current = null;
|
||||
}
|
||||
},
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
[],
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { useCallback, useEffect, useState } from 'react';
|
||||
import { MatrixClient } from 'matrix-js-sdk';
|
||||
import { useMatrixClient } from './useMatrixClient';
|
||||
import { useAccountDataCallback } from './useAccountDataCallback';
|
||||
|
||||
export type Reminder = {
|
||||
roomId: string;
|
||||
eventId: string;
|
||||
timestamp: number;
|
||||
message: string;
|
||||
};
|
||||
|
||||
const REMINDERS_KEY = 'io.lotus.reminders';
|
||||
|
||||
type RemindersContent = {
|
||||
reminders: Reminder[];
|
||||
};
|
||||
|
||||
function readReminders(mx: MatrixClient): Reminder[] {
|
||||
return (
|
||||
(mx.getAccountData(REMINDERS_KEY as any)?.getContent() as RemindersContent | undefined)
|
||||
?.reminders ?? []
|
||||
);
|
||||
}
|
||||
|
||||
export function useReminders(): {
|
||||
reminders: Reminder[];
|
||||
addReminder: (r: Reminder) => Promise<void>;
|
||||
removeReminder: (eventId: string, timestamp: number) => Promise<void>;
|
||||
getReminders: () => Reminder[];
|
||||
} {
|
||||
const mx = useMatrixClient();
|
||||
const [reminders, setReminders] = useState<Reminder[]>(() => readReminders(mx));
|
||||
|
||||
useAccountDataCallback(
|
||||
mx,
|
||||
useCallback(
|
||||
(evt) => {
|
||||
if (evt.getType() === REMINDERS_KEY) {
|
||||
setReminders(evt.getContent<RemindersContent>()?.reminders ?? []);
|
||||
}
|
||||
},
|
||||
[setReminders],
|
||||
),
|
||||
);
|
||||
|
||||
// Re-read on mx change
|
||||
useEffect(() => {
|
||||
setReminders(readReminders(mx));
|
||||
}, [mx]);
|
||||
|
||||
const addReminder = useCallback(
|
||||
async (r: Reminder) => {
|
||||
const current = readReminders(mx);
|
||||
const next = [...current, r];
|
||||
await (mx as any).setAccountData(REMINDERS_KEY, { reminders: next });
|
||||
},
|
||||
[mx],
|
||||
);
|
||||
|
||||
const removeReminder = useCallback(
|
||||
async (eventId: string, timestamp: number) => {
|
||||
const current = readReminders(mx);
|
||||
const next = current.filter(
|
||||
(r) => !(r.eventId === eventId && r.timestamp === timestamp),
|
||||
);
|
||||
await (mx as any).setAccountData(REMINDERS_KEY, { reminders: next });
|
||||
},
|
||||
[mx],
|
||||
);
|
||||
|
||||
const getReminders = useCallback(() => reminders, [reminders]);
|
||||
|
||||
return { reminders, addReminder, removeReminder, getReminders };
|
||||
}
|
||||
@@ -44,10 +44,15 @@ export function ClientLayout({ nav, children }: ClientLayoutProps) {
|
||||
<Box grow="Yes" as="main" id="main-content">
|
||||
{children}
|
||||
</Box>
|
||||
{bookmarksOpen && screenSize === ScreenSize.Desktop && (
|
||||
{bookmarksOpen && (
|
||||
<>
|
||||
<Line variant="Background" direction="Vertical" size="300" />
|
||||
<BookmarksPanel onClose={() => setBookmarksOpen(false)} />
|
||||
{screenSize === ScreenSize.Desktop && (
|
||||
<Line variant="Background" direction="Vertical" size="300" />
|
||||
)}
|
||||
<BookmarksPanel
|
||||
onClose={() => setBookmarksOpen(false)}
|
||||
isMobile={screenSize !== ScreenSize.Desktop}
|
||||
/>
|
||||
</>
|
||||
)}
|
||||
</Box>
|
||||
|
||||
@@ -36,6 +36,7 @@ import { useMediaAuthentication } from '../../hooks/useMediaAuthentication';
|
||||
import { usePresenceUpdater } from '../../hooks/usePresenceUpdater';
|
||||
import { useDeepLinkNavigate } from '../../hooks/useDeepLinkNavigate';
|
||||
import { toastQueueAtom } from '../../state/toast';
|
||||
import { useReminders } from '../../hooks/useReminders';
|
||||
|
||||
function isInQuietHours(start: string, end: string): boolean {
|
||||
const now = new Date();
|
||||
@@ -382,6 +383,50 @@ function DeepLinkNavigator() {
|
||||
return null;
|
||||
}
|
||||
|
||||
function ReminderMonitor() {
|
||||
const mx = useMatrixClient();
|
||||
const { reminders, removeReminder } = useReminders();
|
||||
const setToast = useSetAtom(toastQueueAtom);
|
||||
const mDirects = useAtomValue(mDirectAtom);
|
||||
const firedRef = useRef<Set<string>>(new Set());
|
||||
|
||||
useEffect(() => {
|
||||
const check = () => {
|
||||
const now = Date.now();
|
||||
reminders.forEach((r) => {
|
||||
const key = `${r.eventId}-${r.timestamp}`;
|
||||
if (r.timestamp <= now && !firedRef.current.has(key)) {
|
||||
firedRef.current.add(key);
|
||||
const room = mx.getRoom(r.roomId);
|
||||
const hashPath = mDirects.has(r.roomId)
|
||||
? getDirectRoomPath(r.roomId)
|
||||
: getHomeRoomPath(r.roomId);
|
||||
setToast({
|
||||
id: `reminder-${key}`,
|
||||
displayName: 'Reminder',
|
||||
body: r.message,
|
||||
roomName: room?.name ?? 'Unknown Room',
|
||||
roomId: r.roomId,
|
||||
hashPath,
|
||||
});
|
||||
removeReminder(r.eventId, r.timestamp);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
check();
|
||||
const interval = setInterval(check, 30_000);
|
||||
const onVisible = () => { if (document.visibilityState === 'visible') check(); };
|
||||
document.addEventListener('visibilitychange', onVisible);
|
||||
return () => {
|
||||
clearInterval(interval);
|
||||
document.removeEventListener('visibilitychange', onVisible);
|
||||
};
|
||||
}, [mx, reminders, setToast, removeReminder, mDirects]);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function LotusDenoiseFeature() {
|
||||
const setToast = useSetAtom(toastQueueAtom);
|
||||
|
||||
@@ -417,6 +462,7 @@ export function ClientNonUIFeatures({ children }: ClientNonUIFeaturesProps) {
|
||||
<PresenceUpdater />
|
||||
<InviteNotifications />
|
||||
<MessageNotifications />
|
||||
<ReminderMonitor />
|
||||
<LotusDenoiseFeature />
|
||||
<DeepLinkNavigator />
|
||||
{children}
|
||||
|
||||
@@ -14,10 +14,12 @@ export type MessageSpacing = '0' | '100' | '200' | '300' | '400' | '500';
|
||||
// - 'browser' : WebRTC built-in suppression (Element Call noiseSuppression param)
|
||||
// - 'ml' : client-side RNNoise ML suppression (Lotus denoise shim)
|
||||
export type NoiseSuppressionMode = 'off' | 'browser' | 'ml';
|
||||
// Self-hostable, build-bundled ML models. DeepFilterNet remains excluded — it
|
||||
// loads its runtime/models from external CDNs, which breaks the self-hosted /
|
||||
// Tauri-CSP strategy (see LOTUS_DENOISE_ENGINEERING_REVIEW.md).
|
||||
export type DenoiseModelId = 'rnnoise' | 'speex' | 'dtln';
|
||||
// Self-hostable, build-bundled ML models. DeepFilterNet 3 is included via
|
||||
// deepfilternet3-noise-filter with its df_bg.wasm + ONNX model VENDORED and
|
||||
// self-hosted (its cdnUrl is overridden), so it no longer depends on an external
|
||||
// CDN. Its wasm is single-threaded, so no COOP/COEP cross-origin isolation is
|
||||
// required (see LOTUS_DENOISE_ENGINEERING_REVIEW.md).
|
||||
export type DenoiseModelId = 'rnnoise' | 'speex' | 'dtln' | 'deepfilternet';
|
||||
export type ChatBackground =
|
||||
| 'none'
|
||||
| 'blueprint'
|
||||
@@ -260,12 +262,13 @@ export const getSettings = (): Settings => {
|
||||
? 'browser'
|
||||
: 'off'
|
||||
: (saved.callNoiseSuppression ?? defaultSettings.callNoiseSuppression),
|
||||
// Coerce any retired/unknown persisted model (e.g. 'dtln', 'deepfilternet'
|
||||
// from earlier beta builds) back to the default working model.
|
||||
// Coerce any retired/unknown persisted model back to the default working
|
||||
// model; only whitelisted ids pass through.
|
||||
callDenoiseModel:
|
||||
saved.callDenoiseModel === 'rnnoise' ||
|
||||
saved.callDenoiseModel === 'speex' ||
|
||||
saved.callDenoiseModel === 'dtln'
|
||||
saved.callDenoiseModel === 'dtln' ||
|
||||
saved.callDenoiseModel === 'deepfilternet'
|
||||
? saved.callDenoiseModel
|
||||
: defaultSettings.callDenoiseModel,
|
||||
composerToolbarButtons: {
|
||||
|
||||
@@ -117,7 +117,7 @@ export const CodeBlockBottomShadow = style({
|
||||
pointerEvents: 'none',
|
||||
|
||||
height: config.space.S400,
|
||||
background: `linear-gradient(to top, #00000022, #00000000)`,
|
||||
background: `linear-gradient(to top, ${color.Surface.Container}22, transparent)`,
|
||||
});
|
||||
|
||||
const BaseList = style({});
|
||||
@@ -255,7 +255,7 @@ export const EmoticonImg = style([
|
||||
export const highlightText = style([
|
||||
DefaultReset,
|
||||
{
|
||||
backgroundColor: 'yellow',
|
||||
color: 'black',
|
||||
backgroundColor: color.Warning.Container,
|
||||
color: color.Warning.OnContainer,
|
||||
},
|
||||
]);
|
||||
|
||||
@@ -14,10 +14,10 @@ import { DenoiseModelId } from '../state/settings';
|
||||
const BASE = `${import.meta.env.BASE_URL.replace(/\/+$/, '')}/public/element-call/denoise/`;
|
||||
|
||||
/**
|
||||
* Required AudioContext sample rate per model. RNNoise/Speex (sapphi) assume
|
||||
* 48 kHz. DTLN (@workadventure) targets 16 kHz and does NOT resample internally
|
||||
* — running it at 48 kHz produces robotic/choppy/quiet output, so its whole
|
||||
* graph must run in a 16 kHz context.
|
||||
* Required AudioContext sample rate per model. RNNoise/Speex (sapphi) and
|
||||
* DeepFilterNet 3 are 48 kHz. DTLN (@workadventure) targets 16 kHz and does NOT
|
||||
* resample internally — running it at 48 kHz produces robotic/choppy/quiet
|
||||
* output, so its whole graph must run in a 16 kHz context.
|
||||
*/
|
||||
export const sampleRateFor = (model: DenoiseModelId): number => (model === 'dtln' ? 16000 : 48000);
|
||||
|
||||
@@ -75,6 +75,32 @@ export async function buildModelNode(
|
||||
const handle = await mod.createNoiseSuppressionAudioWorklet(ctx, { bypassUntilReady: true });
|
||||
return { node: handle.node, dispose: () => handle.dispose() };
|
||||
}
|
||||
if (model === 'deepfilternet') {
|
||||
// deepfilternet3-noise-filter ESM: inlines its worklet/wasm-bindgen glue and
|
||||
// fetches only df_bg.wasm + the ONNX model, which we self-host under
|
||||
// deepfilternet/v2/... Override its cdnUrl to our absolute base so nothing
|
||||
// hits the upstream CDN. DeepFilterNet3Core builds the worklet node directly.
|
||||
const dfnBase = new URL(`${BASE}deepfilternet`, window.location.href).href;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const mod: any = await import(/* @vite-ignore */ `${BASE}deepfilternet/index.esm.js`);
|
||||
const core = new mod.DeepFilterNet3Core({
|
||||
sampleRate: sampleRateFor(model),
|
||||
noiseReductionLevel: 80,
|
||||
assetConfig: { cdnUrl: dfnBase },
|
||||
});
|
||||
await core.initialize();
|
||||
const node: AudioWorkletNode = await core.createAudioWorkletNode(ctx);
|
||||
return {
|
||||
node,
|
||||
dispose: () => {
|
||||
try {
|
||||
core.destroy();
|
||||
} catch {
|
||||
/* noop */
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
const cfg = SAPPHI[model];
|
||||
const [, wasmBinary] = await Promise.all([addModuleOnce(ctx, cfg.script), fetchWasm(cfg.wasm)]);
|
||||
const node = new AudioWorkletNode(ctx, cfg.proc, {
|
||||
|
||||
@@ -2,8 +2,10 @@
|
||||
* Detection utilities for Lotus ML noise suppression (RNNoise).
|
||||
*/
|
||||
|
||||
import { DenoiseModelId } from '../state/settings';
|
||||
|
||||
export type DenoiseModel = {
|
||||
id: string;
|
||||
id: DenoiseModelId;
|
||||
name: string;
|
||||
description: string;
|
||||
cpuUsage: string;
|
||||
@@ -40,6 +42,15 @@ export const DENOISE_MODELS: DenoiseModel[] = [
|
||||
transients: 'Excellent',
|
||||
voiceQuality: 'High',
|
||||
},
|
||||
{
|
||||
id: 'deepfilternet',
|
||||
name: 'DeepFilterNet 3 (beta)',
|
||||
description: 'Studio-grade deep-learning model (48 kHz, ONNX). Best quality; highest CPU.',
|
||||
cpuUsage: '25-50%',
|
||||
binarySize: '~18 MB',
|
||||
transients: 'Excellent',
|
||||
voiceQuality: 'Very High',
|
||||
},
|
||||
];
|
||||
|
||||
export const isMLDenoiseSupported = (): boolean => {
|
||||
|
||||
@@ -123,6 +123,43 @@ function lotusDenoise() {
|
||||
}
|
||||
fs.cpSync(dtlnSrc, path.join(denoiseDir, 'workadventure'), { recursive: true });
|
||||
|
||||
// DeepFilterNet 3 (deepfilternet3-noise-filter): the npm package ships only
|
||||
// its ESM (index.esm.js) with the AudioWorklet processor + wasm-bindgen glue
|
||||
// INLINED as a string (loaded via a Blob URL, no CDN for the worklet). The
|
||||
// only runtime CDN fetches are its single-threaded `df_bg.wasm` and the
|
||||
// ONNX `DeepFilterNet3_onnx.tar.gz` model — which we VENDOR locally (under
|
||||
// build/denoise-vendor/deepfilternet/) and serve, overriding the package's
|
||||
// cdnUrl to our self-hosted base. This keeps the feature CDN-free / Tauri-CSP
|
||||
// safe. We copy the ESM (the shim dynamic-imports it, mirroring DTLN) plus
|
||||
// the vendored assets, preserving the package's expected v2/... layout. All
|
||||
// are required — a missing entry means a broken install, so fail the build.
|
||||
const dfnEsm = path.resolve('node_modules/deepfilternet3-noise-filter/dist/index.esm.js');
|
||||
const dfnVendor = path.resolve('build/denoise-vendor/deepfilternet');
|
||||
const dfnAssets = [
|
||||
[dfnEsm, path.join(denoiseDir, 'deepfilternet/index.esm.js')],
|
||||
[
|
||||
path.join(dfnVendor, 'v2/pkg/df_bg.wasm'),
|
||||
path.join(denoiseDir, 'deepfilternet/v2/pkg/df_bg.wasm'),
|
||||
],
|
||||
[
|
||||
path.join(dfnVendor, 'v2/models/DeepFilterNet3_onnx.tar.gz'),
|
||||
path.join(denoiseDir, 'deepfilternet/v2/models/DeepFilterNet3_onnx.tar.gz'),
|
||||
],
|
||||
];
|
||||
const dfnMissing = dfnAssets.filter(([s]) => !fs.existsSync(s)).map(([s]) => s);
|
||||
if (dfnMissing.length > 0) {
|
||||
throw new Error(
|
||||
'[lotus-denoise] DeepFilterNet 3 asset(s) missing — build aborted to avoid ' +
|
||||
'shipping a broken ML feature:\n ' +
|
||||
dfnMissing.join('\n ') +
|
||||
'\n(Run `npm ci`; the vendored wasm/model live under build/denoise-vendor/deepfilternet/.)',
|
||||
);
|
||||
}
|
||||
dfnAssets.forEach(([s, d]) => {
|
||||
fs.mkdirSync(path.dirname(d), { recursive: true });
|
||||
fs.copyFileSync(s, d);
|
||||
});
|
||||
|
||||
const shimSrc = path.resolve('build/lotus-denoise.js');
|
||||
if (!fs.existsSync(shimSrc)) {
|
||||
throw new Error(`[lotus-denoise] Missing shim source ${shimSrc} — build aborted.`);
|
||||
|
||||
Reference in New Issue
Block a user