While debugging our layout shift issue I learned that a single change to the sort order of the participants can cause 3 or 4 redundant emissions of the same items in the same order. Since each of these would cause React to re-render the grid, skipping these spurious emissions seems like an easy performance win.
Instead of tracking for each individual tile whether it's visible, just track the total number of tiles that appear on screen. This ought to make the whole thing a lot less dynamic, which is crucial given that our UI renders asynchronously and RxJS doesn't really support cyclic dependencies in any rigorous way.
In particular this ought to make the following kind of situation impossible:
1. There 3 tiles, ABC. A and B are on screen.
2. Now C becomes important. The requested order is now CAB.
3. To reduce the size of the layout shift, the algorithm selects to swap just B and C in the original order, giving ACB. However, the UI is blocked and doesn't render this order yet.
4. For whatever reason, a spurious update of the importance algorithm occurs. It once again requests CAB.
5. Now because the UI was blocked, the layout still thinks that A and B are on screen (rather than A and C). It thinks that C is some weird island of "off-screen territory" in the middle of the tile order. This confuses it into swapping A and C rather than keeping the layout stable.
The reality is that whenever we think N tiles are visible on screen, we're always referring to the first N tiles in the grid. It's best if the code reflects this assumption.
* Add some quick-and-dirty debug info for TileStore
I'm still in need of more detailed data in order to understand why big layout shifts happen in large calls. This adds a developer option to enable logging and a visual indicator for the state of the TileStore. The indicator should be useful for matching up the behavior I'm seeing in my recordings with the right timestamps.
* Reduce performance impact of checking for whether debug mode is enabled
---------
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
Managing your audio output manually is kind of cumbersome; Chrome creates a default audio output for us, but now that audio outputs are enabled on Firefox as well, I find it necessary for a good user experience that there always be a way to set it to "whatever the default is".
This is closer to what the designs actually want device settings to look like, and it avoids the visual glitch in which the dropdown would render underneath the slider.
Somewhere around version 131 or 132, Firefox has started being more paranoid about media device fingerprinting, and will not even give you the IDs of available devices until you've requested a media stream. Instead you only get a single audio input and video input each with the empty string as their device ID, representing the system's default device. We can recognize this case and avoid resetting the device selection.
As a non-intrusive way to help you remember when you've locally muted a participant (so you're less likely to be confused about not hearing their audio), we can show a different icon on the tile.
By rights, this fix I had made to decrypt reaction events shouldn't have appeared successful, because I was requiring the event to have a certain type before asking matrix-js-sdk to decrypt it, and you can't know an event's type before it's decrypted. Probably what was happening is that another code path was requesting the events to be decrypted so that this mistake didn't matter.
The basic issue here, I think, was that the 'oneOnOne' observable flag and the 'onOnOneLayout' observable could become out of sync, as RxJS does *not* have atomicity guarantees. We can work around this by combining them into one observable.
This fixes a few different usability issues with the footer:
- When tapping one of the footer buttons, the footer would be dismissed rather than activating the button.
- When the footer was hidden, you could still tap the buttons.
- Interacting with controls in the footer would not reset the timer that hides it, leading to a feeling that the footer can disappear out from under you.
The observable-hooks package provides hooks that do exactly the same thing as these custom React hooks I had written a while back. (even the names are the same, wow)
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
* Keep tiles in a stable order
This introduces a new layer of abstraction on top of MediaViewModel: TileViewModel, which gives us a place to store data relating to tiles rather than their media, and also generally makes it easier to reason about tiles as they move about the call layout. I have created a class called TileStore to keep track of these tiles.
This allows us to swap out the media shown on a tile as the spotlight speaker changes, and avoid moving tiles around unless they really need to jump between the visible/invisible regions of the layout.
* Don't throttle spotlight updates
Since we now assume that the spotlight and grid will be in sync (i.e. an active speaker in one will behave as an active speaker in the other), we don't want the spotlight to ever lag behind due to throttling. If this causes usability issues we should maybe look into making LiveKit's 'speaking' indicators less erratic first.
* Make layout shifts due to a change in speaker less surprising
Although we try now to avoid layout shifts due to the spotlight speaker changing wherever possible, a spotlight speaker coming from off screen can still trigger one. Let's shift the layout a bit more gracefully in this case.
* Improve the tile ordering tests
* Maximize the spotlight tile in portrait layout
* Tell tiles whether they're actually visible in a more timely manner
* Fix test
* Fix speaking indicators logic
* Improve readability of marbles
* Fix test case
---------
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
Hugh pointed out that the button labeled "Join call now" on this screen is misleading, because it does not join you to the call but rather send you to the lobby. Meanwhile, the designs call for this screen to use the words "Join as guest" and "Continue", so let's go with that.
Before you're connected to the SFU the local participant object will have the empty string as its ID. This changes to your actual ID once you've connected. Apparently I tried to fix this by forcing the local ID to always be the string 'local' but then I just forgot to use it correctly :)
Previously, dragging it all the way to the left would *not* mute the participant but rather bottom out at 10% volume, and people have found this unintuitive. Let's make it less silly by giving the slider a range of 0% to 100%, and making the mute toggle button have the same effect as dragging the slider to zero. When unmuting, it will reset to the last non-zero "committed" volume, similar to how the volume sliders in desktop environments work.
RobinandGitHub
