Sent to me by Sascha Frietsch on 02.03.2020

(“Spec” for OLD (Twilio based) iOS App)

We are in need of an iOS app that uses the screen sharing capabilities of the Twilio Programmable Video SDK (https://www.twilio.com/docs/video).

We have a web based conference software that uses Twilio Video. Now we want to add screen sharing of mobile devices (phones and tablets), starting with this prototype for iOS devices.

No audio streams, no camera based video streams. This is “screen sharing only”.

I'm going to define the environment and the app requirements describing the user workflow.

• The user is in a call using our conference software running in a Twilio compatible webbrowser (desktop or mobile).
• The user decides to share his mobile phone's screen with the current room. (A SECOND mobile device, if the user already uses one for the call)
• He clicks a “share mobile screen” button
• A generated QR code is shown to the user. This QR code contains a Twilio token, that contains itself the user identity and the target room. (we care about the generation, common encoding that can be generated in JS and read from iOS, tbd)

Specs start here:

• The user opens our shiny new iOS app. On a simple home screen with company logo he then taps the “connect” button.
• The user scans the QR code from the webpage
• The QR code is decoded, the found Twilio token is used to connect to the conference room.
• When successfully joined the room, the app starts publishing a video stream of the device screen into the room.
• The “connect” button turns into a “stop” button
• The screen is transmitted, even if the app itself is in the background.
• Screen sharing continues until one of these happens:
  1. The user brings the app to foreground and taps the “stop” button.
  2. In case the “screen sharing user” forgot about the active sharing running in the background,

the room will be forced to close using the Twilio REST API after the host user of the conference has left the room. When this happens, a room event is fired.

• In both cases the app unpublishes the screen video and leaves the room, cleans up everything and returns to the home screen, which now shows up the “connect” button again.

UPDATE: Maybe I wasn't clear enough with this. The main problem is (Twilio Docs): “The TVIScreenCapturer class captures the contents of a single view at 1x scale. As this class depends on the UIKit and CoreGraphics frameworks to snapshot content, it does not function if the application is backgrounded.” - but we need screensharing to function if the application is backgrounded. You can create a custom capturer that uses ReplayKit screensharing. Official Twilio example here: https://github.com/twilio/video-quickstart-ios/tree/2.6.0-preview/ReplayKitExample Since I'm no “iOS guy” I have no clue if this is the only possibility but it seems that ReplayKit is the preferred one for this use case.

No audio: The app will be used only from users that are already connected to the conference with another device which they use for audio/ video communication. They are only adding the screen video from the second device.

QR: During the conference the user can click a “share mobile screen” button which generates the qr code on the fly and shows it to the user. The user scans it with his second device right off the screen.

The web app notices when new participants join and leave the room and displays published video streams. The token used to connect to Twilio includes the room id and an identity for the participant. So if Mubashir is one of the participants and shares his screen, we will include something like “Mubashir's screen” as the identity within the token that gets QRencoded. These identities are shown as labels beside the videos.

Für Android gibt’s das auch: https://github.com/twilio/video-quickstart-android