iOS Remotely control and mirror an iOS device from your browser with a raspberry pi

[SenorClippy]

I recently published a blog post for my work on how to remotely control and mirror an iOS device from your browser using a Raspberry Pi and two publicly available github libraries. An Apple developer account and Xcode are NOT needed.https://plusqa.com/2022/01/18/control-your-ios-device-over-the-internet/

It turned out to not be that complicated.The first library, ios-screen-mirror, is a Go program that mirrors a usb-connected iOS device via TCP.The second library, keyboard_mouse_emulate_on_raspberry, is a Python program that emulates a Bluetooth mouse and keyboard on a Raspberry Pi, which is then connected to the iOS device via Bluetooth.Then it was just a matter of sending the browser mouse and keyboard events to the Raspberry Pi (I used Flask and Socket.io for that)

https://senorclippy.tumblr.com%2Fpost%2F674041796561633280
Let me know what y'all think!

 

[duanepatrick]

This is interesting. But what's with the timer?

 

[SenorClippy]

This is interesting. But what's with the timer?

It's specific to my use case. The company I work for, Plus QA, primarily does manual testing of apps and websites on real devices. The motivation for me setting this up was to allow testers to test bug regressions on certain edge case devices without physically having the devices. We also didn't want people hogging the remote devices longer than needed or forgetting to end their session so we added a one hour timer that automatically kicks them off lol.

 

[prime17569]

This is cool! It reminds me of appetize.io, which I believe uses a similar method to what you're doing, but with iOS simulators as opposed to real devices.

How did you get iOS to work with mouse and keyboard events? I thought only iPadOS supported keyboards and mice.

Also, if you hold down the command key in an app, does a menu bar pop up as it does on iPadOS?

 

[SenorClippy]

This is cool! It reminds me of appetize.io, which I believe uses a similar method to what you're doing, but with iOS simulators as opposed to real devices.

How did you get iOS to work with mouse and keyboard events? I thought only iPadOS supported keyboards and mice.

Also, if you hold down the command key in an app, does a menu bar pop up as it does on iPadOS?

Thanks!

The Raspberry Pi is emulating a bluetooth mouse and keyboard and is connected to the iphone as such. The keyboard and mouse events from my browser are being sent to the Raspberry Pi where they are converted to hex commands that the emulator uses as traditional mouse and keyboard commands. So to answer your other question, the command key would do whatever a normal keyboard would thats connected to an ipad.

 

[prime17569]

Thanks!

The Raspberry Pi is emulating a bluetooth mouse and keyboard and is connected to the iphone as such. The keyboard and mouse events from my browser are being sent to the Raspberry Pi where they are converted to hex commands that the emulator uses as traditional mouse and keyboard commands. So to answer your other question, the command key would do whatever a normal keyboard would thats connected to an ipad.

Thanks for the detailed explanation!

Do iPad keyboard shortcuts work on iPhone (e.g. command-space on the home screen brings up Spotlight)?

For the command key question, I was referring to how a menu bar pops up on iPadOS 15 or newer when you hold down the command key in an app that supports keyboard shortcuts, and asking if this behavior also occurs on iPhone with a hardware keyboard connected.

 

[Jakexb]

This is amazing, thank you I've been looking for a way to remotely help my mom sort out issues with her iPhone. I'm definitely going to give this a try. Nothing else has been even close to solving my problem.

 

[cslewis2018]

Can this technique work, but with the Raspberry Pi emulating a physical keyboard / mouse via it's USB OTG port?