Hello everyone, I've been thinking about creating a plugin that would allow me to use my iPad as an external controller of sorts for Sketch. I think a multitouch external interface like a tablet could provide a lot of added utility working in tandem with Sketch.
In its most basic form this might be just a grid of shortcut buttons I could quickly tap on a tablet (or phone) next to my computer as I'm designing. It could also have sliders, joysticks, rotary dials, etc to speed up things like nudging layers, expanding/contracting layers, or updating values in the inspector. Those would be what I would want to experiment with for a first prototype, but I have a bunch of other more complex ideas if I can get those working.
So I thought about maybe using WebSocket. I found a thread on here that mentioned websocket, but also said that communication was only possible while the sketch plugin's webview was open. So I'm not quite sure if that limitation will make what I'm trying to do impractical.
Lastly, one other slightly hacky idea I had would be to use Web MIDI to duct-tape everything together. I think this might be the easiest for me to pull off and would satisfy my desire to host the external interface as a webpage. I'm thinking I could have a webpage function as a virtual MIDI controller to send notes and CC messages to my computer. Then by adding the Web MIDI library to my sketch plugin I think I could receive the MIDI data there and trigger actions in Sketch accordingly. I haven't found any record of anybody trying something like this before so I'm not 100% if that would be possible. CC messages are also limited to 1–127, so that would require some workaround to have sliders that adjust larger ranges inside Sketch, but I that's not a showstopper if the basic Web MIDI concept seems like the most viable solution. Using OSC might be a better alternative but then I'd still need to figure out how to send the OSC data between my iPad and laptop.