Hello everyone, I've been thinking about creating a plugin that would allow me to use my iPad as an external controller of sorts for Sketch. I think a multitouch external interface like a tablet could provide a lot of added utility working in tandem with Sketch.
In its most basic form this might be just a grid of shortcut buttons I could quickly tap on a tablet (or phone) next to my computer as I'm designing. It could also have sliders, joysticks, rotary dials, etc to speed up things like nudging layers, expanding/contracting layers, or updating values in the inspector. Those would be what I would want to experiment with for a first prototype, but I have a bunch of other more complex ideas if I can get those working.
I'm wondering if anybody has any ideas on how this might be accomplished? Ideally I'd like the external iPad interface to be hosted on a webpage, so I'm not sure if a Bonjour server would be a viable option. I don't have any experiencing building native apps, so I'm just trying to simplify the amount I'd have to learn for this project. I'm familiar enough with Javascript to be able to build the UI on a webpage. I also always had problems with Sketch Mirror, so even if I put the time into learning how to make an iOS app I'm not sure how reliable that would be.
So I thought about maybe using WebSocket. I found a thread on here that mentioned websocket, but also said that communication was only possible while the sketch plugin's webview was open. So I'm not quite sure if that limitation will make what I'm trying to do impractical.
Lastly, one other slightly hacky idea I had would be to use Web MIDI to duct-tape everything together. I think this might be the easiest for me to pull off and would satisfy my desire to host the external interface as a webpage. I'm thinking I could have a webpage function as a virtual MIDI controller to send notes and CC messages to my computer. Then by adding the Web MIDI library to my sketch plugin I think I could receive the MIDI data there and trigger actions in Sketch accordingly. I haven't found any record of anybody trying something like this before so I'm not 100% if that would be possible. CC messages are also limited to 1–127, so that would require some workaround to have sliders that adjust larger ranges inside Sketch, but I that's not a showstopper if the basic Web MIDI concept seems like the most viable solution. Using OSC might be a better alternative but then I'd still need to figure out how to send the OSC data between my iPad and laptop.
If my ignorance is not obvious from this post, I don't have a ton of Javascript experience and this would be my first Sketch plugin. I thought I'd at least try and explore this idea and see if I can learn enough to figure it out. That being said—if I'm overlooking something obvious that would make this impossible, I would certainly appreciate that being pointed out. OR if there is some technology I'm not familiar with that might make this idea easier, that info would also be very appreciated. Thanks in advance!
