Todd Fisher traces a personal guitar project from a Halloween light display to a plugin that makes an instrument trigger spoken words. The project combines a digital audio workstation plugin with text to speechText-to-speech converts written text into spoken audio by predicting pronunciation, timing, prosody and a waveform or intermediate audio representation. and note driven playback.
Todd Fisher explains why automatically slicing speech into words is difficult and compares silence based segmentation with syllable detection. He then uses pitch detection and a vocoder to map guitar notes onto speech, balancing the synthesized pitch with the voice signal.
Todd Fisher extends the prototype into a conversational loopA voice interface lets people interact with software through spoken input and audible responses.: microphone input becomes text through speech recognitionAutomatic speech recognition converts spoken language into text or structured linguistic output using signal processing and machine learning., a local language modelA local model runs on computing infrastructure controlled directly by the user rather than only through a remote model service. generates a response, and speech is routed back through the guitar. Preprocessed vocal samples and pitch shifting move the project closer to singing, although the live demos expose latency, segmentation, and clarity limitations.
Watch on YouTube
