Video scraping is only one of many new methods potential when the most recent giant language fashions (LLMs), similar to Google’s Gemini and GPT-4o, are literally “multimodal” fashions, permitting audio, video, picture, and textual content enter. These fashions translate any multimedia enter into tokens (chunks of information), which they use to make predictions about which tokens ought to come subsequent in a sequence.
A time period like “token prediction mannequin” (TPM) is perhaps extra correct than “LLM” as of late for AI fashions with multimodal inputs and outputs, however a generalized various time period hasn’t actually taken off but. However it doesn’t matter what you name it, having an AI mannequin that may take video inputs has fascinating implications, each good and probably dangerous.
Breaking down enter obstacles
Willison is way from the primary individual to feed video into AI fashions to realize fascinating outcomes (extra on that beneath, and here is a 2015 paper that makes use of the “video scraping” time period), however as quickly as Gemini launched its video enter functionality, he started to experiment with it in earnest.
In February, Willison demonstrated one other early utility of AI video scraping on his weblog, the place he took a seven-second video of the books on his bookshelves, then bought Gemini 1.5 Professional to extract all the guide titles it noticed within the video and put them in a structured, or organized, record.
Changing unstructured information into structured information is necessary to Willison, as a result of he is additionally a information journalist. Willison has created instruments for information journalists up to now, such because the Datasette venture, which lets anybody publish information as an interactive web site.
To each information journalist’s frustration, some sources of information show immune to scraping (capturing information for evaluation) on account of how the information is formatted, saved, or offered. In these circumstances, Willison delights within the potential for AI video scraping as a result of it bypasses these conventional obstacles to information extraction.
Video scraping is only one of many new methods potential when the most recent giant language fashions (LLMs), similar to Google’s Gemini and GPT-4o, are literally “multimodal” fashions, permitting audio, video, picture, and textual content enter. These fashions translate any multimedia enter into tokens (chunks of information), which they use to make predictions about which tokens ought to come subsequent in a sequence.
A time period like “token prediction mannequin” (TPM) is perhaps extra correct than “LLM” as of late for AI fashions with multimodal inputs and outputs, however a generalized various time period hasn’t actually taken off but. However it doesn’t matter what you name it, having an AI mannequin that may take video inputs has fascinating implications, each good and probably dangerous.
Breaking down enter obstacles
Willison is way from the primary individual to feed video into AI fashions to realize fascinating outcomes (extra on that beneath, and here is a 2015 paper that makes use of the “video scraping” time period), however as quickly as Gemini launched its video enter functionality, he started to experiment with it in earnest.
In February, Willison demonstrated one other early utility of AI video scraping on his weblog, the place he took a seven-second video of the books on his bookshelves, then bought Gemini 1.5 Professional to extract all the guide titles it noticed within the video and put them in a structured, or organized, record.
Changing unstructured information into structured information is necessary to Willison, as a result of he is additionally a information journalist. Willison has created instruments for information journalists up to now, such because the Datasette venture, which lets anybody publish information as an interactive web site.
To each information journalist’s frustration, some sources of information show immune to scraping (capturing information for evaluation) on account of how the information is formatted, saved, or offered. In these circumstances, Willison delights within the potential for AI video scraping as a result of it bypasses these conventional obstacles to information extraction.
