NC_2024_10_20
The episode discusses AI experiences, a synthesized voice journey, Overcast insights, smartphone camera advancements, light pollution, blogging challenges, and reflections on AI's growth plateau and its impact on innovation and collaboration.
Automatic Shownotes
Chapters
Long Summary
In this episode, I discuss my recent experiences with AI technologies, particularly focusing on a synthesized voice I used after losing my own voice to laryngitis. I promised to delve deeper into the process of how I created this AI-generated voice and the implications of using such technology in communications. While I had planned to share detailed insights, my current voice limitations necessitated that I summarize my findings and provide an enjoyable follow-up conversation with Bart, which is sure to pique your interest.
Earlier this year, I wrote an extensive article outlining the complete redesign of the popular podcatcher Overcast, discussing its features and usability. As an enhancement for those who prefer visual content, I also produced a video tutorial for Screencast Online, where I guide users through the app’s functionality. This is a valuable resource for anyone looking to improve their podcast listening experience with customized playlists and playback settings. Featuring sophisticated tools and a user-friendly interface, Overcast is a versatile choice for podcast consumption.
In an unexpected twist, Bart decided to invite me onto his photography podcast, Let's Talk Photography. Our discussion revolved around the remarkable advancements in smartphone camera technology that enable users to capture stunning images that surpass what our naked eyes can perceive. This fascinating dialogue, titled "Better Than Your Eyes," explores how devices like our new telescope enhance our view of the cosmos, even from heavily light-polluted locations like Los Angeles. Bart shared his insights on tracking celestial objects using various apps, making this episode a rich exploration of both technology and astronomy.
As I navigated some technical challenges in creating blog posts, I found myself relying on my go-to blogging software, MarsEdit. This episode includes a segment detailing my struggles and solutions in uploading different media types and properly formatting them for my blog. I also took on the task of creating an automated solution using JavaScript within TextExpander, which simplifies the tedious process of formatting URLs for my blog posts. By sharing my interaction with ChatGPT during this development process, I illustrate the practical uses of AI in programming tasks while also highlighting the lessons learned along the way.
Moreover, I thoroughly examined a recent journal article exploring the impact of large language models on public knowledge sharing across online platforms like Stack Overflow. The study, which analyzed engagement before and after the emergence of ChatGPT, revealed troubling trends in knowledge sharing now increasingly reliant on AI rather than human contributions. Bart and I engaged in a captivating dialogue on what this means for the future of programming help, the necessity of continuing human engagement, and the broader implications for technology and society.
As we navigated these discussions, we pondered the concept of artificial intelligence's rapid growth and the potential for it to reach a plateau. Bart introduced the idea of a sigmoid curve of technology growth, asserting that while the advancements are significant, we may soon find ourselves in a phase where the acceleration of progress slows. By dissecting the interplay between existing knowledge bases and AI's capacity to learn from them, we engaged in a thought-provoking conversation about the state and future of technology and what that means for innovation and human collaboration.
Throughout this episode, I aim to nurture a dialogue with all of you about the evolving landscape of technology, the advantages and challenges of AI assistance, and how we can best leverage these tools moving forward. Your feedback is invaluable, and I encourage open conversations regarding these topics as we continue our journey into the future of technology.
Earlier this year, I wrote an extensive article outlining the complete redesign of the popular podcatcher Overcast, discussing its features and usability. As an enhancement for those who prefer visual content, I also produced a video tutorial for Screencast Online, where I guide users through the app’s functionality. This is a valuable resource for anyone looking to improve their podcast listening experience with customized playlists and playback settings. Featuring sophisticated tools and a user-friendly interface, Overcast is a versatile choice for podcast consumption.
In an unexpected twist, Bart decided to invite me onto his photography podcast, Let's Talk Photography. Our discussion revolved around the remarkable advancements in smartphone camera technology that enable users to capture stunning images that surpass what our naked eyes can perceive. This fascinating dialogue, titled "Better Than Your Eyes," explores how devices like our new telescope enhance our view of the cosmos, even from heavily light-polluted locations like Los Angeles. Bart shared his insights on tracking celestial objects using various apps, making this episode a rich exploration of both technology and astronomy.
As I navigated some technical challenges in creating blog posts, I found myself relying on my go-to blogging software, MarsEdit. This episode includes a segment detailing my struggles and solutions in uploading different media types and properly formatting them for my blog. I also took on the task of creating an automated solution using JavaScript within TextExpander, which simplifies the tedious process of formatting URLs for my blog posts. By sharing my interaction with ChatGPT during this development process, I illustrate the practical uses of AI in programming tasks while also highlighting the lessons learned along the way.
Moreover, I thoroughly examined a recent journal article exploring the impact of large language models on public knowledge sharing across online platforms like Stack Overflow. The study, which analyzed engagement before and after the emergence of ChatGPT, revealed troubling trends in knowledge sharing now increasingly reliant on AI rather than human contributions. Bart and I engaged in a captivating dialogue on what this means for the future of programming help, the necessity of continuing human engagement, and the broader implications for technology and society.
As we navigated these discussions, we pondered the concept of artificial intelligence's rapid growth and the potential for it to reach a plateau. Bart introduced the idea of a sigmoid curve of technology growth, asserting that while the advancements are significant, we may soon find ourselves in a phase where the acceleration of progress slows. By dissecting the interplay between existing knowledge bases and AI's capacity to learn from them, we engaged in a thought-provoking conversation about the state and future of technology and what that means for innovation and human collaboration.
Throughout this episode, I aim to nurture a dialogue with all of you about the evolving landscape of technology, the advantages and challenges of AI assistance, and how we can best leverage these tools moving forward. Your feedback is invaluable, and I encourage open conversations regarding these topics as we continue our journey into the future of technology.
Brief Summary
In this episode, I discuss my experiences with AI, including the creation of a synthesized voice after losing my own. I also explore the redesigned podcatcher Overcast, sharing insights from my accompanying video tutorial.
Joining Bart on his photography podcast, we delve into advancements in smartphone camera technology, as well as the effects of light pollution on our cosmic observations. I address technical challenges in blogging, automation solutions using JavaScript, and a study on the impact of large language models on knowledge sharing.
Finally, we reflect on the potential plateau of AI growth and its implications for innovation and human collaboration. I invite you to join the conversation on the evolving landscape of technology and its responsible use.
Joining Bart on his photography podcast, we delve into advancements in smartphone camera technology, as well as the effects of light pollution on our cosmic observations. I address technical challenges in blogging, automation solutions using JavaScript, and a study on the impact of large language models on knowledge sharing.
Finally, we reflect on the potential plateau of AI growth and its implications for innovation and human collaboration. I invite you to join the conversation on the evolving landscape of technology and its responsible use.
Tags
AI
synthesized voice
Overcast
video tutorial
smartphone camera
light pollution
blogging challenges
automation
JavaScript
large language models
innovation
human collaboration
technology landscape
responsible use
Transcript
[0:00]
NC_2024_10_20
[0:00]Hi, this is Allison Sheridan of the NoCillaCast podcast, hosted at Podfeet.com, a technology geek podcast with an ever-so-slight Apple bias. Today is Sunday, October 20th, 2024, and this is show number 1015. Well, I've got a great show for you this week, but it's not exactly what I had planned. Last week, you'll remember that I had to use an AI synthesized voice to read one of the articles for the show, and I promised you I'd talk about how I did it this week. While I did write up an extensive article explaining the two services I worked with and the pros and cons of both, you can tell by listening to me that my voice isn't yet strong enough to go through this whole thing. Don't worry though, I do have one article from me and a follow-on conversation with Bart that will entertain and perhaps even educate you. And then next week, you'll learn all about how I created my fancy new laryngitis-replacing voice.
[0:54]
ScreenCastsONLINE Tutorial – Overcast
[0:54]In July, I wrote an article walking you through the complete redesign of the podcatcher Overcast. If you like learning from text and screenshots, then you're in luck. It's a great article. But if you prefer to learn through video tutorials to actually see the app in action, you are also in luck. We've just published my latest tutorial on Screencast Online about Overcast. I'll teach you how to find podcasts, set up playlists, tailor each podcast playback and episode counts, and even how to make the theme match your taste. Overcast is a surprisingly deep and powerful podcast player, and I learned even more in creating the Screencast for you. Screencast Online is a subscription service, and if you value learning new tools, you'll find that all of the tutors at Screencast Online are actually fantastic. With a full tutorial and a weekly tip video, it's a lot of content for the money. You can get a free seven-day trial at screencastonline.com, and that'll let you watch the entire back catalog of current tutorials, including my tutorial on Overcast. If you want to check it out, you can go to Screencast Online and find the Overcast tutorial, and you can watch a teaser video even before you open your free trial.
[2:00]
Let’s Talk Photography 133: Better than Your Eyes!
[2:04]This week, Bart was doodling what he should talk about on his next episode of Let's Talk Photography, and he asked me if there's anything I'd like to hear about. I often come up with ideas for him, so this is always fun. Well, the answer I suggested this week turned into him inviting me onto the show to talk with him. The title of the episode of Let's Talk Photography is Better Than Your Eyes, and it's a perfect description of our discussion. We talked about how our phone cameras can now take better photos than we can even see with our own eyes. I learned this when viewing the recent Aurora photos posted online, as people explained that the color wasn't really there to their eyes, they were just able to capture it with their phones. And while it is possible to see the color in an aurora, it needs to be quite bright, and Bart explains why this is true. Then we move on to talking about Steve and my new telescope, the Unistellar Equinox 2, and that allows us to view and take photos of distant astronomical objects even from the light pollution capital of the world, Los Angeles. Again, another example of better than our eyes, things we can't see with the naked eye, but we can see with this telescope. Now, as Bart and I always do, we had great fun sharing stories and even talking about the apps we use to track celestial objects in the sky. You can find this episode at lets-talk.ie, and of course, there's a link in the show notes.
[3:25]
Quick Use of ChatGPT to Write JavaScript for TextExpander Snippet
[3:25]When I add images to podfi.com for blog posts, I use my blogging software of choice, MarsEdit. You've heard me mention it a lot of times. I simply drag it in image to the text of a post, and MarsEdit does all of the formatting automatically, uploads it to my server, and links it all up. But on occasion, I want to upload a different kind of media, and then I have to do things by hand. For example, take last week, when I played an audio excerpt of my AI voice from early 2023, created by Eleven Labs. To make this available to you for playback from within the blog post, I had to go through some manual gyrations. First, I launched my FTP client of choice, Transmit from Panic. I have a bookmark and transmit to the folder on my server, where uploaded media belongs. All of the images in this audio file go into blah blah blah blah slash upload slash year slash month. Then I dragged the audio file up to blah blah blah blah slash upload slash year slash month and that made it available to link within a blog post. The problem is I don't have memorized what the URL would be for this file. I'm calling the folder blah blah blah slash upload slash year slash month on purpose because it's the blah blah blah part that's the problematic part. I can right-click on the file in Transmit that I've just uploaded, and it offers to copy the URL for me.
[4:43]Unfortunately, Transmit doesn't know about my website. It knows just where the file is on my server. So WordPress takes all these files on my server and builds a website, but Transmit doesn't know anything about that. So if I copy the URL from Transmit, it doesn't even start with HTTPS colon slash slash. It starts with SFTP colon slash slash, which stands for Secure FTP.
[5:06]That SFTP colon slash slash is followed by the IP address of my server, not the name of my website, so it doesn't have either of the pieces I needed. There's a few more annoying folders in the URL from Transmit that only someone who works on web servers would appreciate.
[5:21]After all of this server glop information I just described, the rest of the URL is actually helpful to determine the location of the URL on the website. So the last part of it, after the blah, blah, blah, says slash blog slash wp-content slash upload slash 2024 slash 10 slash some file dot mp3. So all that part is actually going to be useful later. That might sound like nonsense to you, but if you right click on any media on my website, it'll have that same kind of information. So the beginning server glop, all that sftp colon slash slash IP address and all this other nonsense, that needs to be replaced with podfeed.com, okay? Now that I explained it, it really isn't that hard to fix the URL so I can plop it into a blog post, but it's hard to remember and it's tedious to do. I've been doing this for 19 plus years and just last week I had to go study an old embedded file to figure it out yet again because I just can't remember what it's supposed to look like. Bert and I noodled the problem in a little text message thread, and we came to the conclusion that a JavaScript string replacement script embedded inside TextExpander might make my life easier. My goal was to be able to right-click on the file I've uploaded in Transmit, you choose copy URL, and then use a TextExpander snippet to paste it that would be, you know, in this web server-friendly format. That's what I want to have, just right-click and copy, and then I go over and I do my text expander snippet and it pasted in the right way.
[6:47]Now, while I call myself a developer, my JavaScript has gotten rusty for misuse. I recently decided to start paying for ChatGPT 4.0, so I thought, hey, I'll ask for a wee bit of assistance in writing the script. Spoiler, in a small handful of conversational interactions with ChatGPT, I have a working text expander script. I want to walk through it as a way to illustrate what I mean by conversational, and how you kind of have to hone your question to get the right answer. Here's my exact conversation. My initial query went like this. Starting with the first text string that I'm going to give you, write a JavaScript script that strips off the front matter and changes it to the second string I'm going to give you. I then gave it the SFTP server location of the file and the HTTPS web server URL with which I wanted it replaced. I used a specific example for this first foray with ChatGPT.
[7:40]ChatGPT assigned variable names to the first string and then defined the second string as the first string but replacing that front matter with the front matter from the second string. I know that sounds funny, but it's doing exactly what I wanted it to do. So in other words, it said, replace the SFTP glop with the HTTPS glop. It then popped the second string into a console command to print it out. So here's a classic case of a computer doing exactly what you asked it to do, not what you meant to ask it to do. Since ChatGPT actually has memory of the continuing in conversation, I was able to adjust my query by saying, I need this to be generic. The first text string will be in the clipboard, and the front matter will always be, and I gave it the glop, and I said the second part will always be replaced, and I gave it the other glop.
[8:27]ChatGPT happily answered, here's a JavaScript script that works generically, assuming you're reading the first text string from the clipboard. This script will always replace the specific front matter you mentioned. The new script was only slightly more complicated. Since I told it I was starting with something in the clipboard, it helpfully put the final URL in the clipboard for me as well. Now one of the helpful features of ChatGPT when you do programming stuff like this is it not only gives you the code, it also explains how it works. It said, number one, it reads the current clipboard content using navigator.clipboard.readtext. Number two, it replaces the front matter and it gave the two replacement strings, and then it logs the new string to the console and optionally it writes the modified URL back to the clipboard. Now we're getting somewhere. I created a new snippet text expander, changed the text format to JavaScript, and I splatted in the code ChatGPT gave me. I tried the snippet, but it threw an error. I could have tried maybe trying to figure out what that error meant, but why should I figure that out by myself. I added it to my conversation with ChatGPT. I said, this throws the following error, and I dropped in the error. Notice I didn't ask it a question. I just said, there's an error.
[9:41]ChatGPT simply explained why it didn't work. It said, the error you're encountering is because navigator.clipboard.readtext is part of the Clipboard API, which only works in secure contexts, that is, over HTTPS, and in certain browser environments. If you're running this outside of such an environment, you can use a different approach to access Clipboard data depending on the platform.
[10:03]Well, that's interesting. I didn't know that. It went on to explain how I could fix the problem by installing a package through the Node Package Manager, but I figured this could be solved more simply and elegantly, if I just gave ChatGPT the information I should have given it in the first place. You see, I never told ChatGPT I wanted this to run inside TextExpander. Once I told ChatGPT the context, it splatted out a new set of code for me, and a new set of how it works explanations. I copied the new code, I splatted it into a TextExpander snippet, assigned an abbreviation to trigger it. I opened Transmit, copied the URL of a file, and then pasted it into a text editor, and there was my happy web-friendly URL. Now, I haven't made fire here or anything. Lots of people have been using ChatGPT for coding for quite some time now. But I wanted to walk through the experience to stimulate you to think about the lessons learned. First of all, I was able to save myself a lot of time in creating the JavaScript text expander snippet because I use ChatGPT. Because I use ChatGPT, though, I didn't actually learn how to do it myself. Now, if I thought ahead and made my initial query more specific, I could have gotten to the right answer more quickly, so I learned a little bit about how to work with ChatGPT.
[11:16]Now, perhaps it was quicker to let ChatGPT, though, highlight the problems in my query, so maybe that was quick enough. Being able to have a conversation, including memory, with a large language model, that's the dream we've always had of how a digital assistant should be able to help us. Finally, having access to ChatGPT makes coding more accessible to the novice, and I think it can probably speed up laborious typing of code, even for the accomplished programmer. Now, a lesson we've all probably learned by now is you have to test and verify whatever ChatGPT tells you. This really isn't all that different from getting an answer from a human on the web, though, is it? You want to study and understand what you're being told, no matter the source. I'm sure my perspective will rankle some folks, and if you're one of those people, I hope you'll tell me where you think I've gone astray. I truly welcome the discussion because, you know, we're all figuring this out right now.
[12:12]
Support the Show
[12:12]Well, our hero of the week is Keith Kuby. Maybe he heard about the cost of an AI voice for last week, or maybe just now realized he gets a lot of education and enjoyment out of the shows we produce here at the PodFeed podcast. Either way, he took his hard-earned dollars and he took them to podfeed.com slash Patreon and he signed up to support the show. I can't thank Keith and all of the other Patreon members enough for their patronage of the shows. It makes me feel good and it helps pay for things like AI-generated voices when I get sick.
[12:45]
Plateaus Coming on LLMs – Discussion with Bart Busschots
[12:46]Well now you've just heard me talk about how i use chat gpt 4.0 to write a javascript script for text expander to copy the url from transmit to a usable url for my blog post on social media when i posted the article that you just heard i wrote i hope bart isn't disappointed in me on the on the post so he responded in slack and uh i thought it'd be fun for you to hear it firsthand, what his reaction was to it. So Bart, tell the audience, were you disappointed in me for using AI to do JavaScript?
[13:17]Well, no. And OK, I'm going to be really, really, really honest. So what I wrote in Slack is what the adult me actually thinks. But there was a little voice in my head that was nowhere near as grown up. A little childish version of me, which would have been the version of me that opened my mouth 20 years ago. And the very, very, very, very, very first thing I thought was, well, that works, but it's three function calls. I could do it on one line, which I did. And I posted it into Slack. but that's not actually what a real me thinks anymore. It's what I used to think. So that's actually, there's three levels then. There's, what I meant was, were you disappointed in me for not doing the work of doing it myself? But your answer to a different question was, well, yeah, but it's, as solution wasn't as good, which is your childish solution. And then there's the third answer, which is another more nuanced view of it. That's a good point. Well, no, because I've never disappointed when, because ChatGPT 4.0, that's cutting edge tech. So being disappointed that the person I love because they love getting stuck into new tech is using new tech would be weird. Yeah, but I sort of feel like, you know, I should have done it the hard way, which would have taken me like three weeks to write a JavaScript script to do that. And I just went, write it for me. And it did.
[14:36]But no, I wouldn't look at it that way at all. I would look at it as if after ChatGPT4O finished and you didn't understand what the code did and you ran it anyway, then I'd be disappointed with my security hat on. Okay, okay. Because that's one of the most terrifying things is people, particularly people who may have privileges on a device, they may be running as an admin and they go to ChatGPT and they ask it to write a PowerShell script where they're like a domain admin and they run it. That makes me cringe every which way. But no, if ChatGPT helps you get from an empty page... Can we say I understood the gist of it? Is that good enough? Yeah. Yeah.
[15:16]It is, because the empty page problem is hard. I hate empty pages. I'm terribly bad at getting started on something. So I'm never going to judge anyone for using a tool to go from zero to anything. Okay. Once you have something, you can finesse and you can refine and you can keep going. But it's so hard to get to zero. Or sorry, from zero to not zero. Something. Anything else. Right, right. So, yeah, what I actually said was, because I decided to give my childish self some presence in what I actually said. So what it actually said was the immature thing to do would be to look at what AI does and to judge it by, well, people always judge AI or, no, people are tempted to judge AI by asking it to do something they're really good at and then comparing AI's output to what they can do and the thing they're really good at. Which is what your childish version did.
[16:11]Right, exactly. Exactly. Inevitably, the GPT will lose to the thing you're really good at, because it's not an expert at everything. It's good at everything. I think you used the word mediocre, which I think is a good thing. Mediocre is a good word for it, right?
[16:30]I think what I said was excellently, no, impressively, brilliantly, universally mediocre. And it's the universal bit that impresses me so much it's you know it's fine at everything, and i'm really good at one or two things and terrible at many many many many many many many many many things so when i judge chat gpt by the one or two things i'm really good at it's terrible when i judge chat gpt against the infinity of things i'm awful at it's brilliant it is way better at drawing than me it's way better at japanese than me it's way better at frankly english unfortunately than me so i think if you judge chat gpt by how it is it's something you're not good at and how it gets you from something you could never even start to something passable or at least something that gives you a starting point i think that's really impressive the fact The fact that it's able to do that for everyone, like the thing it's good for you is not the thing it's good for me, it's not the thing it's good for my mum, it's not the thing it's good for my boss, it's not the thing it's good for my brother.
[17:45]It's good for something for everyone. It's just a different something. And the fact that one tool can be able to help everyone for something is kind of impressive. Not kind of, genuinely. I'm glad you described it that way because what I'm good at is writing. I can construct really good sentences. I had 5,000 words a week for 20 years. I've gotten pretty good at it. And so when I look at people having it right for them, I'm like, well, that's stupid. Why would you ever have it do that? But that's because that's not their strength. So it's exactly what you're talking about here.
[18:20]This is going to pull back the curtain a little bit, and you'll hear this whole story later because I'll be so excited if I get this to work. But I'm working on this idea for something about cross-stitching. And what I wanted was to be able to draw on screen a rectangular box with kind of a squiggly shape in because a cross-stitch pattern is some amorphous. It's usually not a rectangle. It's some sort of shape. And so I thought, well, how am I going to write that on screen? And so I asked Chachi Padilla, I said, hey, draw me a squiggly shape on screen. And it just basically drew scribbles. And I said, okay, well, it needs to be an enclosed shape. It says, okay, so it did an enclosed shape and it made it blue. But the lines were crossing over each other. So I said, okay, don't do that. I want you to not have the lines go over each other. And it went, oh, okay. And by the way, when I described it, I said I wanted it to be a squiggly shape. The entire code, which is now quite lengthy because I've done a lot of iterations, is filled with comments and variable names that all use the word squiggly. So there's like squiggle factor. And it's like from 15 to 20, you can change this little thing. How squiggly do you want it to be? And all the comments have squiggly in it. And it's just, it's delightful. And so that's the blank page to, I've got something to start working with. And now I've had it draw another piece of it. And then I'll be able to start putting those pieces together. But I didn't have to figure out, how do I draw a squiggly shape? That would take me about a year and a half, Bart.
[19:40]Right. That is the ultimate empty page, because you don't actually know what pieces are available to you to put onto that page. So even if the only thing you end up keeping from this very, very first prototype of a prototype is the universe of pieces that exist, that you're probably going to have to spend some time learning. Like the ability to draw with JavaScript, there are a few different ways ChatGPT could have done it for you, but whichever one of them it picked, it's perfectly fine for you then to go and learn that one, because you are going to have to learn a bit more of it to be able to make it do your cross-stitch pattern. But you wouldn't have known where, if you'll excuse the analogy, what thread to pull. Yeah, yeah.
[20:21]Where? Through the universe of possibilities. That is what's great is because you can give it the context. Today we were fighting with our nano leaf shapes, trying to get them into HomeKit, and they were in. We had one scene in, and the other scene wouldn't come in, and we couldn't figure out how we did it the first time, so I just asked ChachiPT how to do it. And it was just like, oh, it's okay. Do this, this, this, this. If you tried to search for that on the web, it would take you years to get to the actual answer because you have to give it so much context that ChatGPT just jumps right into the context. You don't have to, oh, I got to put it in quotes so that it knows I mean to always include results from Nanoleaf, you know, things like that. So well it's it's the fact that there's memory right because you're refining the answer with every iteration instead of starting fresh which is what like a search engine is like current siri no memory no idea what happened before whereas with a chat in chat gpt you're refining refining refining or correcting you have gone down the complete garden path go right back up there Mr. ChatGPT or Mrs. ChatGPT and go down his other path instead. But it remembers. That's very powerful. I want to close this part because we have a whole other thing to talk about. But I found something that it can't draw. Cannot draw. A man either in the mountains or wearing a backpack who is clean shaven.
[21:44]It can't do it. It was so amazing how much it couldn't do it that I spent like 25 minutes trying to get it to do it, going, okay, no, I mean without a beard. No, no beard and no mustache. No, no, no, put him back in the mountains. No, no, he's not supposed to have... Put the baseball cap on. You had the baseball cap a minute ago. Put that back on. No, clean shaven. Nope, couldn't do it. Every single man... I tried another service and it would do it sometimes, but sometimes they were women it was it was very very odd but so the corpus of imagery of hiking men is full of people like me who are a long way from clean shaven yeah yeah apparently men can't shave can't uh hike without beards can't be done it's too cold i guess so all right let's switch gears because we got another really fun thing to to talk about i came across a journal article entitled large language models reduce public knowledge sharing on online Q&A platforms. I'm not really one for reading long-form research papers, but Dr. Gary is always sending me papers and saying, read this. I'm like going, no, it's more than like three paragraphs. I can't possibly do that. But I found this one from the U.S. National Academy of Sciences really fascinating. So, We know that the use of LLMs for programming tasks is on the rise, and I can personally attest to this. We've just been talking about it, and that the capabilities in this area have come from absorbing human knowledge on program-specific help sites like Stack Overflow.
[23:12]So at Stack Overflow, it's a place where you can post programming questions, and people of all skill levels will jump in and try to help you. And people's answers, and questions for that matter, get up and downvoted so you can tell which answer is probably the one you should follow. If you look at, you know, somebody's got 78 upvotes and some weird answer gets a minus three, you know, probably don't want to use that other one. So the research in this paper analyzed the post-activity on stack overflow from 2016 until the middle of 2023. And then they analyzed the changes seen since ChatGPT entered the scene at the end of 2022. So they have about a half a year where ChatGPT is around and 2016 to the end of 2022, that's six years without ChatGPT. What they found was that the number of weekly posts decreased at a rate of about 7,000 posts per year from 2016 to 2022. So this is pre-ChatGPT. It's been going down 7,000 posts per year, which is interesting in its own right, but we don't know why that is. But in the first six months after the release of ChatGPT, the weekly posting rate decreased to, well, yeah, the posting rate decreased by 20,000 posts. So 7,000 posts per year before GPT, and in six months, down 20,000. So that's extraordinary, but we all know that correlation is not causation. And if you haven't been there in a long time, it's always worth going to Tyler, I don't know how you pronounce his last name, Vizgen?
[24:36]V-Z-G-E-N. Anyway, he's got a website called Spurious Correlation Graphs, and you've probably seen it before, but he correlates things like the number of fire inspectors in Florida against solar power generated in Belize. And there's a complete and perfect correlation. Anyway, correlations, not causation. So.
[24:54]To try to determine whether ChatGPT was the cause of this dramatic change in the downward slope of posts on Stack Overflow, the researchers gathered data on similar sites in regions where access to ChatGPT is limited. They found that activity on Stack Overflow decreased by 25% relative to its Russian and Chinese counterparts, where access to ChatGPT is limited and to similar forms for mathematics where ChatGPT is less capable. So, you know, if ChatGPD isn't really making inroads yet on mathematics, which is so funny, it can't do math. That's hilarious. It really gets into... No, it makes perfect sense to me. Right, but not intuitive, but when you know how it works or you have some idea, it does make sense. And we can talk about that in a second. But so I linked to both of the graphs in the show notes if you want to see a visual representation of these results, because it's really dramatic. But the significance of this research is that by its very ability to consume the content where programmers are sharing knowledge and solving problems, it appears that ChatGPT is destroying the creation of that very content. So I think we're talking about a snake eating its own tail here thing. What do you think, Bart?
[26:05]Well, yes. So I have two pet theories about AI. I actually know I have a lot of pet theories about AI because it's fascinating. But my first pet theory is that people judge it wrong. And we've already talked about that in great detail. But my second pet theory, which is amazing that within the one week you hit both of my pet theories. But my second pet theory is that we are running out of acceleration. So way, way, way back, I posted in Security Bits a video from Tom Scott, who is an epic YouTuber for sciencey fun stuff. And he did a video explaining how to think about AI. He didn't have any answers. The video is a year and a half old. He didn't have any answers, and we still don't have the answer. But the question he asked was, where are we on the sigmoid curve of technology? Because all of our new technologies, they start slow, they speed up, and then they plateau. And everything, if you graph it out, takes that shape. It's like a stretched out S shape, you know, flat, flat, flat, way up, and then it flattens off again. So it sort of goes asymptotic at the top?
[27:10]Yeah, it's slow to start, and then it gets into really fast gear, and wow, and you think it's going to go on forever, but it doesn't. At some point, the curve kinks, and you go back to asymptotic, and it all plateaus off. And the question is, was ChatGPT3 the very, very start of the kick up, and it's going to go way, way, way, way, way, way, way, way up? Or was ChatGP3 the steepest part of the curve? And we didn't know a year and a half ago. ChatGPT4, you mean? Well, no, at the time, it was three, right? So at the time it was 3 and 3.5 a year and a half ago. Oh, okay. I thought you were talking about this study.
[27:49]No, no. Well, yeah. Well, I'm leading up to it. Okay. So the question he posed a year and a half ago was, where are we on the graph? And that's been in the back of my mind. That video has been in my mind for the last year and a half. And I'm still trying to figure out, where are we? And obviously ChadGPT3 wasn't the steepest part of the curve because 3.5 was, oh yeah, okay, big improvement. 4, yeah, still a big improvement. And 4.0, yeah, we're still, so we're, you know, we're still going up. But I think we're starting to see the first hints of the end of the up. And this study is one of the data points that leads me to think that we're about to kick in that curve. Because snake eating its own tail is a very interesting analogy because to keep making each version of the chat GPT better, it needs a lot more training, a lot more training. And we're running out of public internet. And not only are we running out of public internet, but we're stopping the creation of more internet.
[28:52]So we've hoovered up everything we have, and we've slowed down the creation of more. That can't continue, let alone accelerate. I see a couple of different things in this. In the research paper, they looked at this effect by language. And the newer languages are not going through this reduction in stack overflow comets. But the more mature ones like JavaScript and Python, those are slowing way down. So what if a new version, what is it, ES6 we're on, ES7 comes out or whatever, you know, a new version that we all have to learn, the answers aren't going to be in ChatGPT because it doesn't know it yet. So we will go back to Stack Overflow and generate it if we remember the URL by then and start generating stuff. Or if there's a new language, or there's languages today that you can't go search in ChatGPT. So people try ChatGPT, can't find it. They keep working in Stack Overflow. Eventually, ChatGPT, all AIs, eat that knowledge. And now we don't need to go to Stack Overflow anymore because of the maturity of the information. So maybe the experts answering dumb questions like mine when, hey, just go ask AI. Why are you bothering the experts here? We need the tough questions here that ChatGPD doesn't know yet. I mean, maybe that's just an evolutionary cycle.
[30:20]Right, but to me, that's part of flattening of the curve, because you're not accelerating anymore, you're achieving a steady state. You're coming to a balance where the rate of improvement of the GPTs is starting to come into some sort of a balance with the creation of new content. And the two are feeding into each other, because once all the obvious questions have been answered, the GPTs get to pick up again. But then that slows down the rate of new material for them to ingest, so they can't accelerate anymore. So they seem to be heading into some sort of a the growth can't go on exponentially but i think what what i'm suggesting and i've no facts to go with this but what i'm wondering is whether, what's it's by category so javascript yes it's it's let's say it's flattening out okay that one's done python yep that one's done but now we're gonna have ruby on rails or whatever the the new kids, the young kids today are using, or maybe it is math that gets figured out next, or maybe it's, it's Latin, you know, has already peaked out, you know, you've got that one figured out, but there's so many categories of knowledge that maybe all of these just keep rolling until they hit that flat curve and then something else gets, needs to get consumed.
[31:33]But that's a linear steady state sort of a growth. That's not an exponential. Like what we've had up to now has been exponential. It's not just like twice as good. Oh, because it hoovered up so much. Yeah. So the exponential can't keep going. I'm not saying it stops growing, but I'm saying it's like at the moment, the acceleration is mind boggling. It's not just accelerating. It's like the acceleration is accelerating. It's because everything really, really, really kicking off. Every question anybody's ever posted in Stack Overflow was eaten in one big jump. And now over the next six months, let's say it learns Ruby on Rails or whatever it is, you know, okay, that's going to be more linear in that category. Yeah, yeah, that makes sense.
[32:10]Yeah, that's kind of interesting. When you look at the history of all of AI, you tend to have these jumps. And at every point in the jump, someone thinks the jump is going to go on forever. And up until now, it hasn't had a societal impact because the jumps have been from absolutely useless in the real world to pretty useless in the real world to still quite useless in the real world. But with the Transformers, we finally landed on to actually useful in the real world. So now we're all paying attention to this jump. but since the 70s the joke has been like cold fusion that true ai is just a decade away and every time we find the new technology it grows exponentially with it yes finally it's here it's here and then it levels off and i think these transformers have given us a really big thing that we never had before with the gpts understand language deeply like they don't like they actually understand that last and lasted are the same word and they have a relationship to each other the same as post and posted. They don't understand anything. They can correctly use those in context because of predictive tech.
[33:21]They don't know anything. No, it's deeper. It's actually deeper because they actually represent language in a multidimensional space. And so if you take son and father, the relationship between those two words and their representation of language is the same as the relationship between daughter and mother. And so you literally do a math on the vector, and the same vector gets you from son to daughter as from mother to... Ah, I'm messing up my words here. But they don't know it. They don't know anything. But their representation is deep. Their representation of language is deep. Yeah, right, right. It's not just what's frequency analysis. Saying they know something is not ever really true, right? You use the word understand Okay, yeah, philosophical You use the word understand And they don't understand.
[34:11]They can represent. They have a deep representation. Yes. A deep representation. Yeah. And for the first time, that allows these large language models to deal with language at a human level. They're not worse at English than us. Whereas up until now, and all of the time I've been watching AI grow, language has flummoxed previous AIs. Completely flummoxed previous AIs. And the closest they got to being useful before was the previous big step forward was their ability to recognize.
[34:43]So the previous big AI jump was pattern matching, where suddenly the Photos app could say, this is you and this is your daughter and this is your brother. That was the previous jump was the ability to actually recognize things, pattern matching. And that got really, really good very, very quickly and then plateaued off. But they couldn't make stuff. You couldn't say, it could recognize a cat, but it couldn't draw a cat. It could recognize you were talking English, but it couldn't talk any English. And so this deeper understanding of language is a huge leap forward. So why is it... But it's not... Why is it difficult for it to have a deep understanding of math? And I want to tell the anecdote that I heard. The first time I ever knew that it couldn't do math was yesterday, and you clearly knew this before yesterday. But they were talking about ATP on Accidental Tech Podcast about how if you give it a word problem where it says, Bart walked three miles and Allison walked five miles, how many times more did Allison walk than Bart?
[35:45]And let's say it gets the right answer this time by some miracle. If you then change Allison's name to Alistair, it might get the answer wrong that time just because the name changed. So it doesn't understand math. And why can't it or why doesn't it yet do that? You acted like maybe you had knowledge on this.
[36:02]Well, it's because of what the transformer is. It's because of how the transformer works and because of how the transformer breaks symbols into a representation and then spits out an answer from that representation. That representation is really good at figuring out without being told. So you throw it half the internet and it builds a representation of the structure of language. But it has no concept of epistemology. It has no concept of relationships, truth, and so it can't tell at a deep level that if I take... So you're saying they're using language. So we're trying to use a large language model to solve math. That's the wrong model. Can't they make models that do math? I'm sure they can, but they haven't. Okay. That is a problem that is unsolved. That is... That one seems so much easier. It's so much more constrained. Math is...
[37:01]You say that, but it involves different kinds of insight. We can... If the universe expressed everything in pure logic, yes. The problem is the universe doesn't do that. And so to try... I'm not saying that it could develop equations for, you know, working on astrophysics. I'm saying that there's an awful lot of math that has already been written and figured out. You know, it should be able to do multiplication. It should be able to solve a word problem that you get in seventh grade. Well, it can do multiplication.
[37:37]Right, but we can do multiplication. We can even do stuff like equation solvers where they are getting pretty good, where you can give them a bunch of equations and tell them to make them like simultaneous equations and make them solve them. Those kind of things. Oh, it can. We haven't got, yeah, but they're different models, right? They're not large language models. And so someone finding a way to connect the pieces, like we figured out a really good carburetor and we figured out a really good crankshaft, but no one's quite figured out what goes between the carburetor and the crankshaft to make it all work together. So, that's kind of why AI is so fascinating, we get really good at figuring out, you could describe the pattern matching stuff from a decade ago as we developed really good eyes. So we gave AI eyes and they're pretty darn good. And now we've given AI the ability to assemble and disassemble language, but not the ability to understand the universe, which is why it hallucinates.
[38:35]If they're valid grammar, they're valid. The fact that it doesn't make sense to have a purple dishwasher, that does actually make sense. You know, what was it? Cold clouds sleep gently. Perfectly valid English. Nonsense, right? But they have no representation of why that might be nonsense. Why shouldn't those phonemes go together? So they don't have an epistological model. So someone has to invent that. I don't know what that word means. You keep using epistological? Epistemological is the study of truth So How do you decide what is and isn't true Is epistemology It's hard It's really bloody hard Bart, we don't want to get into politics now.
[39:18]It's a pretty good example of how it's hard It's hard and no one's figured that out So it's like we're trying to build a person And we have good eyes And we have the ability to speak But we don't have the ability to sense things We don't have the ability to understand certain types of things, you know there's so many more pieces and as we build the pieces we then getting each piece is already cool and powerful but then we have to figure out how to make all the pieces work together and that's not going to be easy either so there's a lot more there's a lot more giant leaps of ai to go yeah and i think the people who think that a large language model is going to get us straight from here to everything i've seen that before and it's like i love your optimism So I hope I'm wrong. I hope I'm wrong, but I don't think so. I want to end this on a really bad note. The final, one of the main conclusions of this article was something that escaped me until I read the article was that all of this knowledge that was open source, because everything in Stack Overflow was created by humans for humans, all open source. It's now, if we lose that, it's now in a closed source database you have to pay for, or not database, but a closed source system owned by OpenAI. That's where that knowledge is now.
[40:37]That is true. But if you watch what's happening in the open source language models, none of them are as gigantic as ChatGPT 4.0 because the open source community doesn't have the same resources yet. But the open source models tend to be more efficient, but with way smaller resources. And that's all out there too. So I wouldn't assume that today's reality is tomorrow's certainty. So what I think you're talking about with the more efficiency is what is true is if you write a large language model that has to solve every single problem, then that's really inefficient in terms of resources. But if you say, I want a large language model that only does programming, that's going to use a lot less resources. So that could be an advantage. As long as that data continues to exist...
[41:29]Look, I'm not saying we're definitely in a utopia, but I'm also not seeing the sense of doom that others are, because I am watching the open source world, and it's a lot less dramatic and a lot less shiny, but also evolving quickly. So I haven't lost hope. Okay. I may be Pollyanna here, and this may be my turn to be Pollyanna, but I wouldn't assume the worst immediately. All right. Well, hey, this was fun to jump on and get to noodle this out with you right after I've been having so much fun with it. I appreciate you coming on and just joining us for a spontaneous talk, Bart. This was fun. Well, it's great to be able to come on and just like spout about theoretical computer science. I don't get to do that very often. Yeah, I said, hey, Bart, you want to come on where you don't have to do any prep? Yes, I do. What's your answer?
[42:16]Low impact. All right. Well, I'm going to say it until next time. Let's happy computing, I guess, right? Amen. Well, that's going to wind us up for this week. did you know you can email me at allison at podfeed.com anytime you like? If you have a question or suggestion, just send it on over. Remember, everything good starts with podfeed.com. You can follow me on Mastodon at podfeed.com slash Mastodon. If you want to listen to the podcast on YouTube, you can go to podfeed.com slash YouTube. If you want to join the conversation, you can join our Slack community at podfeed.com slash Slack, where you can talk to me and all of the other lovely Nocila castaways who play around in there. You can support the show at podfeed.com slash Patreon, like Keith did this week, or with a one-time donation at podfew.com slash PayPal. And if you want to join in the fun of the live show, head on over to podfew.com slash live on Sunday nights at 5 p.m. Pacific time, where we might be talking about cars in the chat room while I'm talking, but you can join the friendly and enthusiastic Nocilla Castaways.
[43:12]Music.