Chit Chat Across the Pond
Adam Angst explores AI transcription tools, discussing accuracy challenges, 'Word Error Rate' (WER), and AI's impact on information retrieval and cultural understanding.
Automatic Shownotes
Chapters
0:09
Introduction to AI Transcription
0:38
The Challenge of Accuracy
3:32
Metrics for Comparison
6:51
Insights from Transcription Tools
8:12
Exploring AI Search Engines
13:50
Evolution of Search Technologies
20:55
Conversational AI and Information Gathering
24:34
The Role of Context in AI Responses
30:01
Cultural Technology and AI
34:43
The Stone Soup Analogy
40:36
The Impact of AI on Content Creation
47:27
Embracing Uncertainty in Technology
48:13
Supporting the Podcast
Long Summary
In this episode, I engage in a fascinating discussion with Adam Angst from Tidbits, focusing on the efficacy of various AI transcription tools. Our conversation begins with Adam sharing insights from his recent experiments comparing the accuracy of different AI-driven transcription methods. We delve into the complexities of determining what 'accuracy' truly means in this context, especially when contrasting official transcripts with AI-generated ones.
Adam recounts his initial enthusiasm when Apple introduced audio transcription features and how it prompted him to explore other options like Audio Hijack and Mac Whisper. We discuss the recording process he undertook, using clean audio samples from official Apple presentations and NPR’s podcasts to measure the efficiency of the transcription tools. However, the journey quickly became more intricate than anticipated as we confronted the ambiguity surrounding the term "accurate" in relation to transcription outputs.
Through our exploration, we uncover the challenges posed by variations in transcription from clean audio sources versus those with mixed dialects and interruptions. Adam discovered the importance of assessing the transcripts against the source audio itself, leading to surprising revelations about inaccuracies in official transcripts. We analyze the concept of 'Word Error Rate' (WER) and how it plays into evaluating the effectiveness of each tool, emphasizing the need for consistency in transcription metrics.
Our discussion takes a philosophical turn as we ponder the blurred lines between human-generated and AI-generated content. We examine the implications of AI tools in our everyday lives, particularly in the context of research and information retrieval. Adam reflects on the generative nature of AI and its transformative impact on how we seek and utilize information, comparing traditional keyword searches to more conversational, nuanced inquiries we can make with AI technologies.
Furthermore, we dive into distinctions between different AI systems, such as Perplexity, ChatGPT, and the Arc browser. Each has its unique take on searching and providing answers, often leading to different user experiences and outcomes. The nuanced differences in how these platforms synthesize information compel us to rethink the nature of searching in our digital age.
As we navigate through these themes, I share personal anecdotes about utilizing AI for practical problem-solving, like diagnosing issues with household appliances. The ability of these AI systems to handle nuanced queries is a significant leap from traditional search engines that might have constrained us to simple keyword prompts.
We conclude our conversation by addressing the cultural implications of AI technologies, framing them as tools for extracting and engaging with the vast sum of human knowledge. Adam references the concept of ‘cultural technology’ to describe how generative AI acts as a vessel for collective human understanding, enabling richer and more meaningful information exchanges. This leads us to consider what the future holds as we embrace these technologies, asking whether they will enhance our ability to seek truth in an increasingly complex digital landscape.
Join us to unpack these compelling ideas about AI, transcription accuracy, and the evolving nature of information retrieval in our lives.
Adam recounts his initial enthusiasm when Apple introduced audio transcription features and how it prompted him to explore other options like Audio Hijack and Mac Whisper. We discuss the recording process he undertook, using clean audio samples from official Apple presentations and NPR’s podcasts to measure the efficiency of the transcription tools. However, the journey quickly became more intricate than anticipated as we confronted the ambiguity surrounding the term "accurate" in relation to transcription outputs.
Through our exploration, we uncover the challenges posed by variations in transcription from clean audio sources versus those with mixed dialects and interruptions. Adam discovered the importance of assessing the transcripts against the source audio itself, leading to surprising revelations about inaccuracies in official transcripts. We analyze the concept of 'Word Error Rate' (WER) and how it plays into evaluating the effectiveness of each tool, emphasizing the need for consistency in transcription metrics.
Our discussion takes a philosophical turn as we ponder the blurred lines between human-generated and AI-generated content. We examine the implications of AI tools in our everyday lives, particularly in the context of research and information retrieval. Adam reflects on the generative nature of AI and its transformative impact on how we seek and utilize information, comparing traditional keyword searches to more conversational, nuanced inquiries we can make with AI technologies.
Furthermore, we dive into distinctions between different AI systems, such as Perplexity, ChatGPT, and the Arc browser. Each has its unique take on searching and providing answers, often leading to different user experiences and outcomes. The nuanced differences in how these platforms synthesize information compel us to rethink the nature of searching in our digital age.
As we navigate through these themes, I share personal anecdotes about utilizing AI for practical problem-solving, like diagnosing issues with household appliances. The ability of these AI systems to handle nuanced queries is a significant leap from traditional search engines that might have constrained us to simple keyword prompts.
We conclude our conversation by addressing the cultural implications of AI technologies, framing them as tools for extracting and engaging with the vast sum of human knowledge. Adam references the concept of ‘cultural technology’ to describe how generative AI acts as a vessel for collective human understanding, enabling richer and more meaningful information exchanges. This leads us to consider what the future holds as we embrace these technologies, asking whether they will enhance our ability to seek truth in an increasingly complex digital landscape.
Join us to unpack these compelling ideas about AI, transcription accuracy, and the evolving nature of information retrieval in our lives.
Brief Summary
In this episode, I chat with Adam Angst from Tidbits about the effectiveness of AI transcription tools. We discuss his experiments comparing various methods, the complexities of defining 'accuracy' in transcription, and the challenges posed by different audio sources. Adam highlights the importance of evaluating transcripts against their originals and introduces the concept of 'Word Error Rate' (WER). Our conversation also touches on the implications of AI in everyday information retrieval and the varied experiences offered by systems like Perplexity and ChatGPT. We conclude by reflecting on the cultural impact of AI technologies and their role in enhancing our understanding of knowledge.
Tags
AI transcription
Adam Angst
effectiveness
accuracy
audio sources
evaluating transcripts
Word Error Rate
information retrieval
Perplexity
ChatGPT
cultural impact
knowledge understanding
Transcript
[0:00]Music
[0:09]
Introduction to AI Transcription
[0:07]Well, it's that time of the week again. It's time for Chit Chat Across the Pond. This is episode number 811 for March 11th, 2025. And I'm your host, Alison Sheridan. Well, the delightful Adam Angst of Tidbits is back again with us this week. And he, in the last couple of weeks, has let me help him conduct some experiments, trying to find the accuracy levels of different AIs' ability to do transcription. And today we're going to start by exploring his findings, but I have a feeling we're going to wander off that path as well.
[0:38]
The Challenge of Accuracy
[0:35]Gee are you suggesting that i don't usually stay on topic but i.
[0:39]Love our non-linear conversation yeah
[0:41]Non-linear is the way to go man um so okay yeah so so what happened was i is when apple introduced the audio transcription feature in notes i thought well that's a great idea you know why wouldn't you want to record a presentation you were at or you were listening to or something like that and then get a transcript of it i mean i'm a journalist right like i need to check what people say um i not go on my memory and and so i did this and then and then like the rabbit hole just started he was like there's audio hijack it can do transcription too and oh but wait notes works on the mac and the iphone i wonder if those are different and and and then you told me about mac whisper and like well clearly i better check mac whisper too and so it's like kind of got this idea that i could figure out the accuracy level of one of these things this turned out to be, vastly harder than i'd anticipated because like what is accurate right um so what i've been recording was apple presentations you know because they're relatively clean like at first like the wwc you know and iphone releases they're clean right i mean those are scripted but there's no transcripts of them that you can just download so um but then you know like an Apple earnings call. You know, the first part is scripted, but then you get into all the back and forth of the analysts, and it just goes off the rails. So it's.
[2:09]So it all worked, but how do you check how accurate it is? And then I remembered that NPR does transcripts for all their shows, at least some of their shows. And so I found a shortwave podcast and I had transcripts and everything like that. And I recorded it. And of course, I recorded it in multiple ways because if you want to use notes, you actually have to literally record sound coming out of a speaker. Whereas Audio Hijack can just suck in the digital sound without actually ever playing it. Mac whisper needs files. So, so I like record it in all these ways. And then I transcribed it. And then again, like started trying to figure out how do you compare? And I, I actually started with chat GPT, which gave me some answers that were like, they looked plausible at first, but every time I asked in a slightly different way, I got wildly different answers.
[3:03]Oh, you were asking it, which one of these two is more accurate?
[3:06]Yeah, because I had the official one.
[3:07]So did you give it the audio file too?
[3:08]No, no, no. I just gave it the transcripts. So I said, here's the official transcript, and here's five others. Compare them on how many words are different, how the punctuation is different, capitalization, that kind of stuff.
[3:20]Okay. Trying to give it some sort of metrics?
[3:23]Yeah, yeah. And eventually, it said, oh, you know, it was calculating word error rate. And it turns out word error rate is a thing.
[3:32]
Metrics for Comparison
[3:32]So this is what I didn't know. So, it turns out where I write is an equation of, let's see, it's the substituted words, the missing words, and the inserted words divided by the total number of words in the official version.
[3:48]Okay.
[3:49]So, it doesn't worry about punctuation, doesn't worry about capitalization, things like that. Good, good. And so, that helped a lot because then I had something I could tell it to do, which it sort of knew how to do, and then it became consistent. Okay. And it also, I learned that there were calculators that you could just paste into two transcripts and it would calculate this for you.
[4:10]The W-E-R? Yeah.
[4:13]Yeah, the word error rate? None of them agreed. They're driving me up the wall. You know, they were ballpark, and they sort of went in the same order, you know, like this one was more accurate than that one and things like that, but they never agreed on the numbers just about.
[4:30]Let me ask, is it a percentage?
[4:32]Yes.
[4:33]So how close were they in percentage? I mean, a word error rate would be like 5%, 3%. It'd be a small number, hopefully.
[4:40]Yeah, they were all within, you know, 1% to 4% difference.
[4:46]Okay. But you wanted scientific evidence.
[4:49]It just seemed a little fuzzy to me. So I'm like, come on. And what I realized eventually was this was a fool's errand. And I was the fool. Always get it while she's drinking.
[5:04]I was drinking coffee right when he said that. It did not come out of my nose, but close.
[5:11]So, the problem was, I actually did an error check on the official transcript. So, I played the audio and read along with it. Luckily, it was only like 12 minutes or something, so I was able to do this.
[5:27]I was going to say, that's really the only metric that you could do that you could trust is really, I'm reading it, I'm looking at it, I can tell what it's saying, right?
[5:35]And you know what? There were at least four errors in the official transcript. Oh. They were missing words. And those were errors that were true errors. There were other ones which I realized suddenly, well, now there's decisions you have to make. So if I say, just right there, I said, so, so if I make.
[5:57]Right.
[5:58]I repeated the word so. Right. Should a transcript include that?
[6:02]If you want to be technically accurate, yes. If you want it to be good for the reader, definitely not. No.
[6:09]And so, yeah. So it turns out that Notes, both on Mac and iOS, is pretty verbatim. If you repeat a word, it will pick it up twice. All the Audio Hijack and Mac Whisper, which actually both use Whisper behind the scenes, they're like you don't really need to see this.
[6:33]I've actually really appreciated that in the interviews that I do at CES I am not very articulate in those interviews I'm often repeating something or I'm going and it's just not there in the transcript you
[6:48]Sound great I've listened to some of those.
[6:51]
Insights from Transcription Tools
[6:51]I listen to them and I'm just like oh man
[6:54]So yeah so it turns out And I went in thinking this was a computer problem. You know, like one of those, you know, you plug in the numbers and it spits out an answer and you're done.
[7:07]You can do a graph. You can do math.
[7:09]You could do – right. You know, like there should be a number.
[7:13]Sure.
[7:14]And what I came to realize – and I mean there's still some useful information that came out of it. You know, that I did get this nice chart and a table of all the things and was able to come to some conclusions. Mac Whisper is probably the best sort of unsurprisingly and that's what it does, and interestingly notes in iOS and notes on macOS are different macOS does better really yeah don't know why maybe it's same recording locally.
[7:45]Maybe it's running locally and it's RAM-based and that sort of thing.
[7:50]I don't know. Who knows? Very interesting. So, you know, that's on an iPhone 16 Pro versus an M1 MacBook Air. So maybe even a different Mac might make a difference. I don't know.
[7:59]I asked you about that when I did it because I was running it on my M3 MacBook Pro and on my M2 MacBook Air. And I was asking you whether you thought it would be different. And we didn't think it would be, but maybe.
[8:12]
Exploring AI Search Engines
[8:13]Again, I just don't know. And that's sort of what this is coming down to, is that we're seeing that more and more of the world's problems, in some sense, don't have fixed, easy answers. We've moved beyond fixed, easy answers. Those are all – we've solved those. This is computers, though.
[8:36]It's got to be exact.
[8:37]It's got to be exact. No, no, it doesn't. And the fact that there's these decisions that you have to make is part of why it's not exact. Even the two calculators where you just pasted it in your text and they claimed to calculate word error rate, they didn't agree. So clearly they made some different decisions on how to do this. And again, what is right? I don't know. You know, they all agreed that when four errors were just exactly four errors, that they had the same. They all agreed on that. But as soon as you got into, like, actual real-world situations, it was much trickier. And, of course, that is professional audio from NPR. Like, it's as good audio as you're going to get. It's an official transcript, which I believe was created by a person, at least to some extent, because the missing words were not missed by any of the AIs.
[9:34]Oh. And these weren't just uhs and ums.
[9:38]They were actual words missing? Yeah, this was actually—they were talking about something being an annual exercise. And I forget it was both annual and exercise, or just one, or just annual. But basically, that word just disappeared from the official transcript. Okay, which looks like human error now. And it was very clear. Right, it was very clear in the audio. there was no reason for it to have been missed. So there were a few things like that.
[10:06]You do realize how scary this is. Now you can tell the error, you can tell it's human because it made a mistake.
[10:15]So yeah, so it just became this really interesting kind of philosophical issue of like, well, what is the truth in some sense? What is the answer? And and you know I've been thinking about this more and more because I've been using basically like the last year and few months I've been using Perplexity, Arc Search which has this AI driven browse for me and now ChatGPT to do searches, and so I've been I've been I've been looking more into this like into the search space, and not not so much in an official like a comparative way We're like, oh, let me run the search across all the four of them and see what happens. But just like, is it solving my problem? Let me ask you a question.
[11:10]Let me ask you a question. I don't know what people mean when they say using it as search. How is it different than giving it a prompt? It's like the same thing.
[11:21]Well, so perplexity is just a search engine, right? I mean, or 99% of, oh, okay. So perplexity is a search engine. And so when you put a prompt into perplexity, it's like typing a search into Google. It will do a search based on your prompt. Then it will collect all the results and it will use them as context to answer the question in your prompt.
[11:46]How is that not just doing a prompt? That sounds like the same thing to me.
[11:51]Oh, no, no, no. Because imagine ChatGPT before it had access to the web, right? Perplexity is literally doing a search in the background, whereas ChatGPT before it had access to the web, all it could do is base its answer on its training model.
[12:07]Okay. I didn't actually realize that had changed. It's not just using its training data. So it can answer current events now?
[12:15]Well, ChatGPT is doing full web searches when you ask. Perplexity always does them or almost always does them. You can ask Perplexity to do certain things where it will just be generative AI because there's sort of no search involved. Perplexity will do just write me a limerick kind of thing. But ChatGPT in the past was just completely on the other side. They would only do generative AI, and only recently in November, I guess, did they open it up to search.
[12:51]Okay, that explains some of the results I've been getting. I didn't notice a big shift, but now they think about it.
[12:57]You should be able to see it. It should say searching the web when you type in certain kind of prompts in ChatGPT, and then it will give you sources, because both of them will give you sources to what they found and incorporated into their answer.
[13:14]That's interesting. I know when I used to do it from the web interface, I used to see that, but in the app, I don't.
[13:23]I'm not sure. I'm not using the app primarily. I'm not even sure I have it downloaded in my other machine. It only works in Apple Silicon, so I can't run it on my 27-inch iMac. So I'm just using ChatGPT in the browser and an Arc tab. Okay. All right. So, and then just the third one is Arc Search,
[13:50]
Evolution of Search Technologies
[13:47]which is the iOS version of Arc. So, all my tabs are synced at all times so that I can just tap, tap, tap.
[13:57]So, Arc is a browser.
[13:59]Arc is a browser. But they added this feature called Browse for Me, which is you type in your search or they've got a very nice way to talk to it. And it does exactly what the others are doing whereas it goes out and does a search runs your prompt against the context of that search and then gives you answers with resource links.
[14:24]So when I go to just to my URL bar in Safari with my default set as Google for search and I type in a term and it does that Gemini thing at the beginning is that the same sort of search that you're talking about?
[14:38]Honestly, not sure. I've given up on Google so long ago that I'm not really up on what they're doing these days. Brave Search, which I used until I switched to perplexity for a couple of years before that, Brave Search does an AI summary, but it's so short that it's often not useful.
[15:01]Okay. How are you doing, let me guess, in Arc, the Arc web browser, does it allow you to choose perplexity as one of the options for your search? Is that how you're using perplexity?
[15:14]I'm trying to. So I'm currently using ChatGPT. And I'm hesitating because I can't remember whether it lets you choose or if I, like, coded it in slightly. Because it's a Chromium browser, you can set up your own search engines very easily. And I can't remember if I did that. And I think with ChatGPT, I had to have an extension, a Chrome extension.
[15:39]Okay, just because in Safari, my only choices are Google, Yahoo, Bing, DuckDuckGo, and Ecosia, whatever that is. But I don't think I can choose— Oh,
[15:46]Don't use Ecosia. Horrible, horrible.
[15:48]But I can't choose perplexity.
[15:51]No, no. You're limited to what Apple will let you do. Yes. And so, that's one of the reasons I don't use Safari. But what's interesting about these is I feel like there's a bit of a sea change happening. So, I mean, do you remember using Archie and Veronica?
[16:16]No. I remember the comic book when I was a little kid.
[16:20]Yeah. So, Archie was the search engine for FTP sites.
[16:25]Oh, wow.
[16:26]In the early 90s. Veronica was the search engine for gopher sites. And then the web comes along, and then we get AltaVista and Yahoo, and then eventually Google. And, but what was sort of interesting about the jump from, like, you had this sort of interesting jump where, like, FTP sites you had to, you could just navigate, like, just it was literally a file system. And, you know, you just go into folders, and there'd be more folders and documents there. TurboGopher, or Gopher, and then TurboGopher was the client from the University of Minnesota. Sort of made that more fluid in the sense that you could have, you could be navigating through this gopher space, which would have documents and whatnot. But it was still very much this hierarchical list, and you kind of go through. And then, of course, the web pops up, and suddenly we've got full text, right? And with links and everything. And so that was the evolution. And this feels like it's sort of the next thing, where instead of finding a single page with text on it, it's finding a bunch of pages and collating their information in a way that it can then go get to the answer so you don't have to go and look at each page to see if it actually answers your question.
[17:45]Okay.
[17:46]Because, I mean, that was the big win of Google, right? PageRank, the idea of PageRank was that it was, you were likely to get in the first couple of results the answer to your question. Right.
[18:00]That seems to have gone away.
[18:03]Part of the reason why I don't use Google anymore. Yeah, it has gone away. And part of it is also that I haven't thought this through, but I'm just saying this out loud. I wonder if keywords are no longer enough. In other words, we've been trained to do keyword searches. And so a keyword search, you know, Google's going to go like, ah, well, that's a keyword, and that's a keyword, and that's a keyword. Here's the pages that have those keywords on them and that do all my other things in terms of being useful and blah, blah, blah, blah, blah, so that they meet the page rank. And whereas what I find with the more AI-driven search engines is it's much more of a conversation because it's more like a chatbot, right? Sure. So you don't usually give it just keywords. You usually say what it is you're trying to accomplish.
[19:02]And that makes a difference in how it's going to answer.
[19:05]And that makes a big difference because, of course, it's generative AI. So it's going to be doing the next token lookup things. And the more you give it, the more those can be more statistically appropriate to actually meet the needs of the question.
[19:26]Right. So, you're saying this feels like a sea change to you. In what way?
[19:33]Well, because, in fact, and I understand the various issues underlying this, but the fact is, is when you do a search, you're trying to answer a question most of the time. I mean, there are searches that are no question, like our navigation. I just want to go to this website. I don't know what its URL is. So take me there. And then there are simple, very simple fact searches. You know, what is the world record in the mile? You know, 343.13. But then there are a lot of things where the search is really starting something. So, for instance, I am looking for information about this particular model of GE refrigerator because I have one and it's not working. In this particular way. Tell me what might be wrong.
[20:30]Different. Yeah, that's really different. That's not something, that's got to explore a lot to get to that answer.
[20:36]It's got to explore a lot, and then it tells me what it thinks might be wrong, and I'm like, yeah, I don't think that's it, because the freezer's still working. The bottom freezer's still working. So clearly the compressor's not gone.
[20:55]
Conversational AI and Information Gathering
[20:48]Oh, okay, then let Let me adjust my answer to focus in on the other things that could be wrong. And I'm not saying that it's necessarily going to get the answer right, but it's a more exploratory and participatory approach to information gathering.
[21:12]It's also like dealing with somebody who's pretty smart, but also has zero ego. So when you tell them, yeah, no, this model doesn't have that kind of compressor, it goes, oh, yeah, you're right. I'm sorry. I'm sorry. Let me approach this from a different way.
[21:30]Unlike Siri, which never apologizes.
[21:33]Yeah, that's really what it is. We just want her to apologize. I find that interesting because well yesterday i was playing with uh i was using claude to help me with some coding and i told him what was going wrong and it said oh here or no i said i wanted to do a certain thing and it said okay here's how you could do that and i said yeah that didn't work and it goes okay i understand well how about if you try this and i did it again and then it goes yeah clearly none of these paths are going to work let's take a whole new approach and it wasn't you It wasn't upset and saying, well, why didn't it work? I don't know, it just came back and said, let's take another angle on this. You could hard code this thing, and here's how you could do it.
[22:10]Infinitely patient, yep, infinitely patient. And it's also, again.
[22:18]It's very important that these things provide sources because they – I hate the word hallucinant. I think it's a stupid word. I do too. They make mistakes.
[22:29]They make mistakes. And partly they can make mistakes because people make mistakes. Sometimes their data is wrong. One of the things that drove me off of Brave Search to begin with is not that Brave Search was doing a bad job, but just that the stuff I was getting was so stupid. Like the results like the actual pages i was reading i'm like well that's dumb you know like i was i was doing things that are in my field right like they're you know like i was talking about you know apple stuff and iphone and whatnot and so i was usually looking for confirmation, or the nudge of the one thing i'd forgotten those kinds of things so like i had a pretty good sense of what quality was and i was not finding it you know that the stuff that was popping up in the search engines. So it wasn't like the originals, the sources were necessarily good, which can result in bad AI responses. But I still want to be able to check them to see, because sometimes, again, AI just got some bad data somewhere and put it together in a stupid way, and now you've got some really incorrect stuff and you need to check that. But most of the time, I don't care that much, right you know i'm not you know when i'm when i'm asking about what might be wrong with my refrigerator i'm basically trying to determine is this something i can fix myself or do i need to call a repair person right away right.
[23:52]And the chances that it's going to be super wrong and that is probably well it doesn't know your skill set i mean if you tell it that you have a sponge and a piece of electrical tape and it's going to go yeah no you can't do this
[24:04]But in but in fact you can tell it your skill set and it will take that into account don't, as opposed to having to watch a YouTube video and get 15 minutes in to determine that, no, the guy's not going to talk about your particular problem. So, you know, I said, I mean, the people who do YouTube videos on repair stuff, that is actually fabulous. But a lot of the time, it is not actually what you need because it's just a slightly different model or, you
[24:34]
The Role of Context in AI Responses
[24:32]know, your particular problem is enough different, et cetera, et cetera. So there are things like that where – but with the chat pots, the AI search engines, you can literally do things like, I'm having trouble taking this thing apart. Are there any tricks?
[24:54]Hmm.
[24:55]And because it has the context of what you've been searching for all the way back, it can say, oh, well, you know, yes, these kind of connectors, you know, can be a problem and, you know, try doing X, Y, or Z. And again, sometimes like as someone, I do fix a fair amount of things. I'm not great at it. I'm usually successful, but it takes me a while.
[25:20]Successful and great being two different things.
[25:23]Yeah, meaning great, meaning it would just like, oh, you do this, click, click, click, click, and you're done. The success was like an hour later, ah, I finally beat it into submission. But a lot of times it really is the trick. Like it's that experience of, yeah, yeah, here's how you get it off. You know, like if you have to spend 15 minutes trying to figure out how to get one little piece off, then you feel you wasted a lot of time, whereas something can just tell you, oh yeah, there's a little tab underneath you gotta push. Thank you, you know, couldn't see the tab.
[25:51]One of my favorite anecdotes is that I had somebody snapped off the antenna on my old 76 Honda Civic, and I went down to the store and bought another one, and I went under the console, and I unplugged the connector, and I unscrewed it from the roof, and I pulled the cable all the way out before I thought, boy, I bet my dad would have tied something to that before he pulled it out. And I called him and I told him that. He said, Allison, how many times do you think I forgot to do that before I learned to always do that? You know, I've never forgotten that again.
[26:25]That's really true. You know, I grew up on a farm. So, yeah, so I have a fair amount of those kind of skills. But as I said, particularly working with small stuff you can't see real well, it's just, you know, you just like, oh, just tell me the trick. And in fact, so just to go with the story, the refrigerator did die on Monday. The guy actually was able to come on Tuesday, which was great. It was luckily it was cold enough to move everything outside and not have a problem because we live. We can't do that. We don't live where you do that. We don't live in sunny Southern California. And then, but he actually determined that it was the control board that was the problem. And I watched him take it off, including these little clips that you had to like use pliers on. I'm like, oh, that would have taken me, I don't know how long, to not realize those little clips.
[27:14]A lot of times I think what we really want to know is, should I just pull harder or is that going to break it?
[27:19]Yes, yes. Because I, oh yeah, precisely. So in any event, so he eventually, you know, like he figures out the control board. He's looking it up on his phone and having a little trouble finding it. So I look it up on my phone in Repair Clinic and I find the thing. And he's like, well, you know, it'll be, you know, $380, you know, for the part and the service visit. We can probably get it in three or four days. And, you know, I can probably be back another couple days after that. And I'm like, a week without a fridge is not going to be fun. Looks like Repair Clinic can send it to me tomorrow. And i can put it in myself because i have watched you with a little clippy things so right right so by wednesday we had a working fridge again and.
[27:59]So you had gotten to that point with using a an ai search agent
[28:04]In that in that case not uh i'd gotten to the point of calling him um because, yeah well no because i was like okay the the various things that it's talking about are things I can't just solve with Repair Clinic. And it had mentioned the control board, but I was like, control board? Like, I don't do the electronic stuff. I mean, I'm good at plugging and playing, but if it's actually a dead control board, I don't know how to figure that out. And to be fair, like this guy, I mean, the repair guy was great. I mean, like he was looking up wiring diagrams and stuff like that. And he showed me on the back of the control board, like there were two little dark spots at solder joints where things had probably gotten too hot at some point. And it's like, okay. And, and, and, And the part that really pissed me off, I had power cycled the fridge multiple times at this point, right? Because that's what you do. Because it's always just unplug it and plug it back in, right? Right. And so he unplugs the board, all the connectors from the control board. And he's looking at it in the back. And he puts it back together. And he plugs them all in. And the fridge starts working. I'm like, seriously? No. And there were no capacitors on it anywhere. There was no way it could have stored anything. stored anything i was like i have had this thing sitting unplugged for a day and when we plugged it in you know to show him it was still broken so so any event so i still got the new control board and all that because i didn't trust it but uh but yeah so but so but but it did it was it was very helpful um i didn't trust the old control board yeah.
[29:32]You said i didn't trust him i wanted to make
[29:34]Sure he was good yeah no i did you know i trusted him but i didn't trust the old control board to keep working. But again, getting to that point was the tricky part because I was like, well, it has these symptoms. I can only tell you what the symptoms are. And that was a useful, and that was a discussion. It was not a keyword search.
[30:01]
Cultural Technology and AI
[30:02]Right. And the thing is that the ability to go back and forth with it, like you say, you can go to Google search and search for a term, and it'll spit you out some answers. And then you want to ask the follow on question, and you're starting from scratch again. It's sort of like talking to Siri. So, but with these chat clients, you can just say, okay, it assumes it knows all this information. I talked about on my show that I was asking a question and it says, well, you know, you could use TextExpander for that. And I see you've got it on your system. And I was like, what? How do you know that? And it says, because you told me. And it says, go back and look in your search history. And sure enough, I had said, I'm trying to make a TextExpander snippet that does this. And it basically keeps an inventory of everything you've ever told it.
[30:49]Well, in fact, I mean, one of the things as I'm starting to write about this, I'm actually going back and looking. Both Perplexity and ChatGPT keep sort of a library of your past searches, which is kind of interesting because, again, you might pick them up. Like I've gone back and continued a conversation because sometimes I don't want to start from scratch. I'd already had some discussion about something. I've gotten somewhere. And now I want to pick that up and go further. So that was just, it's kind of an interesting thing. The other thing that perplexity does, which actually I find, I find, I've been finding more and more helpful, is it suggests like four or five prompts to continue. You know, like it's helping you continue the conversation.
[31:36]Oh, yeah. It'll say, did you want to know a little bit more about this, this thing I just told you?
[31:41]Yeah. And so I've been finding that more and more that's actually been useful for certain kinds of like open-ended things where I'm like, tell me about this. I don't really know very much about this. And it will tell me something and then it will make some suggestions later on that actually are not bad.
[31:59]Now, in your musings as you're working on this topic, you were starting to talk about this as a cultural technology. Can you expand on what you mean by that?
[32:10]Yeah, so there's a woman named Allison Gopnik came up with this term. And she's arguing that generative AI, in essence, because it's been trained on everything it can possibly find, is a cultural technology in the sense that we are using it to extract information, extract specific bits and pieces of human knowledge. Right? I mean, in that sense, it's sort of like Wikipedia.
[32:44]I mean, Wikipedia is a large collection of human knowledge. And when we go and read an article about something in Wikipedia, we're extracting that information from the sum total of human knowledge that's been recorded in Wikipedia. And it was an interesting way of thinking about it because it gets away from a little bit of the intelligent agent concept, which I've had more trouble with. I mean, I haven't yet seen any of the agent-y stuff where it's like doing something that feels like it may even make sense to me, you know, that, you know, like, why would you have something do this? You know, like, they've given examples of like, book me plane tickets. I'm like, I barely trust myself to book plane tickets. I'm not going to trust some random, you know, like, every time I book a plane flight, I'm like, well, let's say if there's a cheaper one. What about this? What if I do these other hours?
[33:45]That's the wrong seat. That's a stop over to a weird place. That's too close.
[33:50]How could it possibly know those things about me when I barely know them about them myself? And I don't even know them until I'm presented with something. I'm like, oh, that doesn't seem like a good time. It feels problematic to me in that regard. Whereas this way of plumbing human knowledge. I mean, what's Google, if not, you know, this grand sum of human knowledge that you can search for bits and pieces of? Well, now we're not searching for the bits and pieces, we're asking questions of it, which is sort of the same thing with a jump.
[34:32]I see what you're saying. Because we're having a conversation with it,
[34:43]
The Stone Soup Analogy
[34:36]not giving it, You know, Audacity envelope tool version 3.7.1.
[34:43]We're asking, I.
[34:44]Want to change the volume of a track and I want to put control points in it. And I'm using Audacity. How do I do that?
[34:53]I was just helping a friend, controlling his Mac remotely, and he'd been doing something where he'd been receiving spam that triggered more spam to a forum, a web forum that I create, that I manage. And I'm like, how are you doing this? And so I finally, I mean, I sort of figured it out. I mean, I knew roughly what was going on, but he uses Apple Mail, which I don't. And so I'm sort of on the spot. We're literally talking, and I'm controlling his computer. And I'm like, man, how do you find the freaking raw source of an email message in Apple Mail? And so I just asked ChatCPT. And it told me. And I'm like, okay, thank you. Now I know. Yes, I could have done a search in Google and probably found three or four things that might have told me if I went and read them. But I just needed to know what the menu structure was to get to that command. And there were a couple of other little things like that. Because I'm not as familiar with Apple Mail as I could be because it's not the app I use. And so to be able to get to that information, again, I wanted answers. I didn't want references.
[36:11]Yeah, right, right, right, right.
[36:13]Right?
[36:15]I mean, it's doing the same thing. It's going off and reading and finding all of these references, like you say, but then it's telling us the answer in the way we need to hear it, not the way it was typed into a web page the day the person wrote it.
[36:30]And keep in mind, the answer may come from a web page that has almost nothing to do with what you're interested in.
[36:39]Or it's wiki how. You can't see Adam but his head is exploding as I said that I mean I just I just hate it they always come up to the top too that's the thing that drives me crazy and it's literally never the one I want to read precisely
[36:57]So yeah so so that's just it is is that like particularly when you're when you're searching for something I mean that's in some ways a sort of bad example because that probably would have come up relatively quickly in a search. But there's sometimes you're looking for something which no one's written an article about this. It's not that interesting. It's some little piece that you're going to find referenced in something else. I mean, I do this a lot in tidbits. I don't know if anyone notices or appreciates or if the search engines notice or appreciate or the chatbots notice or appreciate. I like to try to put additional information into articles. So, like, the answer, I mean, like, I'm writing about something, but I will mention other things on the way to getting there so that those are recorded in a place that they can be found. And, or just kind of... Absorbed by the reader. Reader's going, oh, they have an article about such and such and such and such. And by the time they're done, they also know these three or four other little facts, which weren't absolutely necessary to know to get to the end, but they're there now.
[38:03]I think about this a lot because when I go to a YouTube video to see how to do X, and I'm taking that link, all I want is that. When I go to a recipe, all I want is the recipe. But when I write, I'm the person who's going to tell you the story of how I got there. I am the exact opposite of the person whose site I would ever want to look for. You know, I've started teasing myself in my own article saying, so anybody else would say, click here, here, here, there's the answer to your question. But that's not what I'm going to do today. Let's start back in 1958 when I was born. And I'm going to tell you how I got to this point and told this story today. But I do feel like giving that context can be important to helping people understand why the answer is what it is.
[38:51]It's a fine line because, right, you don't want to go back to 1958, but you do want to provide some context. And the food blogs are actually a good example of that. So, in the last couple of years, I've become a huge fan of Deb Perlman and Smitten Kitchen. Sorry, I cannot get that right. My wife and I made that mistake very early on, calling it Smitten Kitten, and now we can't stop.
[39:19]We do stuff like that. We always talk about Avocado's number, instead of Avogadro's number.
[39:27]Yes, precisely. So, make a great guacamole for Smitten Kitchen with that. But any event, she does a wonderful job of having, a small amount of a lead-up or verbiage before the recipe, and so that you will read it, and it's amusing and entertaining and well-written, and again, additional facts show up in there. Whereas most of the time when I go to a food blog, I am mashing that jump to recipe button as fast as I can, because I do not want to hear what hubby and the kids think of it. you know so i.
[40:05]Open up the app paprika and i tell it to import the recipe and i get that immediately
[40:11]Well i don't i don't i don't get i don't i don't let paprika import anything until i've gotten to make sure the recipe is okay so that's the problem um so yeah so that's part of it but but anyway yes so i think it is a fine line but but but this is a little bit of of kind of again what we're looking for i mean someone was complaining to me recently or committing a tidbit to talk about how So they did a search for, I think it was like HP printer help.
[40:36]
The Impact of AI on Content Creation
[40:37]And then they called the phone number that Google got, and it was a scammer.
[40:42]Oh, no.
[40:43]And I'm like, that's really, I mean, that's a Google fail. Like, how could Google possibly not put up an official phone number? And I did the search, and I got official phone numbers. But it just goes to show there is no, again, there's no real answer anymore. Everything's a little fuzzy. it's personalized and customized and a little different so who knows what this person had searched for in the past such that Google thought differently about giving them a scammer rather than the right answer, Oh, he really likes the scammers. Okay, then.
[41:18]When you were talking, we were texting back and forth or emailing, when you were talking about this idea of a cultural technology, you quoted the woman you just referred to, Alison Gopnik from UC Berkeley. And she told the story of the stone soup. And that really struck me as a way to explain what you're talking about, about a cultural technology with AI. Can you just talk through what she wrote? And I'll put a link in the show notes to her article.
[41:44]A little bit. I actually didn't, I mean, the stone soup analogy didn't quite work as well for me. But basically the idea was, I mean, the stone soup fable is, you know, people come to a, you know, a traveler comes to a village and he's like, hey, you know, let's make some soup, you know. And people are like, oh, we don't have anything, you know. And he's like, well, I've got these magic stones. And, you know, he puts the stones in a pot with some water. And, oh, you know, boy, this would be great if we just had, you know, a carrot or two. And we're like, oh, I got a carrot, you know. And by the end, people have provided all of the parts to actually make a really good soup. And I think what she's suggesting is that we are adding all the little bits and pieces to make generative AI actually useful. We're turning it into soup by providing all of the extra little bits. Oh, wouldn't it be nice if it knew about such and such? Wouldn't it be great if it was trained on that? You know, I'm going to tell it about my issue over here. And just everything sort of goes into the pot. So what you end up with is more valuable than what you put into parts and pieces.
[42:55]I like that perspective. I'm also choosing to use Claude because it's not training on what I'm asking it.
[43:03]That's usually an option for most of them at this point.
[43:06]Well, it's opt-out on some of them. Like on ChatGPT, it's opt-out, but on Claude, it's opt-in. And so I'm choosing not to train it on my own data, but I probably could because asking stupid questions about how to write something in jQuery is probably not a national secret I need to protect.
[43:25]It is an interesting question. And, you know, like I've been using this Lex word processor, which is kind of an, you know, it's got a chatbot attached to it. You can ask it questions about what you're writing at all times. And you can feed it context. You know, it's like, say, you know, go to these, you know, include as context these web pages. And, you know, in this PDF I've uploaded and things like that. and, you know and it's not doing a sharing stuff but someone was like oh aren't you worried about putting what you're writing you know into a chatbot where you can't control and I'm like everything I write is for public I mean like literally published you know so you know no I'm not in the slightest bit concerned about that you know if I was writing you know my journal, you know that I didn't want anyone to read ever yeah I probably wouldn't put it into even an online word processor, but you know you did have to you have to figure out whether or not you actually care about some of these things because it is a little bit about providing the carrot for the soup.
[44:22]Right. Yeah. One of the things I was thinking about with this is there was a study done recently looking at the level of drop-off in information being provided to Stack Overflow over the course of time and mapping when AI started to kick in. And so there has been a steady decline in the number of people participating in the information that's being added to Stack Overflow. It's been at a fairly constant rate going down. But the minute ChatGPT hits the market, it just plummets. And they did a good job of doing a comparison study in other countries where ChatGPT is not available and the drop-off did not happen at that point. So they're fairly confident of the data that they had. But part of it was like, well, wait a minute. Does that mean we won't have new information if we stop contributing? But what they found was that this drop-off was in languages that are fully established.
[45:22]And so they said, if you look back at JavaScript, it's been around a long time. HTML's pretty much figured out. People are just asking the same questions over and over again, and people are answering the same thing. or saying, oh, you should go to this answer. It's already been answered. Oh, go to this answer. It's always been answered. But if you look at some of the newer languages, they aren't seeing that kind of drop off. So that gave me a little bit of confidence that maybe what we're doing right now is I'm writing an article about how to turn on some accessibility feature in system settings on Mac OS, but you've already written it up. And, you know, Mac Rumors or somebody, you know, one of the other sites has already written it up and it's probably written up 100,000 different places maybe that's not what we need to be doing anymore because that information's no longer, you could just ask an AI about it because the world knowledge exists, shut up, talk about something new.
[46:17]Well, we can always hope. I mean, the cultural well is deep and is getting deeper, but it is definitely a situation where, you know, pretty much anything you think of writing about has been written about already. Very quickly, my wife's aunt is a cookbook writer and has been since the 70s. And she's written many cookbooks. And she said, Google was what got it for her. She's like, I can type in basically the ingredients I want to have, and I'm thinking of a new recipe, and someone's already made it.
[46:54]So there was no point then?
[46:56]She used to invent recipes. Right. Well, she used to invent recipes, but it turns out there are no new recipes. In some basic fashion. So again, what the – Google was one too. I mean it's a cultural technology. It's giving us access to the human knowledge. And the chatbots are just a slightly different approach to that. But yeah, that said, I still believe you want to be careful about when you ask them for recipes. They don't really taste so well.
[47:27]
Embracing Uncertainty in Technology
[47:28]Well i know adam you and i both like to be precise and have data and analysis and we've basically made this a discussion of boy this is a squishy thing now so
[47:40]Squishy so squishy gonna have to learn to live with it learn to live with uncertainty.
[47:45]Oh that's uncomfortable well thanks for coming on as always this is always a blast and everybody should go to tidbits.com and I assume there's going to be an article on this subject coming out eventually as soon as you resolve the squishy bits.
[47:59]Right. Once I come up with the answer, we'll be good.
[48:04]All right. We'll talk to you again soon.
[48:06]Thank you.
[48:07]I hope you enjoyed this episode of Chit Chat Across the Pond Light.
[48:13]
Supporting the Podcast
[48:11]Did you notice there weren't any ads in the show? That's because this show is not ad-supported. It's supported by you. If you learned something, or maybe you were just entertained, consider contributing to the PodFeed podcast. You can do that by going over to PodFeed.com and look for the big red button that says Support the Show. When you click that button, you're going to find different ways to contribute. If you'd like to do a one-time donation, you can click the PayPal button. If you want to make a recurring contribution, click the weekly Patreon button. You're only charged when I publish an episode of the Nocilicast, which, let's face it, it's every single week, so I don't charge Patreon for Chit Chat Across the Pond Lite or Programming by Stealth episodes. Another way to contribute is to record a listener contribution. It's a great way to help the Nocila Castaways learn from you and takes a little bit of the load off of me doing all the work. If you want to contact me for any reason, you can email me at alison at podfeed.com and I really encourage you to follow me on Mastodon.
[49:07]Music