Paul Frazee joins Bryan, Adam, and the Oxide Friends to talk about the inner workings of Bluesky and the AT Protocol. Paul and the Bluesky team have been working on decentralized systems for years and years--very cool to see both the next evolutionary step in those ideas and their successful application in Bluesky!In addition to Bryan Cantrill and Adam Leventhal, speakers included our special guest, Paul Frazee, and slightly-less-special guest, Steve Klabnik.Some of the topics we hit on, in the order that we hit them:ScuttlebuttBluesky FirehoseBluesky JetstreamBluesky and the AT ProtocolBluesky Feed: Quiet PostersBluesky's bot invasion: AI accounts argue with everything you postAI Imagery labeleratprotoOxide starter packIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Is that true? I can join up. I think I've always joined with putting up my hand. Have I always done this? I thought that's how you joined with, but I thought I put up my hand and then I joined. It's like, are you just like core Brian always raises his hand before. It's like that guy in section.
It's like, dude, we are in college. We're not in middle school. You don't have to raise your hand.
Just like, I'm the guy that likes to stand while he's presenting a section. Is that wrong? I guess so. I mean, I thought everyone did. I guess. Okay.
Holy smokes. All right. There is Paul. Not even raising his hand. He's just waiting patiently to be called on.
Paul, thank you very much for joining us. I am super stoked for this today.
Thanks for having me.
Oh, I... I'm not convinced that there's only one of you. Especially in those early days of Blue Sky, you were everywhere at once, I just felt. It's been remarkable. I'm not sure how many of you there are, but I appreciate your collective allowing one of you to come speak with us today. Yeah.
There we go.
Exactly. Paul, there's so much to talk about here. But, I mean, you all have been on just an incredible ride. I mean, it's a hard... I mean, having watched this thing from when it was invite-only for kind of a protracted period of time, and then just going, I mean, non-linear, not in the metaphorical sense, but the actual literal quantitative sense, has just been amazing.
And I want to get to a bunch of that, but before, do you mind, can we talk about Scuttlebutt? Is that inbounds?
Totally. Yeah, that's totally inbounds. Let's do it.
Because I want to talk about Scuttlebutt and Beaker. People think they're in here to talk about Blue Sky and Blue Sky, but we're actually talking about Scuttlebutt and Beaker. So, Scuttlebutt, could you describe what Scuttlebutt is and how you got involved in Scuttlebutt?
Because this is obviously a problem that you have been thinking about, and I think this is true for a lot of folks on the Blue Sky team. People have been coming from different angles thinking about this problem for a long period of time. How did you get into Scuttlebutt?
Yeah, I mean, so, like, everybody on the team has been at this for years. So, like, collectively, I think we have 20 to 25 years of, like, decentralization work, like, leading into the team. Skullbutt was my first one. That was back in 2012-ish. And that was a technology invented by Dominic Tarr. And...
it could fit into actually all of us came everybody on the blue sky team came from like the p2p world right which was kind of like hey could you take some of the techniques from bittorrent and then do some modifications and actually try to build sort of real time or or large scale or social applications depending on what you were up to using those kinds of p2p techniques and secure skull that was directly geared towards social networking so it was a peer-to-peer social network
And it had this very aggressive kind of local first mentality. It was called Secure Scuttlebutt because actually it was based on a Scuttlebutt gossip protocol where you're just like having each node kind of like rebroadcast logs to each other in a kind of best effort way.
And that meant that it had a very fluid topology of connecting together, like any time you were able to catch up from one node that you were able to connect to, you'd be able to, which makes it actually quite ideal for even extreme cases like a sneaker net if you were so inclined.
And then what was sort of interesting about it was that we merged together the gossip protocol and the social layer where it would use your follow graph to decide which account logs to synchronize.
So if I was following five people, the default kind of intuition there is whenever I would connect to a node on the system, I would ask those nodes like, okay, here are the five feeds that I'm following. Can you like catch me up on anything that you have for them? And since you're connecting sort of like to a forest of different nodes, you would...
presumably you know at some point catch up to everything in a kind of eventually consistent way and uh we actually would take it further where you would do friend of a friend expansion so you would actually ask for like the five people you're following plus like two hops out if i remember right Really sort of wild way to do it, but it was aggressively decentralized.
In fact, I would call it an anarchy, and not in like a pejorative sense, but like quite literally no authorities were encoded in the system. Interesting.
Yeah. Was that as a kind of a response? Because I mean, this is like Pretty early. I mean, this is before the animals were walking on their hind feet in terms of social networks. Or maybe they were already. Maybe they were born on their hind feet.
Yeah, what's the era? When was Scuttlebutt?
It was 2012. So I can say that that vibe at the time was correlated probably to two different things. For one, it was pretty clear by then that the major social players were sucking all of the oxygen out of the room. And so there was a collective of open source hackers who were feeling really excluded from what was exciting about social computing.
and wanting to be able to get in there so you have a general the through line throughout all these projects has always been like frustration with monopoly and feeling like you're not able to get in there and make meaningful change because we're talking about internet technology we're all programmers let's get in there like let's have that Linux philosophy or something like that being applied and so that bugged all of us and that was a big part of it and then you know you also had um
It's still, I think even then there was starting to be a little bit of initial disillusionment with the major social companies. Who knows if we were kind of like on the right target with that or not, but we, in general, were starting to, you know, this was like... Not that far after Occupy Wall Street, which was a big animating activist era.
And a lot of us were thinking a lot about where we expected... People still use the term new media in 2012, if anybody remembers saying that nonsense. So there was quite a bit of thought being put in at that point about what are the power structures that are going to be a part of the future of internet and social media. And how are we designing systems to be smart about that?
So we were thinking pretty heavily about that stuff right from the get-go.
Pretty prescient, I have to say. Because I feel like in 2012, honestly, maybe I'm just a dunce, but it felt like it was kind of the good old days for Twitter. I mean, it felt like it was fun and light. And maybe Gamergate changed that. I'm trying to kind of replay that era in my head. But definitely, this was... I mean, this is like, why wouldn't anyone like Facebook?
Facebook just feels like a net good for society in 2012, says I. I mean, I just feel like I was, anyway, I was very naive. I mean, I felt like at that era... the, I mean, it was more like Twitter was finally kind of functional. I mean, there was such a long period of time where it was just not very functional. Um, and it could kind of keep up with everything and everything else.
So I really admire the prescience of people saying like, no, there's like a, there's a, there's a centralization of authority here. That's actually problematic. Um, and, Then I got to ask you, were folks like, you know what, I'm burning my boats on Secure Scuttlebutt and I'm now like, you can't find me on any other social network? Or was...
We never got that far. It was janky at best. And I say that with all love. But we had a lot of challenges that we just did not get through. And if we're going to go through this history, one of the meta arcs I would follow throughout all of it was learning how to...
serious about delivering, you know, the level of quality that's necessary for something like this, because you're on the one hand, you're pressing really hard on like, okay, novel technology, novel way to do things, and throwing out a lot of assumptions.
And along the way, everybody, you know, by the time we get to blue sky, almost everybody at the team had been spending, like I said, years working on this stuff, and nothing was quite working.
And it wasn't until we got to this project, we all had a bunch of kind of collective, collective realizations as we came together about like, you know what, okay, if we keep these pieces, but then like throw in a lot of the kind of complexity and the novelty that what we're doing, we can keep what we think actually is important about the systems, but have this work actually be usable to end users.
Not working and making it work. Are you talking more from like a product perspective rather than like a technical perspective?
Really both, to be honest. Oh, yeah. Yeah. Yeah. But I don't know if I would give more emphasis to one or the other, because it would affect on both sides. Reliability, scale, and performance on the technology side, and usability on the product side. almost every project that the group of us worked on all use like client side signing keys up until blue sky, you know what I mean?
So you had like key management problems and you would also have, uh, it was all local first, all these projects. So like you would have device synchronicity and like, um, essentially like you'd have to get them like CRDT territory to just make a post. It felt like rocket science to try to implement a content section. Yeah. It was, it was very complex software that we were messing with. Yeah.
with blue sky yeah we were just like what are we doing to ourselves i mean honestly it explains so much too because i think and i just want to like not lose us say kind of off the jump the blue sky is so remarkably available and remarkably usable in a remarkably short period of time and
You know, it's not surprising, of course, that you all come with this scar tissue from these other endeavors where you had tried other things. Because, I mean, and I think that, I mean, speaking as a user, I mean, social networking is really important to me personally. And what you all have done is so extraordinary. And, like, there's been no fail whale.
You know, I mean, it's like the closest we get is, like, Health Red, which obviously I love. But the... I'm sure you're like, listen, let's not paper over it too much.
There have been plenty of... Just when you're adding millions of users a day and you are retaining... It's so responsive and it feels quick and lively and you're not getting... It's not at all a surprise that, of course, you're learning a lot from these previous systems and
So you said there were a couple of things that you knew you wanted to, that you all kind of collectively from your experience knew that you wanted to replicate and some things you wanted to improve upon. What were some of the things you wanted to replicate?
Well, like the... One that we knew for sure was what we call account portability. And then that one comes to mind first because it has to do with like why you bother with peer-to-peer in the first place.
You're trying to turn the services that you use into something that is actually like, oh God, I'm going to say it, fungible, that you could actually replace a hosting service or like an application service, but keep all your data. And we knew that was one of the really interesting properties of peer-to-peer because like, you know, their client side, right?
So like your storage and your signing keys were being kept on your device. And so the first thing we sort of figured out was, okay, how can you take advantage of using a server, but like maintain that ability to like adversarially move away from a server. And that's what the entire cop portability system is for.
And that was a pretty big move forward because using a lot of the same techniques, but putting it on the server and not losing those properties was like the, kind of the biggest like scale and ease of use unlock that we had throughout the whole system compared to our previous work. We knew we wanted to keep that.
The other thing that we wanted to keep for sure was what is now starting to get called a shared heap model, this notion of an open broadcast of all of the public data so they can be repurposed, because that was a pretty big design element that we had landed on in the peer-to-peer stuff that we'd been doing about how to build these large-scale applications, is this notion of...
It works actually quite a bit like the web where it's almost like everybody's publishing JSON of like posts and their profile and their likes and stuff like that. And then you just like aggregate them in the applications. And that's how you build out an application.
We knew we wanted to keep that because that's what makes it possible to build other applications without some kind of a hard binding to the hosting. So you're really separating the hosting and the application layer. And that maintains that hackability and like capacity to repurpose these applications in interesting ways. Those are probably the two that come to mind.
And I can say, I mean, I love both of these things I think are extremely important. I think that the, obviously, I mean, the portability, it should go without saying, but one of the ways that I realized how just viscerally important this is to me, and this is a super stupid thing, but just in terms of my own age and social networking, tracking my own life,
my first two kids I announced with a blog entry, but my third kid born in 2012, I announced for the tweet. Um, the, so that tweet, for whatever reason, like the replies on that tweet have been deleted by Twitter and, or I can't find them anyway. And it's like, damn it.
In terms of that portability, I feel like, and of course, what idiot would announce the birth of their children on social media and expect it to be retained anywhere? Look, she doesn't have a baby book. She's a third kid. This was her baby book. There's going to be the replies on this tweet. Okay. If, if you are a third kid or you have three kids, you know what I'm talking about?
She's, she's lucky. She's got a tweet. Okay. She knows that she knows she's lucky. She's great. But the, so, I mean, just like having that portability is, is, is really, really, really important to people. And it is a, in order for us to really engage in social networking, we absolutely have to have that portability. So, I mean, obviously I love that. Hey, I don't know what her name is.
You're saying I do know her name. Hold on just a second. Let me actually, you will add up my nine. I've said this before, but she, the day she was born, we, the CEO of the startup I was at was fired. Yes. And fortunately she's got a very good sense of humor. So she's like, Hey dad, it was 12 years ago today. Your CEO was fired. I'm like, it's, Is this going to go on for your entire life?
Probably is. But it's true. That is how I remember it. It's just easier to remember the firing date and then I get to her birthday in one hop. But Paul, I also love what you've done with the fire hose because when did Twitter turn off fire hose access? I want to say that was like in, when was that? 2013, 2014, maybe earlier. I don't know.
I don't know the exact timeline. I know it got a lot more turned off recently. Yeah. More turned off.
But the ability to sit on the fire hose is just, I think, amazing.
It's super cool, yeah. And that's also what a lot of people have started to play with first, because it's just so obviously cool. Like, okay, what can I do with this? It's cool in two fronts, actually, because actually, it's really like you're just getting into the innards of our data center, because this is the backbone. You normally...
like an event processing architecture in a way and like normally you would use like a kafka or something like that to like send all these things through that's what this thing is that's what this firehose is all of our application is downstream of that just running computed views off of that firehose so whenever you're tapping into it you're just jumping straight into our data center to tap into the data set so that's really fun it's it's amazing yeah like a meaningful thing not just like a replication of all the activity but like that's it that's the network
So, Paul, that seems tough to handle as a consumer and as a sender. Do you have a sense of, like, is there a scale at which that no longer becomes feasible? And might that have contributed to Twitter kind of limiting that and then turning it off? Oh, yeah. What are your thoughts on that?
Yeah. There's a... There are going to be some interesting learnings about the economics of this over time, to be sure. So I don't want to be walking around as if that's not a big reality about all this. That said, it is a lot.
There is a version of the Firehose that we offer that is a kind of a reduced form that drops all of the cryptographic proofs to make life a little bit easier and allows you to do filters on it and things like that. But one thing to also remember is that the Firehose is kind of a convenience service, to be honest. So if you, like I said before, it works kind of like the web.
You broadcast or publish these records, and then those flow into the applications to be processed into the views. The Relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the world and kind of goes ahead and grabs their replication streams and puts them into one stream for you.
If we decide at some point what we can't afford, like all the consumers on this thing, you can run your own relay and directly do those crawls. So again, we're trying to make sure that there is openness to the system so that if there gets to a point where we're like, you know what, we can't afford passing this along to everybody, somebody else can do it. You can directly do it yourself.
Well, and I, I mean, I got to believe, I don't want to editorialize too much, but based on what I know of Twitter at the time, if I think that there are the, the, the fire hose were kind of the crown jewels of Twitter. They did not want to let people on that. You had to pay a lot of money to get onto the fire hose.
And I don't think that that was a, I mean, I I'm sure folks will correct me if I'm wrong, but I question whether that was a technical limitation. I think that was a, a business decision. And I think as folks saying that.
An injection of ads decision is how you're.
I think it was an injection of ads decision. Yeah.
And I also think they had also made, I mean, and this is another thing that I think that, you know, as kind of reflecting Paul on the terrific work that you all have done, there is an advantage to you coming after having the world having figured out one, how to architect a Twitter that actually performs, which Twitter itself spent half a decade doing.
And then two, the systems are just better that you can go use, right? You don't have to deal with a Ruby VM, for example. I mean, you've got other things that you can go... I mean, even Ruby has improved. I know I'm going to hear it from the Ruby folks. But all of that software has really improved a lot. So you've just got...
And then that fire hose is so extraordinarily valuable to everyone, I feel. I mean, I think that I love what you had said in the, and I want to drop a link into the paper that you all recently published, which is terrific, by the way. I love this ACM paper that you all wrote on Blue Sky and the app protocol.
And just in terms of the value to researchers to be able to get this whole thing is extraordinary.
Yeah, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, it does form the basis for building applications beyond just the kind of Twitter style of application. So I'm pretty excited about that.
You just kind of nailed what was on my mind as you were talking about us scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out. And in fact, Martin Kleppman, who actually helped us with that paper quite a bit, really helped drive that paper. He's been consulting with us from the beginning.
And that was a huge asset, an incredible win. I was... when we first got him i was over the moon um that's the kind of you know he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications these kind of high-scale data heavy applications and so we were able to really apply
just image everything the industry had learned about how to do this sort of stuff both to like our internal systems and also just to the protocol since scaling was always a big part of what we were trying to do we wanted this to be able to be at the scale that you expect out of these kinds of applications so that i think you know that plus just a really great team that you know did some great work to keep those servers running throughout all that growth that's that's how this
Okay, but I've got to say the level of growth, I mean, and maybe Elon called you up and like, look, I want to help Blue Sky out all I can. Please tell me the things that I should do to my own social network to drive people to you as quickly as possible. I mean, that actually a lot more makes sense, honestly, if I knew that. But I'm not saying for you to confirm or deny that.
I'm just saying it's like kind of consistent with what's happened. I mean, the level of scaling that you've seen, were you expecting that in your kind of wildest dreams? Oh, man. Yes, this is going exactly according to plan. This is what I expected.
Totally, 100%. Yeah, just everything, every moment. No, I mean, I feel really pretty fortunate across the board. And honestly, when we were starting out the project, we were originally just a protocol consultancy for Twitter, which is a complicated and really interesting history.
But when we started on this thing, we didn't expect there to be either the relevance or the market opportunity to actually have this thing go into production quite like it did. So every step of the way has been really surprising on that front and exciting. But when we first started to do the beta and open it up,
I remember people were starting to come on, and this was, again, like, I don't know, the third, probably more like the fourth time I had launched a product in the decentralization space. And up until then, none of them had worked. So we had these users coming on. I was like, okay, cool. You know, they'll be here for a couple of days, then they'll leave.
You know, like I had really, really learned that, like, yeah, everything you do fails, of course, you know.
Right. The crops have failed again, of course.
Yeah, exactly. Like that's when they die. Yeah, exactly. But no, they stuck around. And it was like, that was the moment that really shook me the most. It was like, oh my God, they came back today. And it's been kind of that level of surprise ever since.
That's amazing. And so you all, because I mean, I originally, I mean, you launched your private beta in March, April of 2023. When was that? Somewhere in there. Yeah, it was around. Um, the, um, and which was great. We got, I got lucky enough. Fortunately, we've, um, I always felt Steve Klabnik is here.
I always felt I may have put Steve in an uncomfortable position when I was asking him for a blue sky invite. He's like, I mean, I'm like, I guess I'm technically your boss, but like, and I know, I know this sounds like, but you know, I mean, Steve is here either to defend himself or to, to bring up his attorney on stage. Yeah.
Hi, Paul. Good to see you. So what I remember it being like is like somebody had posted like, thank God this website is so small that my boss isn't on here and can't see what I post. And it was like three days after I invited you. And so I was like, oh, shit. That's my recollection of this story.
Right. Well, I really appreciate you sharing an invite with me. And invites were super precious. And then, Paul, somewhere along the line, the decision is made to like, really, okay, we need to like, we've got enough users on here to know what we need to go do. And we're going to more or less shut it off for a little bit. I mean, keep the site up, but we're not actually adding more invites.
Or was Steve lying to me this entire time when he said he was out of invites?
Sorry, boss.
That's a good lie. I'm going to try to give an answer that doesn't blow that up.
Steve is furiously DMing you.
Yeah, no, so that's actually not an unfair characterization. Everything moves so much faster than we were ready for. We started working on this protocol in 2022 at the beginning of the year, and then by the October of 2022, we were starting to realize, and you may understand the timing if you can play back everything that happened.
October 2022, what was going on? Yeah, what happened then? What happened then?
So by then we were going, well, we're going to need to make an app. And suddenly we were productionizing the protocol, which the org wasn't designed to do. We were not prepared really for that. And so we were kind of initially, we did have a client, I think, already going, but it was really just like a test bed to make sure that the technology worked.
We suddenly were hitting the gas to get features in there. And get the servers to a place where they could start to accept users. And then once we started to have them on, you know, the single Postgres era lasted, you know, for a lot of that private data, right? So there was a... Yeah.
So... Welcome to Oxide, friends. You don't have to dig too far to get to the Postgres era for anyone. So this is good. This is very on-brand.
Oh, if you don't start that way, you're doing it wrong. You definitely should. Because the transition after that is into the ScyllaDB and the event pipeline era where the engineering gets way harder. So you only want to do that once you actually are scaling. But that's what we had to transition to.
So by the time we're in the fall of 23, we're looking at this and going, we got to stop everything and re-architect this thing or we're not going to be able to handle this growth. It was just the demand was too high. One thing I know that Jay would want me to clarify about the invite period is that that was always... There were some people who were like, it was about exclusivity.
It was not about exclusivity. We were more trying to make it like a hip thing. It was very much just that the servers could not handle the load and we needed to get our TNS team scaled up. And that stuff takes time. We just weren't ready. So the demand was way higher than our supply. and that's what we were able to buy ourselves by doing that in my only period.
And, but you know, we moved as fast as we could and it wasn't until February of 23, 23, 24 that we could, yeah. Right. You're in the post in, in the, um, yeah.
So what I can say, I really admire the discipline of the team because when you, I mean, what would, At some level, and I know Blue Sky is interesting because you're a public benefit corporation. There's a bit of both sides to this in terms of you're seeking to make a profit and to be a financially viable company, but you're also seeking to serve the public good.
I know it's a little bit different, but I think it still takes a lot of self-control around here because everybody wants product market fit. And when you have product market fit, which I think we could not, not to disparage secure scuttlebutt or beaker, but this was, this was like probably things that didn't have as much product market fit. Correct. I see what you're getting at.
You can see we're having this conversation about Beaker, but poor Beaker. You've got this enormous product market fit and to have the discipline to say, if we don't fix this, we can't fake it till we make it, move fast and break things, all these other kind of Silicon Valley mantras that actually you're going to end up with something that's broken at the end of it.
You actually won't have something that works.
That is completely correct. Yeah. And it's actually, well, first of all, this is all pretty integrated. So the PBC specifically, what that does is it gives us space to pursue a mission in addition to the profit motive. And so the mission is to create this decentralized protocol for public conversation and for applications related to it, right?
And that's actually just remains like the main goal, product market fit, scaling of having a profit come in so that we can sustain the organization that is still all in service of that mission. It's just the vehicle that we've chosen to do it by.
But that, you know, was something that when we were sitting down to attack this project, we had all realized like, okay, we are all trying to get a change in how technology works, but what's your theory of change? How do you expect to reach the market? And the way you do that is with a product.
So all the mechanisms that come along with the startup are really, we try to maintain that that is the vehicle by which we're doing, accomplishing this mission. That's not the mission itself. And either the discipline around like the scaling side of it or the moderation side, that was tough. But the other one that we maintained was making sure that the protocol was consistent
sufficiently out of 1.0 that like, at least the self hosting part of it was active before we launched, we did not want to go into the public launch prior to having that set up because we wanted to make sure that we hit those targets for what. So that was another thing that we delayed launch for to make sure that we were actually acting on the protocol. And it's kind of always that balance.
which I got to say is another act of real discipline, because I think that part of what makes this challenging is that you are trying to build blue sky as kind of an exemplar of what's possible on at proto. And it's in any of these things, it's tempting to just be like, ah, just like, okay, can we just get like blue sky working? And then we'll, we'll quit.
We'll figure at proto out later and allow people to do this kind of on their own later. I mean, it must be, I mean, obviously that's not a temptation that you all had because of the strong call to mission, but I just think it takes an admirable amount of discipline to really say like, no, we actually need to do this the right way. And we need to allow the,
We want to be sure that this thing is actually, that at Proto is the actual basis that we're building this on, and that Blue Sky is but an instantiation of it. I'm really impressed about that. Was there... I mean, there must have been some level of tension of like, hey, we've got to, like, we're missing a window of opportunity. Did people feel that way?
Or did you feel that the window of opportunity, did you... know that your agent on the inside of X, Mr. Musk, is going to continue to operate per your explicit wishes?
No, we had no guarantees of what was going to happen, for sure. And there was certainly – we had the – it would be a lot if I said we didn't have that anxiety in the conversations about – making sure that we're capitalizing on opportunities, but I would never say that the protocol or mission focus was ever in question.
That was just something that this team never really is willing to throw overboard. You do get into some complicated conversations about capacity assignment towards protocol work versus product work. And there are definitely some times where we're kind of looking at the quarter and going like, okay, what are we gonna focus on? In fact, a lot of last summer,
We were sitting there and going, you know, there's just a lot of features that are missing if we want to be, you know, what users are looking for. We just had a lot of requests and things that were absent. And one area that we did, I think...
you know make a i i believe defensively pragmatic choice was with dms we knew that not having dms was killing us and the protocol is not ready for dms it's really geared towards public posting at the moment so we're like all right we're going to give ourselves a cheat on the dms we'll go back and clean that up later but for all of the summer yeah that was like a
we're going to make incremental progress. We're going to do a little stuff on the protocol, but we really got to get this product shipped up because if it's not working, then again, that like, then the protocol may not get its chance. Right. And I think we're in a better place. Yeah. I think we're in a better place now.
I got to say, I really... Just speaking as a consumer, I really appreciate that choice because there were very few... One of the reasons that I was still sadly on the fucking bird site was because of DMs. And I actually do need DMs. And this... But DMs are a place, and my DMs have always been open on both platforms. It's been very interesting to watch the supposedly bot-free Twitter.
I was going to say, wow. Oh, my God.
I was going to do it on Blue Sky because I'm not sure our DMs are bot-free.
either right now i thought it was great when blue sky it's like oh i got some i got like unsolicited crypto and and oh without noise and some porn is here too it's like this is great like this guy's gonna make it this is so this is all warming mods uh this is like three quarters of them is like report a spam report a spam but it's so essential for the times when it's not that though so you know it's good yeah
Yeah, and I also feel it's like DMs are... I think that you all made the right call there in that DMs are a good place where it's like, yeah, I try... This does not need to be a part of that proto for this. I actually just need the Blue Sky app to have this functionality. I guess you all have done such a good job... mimicking and it's Steve did the right thing.
So Steve was a Twitter dead ender until he wasn't. And I thought like, there was a time when Steve's like, yeah, I'm like, I'm just, I'm going down with the ship, man. Like, I just don't care how bad this place gets. And I'm like, I don't know, man. I'm kind of, and then I'm like, you know what? I am. I'm going to go down with the ship too. And then I look over and I'm like, where's Steve?
Steve's gone. And I'm like, Steve is Steve has like deleted his Twitter. Steve is gone. Gone. And I'm like, oh, okay, this is actually, maybe I'm being liberated to get Gone Gone. So, I mean, I desperately wanted to get off of that thing as it was just like absolutely descending into the muck.
But the thing that has happened to me a couple of times, and I don't know if this has happened to you, like, look, look, I would love to tell you that I've been off for a long time, but it's a process, man. One day at a time. Okay. Like every day is a struggle.
Some days I got to call my sponsor and the number of times that I have, you've got, you all have done such a good job on just like the look and feel the number of times I have been like, Hey, you know what? Actually looking at these, these posts are actually surprisingly good for the hellscape that Twitter is. Oh no, wait a minute. I'm blue sky.
Or then also what has often happened is that I will get a... You know who's not on Blue Sky? Only the good VCs are on Blue Sky right now. This will change very shortly. But the VCs that really troll the hell out of me are all like have shackled themselves to the mast of Twitter. And I just don't have the self-control to not actually go and go to the rage bait.
Um, but then I will, the, the, the, the, the inverse also happens where, um, I will accidentally leave a Twitter window up and I'll be like, oh man, blue skies. So you've done an excellent, excellent, excellent job of.
I would say as distinct from Mastodon where you knew you were there because it was boring as fuck.
Yeah, Mastodon. The problem with Mastodon, I love Mastodon. Do you? I love aspects of Mastodon. No, no, no, I don't love Mastodon. I was trying to find, this is like where I'm sorry for not being sorry. I love the idea of loving Mastodon. That's what I love. Yeah, I think that's right. But Mastodon has got the usability of a BBS. And I love BBSs.
Paul, I'm not sure how much... This is like modem era social networking. But the problem was... And this is part of why it's really important to... And I thought you all did actually a really good job of... Talking about why not ActivityPub and why not Mastodon. And there are some really good reasons for that.
And I think that Mastodon should... I mean, I think it's going to be... We're going to continue to be on Mastodon. I think it's great. I think that it serves a great purpose, but... the problem is that the, uh, like I'm a, I'm a, I am a baseball and football fan and I like social networking because I, I like sports Twitter. What was sports Twitter?
And, uh, like you're just never going to end up with that on Mastodon. Um, it's just not going to happen and you need, and also, or, or, or Paul, the, the shake test, are you, I'm not sure if you, uh,
so the the real acid test for social networking is like if you're in the bay area and you feel that you just felt a quake oh yeah and um and i'm sure you know this but um the uh you should know that blue sky has passed this test with flying colors because we didn't just have a quake we had a tsunami warning recently
And my phone, like every other phone in the Bay Area, exploded with what I assumed was an Amber Alert. You're like, tsunami warning on a quake? I didn't feel. My first thought is like, this is like an IT error somewhere. This is like a missile alert in Hawaii. And, uh, but it's actually obviously real.
And it was, what was really interesting was blue sky was just, just exploded with, you know, people talking about it, talking about the, the, uh, tsunami know your zone site, uh, is down. You can't get a database handle to know your tsunami zones and no one can figure out if they're going to die in a tsunami or not. The, and blue sky.
And I, and what I also really loved is then my geo sciences feed on blue sky, uh,
was great i mean it was it and the and meanwhile on the other side it was just like they're all it's all garbage so i mean it was it really passed this test with flying colors um could you talk a little bit about the feeds because this is something that again i've really i i think feel i've started to make use of i haven't done enough with but i absolutely love and it's really kind of a first class thought in the way you built the thing
Yeah, yeah. It is an interesting area of the product that actually we're going to be investing even more in the future. I'm really excited to get into it more. Yeah, so like feeds, gosh, the basic idea with feeds was just how could we, I'll tell you the real mentality to it.
We knew going in that we liked some of the ideas about like improving how there's back pressure to the organization beyond just at the really kind of systemic level. Because the entire protocol that we've created, it makes it possible to make alternative applications and have alternative hosting, but those aren't super tangible in the day-to-day to people's lives.
So we sat down early on and asked, okay, well, what could we do with this that would allow people to more tangibly experience an open network and decentralize what is meaningful to the average person? And so then the two answers that came up were, could we decentralize moderation in some way, and could we decentralize algorithms in some way?
And so the feeds came out of that and the feeds, I think that they can, there are ways that I think we can improve on them and make them even kind of more central to the experience and really get the most juice out of them. But I will say that from a kind of like validating the concept perspective, man, they totally struck gold right from the get-go.
Absolutely struck gold. Yeah.
yeah and i'm really pleased about that it's actually it's really fun how they work they are hosted on other people's servers they do not run on our systems um but they feel integrated into the app and with that we kind of invented this whole principle that we call third party is first party with this idea of being able to integrate in systems
posted outside of the application, but make them feel native. And so the way they work is the simple answer is that you contact the feed server and then it just sends back a list of URLs. And then those list of URLs get hydrated into the feed.
the way they have those urls is they tap into that firehose and they use whatever kind of algorithm that they want to use to decide uh which posts to select uh to make the feed and that ends up being a really nice compositional uh boundary for uh that thing they can use any kind of logic they want from social graph you know queries to computing ranking off of likes and reposts and
other kinds of signals, you can get, you know, text queries running in there, which a lot of them are a lot of them are kind of operating off of like a smart hashtag in a way where they're just looking for particular topics that are being discussed. Honestly, you can get really far with just social graph limitations plus text query, right?
Like the basic object being these are these are people that are experts in this topic. And here's the topic and let's get a feed going.
it must've just felt so right. Like it just, it just feels like so spot on from an architectural perspective, from a usability perspective. And it's, it's got to clear up both the, some of the, the, the real concerns around Twitter, around Facebook, around threads, around these other social sites where you're like, as you say in the paper, like,
They're prioritizing engagement over all else, like over the mental health of the users. And then the other side, you got Mastodon saying, look, the algorithm is timestamp ordering, which leads to a less than simple experience.
Timestamp ordering of the people you follow. It's like, oh, you're interested in the tsunami that may be coming to your place in an hour? I don't know. You should follow more geologists. Maybe in the next life, you'll follow more geologists.
You should have done that in time for you.
You should have done that. You should have been carefully following geologists. You're like, okay.
I got, I got, I got a timeout and give like a, you know, I'm glad Mastodon is there pursuing the same mission. I'm just going to say that. And then we could move on.
Yeah. Yeah. Look, I think I continue to like and use Mastodon. I think I actually like it more than you do, Brian. I wouldn't say I love it, but I still, I still love Mastodon.
You can't say I, but we know that I actually just love the idea of loving Mastodon.
Yeah, exactly. And actually for a while, I liked that it just helped me put down my phone and not keep on scrolling. So there was that benefit too. But it just feels like it plugs such a nice hole. It's a really lovely design that you have.
Well, in the feeds also, Adam, what you're saying about like, it's one of those ideas that you know is an important idea whose time has come because you're immediately like, why are we doing it some other way before? Like, this is, you know what I mean? It's... Not that it's not at all obvious because it obviously wasn't.
But the fact that you – I mean, it's one of these things where you – Paul, you all made a bunch of good decisions that built on one another. And then out of this comes this, I think, artifact that's going to be, I think, more and more important as time goes on because I think that –
you get to bring your entire self because all of us consist of like different things and different things at different times. And the feeds allow to allow, give us agency over that, which I think is extremely important.
Yeah, I mean, I have to agree. I mean, there are also times where I'm just kind of like, you know, today I'm in the mood for a more quiet thing, you know, and that's, I've got feeds for that, you know, or today I'm actually like voraciously consuming the stuff. I read all my fees and now I'm going over to the one that shows me more of the fire hosey kind of experience.
Like just whatever you're vibing on that you can, you can get there. Yeah.
Yeah.
Steve, can you explain what that feed is? Because I love that and you had talked about it in your post about how AdProto works.
So Quiet Posters is basically a feed that takes the people that you follow who don't post very often and only shows the posts from them. So some people who are mentally healthy don't post as much as someone like, say, me, who is horribly broken as a human.
And so if you want to see those good, healthy posts of the people that post, say, maybe once a day instead of once every 10 minutes, you can just load up Quiet Posters. And that way you can keep up with friends who maybe get lost in your stream because you're following too many people or whatever else.
I love that. Yeah. And I, I think it's, it's so, and I think we're just going to see a lot more experimentation in that. And again, giving people more agency, you know, Adam perversely what it reminds me of, and sorry, Paul, just to go overly Gen X on you here, but the, the day MCA died, I wanted to just listen to all of his music from beginning to end and,
And Pandora is like, let me give you something that sounds like the Beastie Boys. I'm like, a man is dead, Pandora. I'm not interested in your algorithm right now. I am mourning a dead musician you need to actually give me. And that's when I'm like, I'm going to Spotify. I'm not coming back.
I'm a little embarrassed that what Steve's anecdote just triggered in me is the recollection of on Facebook. I had a friend who would post every 20 seconds, and I only just now remembered that I muted him 15 years ago. So I need to go take a look at Henrik and see if I can unmute him.
Yeah. He's lived this extraordinary life that you've missed because you just couldn't take it anymore. It's like Mr. Peanut on Twitter. I finally muted Mr. Peanut because I couldn't take the ads. Jesus. So, I mean, Paul, this must have been one of these breakthroughs that you just kind of like...
feeds must have felt like a lot of things were clicking I mean it's such a I mean actually let me ask this did you kind of know immediately like oh of course this makes a ton of sense or was it when you started experimenting with it you saw it
Yeah, I mean, I wish I could say it was like, aha, you know, like we're in a way. I will say from a technical point of view, it felt really good. Actually, this is we're kind of talking about like the experience of like the engineering side, the product side. I'll just actually tell the whole story. Like we initially did not do custom feeds. Initially, we did something called scenes.
Scenes were like a hybrid between a community and a custom feed because we knew what we were aiming for, but we didn't quite have it right. And the way that a scene worked was that the members of a scene, when some number of them liked a post, it would trend automatically within the scene.
So you can kind of see there's some approximate thinking there and you would actually get like a notification whenever your post trended within a scene. And we obviously didn't think this through very well because people would create scenes of like three people. And so then suddenly you're getting like your post is trending, like the same two people would like a
post and then he would trend in five different scenes that they're all in with just three people in it. So that wasn't brilliant. But the other thing that also jammed that up completely was nobody understood this idea. It was like weirdly like both a community and not a community. And people were like, how do I post into it? You don't, no, you don't do that.
It just shows things that you're liking. So like we did that first. And I are probably the very many people seeing the product. We pulled it out. We just like, this is quite terrible. We pulled it out and we brought in the custom feeds thing as an alternative. I think in many ways it was kind of taking a bigger swing.
on a technical level because it is, yeah, because it's, you know, it's a more sort of nuanced, like we were at that point doing a at runtime request out to a third-party service, which in and of itself, I think, is going to be an interesting thing to kind of see how that plays out because, you know,
Anytime you're doing that, you got to talking about like out to these services that are not under, you know, we have no SLA with them or anything like that. They're like, how's the performance going to hold up things like that. So far, it seems to be actually working. But that definitely was a little bit more of a swing.
And then once we got it in and saw it working on a kind of design of the technology side, we were like, yeah, okay, this feels pretty great. And that side of it, the compositional side, that felt good right from the get-go. I think we still have some open questions about how we can really make the experience of using them as good as we want them to be. We have this like...
sticky problem around the management of them where there's more feeds that are interesting than you can stick onto your home tabs. And how do we find the right way to surface them to like easily like, oh, right, right. I've got this great feed over here. I want to go to it, especially on mobile. So we're still, I think, picking through the right UX to really make these things shine.
But I would say that somewhere around maybe a year ago, we were starting to kind of realize like, no, this is like pretty solid though. Like even though we had gripes about it, we were pretty happy with the outcomes that we were getting out of them.
Yeah, I agree with you that there's some UX things that you could probably do to... But I think that the bones are really, really sound on this one.
And I also think that you've got... The thing that I also loved about it, just like being able to follow my geosciences feed and my earthquakes feed during the tsunami warning, is that you are able to... I mean, there's so many... Part of what made Twitter great
when it was great was you would stumble into one of these like delightful communities and of like super nerds um and you know suddenly you've got you know top scientists in the field who are i mean just absolutely nerding out and going very deep on a particular subject and it's very nice to be able to like when you find those you feel like you've tripped over something
And I think it's like feeds are going to make it way easier to find those. And I just think that like the, there's so much we can go do to find amazing feeds. I also want, I want feeds for podcasts too. It's like my own, now we're getting into my own like weird agendas, but like podcast search is so bad. And I, I,
I think I want feeds for other things in my life, not just posts, not just skeets is what I got to say. I think it's amazing.
I got three things. One, scenes sound amazing. Two, I understand why you removed them. Three, naming a feature scenes is not beating the blue skies made for 38-year-olds allegations.
Yeah, yeah. Every time somebody comes in with... There are two critiques that feel really personal. And they're not really critiques, observations. And that is that it's elder millennials and then it's theater kids. And seeing as I'm both of them, every time I'm like...
I got a month left of being 38, but I've only been theater kid adjacent, so I get hit 1.5 times than a 2X on that one.
Yeah, but you're going to turn 39 when all the other millennials turn 39, Steve. You're not beating the allegation here. Listen, Paul, this is why we're trying to talk about BBSs and the Beastie Boys, just to get Gen X here pulling on it a little bit. I appreciate that. Yeah, yeah. So feeds were obviously, I think, really important. And I think feeds are going to be even more important.
I think we're going to see a lot of creativity. I don't know. I'd be interested to get your take on it. But I just feel like, boy, there's so much stuff that's possible there, especially when you go to look at things like LLMs. I mean, there's a lot you can go do there that hasn't been done yet that would be really exciting. Yeah.
You know, so to your point, our biggest problem is that we have too many ideas about what we want to do with them. And it's hard to pick about like where to take it next. We finally started to make some progress once we like broke it down into like bite-sized chunks. But yeah, I agree. There is a lot of really cool ideas that we're kicking around about what to do with feeds.
Well, and I think that also part of what I'm really excited about is that I think that we've kind of had this era of maximum engagement. And we are seeing kind of the logical conclusion of that in Twitter. And I think I actually want a world where we're actually able to find the hidden nuggets a lot more easily.
And we're able to find that the delightful bits that are at the edges are, to me, way more interesting and way more delightful. And I just feel that feeds are an important way of getting that stuff. It's just like the quiet feed, Steve. Let's get to the people that don't say very much, but when they say something, it's meaningful.
I hesitate to mention the R word, but it seems like there are even third-party revenue opportunities. Would people pay to get an improved feed, whether it's from a first-party thing from Blue Sky or others? I'm sure it'd be great to see, I mean, obviously you folks making money off of it, but other folks making money off it as well. Meaning the platform writ large. Yeah, yeah, yeah.
I do think, by the way, that there are a lot of opportunities. I think people will pay for things that add value, and I think there is opportunity to monetize a bunch of this stuff. Adam, this is where our inner VCs are coming out. What's wrong with us? We're broken human beings. That's right.
You do have to think about that stuff, especially when you have other people making this thing. You want to think about what the incentives are. If they can make money, that'd be great. We want to make sure that the whole thing sustains itself.
Totally. So you mentioned feeds, and then the other thing that you wanted to, that you were rethinking kind of at the same time was moderation. And what was your, obviously, another, I mean, just age-old issue in social networking and now a hotter issue than ever, just with our divisive era. What was your thinking on moderation?
Because when you rolled out the beta, moderation was still pretty primitive in terms of the tools that were offered for folks.
Yeah, this one is hard, and it's going to be probably a very, very long time before I'm willing to say, yeah, we totally nailed that one. There's going to be a lot of trying different things to find out where it can go. We've gone, I think, we've taken some interesting steps
stabs at it and i'll kind of get through uh all that in a second but the uh but i think there's probably many more uh things that we have to play with probably speaking the core of it i think the thing you just got to realize is that you really you cannot make the world happy with a one-size-fits-all moderation at this point It just can't be done.
And I think a big part of it is that people come into the world with different values and expectations about how those values should be upheld. And I think it's very painful to see those values not being reflected back in the spaces that you're in.
And it's, you know, I think it's quite important that we find a way to get there so that communities can, you know, I kind of really feel this is really at the heart of like why social media can be so painful for in general is that we just haven't found the right way to do this for large scale, you know, public spaces like this and what you might, you know, kind of public square mentality about social networking.
So anything we can do to get us closer to that is, I think, kind of the right sort of challenge to be trying to dive into with this sort of stuff.
Where can you move forward the experience of really the internet at large would be finding a way to create these spaces that I think give people a better experience with that element of it, the moderation of it, in some ways curatorial or editorial side of things.
Like feeds, you're saying, look, there's not going to be one way to moderate. Can you talk about that approach? Because I think that's a really interesting approach. We're going to allow people to moderate in different ways and allow people to give people some transparency and some choice among different approaches to moderation. I'm probably being overly reductive there, but...
I mean, that's probably right. I mean, the basic thing is, first of all, we start from the assumption that even if we did come up with a really great execution of an algorithm or moderation or things like that, you have the long term to consider. And do organizations stay effective at what made them successes in the first place?
And so coming into this, we have always gone in with that belief that the answer is no, right? That you kind of have to be planning for...
when our organization or any other organization starts to uh wither from its original purpose or or just maybe not be fit for the time anymore and be ready for for like the world to be able to change without too much loss right that's really deeply embedded in everything that we do from the application side of it to the you know the hosting side to the moderation of the algorithms so
Uh, as much as we do our best to try to create good versions of like algorithms and good versions of moderation, uh, we just kind of know that first of all, long-term systemically, you gotta be ready for that to not always be the case. Um, But beyond that, I think it's just quite clear that people have very different tastes when it comes to both algorithms and moderation.
And people have different needs for this stuff. And so how can we get everybody to a place where they are experiencing the life that they want to live on the internet? And that's only going to happen if we start to allow... A company like ours, we're in this kind of challenging position of balancing a lot of different voices. Users don't have the same challenge.
They can just choose to be parts of communities that actually are more connected to what they personally want as opposed to what an organization wants. So we should enable those groups to be able to get that specific with what their communities are like online. That's the basic mentality of it.
I have to say it's kind of remarkable. And as you said, it permeates your thinking of like thinking, and you call it the credible exit in your paper in terms of thinking beyond blue sky organizationally, I think is kind of remarkable and unusual, I think, for an organization to think we need to build in the things that
that will survive an arbitrary change in management and arbitrary, I mean, which is kind of, I mean, it reminds me of open source in that regard that where, you know, we are open sourcing this to assure basically that it, that it can endure, it can endure beyond its corporate vessel.
And, you know, not that Adam and I might be speaking from personal experience a couple of times over, but the corporate, corporate vessels don't endure folks. The, except for this one, of course, for whatever one you happen to find yourself in. Right.
exactly um so i i think that that that is uh the guy how do i boot some out of a discord larry ellison buys blue sky who would say such a terrible thing i unfortunately oh my god i'm so sorry paul this is the the demographic that we attract you know these are um um But I think that like being able to kind of think beyond yourselves in that way is, is really pretty interesting.
Can you speak to some of the specific tools that you books? I mean, in particular, like I love the ability to detach a quote tweet, I think is a really, I mean, that's not one that I've seen before. Could you talk about some of the stuff that you've done to just give some tools out there for folks?
Yeah, I mean, generally speaking, it's funny, honestly. Like, moderation is a tough job, as you might expect. And things like that... I don't know why nobody else has done it before because, gosh, anytime people can fix the problem for themselves, that actually makes our lives easier. So we have a pretty good incentive to do this sort of stuff.
Removing the quote posts, that one was like, we didn't do it straight away, but that was one that I had personally right from the get-go. Because we all know that the quote dunking is one of the big generators of just some of the most painful experiences on...
twitter at least and i imagine anywhere else that has a similar dynamic and so the detaching a quote post is essentially like it it's it just takes away whatever the quote quoting post is it just detaches it it's just whatever they've quoted it now just says content no longer available and so now you know of course what people do instead is they'll screenshot whatever it was that they originally of course of course they will but that's actually still better because
the things that you got to think about with large-scale social spaces is friction friction matters so much it matters so much and when you can introduce points of friction you're actually able to because it is such a game of numbers you actually will dramatically reduce bad outcomes and If you can just slow people down.
And the reason that the quote posts are so painful whenever they're quote dunks is that it sends a horde of people with one tap into the replies to start dogging you for whatever it is that you're getting dunked for. And honestly, if you've ever... I've witnessed a friend of mine get a really aggressive raid army kind of directed at them with a quote tweet back in, I want to say, 2017.
2019 somewhere around there and like you experienced that once and i think you'll realize like oh this is you gotta have something to slow this down um yeah so yes i adam actually once gave me the book so you've been publicly shamed which i would highly recommend not necessarily being publicly shamed but uh adam obviously thought of me immediately when he saw the book so i i think that's uh nobody on this voice call has been publicly shamed at all definitely not never before
Yeah, I mean... So are there other things that you all did like that? I mean, I think it's like one of the things that I think is interesting is that mute lists and block lists are public. And I'm sure this is an issue that is controversial. Oh, yeah. Not to have you weighted to every issue of controversy, but I mean, one of the things I really didn't like or I thought was lame about Twitter...
is when you had very powerful people that would block people that they disagreed with. And I was like, that's not kind of not what it's for. Mark Andreessen, the blocking functionality is not so you can block me. Sorry. Not to overly personalize it, but I've always been like... I've always wanted to be like, Hey, so I find Mark Andreessen, you get to block little old me.
I also like, I kind of want to follow everyone that you're blocking. So, cause I think I would like, like them a lot. So I just want to like, have a feed of people that Mark Andreessen blocked and that requires public block list.
I'm in the brotherhood too. And I have no idea why, but yeah, I'd love to, I'd love to get together for meetups and stuff too.
Oh, but that's not why we, Yeah, I know. So do I get kicked out? Yeah, you're out. Of course you're out. Can't you read the sign? Get out of here.
Well, we didn't make them public for that reason. I'll start. But actually, the blocks are, mutes are private, blocks are public. The reason that is, is actually kind of a limitation of the design of the protocol because you need the blocks to be public for the applications and really all applications to be able to respect them.
And so you can do the mutes privately because only the application needs to know. It's just filtering things to the recipient, but the blocks kind of had to get broadcast.
And you could come up with a privacy scheme for that where you just give the blocks to the applications in the world, but there's no well-defined set of applications without creating some kind of a organization which has like membership. And at that point, you're essentially starting to create a cartel around the protocol.
And maybe that's where all this goes someday, but for now we're not about to start that action. So we just decided, you know what, we can't find a better way to do blocks. They're going to have to be public. So that's why they're public. That said, there's a lot of other interesting things about blocks and the fact that we implemented what we call moderation lists.
So these are lists that you can mute or block. implemented that pretty early on, again, as a part of the application's kind of decentralized moderation. And we also made blocks really aggressive. Sometimes people call them the nuclear block. Where not only do they sort of like in Twitter, as I recall, it would sort of freeze the interactions between you two.
So you would no longer see each other's stuff, but like your previous interactions would just kind of be frozen in time and public. On Blue Sky, it actually causes all interactions previously between you to also essentially go away. They just show content not available.
And so we actually have a very sort of really aggressive toolkit for blocking, and we're encouraging people to put together block lists and to subscribe to them. And again, that kind of gets into the point of view that we have, which is just that there are just some people who can't coexist very well. And I don't think trying to force them to do that is a great idea.
I think that's why you get so much toxicity in social networks. And it felt even more pressing when we're creating this open design and sort of like
loosening up the control that we have over participation, we wanted a way to articulate really aggressively, almost like your own perimeter, since the website is no longer a perimeter for the social network, being able to say, no, this is kind of a group that I'm keen to be hanging out with. And if you're in this list, I'm not, I just don't want to be in the same public space as you.
And it comes with pros and cons, you know, I won't say that it's like a pure win. Sometimes it can be really frustrating for people, I'm quite sure. But I think it's important that people are able to make that choice for themselves. I don't really like the idea that people should be forced to share space with each other. So that's the mentality.
And I mean, obviously you've seen that, I mean, as the popularity and it's just, it must be still a bit surreal to watch the popularity of blue sky really stumble into like, I mean, not stumble, but really become, I mean, like for whatever reason, Adam, you said ESPN has got a blue sky account now. I mean, I just felt like, wow, that is a blue sky has really arrived. Um,
But presumably with that has become, I mean, you get a lot of people who are trolls in the kind of the oldest sense of really deliberately wanting to get a rise out of people who are now attracted to the social network because of its very success. I mean, I assume, is that fair that you've seen more users, more problems, I assume? Oh, yeah. Oh, yeah.
Yeah. Yeah. Now we're attracting raids, right? Like, the persistent, automated, like, harassment campaigns. And so we've had to really aggressively scale up our tooling to deal with that. And, you know, that is... Thankfully, we designed the whole system to be prepared for that. We just had to get the tooling in place. But then, you know, on a long enough timeline, we all know the game.
Like, the bot fight is hard, and it just got a whole lot harder with AI. And we're starting to putz around with a couple of things that might help with that. But yeah, somebody, some jerks made a... AI bot army that just would disagree a little bit with people. That was their angle.
Oh, my God.
Oh, my God. Thanks a lot, you know?
It's some real stick-out stuff. Like, they know how to make people upset, which is not like a giant, like, fuck you at somebody just to be like, you know, you make good points, but I think you're just a little wrong about this. And it's like, oh, man. Yeah.
Is it possible that their training data is only older siblings? God, I know. Yeah.
they've tapped into something deep in the older sibling brain.
The older siblings, like, I mean, cause older siblings do. And I look, I say this as an older sibling. I mean, you just like, you optimize the least amount of effort for the most amount of, of reaction. And it's like,
yeah i mean it's they're very very good at it um especially you know when the i when my oldest has like the countdown to when my 17 year old is going to crash out is the new term for like going it's just like it's you know uh so i think this is very deep in the human psyche is what i'm saying it's like you're trying to contend with something that is uh very very deep um
And so, I mean, how do you... I mean, you said that the things that you've deployed seem to be so far holding up. Have there been any surprises there in terms of... Or maybe you've just been surprised by the scope and depth of some of these efforts.
Oh, gosh. I wouldn't say it's terribly surprised. I mean, it's discouraging. It definitely makes you go, ah, I wish better of the world, but we all know it's a bit of a dark forest, so...
um no i don't think i'm terribly surprised right now i i'm still i think in the zone of like yeah i think we could do novel things and i'm like uh it's it's only speaking somebody that builds things like it's only once i'm out of ideas completely and like feeling defeat that i'm really like bummed out um but i'm still i still see some moves that we can try so i'm feeling good
Well, that's great. And I mean, I think also, is this where I presumably at proto is helpful where you think like, this is not like your fight alone, that there are other folks that can like start implementing some of their own ideas. I mean, if you, in terms of like getting into the, the, like the, the technical details of how those moderation servers actually work.
Because I think one of the interesting things, the observations you had is that the skill set to like run one's own server and the skill set to moderate don't overlap very much. Yeah. Can you speak to some of the innovation that you're kind of, I mean, are you seeing some innovation there as kind of third parties experiment with different ways to moderate?
A bit, you know, the we are starting to see researchers just sit on the fire hose and there have been a couple of catches they've made that we didn't make. And that's awesome. That's like, you know, that's a great benefit. Hasn't gone like crazy on that front yet because we're we're pretty active in there.
So we're, you know, thankfully, it's not like we're getting a ton of like, oh, my gosh, I had no idea. But we are starting to see some folks do it. In terms of actual application, interestingly, I don't think I've seen anybody deploy a bot labeler that I think is totally crushing it yet. But one of the things that we did put in is the ability to run what we call labelers.
And labelers are essentially moderation services on the network. It is how our moderation works.
uh when somebody else is running it the application privileges our moderation so that like only ours is able to do what the application considers to be a full takedown but the uh other the user run or what we call community labelers are still able to do some pretty direct interventions into the application in terms of hiding and filtering out content and where we do see like really good pickups on that i think or like for instance the um
screenshot labelers for, uh, screenshots from like other social networks. Uh, yeah, that was a good one. Right. Because like, sometimes you really get, I think it's kind of like whenever a cat keeps showing up with a dead bird on the doorstep, you know, to show you the dead bird. So that's dead birding. And, uh, so a lot of, a lot of content from the dead birds coming onto the website.
And, you know, sometimes you're like, you know, I really don't. see that or at least I want like a little warning so I can kind of check in like, am I in the mood for that? So somebody was able to apply just a nice little AI modeler to be able to detect those things and then label them. Um, that is pretty great for improving the experience.
Uh, another one that somebody put out there that I don't think I've, I haven't checked in on it to see how it's doing, but, uh, like an AI slop detector. Yeah. Are you tired of AI-generated images? Well, here's a little labeler that at least, you know, you can filter it or it could just give you a little warning notice so that you could just decide.
Or, you know, if anything, in some ways that kind of works well as just a protection against misinformation. If it's, you know, I assume if the AI model can figure out that it's AI slop, it's probably pretty obvious. But still, you know, it's in the zone. Another one, this is not automated, but actually a decent one is a... uh, a politics labeler.
Um, and this one's spicy cause like, yeah, cause like what gets characterized as politics is, you know, it's a little bit, a little bit spicy, but, um, definitely not the kind of thing that we as a company would necessarily want to be doing, uh, for, because it is spicy, but for somebody else out there to be doing their best, you know, like, yeah, we think this is kind of political and like, it's not actually, you can have it just like completely hide this stuff, but honestly just having it behind a little interstitial.
And so you could just decide, do I have really want to read something about the election going on right now. People need a break. You know what I mean? So being able to just have this kind of thing to, you know what? I need a break. I'm turning this on. Let's cool down the politics for a little bit. That's the kind of stuff that I get pretty excited about because these things do psychic damage.
So any tools to kind of reduce it.
Totally. So how do these labels work? Where are the labels actually stored?
Yeah, that works on actually a completely separate synchronization. So we have two, I guess, main kind of data structures in the ad protocol. The first one is what we call the repositories. Those are what represent users, and that's where all your data goes. So repository, like a Git repository, but in this case for like JSON. And then we have the labelers and the labels.
And so those exist kind of in their own system. We didn't put them in repositories because it's kind of a giant long list of just strings. So it does not work well with the cryptographic structure that resides underneath. The repositories are a variation of a Merkle tree. And we, with the labels, were like, you know, that's just not going to work.
So we just decided to have them in their own system. So they kind of synchronize between the labeler and any kind of services that are interested in subscribing to them in a slightly dumber way, to be honest. Yeah, that's it. That's how that works.
So our app view, our application, whenever a user subscribes to a labeler, our application, if it isn't already subscribed to the labels list, we'll find it and start syncing it. And then we do a little runtime join based on your currently... Actually, it's a header in the request. We look at a header of which labelers you want to apply labels for, and then we just pull them in and attach them
through the response, and then the client observed them based on the user settings. And that's it. I mean, a label really is just a label. It's just a string saying, hey, this is like NSFW, or this is like a spider. Are you arachnophobic? Great. Don't show the spider. That's about it.
And then, so I subscribe, though, to these labeling services. So I get to choose, like, if there's a label that I really like, I can subscribe to that. Is that right?
Yeah. Yeah, the way they, right now, the way that it feels in the application, I wouldn't say that we have, like, nailed this UX. Maybe. But the way they work is they're kind of like special accounts. Like you can kind of upgrade an account into becoming a labeler, at which point the labels that it offers get turned into the primary profile screen.
And rather than the primary action being follow, you know the primary action is subscribe, and you subscribe to it. It gets added to your moderation screen, and you then configure each of the labels and how they're supposed to operate.
Well, that UX definitely doesn't clutter anything else in that it is definitely... That's great. I actually did not know this at all. And going to, like, someone in the chat had linked to the U.S. government contributions labeler. Yeah, I think so. And this seems like, I mean, this seems like a really good experiment.
Because then, of course, I want to know, okay, what are the labelers that Adam subscribes to? Because I may want those same labelers, or maybe those are labelers just to avoid, because, you know, that guy, I don't know. I mean, he just kicked me out of the Blocked by P. Marker Club because I'm no longer blocked by P. Marker. So I don't know. I've got kind of some axes to grind over there.
But I mean, it just feels like you're giving people kind of another tool to go experiment with this problem that you did not create called bad humans. But also like good humans that actually want to come in and help offer some clarity on some of this stuff. I don't know. I think it's really... To me, it's really inspiring.
I know it's kind of strange to be inspired by the encroaching 8chan of the internet, but I really do think it's inspiring.
Well, I hope that we can get there as far as we want to with it. I think there's a lot of good things that are happening with it, and there are aspects of it that I'm really satisfied with. I will say that the labelers so far have been a bit of a mixed bag in terms of their practical reality because...
It's pretty hard to run one for the non-trivial use cases if you're trying to actually do what you might classically consider to be moderation. I think that part's been a little bit tough, and that is one of the things that we've been sort of sitting back to evaluate before we kind of push on recommending them more, is to see how this is affecting communities.
And I don't think I would give that one 10 out of 10 marks. we're hearing in some cases it's gone pretty well, but in some cases we're hearing that it's a pretty stressful position. There's a lot of responsibility that comes along with it. And, uh, and I don't know if they're empowered in all the ways they need to be to handle that responsibility or the expectations that come along with it.
So, yeah, I don't want to, I don't want to, uh, walk in here and, and ignore that. You know what I'm saying? Like, I do want to cop to that. It's, it's not quite 100%, I think at the place that I'm, I would love for it to be, but I, And I'm glad that we're giving it a shot, you know what I mean? And I'm glad that we're learning off of it.
And I think you'll probably see more updates to it based on some of the challenges that we've had over the coming year.
Well, that's exactly it. I feel like it's the giving it a shot bit, and I feel that learning from so many previous experiences, and then also doing it all in the open, because this is all... I mean, you're all open source, right? In addition to having these open specifications, you've got... all or much of your software is actually out there in the open.
Yeah, I would actually like for it to be more of it. There's still a couple of things that we've still got closed, like our main algorithm is still closed. We'd love to get that open at some point. We have some of our backend that's, you know, you can get the backend running, but it's not to the 100% ready-to-go place.
And so I'd love to get all that totally open and in a good package so that people can run it. But... a lot of it is open source. It was really fun to be able to share, you know, when we got to the, we got, we managed to hit, you know, number one in the app store, really proud moment. And that's awesome. I was able to share the source code and say, here it is. Here's the source code.
If you want to do it, you know, have one yourself. So that I'm really proud of that. You know what I mean? Like that we were able to take an open source application to this place. And I really liked that. We have stayed with that throughout.
That is awesome, and congratulations on that. That is really terrific. Close to without precedent, I assume? Certainly for a social app, it feels like it's without precedent.
I'm going to Steve Jobs it and say, yes, it's never happened before.
This is the very first time. I was just going to ask you, you've obviously got a lot of challenges still ahead, in part because of the popularity. What have been some of the surprises along the way of things that maybe have worked better than you thought, or worked much worse than you thought?
I don't know, maybe you were putting scenes into that category, at least in terms of nomenclature, but what are What are some of the things that you were or are there things that you've been kind of surprised by along the way?
Yeah, I may only restrict this to the things that I'm proud of, to be honest. But one, I'll say the one that comes to mind that actually worked so much better than I ever expected was the domain names for usernames. I'm glad to bring that up. Yeah, we put that in there, honestly, for a very pragmatic reason. We just needed a namespace that was decentralized, right?
And which would work for the open network and which had the affordances that we needed. And so I was like, well, okay, domains, right? What else are you going to use?
Yeah.
I mean, there is something kind of cool about that. Believe it or not, I do really like... You could just sort of turn the host into a pill and then just have the username, and you could do maybe some nice things with the UX, but anyway...
I'm glad we made this choice where I was really shocked when it came to the domain names was the uptake of the custom domain names and how that has played out has been way beyond expectations. Cause when we did that, we would put together like little mock-ups with like the Washington post using that as their handle.
And it was kind of like the, again, it was one of those things that was like, yeah, right. You know, someday maybe, you know, who knows? And then they did like when ESPN came on, they used ESPN.com. Yeah, that is so cool. I can't believe that these organizations are going to the trouble of configuring DNS for us.
Well, it was even better than that. It was when ESPN came on, they were ESPN.bsky.social for like a hot second. And literally their first post was like, no, no, no, we know, we know, we know, we're changing. I mean, everyone is dogpiling them being like, come on, you got to change the domain name. You're ESPN.com. Yeah, yeah.
I thought it was cool to use the domain names, but at one point it really hit me. I don't remember which account it was, but during the massive uptick in growth, people join and you always wonder if it's really them or not. And I remember seeing a post that was by the domain name of some news organization or something probably, but just being like, oh, I know that's their account.
And that's actually really, really cool. And obviously... I don't know how lay people, you know, know about that kind of thing or whatever, but it definitely like really, there was a specific moment where it really sunk in for me as like, this is more than just, you know, piggybacking off of another internet protocol to, you know, name things.
That is a very elegant solution to the problem. And I mean, and people are obviously calling out like the fact that you get, because you're getting this verification basically, and you're verifying it the way we verify the internet, which is like, why wouldn't you do it that way?
Yeah. Yeah. I mean, I, again, that wasn't like the driving decider on that, right. It was really, Oh, well we need something, you know, but then I think, you know, it does, it has helped us a lot. I think we're probably going to need something more both because it is a little bit wonky there.
You know, I don't want to move too quickly on that, on that aspect of it, but there's definitely some folks that may not totally understand it, but then you also just have folks that don't have like high reputation and, domain names. A good example of that would be Flavor Flav came on a bit ago, and it was so enthusiastic, love it, and he doesn't have a website. He just doesn't.
You also start to run into the, okay, he could set one up. He could set up FlavorFlav.com, but then how do you know that that's the right one versus FlavorFlav.net or .io or whatever? There's going to probably need to be some layer in there at some point, but no matter what it is that we do, The well-known domain names are a really fantastic anchor for it.
And boy, is it great for the government stuff because they've all started to use like .senate.gov and .house.gov. Like Tim Walls just came on and became, I believe, governor.mn.gov. Just, you know... Totally, totally solving that problem for some of the more high stakes ones, which is political leaders. And those are under that tight governance, right?
Like the .gov domain name is the US government highly regimented. And it's so much better than anything that we as an organization could be doing. So definitely some wins in there.
Big win, I think. Can I ask you just a very pedestrian question? Because this is the only thing that has kept me, so I'm currently bcancelbsky.social, but I definitely want to, I actually, I don't know if I want to bcancel.dtrace.org or bcancel.oxide.computer. So this is my own thing I need to go grapple with.
I'm sure Paul can tell you this, but you could move it. You could change it later, right? This is one of the frustrations with the Fediverse and one of the liberating things about Blue Sky.
You can, but the only thing that's hung me up is that I need to go immediately squat on my old name, right?
Oh, I got good news. We fixed that. We just fixed that. So yeah, yeah, yeah, yeah. We, we haven't done clams on it yet, but we, um, are, uh, planning to, to work scrunch together those columns. But yeah, um, you now, if you have a piece of it, my friend, It's out.
Actually, ironically, somebody ran into it in the reverse because they were doing the squatting move that people have been doing as a patch. They're like, I can't register it. No, no, it's working right. What it does now is we just keep in our database of the BSky.social stuff. We just remember when somebody moves off what your original one was. You can go back to it.
If you were the original holder of it, you can go back. Nobody else can, though. And so, and I guess if you delete your account, then that gets relinquished. But yeah, yeah. So that is now fixed. Go wherever you like.
Oh my gosh, that is great. That is terrific. And I'm sorry to disrupt your comms plan on that. Thank you very much for breaking news here on Oxide and Friends. That's just being handed to me. That's terrific. And one of the things that you all are doing so, so, so well, and I just cannot thank you for enough is
is the degree to which you're listening to people who use the product and you're using the product yourselves. It feels basic, but it's amazing how often this doesn't happen. That's a great example. I know you've got so many... terrific ideas for things that can be done.
But that's just like, that's a really kind of like a basic, simple kind of pain point that is not debilitating, but boy, thank you so much for, for fixing it. I think it's just, it's part of what makes it as a user, a blue sky. It makes me so enthusiastic for the future of this platform and, One thing I do, I won't be mindful of time because I know Adam's going to have to bolt here.
His no longer toddler is now, is that kid in high school yet? First grade? What is he in? Second grade. Second grade. God, yeah, built like a high schooler. But there's one last feature I wanted to ask you about, and that is starter packs. Um, because another thing that was, I know is basic. Uh, it was really important.
I feel, um, could you just describe a little bit like kind of the, the, the thinking behind starter packs?
Okay. This one's pretty fun for a couple of reasons. Um, so first of all, let me start with the context of how that got built. We, um, The full context, you have to start with the fact that we are not an algorithm company. And so our initial algorithm that people are getting landed in, we're working on it all the time and we're making some great progress on it.
And I don't want to harsh the team that's working on it. They're really cruising, but it is not great yet. And so new users coming in, it's really important that they are able to get their social graph built so that the following feed could start working for them. That's point number one. Point number two is building your social graph on the app. also a bit challenging.
Getting good recommendations for follows, that is quite hard, especially when you're seeding communities. Like, here's a design challenge for you. Make a good recommender when you don't have social clusters on there yet. How do you even evaluate if this thing is doing its job correctly? You kind of can't do it. Like, we're learning a lot about algorithm design without content to work against.
Thankfully, that is less a problem now, but that was the kind of... We had a real chicken and egg problem. Okay, your obvious thinking is...
yeah so so okay so you're obviously thinking you're you've got this design problem well let's do like a contact import of some kind either from an existing social network or like from the address book well you can't do the address book because that has privacy problems that may not be fixable as far as i know last i heard you can't come up with an anonymized upload your address book thing without creating triangulation of people's contacts so we just said hard no we're just not comfortable with that we're not there's a
dirty tactic anyway i think that is so damn dirty i mean i agree yeah well and like it gives an ick vibe that like anytime i get up to a nap now and i think everybody has it you know he's like oh no i'm not gonna give you my contact actually get out of there no don't even ask it's like i don't want to peek like no no peaking gross get out of here yeah
It's a real shame. Now that I'm on the other side of that, I get why people go ahead and do it, but we still want to cross that line. It is a shame because we're sitting there going like, we're not trying to be jerks. We're actually trying to help you, but here you are. We didn't cook up some kind of breakthrough on the privacy solution there, so we didn't bother. So, okay, that's out.
Well, then you can get into, can you try to get the social graph from another social network? And the reality is none of them want to give that up. And we're really in an era where like these APIs are getting closed down. You can't do it off of the meta products and you sure as hell can't do it off of Twitter. I guarantee I know how that would go. So we, that was a non-starter too, right?
So we're sitting around going, how the heck do we solve this problem of getting people's social graphs seeded? The original idea for starter packs was actually not just to have like a list of people to follow. It was actually to create an invite link because we have had a lot of success seeding the social network back when we were invite only.
And ironically, when we switched to the public thing, we lost one of the best organic mechanisms for people to get worked into their social graphs whenever they joined. So we were looking at that and asking, how can we make that better? And the initial impetus for this was actually just to create an invite link that you could share with friends.
They would take them through the install flow and then get you into a bi-directional follow relationship with whoever it was that invited you. And that would help seed the network. And then we were like, OK, well, let's go ahead and add in suggested follows and some suggested feeds and things like that so that you can actually hydrate in the experience.
Well, we launched this thing and we also included the ability to just like follow all of them from within the application. So it would work also if you were an existing user, but that was just kind of like, yeah, just in case kind of thing. We launched starter packs and people were like, oh yeah, that's really cool. But like, they just didn't get used.
So we ended up spending like two months going like, ah, there was so much promise with starter packs, but that really didn't land with folks. And then the big influx came in November and man, did they pop off. They ended up becoming the absolute most potent mechanism for getting people onboarded into the application. And again, rarely through the install flow.
it's very rare that people were using like clicking into them and then like kind of having the app and we would use an app clip to like do the install flow and then get you creating your account stuff like that that was very really it was almost always people just sharing them in uh in feeds or like installing the app first and like you know messaging each other with these things or however it was they were doing it and that ended up helping the social graphs and all the clusters form really rapidly and get people's following feeds into an interesting place
They ended up being really strong proxies to topic interest, which is an area that I think we're still pretty weak on, is giving new users a fast access to a topic. So you would find like a journalism starter pack or an art starter pack, and that would... Boom, now your following feed has given you some great stuff on that front. So...
Yeah, it was not a linear, like a A to B thing with starter packs. The core idea, I think maybe there was, but in terms of like how it got used and what we thought we were going to be pulling off of it, it had some twists and turns.
And for whatever it's worth, just my own personal experience with starter packs, I was trying to get... I'm an Oakland A's fan. The hashtag sell the team. And we kind of got this now beleaguered lost tribe that is still very much on Twitter, was very much on Twitter.
And talking to the folks who are kind of like the leaders of that community being like, we've got to get one of the starter pack because the entire community wanted to leave, but you kind of needed everyone to go at once. Yes. And that is what starter packs – starter packs were a lifeboat for people who wanted to – and I saw this in various scientific communities.
I saw it in certainly in Oakland A's Twitter where it's like, okay, this is now the opportunity. And I think actually another thing that I love about just like the dynamic was people would start a starter pack with 60 people. And those people would kind of come on and the person who started the starter pack would always say, hey, like, let me know if I forgot anybody.
And you've got like a bunch of people raising their hand being like, oh, God, I'd love to be included in that starter pack. And they ended up being very inclusive and uplifting and a great way to move communities across. So it doesn't surprise me because in that multi-month gap where you're like, why isn't this working? Yeah.
I, and I'm sure other people in various other communities are kind of lobbying people who are still on Twitter to be like, no, like let's get one of these together so we can get everyone over at once. Um, yeah, it's been really great. So thank you very much for that. It's a good, it's been a, it's been a great feature.
Oh, really happy to hear it. Yeah.
And just again, thank you. And thank you, Paul, for taking all the time with us tonight. And we are huge fans. Love what you're doing, what you and the team have done. Love that.
I mean, we feel like very much kindred spirits and that like this is a problem that you all have as a team have been thinking about for many, many, many years and watching all of that wisdom get expressed in what you've done. And you've created something. I know it's challenging. And the more people you get in there, there's going to be challenges that are going to arise with that.
But boy, what you're doing is so important and special. And we're really deeply, deeply, deeply appreciative. You've given, I think, a lot of people, certainly me anyway, real hope in what we can have out of social networking. So really deeply appreciated.
Well, thanks. I mean, I got to be honest, that level of supportiveness that you're showing here really does help at any given moment. It's been really nice to see people get that level of interest in it, too. So the feeling is very much mutual. When it gets tough, the folks that are there and like, yes, let's keep moving, that kind of buoys us up for those tough points, too.
Absolutely. Well, and I'm sure again, humans are tough, so more challenges to come, but really appreciate it. And thank you again for taking the time to come here. And we'll be glad that no one has to hit up Klavnik for invites anymore. Everyone can hop on there.
And Adam dropped an Oxide computer starter pack if you want to follow folks who are, we got a lot of folks from Oxide, obviously, who are over there. Paul, thanks again. Thank you very much. Um, welcome back anytime, obviously. Um, and, uh, keep up the great work. We'll be, um, we'll be there and in the thick of it. So thanks again. Sounds great. Thanks for having me. All right. Thanks everybody.
And next time we're going to be doing our, our year end wrap up. So, uh, bring your, your, your, your best and worst from, uh, from 2024. Um, so Adam looking forward to that. It's going to be fun. Should be great. All right. Thanks, everyone. See you next time.