Paul Frazee
Appearances
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. There's a... There are going to be some interesting learnings about the economics of this over time, to be sure. So I don't want to be walking around as if that's not a big reality about all this. That said, it is a lot.
Oxide and Friends
Scaling Bluesky with Paul Frazee
There is a version of the Firehose that we offer that is a kind of a reduced form that drops all of the cryptographic proofs to make life a little bit easier and allows you to do filters on it and things like that. But one thing to also remember is that the Firehose is kind of a convenience service, to be honest. So if you, like I said before, it works kind of like the web.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You broadcast or publish these records, and then those flow into the applications to be processed into the views. The Relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the world and kind of goes ahead and grabs their replication streams and puts them into one stream for you.
Oxide and Friends
Scaling Bluesky with Paul Frazee
If we decide at some point what we can't afford, like all the consumers on this thing, you can run your own relay and directly do those crawls. So again, we're trying to make sure that there is openness to the system so that if there gets to a point where we're like, you know what, we can't afford passing this along to everybody, somebody else can do it. You can directly do it yourself.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Totally. Yeah, that's totally inbounds. Let's do it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, it does form the basis for building applications beyond just the kind of Twitter style of application. So I'm pretty excited about that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You just kind of nailed what was on my mind as you were talking about us scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out. And in fact, Martin Kleppman, who actually helped us with that paper quite a bit, really helped drive that paper. He's been consulting with us from the beginning.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And that was a huge asset, an incredible win. I was... when we first got him i was over the moon um that's the kind of you know he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications these kind of high-scale data heavy applications and so we were able to really apply
Oxide and Friends
Scaling Bluesky with Paul Frazee
just image everything the industry had learned about how to do this sort of stuff both to like our internal systems and also just to the protocol since scaling was always a big part of what we were trying to do we wanted this to be able to be at the scale that you expect out of these kinds of applications so that i think you know that plus just a really great team that you know did some great work to keep those servers running throughout all that growth that's that's how this
Oxide and Friends
Scaling Bluesky with Paul Frazee
Totally, 100%. Yeah, just everything, every moment. No, I mean, I feel really pretty fortunate across the board. And honestly, when we were starting out the project, we were originally just a protocol consultancy for Twitter, which is a complicated and really interesting history.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But when we started on this thing, we didn't expect there to be either the relevance or the market opportunity to actually have this thing go into production quite like it did. So every step of the way has been really surprising on that front and exciting. But when we first started to do the beta and open it up,
Oxide and Friends
Scaling Bluesky with Paul Frazee
I remember people were starting to come on, and this was, again, like, I don't know, the third, probably more like the fourth time I had launched a product in the decentralization space. And up until then, none of them had worked. So we had these users coming on. I was like, okay, cool. You know, they'll be here for a couple of days, then they'll leave.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You know, like I had really, really learned that, like, yeah, everything you do fails, of course, you know.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, exactly. Like that's when they die. Yeah, exactly. But no, they stuck around. And it was like, that was the moment that really shook me the most. It was like, oh my God, they came back today. And it's been kind of that level of surprise ever since.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I mean, so, like, everybody on the team has been at this for years. So, like, collectively, I think we have 20 to 25 years of, like, decentralization work, like, leading into the team. Skullbutt was my first one. That was back in 2012-ish. And that was a technology invented by Dominic Tarr. And...
Oxide and Friends
Scaling Bluesky with Paul Frazee
That's a good lie. I'm going to try to give an answer that doesn't blow that up.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, no, so that's actually not an unfair characterization. Everything moves so much faster than we were ready for. We started working on this protocol in 2022 at the beginning of the year, and then by the October of 2022, we were starting to realize, and you may understand the timing if you can play back everything that happened.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So by then we were going, well, we're going to need to make an app. And suddenly we were productionizing the protocol, which the org wasn't designed to do. We were not prepared really for that. And so we were kind of initially, we did have a client, I think, already going, but it was really just like a test bed to make sure that the technology worked.
Oxide and Friends
Scaling Bluesky with Paul Frazee
We suddenly were hitting the gas to get features in there. And get the servers to a place where they could start to accept users. And then once we started to have them on, you know, the single Postgres era lasted, you know, for a lot of that private data, right? So there was a... Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh, if you don't start that way, you're doing it wrong. You definitely should. Because the transition after that is into the ScyllaDB and the event pipeline era where the engineering gets way harder. So you only want to do that once you actually are scaling. But that's what we had to transition to.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So by the time we're in the fall of 23, we're looking at this and going, we got to stop everything and re-architect this thing or we're not going to be able to handle this growth. It was just the demand was too high. One thing I know that Jay would want me to clarify about the invite period is that that was always... There were some people who were like, it was about exclusivity.
Oxide and Friends
Scaling Bluesky with Paul Frazee
It was not about exclusivity. We were more trying to make it like a hip thing. It was very much just that the servers could not handle the load and we needed to get our TNS team scaled up. And that stuff takes time. We just weren't ready. So the demand was way higher than our supply. and that's what we were able to buy ourselves by doing that in my only period.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And, but you know, we moved as fast as we could and it wasn't until February of 23, 23, 24 that we could, yeah. Right. You're in the post in, in the, um, yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
it could fit into actually all of us came everybody on the blue sky team came from like the p2p world right which was kind of like hey could you take some of the techniques from bittorrent and then do some modifications and actually try to build sort of real time or or large scale or social applications depending on what you were up to using those kinds of p2p techniques and secure skull that was directly geared towards social networking so it was a peer-to-peer social network
Oxide and Friends
Scaling Bluesky with Paul Frazee
That is completely correct. Yeah. And it's actually, well, first of all, this is all pretty integrated. So the PBC specifically, what that does is it gives us space to pursue a mission in addition to the profit motive. And so the mission is to create this decentralized protocol for public conversation and for applications related to it, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And that's actually just remains like the main goal, product market fit, scaling of having a profit come in so that we can sustain the organization that is still all in service of that mission. It's just the vehicle that we've chosen to do it by.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But that, you know, was something that when we were sitting down to attack this project, we had all realized like, okay, we are all trying to get a change in how technology works, but what's your theory of change? How do you expect to reach the market? And the way you do that is with a product.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So all the mechanisms that come along with the startup are really, we try to maintain that that is the vehicle by which we're doing, accomplishing this mission. That's not the mission itself. And either the discipline around like the scaling side of it or the moderation side, that was tough. But the other one that we maintained was making sure that the protocol was consistent
Oxide and Friends
Scaling Bluesky with Paul Frazee
sufficiently out of 1.0 that like, at least the self hosting part of it was active before we launched, we did not want to go into the public launch prior to having that set up because we wanted to make sure that we hit those targets for what. So that was another thing that we delayed launch for to make sure that we were actually acting on the protocol. And it's kind of always that balance.
Oxide and Friends
Scaling Bluesky with Paul Frazee
No, we had no guarantees of what was going to happen, for sure. And there was certainly – we had the – it would be a lot if I said we didn't have that anxiety in the conversations about – making sure that we're capitalizing on opportunities, but I would never say that the protocol or mission focus was ever in question.
Oxide and Friends
Scaling Bluesky with Paul Frazee
That was just something that this team never really is willing to throw overboard. You do get into some complicated conversations about capacity assignment towards protocol work versus product work. And there are definitely some times where we're kind of looking at the quarter and going like, okay, what are we gonna focus on? In fact, a lot of last summer,
Oxide and Friends
Scaling Bluesky with Paul Frazee
We were sitting there and going, you know, there's just a lot of features that are missing if we want to be, you know, what users are looking for. We just had a lot of requests and things that were absent. And one area that we did, I think...
Oxide and Friends
Scaling Bluesky with Paul Frazee
you know make a i i believe defensively pragmatic choice was with dms we knew that not having dms was killing us and the protocol is not ready for dms it's really geared towards public posting at the moment so we're like all right we're going to give ourselves a cheat on the dms we'll go back and clean that up later but for all of the summer yeah that was like a
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it had this very aggressive kind of local first mentality. It was called Secure Scuttlebutt because actually it was based on a Scuttlebutt gossip protocol where you're just like having each node kind of like rebroadcast logs to each other in a kind of best effort way.
Oxide and Friends
Scaling Bluesky with Paul Frazee
we're going to make incremental progress. We're going to do a little stuff on the protocol, but we really got to get this product shipped up because if it's not working, then again, that like, then the protocol may not get its chance. Right. And I think we're in a better place. Yeah. I think we're in a better place now.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I was going to do it on Blue Sky because I'm not sure our DMs are bot-free.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And that meant that it had a very fluid topology of connecting together, like any time you were able to catch up from one node that you were able to connect to, you'd be able to, which makes it actually quite ideal for even extreme cases like a sneaker net if you were so inclined.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And then what was sort of interesting about it was that we merged together the gossip protocol and the social layer where it would use your follow graph to decide which account logs to synchronize.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, yeah. It is an interesting area of the product that actually we're going to be investing even more in the future. I'm really excited to get into it more. Yeah, so like feeds, gosh, the basic idea with feeds was just how could we, I'll tell you the real mentality to it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
We knew going in that we liked some of the ideas about like improving how there's back pressure to the organization beyond just at the really kind of systemic level. Because the entire protocol that we've created, it makes it possible to make alternative applications and have alternative hosting, but those aren't super tangible in the day-to-day to people's lives.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we sat down early on and asked, okay, well, what could we do with this that would allow people to more tangibly experience an open network and decentralize what is meaningful to the average person? And so then the two answers that came up were, could we decentralize moderation in some way, and could we decentralize algorithms in some way?
Oxide and Friends
Scaling Bluesky with Paul Frazee
So if I was following five people, the default kind of intuition there is whenever I would connect to a node on the system, I would ask those nodes like, okay, here are the five feeds that I'm following. Can you like catch me up on anything that you have for them? And since you're connecting sort of like to a forest of different nodes, you would...
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so the feeds came out of that and the feeds, I think that they can, there are ways that I think we can improve on them and make them even kind of more central to the experience and really get the most juice out of them. But I will say that from a kind of like validating the concept perspective, man, they totally struck gold right from the get-go.
Oxide and Friends
Scaling Bluesky with Paul Frazee
yeah and i'm really pleased about that it's actually it's really fun how they work they are hosted on other people's servers they do not run on our systems um but they feel integrated into the app and with that we kind of invented this whole principle that we call third party is first party with this idea of being able to integrate in systems
Oxide and Friends
Scaling Bluesky with Paul Frazee
posted outside of the application, but make them feel native. And so the way they work is the simple answer is that you contact the feed server and then it just sends back a list of URLs. And then those list of URLs get hydrated into the feed.
Oxide and Friends
Scaling Bluesky with Paul Frazee
the way they have those urls is they tap into that firehose and they use whatever kind of algorithm that they want to use to decide uh which posts to select uh to make the feed and that ends up being a really nice compositional uh boundary for uh that thing they can use any kind of logic they want from social graph you know queries to computing ranking off of likes and reposts and
Oxide and Friends
Scaling Bluesky with Paul Frazee
other kinds of signals, you can get, you know, text queries running in there, which a lot of them are a lot of them are kind of operating off of like a smart hashtag in a way where they're just looking for particular topics that are being discussed. Honestly, you can get really far with just social graph limitations plus text query, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Like the basic object being these are these are people that are experts in this topic. And here's the topic and let's get a feed going.
Oxide and Friends
Scaling Bluesky with Paul Frazee
presumably you know at some point catch up to everything in a kind of eventually consistent way and uh we actually would take it further where you would do friend of a friend expansion so you would actually ask for like the five people you're following plus like two hops out if i remember right Really sort of wild way to do it, but it was aggressively decentralized.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I got, I got, I got a timeout and give like a, you know, I'm glad Mastodon is there pursuing the same mission. I'm just going to say that. And then we could move on.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I mean, I have to agree. I mean, there are also times where I'm just kind of like, you know, today I'm in the mood for a more quiet thing, you know, and that's, I've got feeds for that, you know, or today I'm actually like voraciously consuming the stuff. I read all my fees and now I'm going over to the one that shows me more of the fire hosey kind of experience.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Like just whatever you're vibing on that you can, you can get there. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
In fact, I would call it an anarchy, and not in like a pejorative sense, but like quite literally no authorities were encoded in the system. Interesting.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I mean, I wish I could say it was like, aha, you know, like we're in a way. I will say from a technical point of view, it felt really good. Actually, this is we're kind of talking about like the experience of like the engineering side, the product side. I'll just actually tell the whole story. Like we initially did not do custom feeds. Initially, we did something called scenes.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Scenes were like a hybrid between a community and a custom feed because we knew what we were aiming for, but we didn't quite have it right. And the way that a scene worked was that the members of a scene, when some number of them liked a post, it would trend automatically within the scene.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So you can kind of see there's some approximate thinking there and you would actually get like a notification whenever your post trended within a scene. And we obviously didn't think this through very well because people would create scenes of like three people. And so then suddenly you're getting like your post is trending, like the same two people would like a
Oxide and Friends
Scaling Bluesky with Paul Frazee
post and then he would trend in five different scenes that they're all in with just three people in it. So that wasn't brilliant. But the other thing that also jammed that up completely was nobody understood this idea. It was like weirdly like both a community and not a community. And people were like, how do I post into it? You don't, no, you don't do that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
It just shows things that you're liking. So like we did that first. And I are probably the very many people seeing the product. We pulled it out. We just like, this is quite terrible. We pulled it out and we brought in the custom feeds thing as an alternative. I think in many ways it was kind of taking a bigger swing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
on a technical level because it is, yeah, because it's, you know, it's a more sort of nuanced, like we were at that point doing a at runtime request out to a third-party service, which in and of itself, I think, is going to be an interesting thing to kind of see how that plays out because, you know,
Oxide and Friends
Scaling Bluesky with Paul Frazee
Anytime you're doing that, you got to talking about like out to these services that are not under, you know, we have no SLA with them or anything like that. They're like, how's the performance going to hold up things like that. So far, it seems to be actually working. But that definitely was a little bit more of a swing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And then once we got it in and saw it working on a kind of design of the technology side, we were like, yeah, okay, this feels pretty great. And that side of it, the compositional side, that felt good right from the get-go. I think we still have some open questions about how we can really make the experience of using them as good as we want them to be. We have this like...
Oxide and Friends
Scaling Bluesky with Paul Frazee
sticky problem around the management of them where there's more feeds that are interesting than you can stick onto your home tabs. And how do we find the right way to surface them to like easily like, oh, right, right. I've got this great feed over here. I want to go to it, especially on mobile. So we're still, I think, picking through the right UX to really make these things shine.
Oxide and Friends
Scaling Bluesky with Paul Frazee
But I would say that somewhere around maybe a year ago, we were starting to kind of realize like, no, this is like pretty solid though. Like even though we had gripes about it, we were pretty happy with the outcomes that we were getting out of them.
Oxide and Friends
Scaling Bluesky with Paul Frazee
It was 2012. So I can say that that vibe at the time was correlated probably to two different things. For one, it was pretty clear by then that the major social players were sucking all of the oxygen out of the room. And so there was a collective of open source hackers who were feeling really excluded from what was exciting about social computing.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, yeah. Every time somebody comes in with... There are two critiques that feel really personal. And they're not really critiques, observations. And that is that it's elder millennials and then it's theater kids. And seeing as I'm both of them, every time I'm like...
Oxide and Friends
Scaling Bluesky with Paul Frazee
You know, so to your point, our biggest problem is that we have too many ideas about what we want to do with them. And it's hard to pick about like where to take it next. We finally started to make some progress once we like broke it down into like bite-sized chunks. But yeah, I agree. There is a lot of really cool ideas that we're kicking around about what to do with feeds.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You do have to think about that stuff, especially when you have other people making this thing. You want to think about what the incentives are. If they can make money, that'd be great. We want to make sure that the whole thing sustains itself.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, this one is hard, and it's going to be probably a very, very long time before I'm willing to say, yeah, we totally nailed that one. There's going to be a lot of trying different things to find out where it can go. We've gone, I think, we've taken some interesting steps
Oxide and Friends
Scaling Bluesky with Paul Frazee
and wanting to be able to get in there so you have a general the through line throughout all these projects has always been like frustration with monopoly and feeling like you're not able to get in there and make meaningful change because we're talking about internet technology we're all programmers let's get in there like let's have that Linux philosophy or something like that being applied and so that bugged all of us and that was a big part of it and then you know you also had um
Oxide and Friends
Scaling Bluesky with Paul Frazee
stabs at it and i'll kind of get through uh all that in a second but the uh but i think there's probably many more uh things that we have to play with probably speaking the core of it i think the thing you just got to realize is that you really you cannot make the world happy with a one-size-fits-all moderation at this point It just can't be done.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I think a big part of it is that people come into the world with different values and expectations about how those values should be upheld. And I think it's very painful to see those values not being reflected back in the spaces that you're in.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it's, you know, I think it's quite important that we find a way to get there so that communities can, you know, I kind of really feel this is really at the heart of like why social media can be so painful for in general is that we just haven't found the right way to do this for large scale, you know, public spaces like this and what you might, you know, kind of public square mentality about social networking.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So anything we can do to get us closer to that is, I think, kind of the right sort of challenge to be trying to dive into with this sort of stuff.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Where can you move forward the experience of really the internet at large would be finding a way to create these spaces that I think give people a better experience with that element of it, the moderation of it, in some ways curatorial or editorial side of things.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I mean, that's probably right. I mean, the basic thing is, first of all, we start from the assumption that even if we did come up with a really great execution of an algorithm or moderation or things like that, you have the long term to consider. And do organizations stay effective at what made them successes in the first place?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so coming into this, we have always gone in with that belief that the answer is no, right? That you kind of have to be planning for...
Oxide and Friends
Scaling Bluesky with Paul Frazee
when our organization or any other organization starts to uh wither from its original purpose or or just maybe not be fit for the time anymore and be ready for for like the world to be able to change without too much loss right that's really deeply embedded in everything that we do from the application side of it to the you know the hosting side to the moderation of the algorithms so
Oxide and Friends
Scaling Bluesky with Paul Frazee
Uh, as much as we do our best to try to create good versions of like algorithms and good versions of moderation, uh, we just kind of know that first of all, long-term systemically, you gotta be ready for that to not always be the case. Um, But beyond that, I think it's just quite clear that people have very different tastes when it comes to both algorithms and moderation.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And people have different needs for this stuff. And so how can we get everybody to a place where they are experiencing the life that they want to live on the internet? And that's only going to happen if we start to allow... A company like ours, we're in this kind of challenging position of balancing a lot of different voices. Users don't have the same challenge.
Oxide and Friends
Scaling Bluesky with Paul Frazee
They can just choose to be parts of communities that actually are more connected to what they personally want as opposed to what an organization wants. So we should enable those groups to be able to get that specific with what their communities are like online. That's the basic mentality of it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
It's still, I think even then there was starting to be a little bit of initial disillusionment with the major social companies. Who knows if we were kind of like on the right target with that or not, but we, in general, were starting to, you know, this was like... Not that far after Occupy Wall Street, which was a big animating activist era.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I mean, generally speaking, it's funny, honestly. Like, moderation is a tough job, as you might expect. And things like that... I don't know why nobody else has done it before because, gosh, anytime people can fix the problem for themselves, that actually makes our lives easier. So we have a pretty good incentive to do this sort of stuff.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Removing the quote posts, that one was like, we didn't do it straight away, but that was one that I had personally right from the get-go. Because we all know that the quote dunking is one of the big generators of just some of the most painful experiences on...
Oxide and Friends
Scaling Bluesky with Paul Frazee
twitter at least and i imagine anywhere else that has a similar dynamic and so the detaching a quote post is essentially like it it's it just takes away whatever the quote quoting post is it just detaches it it's just whatever they've quoted it now just says content no longer available and so now you know of course what people do instead is they'll screenshot whatever it was that they originally of course of course they will but that's actually still better because
Oxide and Friends
Scaling Bluesky with Paul Frazee
the things that you got to think about with large-scale social spaces is friction friction matters so much it matters so much and when you can introduce points of friction you're actually able to because it is such a game of numbers you actually will dramatically reduce bad outcomes and If you can just slow people down.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And the reason that the quote posts are so painful whenever they're quote dunks is that it sends a horde of people with one tap into the replies to start dogging you for whatever it is that you're getting dunked for. And honestly, if you've ever... I've witnessed a friend of mine get a really aggressive raid army kind of directed at them with a quote tweet back in, I want to say, 2017.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And a lot of us were thinking a lot about where we expected... People still use the term new media in 2012, if anybody remembers saying that nonsense. So there was quite a bit of thought being put in at that point about what are the power structures that are going to be a part of the future of internet and social media. And how are we designing systems to be smart about that?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, we didn't make them public for that reason. I'll start. But actually, the blocks are, mutes are private, blocks are public. The reason that is, is actually kind of a limitation of the design of the protocol because you need the blocks to be public for the applications and really all applications to be able to respect them.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so you can do the mutes privately because only the application needs to know. It's just filtering things to the recipient, but the blocks kind of had to get broadcast.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And you could come up with a privacy scheme for that where you just give the blocks to the applications in the world, but there's no well-defined set of applications without creating some kind of a organization which has like membership. And at that point, you're essentially starting to create a cartel around the protocol.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And maybe that's where all this goes someday, but for now we're not about to start that action. So we just decided, you know what, we can't find a better way to do blocks. They're going to have to be public. So that's why they're public. That said, there's a lot of other interesting things about blocks and the fact that we implemented what we call moderation lists.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So these are lists that you can mute or block. implemented that pretty early on, again, as a part of the application's kind of decentralized moderation. And we also made blocks really aggressive. Sometimes people call them the nuclear block. Where not only do they sort of like in Twitter, as I recall, it would sort of freeze the interactions between you two.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So you would no longer see each other's stuff, but like your previous interactions would just kind of be frozen in time and public. On Blue Sky, it actually causes all interactions previously between you to also essentially go away. They just show content not available.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so we actually have a very sort of really aggressive toolkit for blocking, and we're encouraging people to put together block lists and to subscribe to them. And again, that kind of gets into the point of view that we have, which is just that there are just some people who can't coexist very well. And I don't think trying to force them to do that is a great idea.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we were thinking pretty heavily about that stuff right from the get-go.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I think that's why you get so much toxicity in social networks. And it felt even more pressing when we're creating this open design and sort of like
Oxide and Friends
Scaling Bluesky with Paul Frazee
loosening up the control that we have over participation, we wanted a way to articulate really aggressively, almost like your own perimeter, since the website is no longer a perimeter for the social network, being able to say, no, this is kind of a group that I'm keen to be hanging out with. And if you're in this list, I'm not, I just don't want to be in the same public space as you.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it comes with pros and cons, you know, I won't say that it's like a pure win. Sometimes it can be really frustrating for people, I'm quite sure. But I think it's important that people are able to make that choice for themselves. I don't really like the idea that people should be forced to share space with each other. So that's the mentality.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. Yeah. Now we're attracting raids, right? Like, the persistent, automated, like, harassment campaigns. And so we've had to really aggressively scale up our tooling to deal with that. And, you know, that is... Thankfully, we designed the whole system to be prepared for that. We just had to get the tooling in place. But then, you know, on a long enough timeline, we all know the game.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Like, the bot fight is hard, and it just got a whole lot harder with AI. And we're starting to putz around with a couple of things that might help with that. But yeah, somebody, some jerks made a... AI bot army that just would disagree a little bit with people. That was their angle.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh, gosh. I wouldn't say it's terribly surprised. I mean, it's discouraging. It definitely makes you go, ah, I wish better of the world, but we all know it's a bit of a dark forest, so...
Oxide and Friends
Scaling Bluesky with Paul Frazee
um no i don't think i'm terribly surprised right now i i'm still i think in the zone of like yeah i think we could do novel things and i'm like uh it's it's only speaking somebody that builds things like it's only once i'm out of ideas completely and like feeling defeat that i'm really like bummed out um but i'm still i still see some moves that we can try so i'm feeling good
Oxide and Friends
Scaling Bluesky with Paul Frazee
A bit, you know, the we are starting to see researchers just sit on the fire hose and there have been a couple of catches they've made that we didn't make. And that's awesome. That's like, you know, that's a great benefit. Hasn't gone like crazy on that front yet because we're we're pretty active in there.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we're, you know, thankfully, it's not like we're getting a ton of like, oh, my gosh, I had no idea. But we are starting to see some folks do it. In terms of actual application, interestingly, I don't think I've seen anybody deploy a bot labeler that I think is totally crushing it yet. But one of the things that we did put in is the ability to run what we call labelers.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And labelers are essentially moderation services on the network. It is how our moderation works.
Oxide and Friends
Scaling Bluesky with Paul Frazee
uh when somebody else is running it the application privileges our moderation so that like only ours is able to do what the application considers to be a full takedown but the uh other the user run or what we call community labelers are still able to do some pretty direct interventions into the application in terms of hiding and filtering out content and where we do see like really good pickups on that i think or like for instance the um
Oxide and Friends
Scaling Bluesky with Paul Frazee
screenshot labelers for, uh, screenshots from like other social networks. Uh, yeah, that was a good one. Right. Because like, sometimes you really get, I think it's kind of like whenever a cat keeps showing up with a dead bird on the doorstep, you know, to show you the dead bird. So that's dead birding. And, uh, so a lot of, a lot of content from the dead birds coming onto the website.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And, you know, sometimes you're like, you know, I really don't. see that or at least I want like a little warning so I can kind of check in like, am I in the mood for that? So somebody was able to apply just a nice little AI modeler to be able to detect those things and then label them. Um, that is pretty great for improving the experience.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Uh, another one that somebody put out there that I don't think I've, I haven't checked in on it to see how it's doing, but, uh, like an AI slop detector. Yeah. Are you tired of AI-generated images? Well, here's a little labeler that at least, you know, you can filter it or it could just give you a little warning notice so that you could just decide.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Or, you know, if anything, in some ways that kind of works well as just a protection against misinformation. If it's, you know, I assume if the AI model can figure out that it's AI slop, it's probably pretty obvious. But still, you know, it's in the zone. Another one, this is not automated, but actually a decent one is a... uh, a politics labeler.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Um, and this one's spicy cause like, yeah, cause like what gets characterized as politics is, you know, it's a little bit, a little bit spicy, but, um, definitely not the kind of thing that we as a company would necessarily want to be doing, uh, for, because it is spicy, but for somebody else out there to be doing their best, you know, like, yeah, we think this is kind of political and like, it's not actually, you can have it just like completely hide this stuff, but honestly just having it behind a little interstitial.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so you could just decide, do I have really want to read something about the election going on right now. People need a break. You know what I mean? So being able to just have this kind of thing to, you know what? I need a break. I'm turning this on. Let's cool down the politics for a little bit. That's the kind of stuff that I get pretty excited about because these things do psychic damage.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, that works on actually a completely separate synchronization. So we have two, I guess, main kind of data structures in the ad protocol. The first one is what we call the repositories. Those are what represent users, and that's where all your data goes. So repository, like a Git repository, but in this case for like JSON. And then we have the labelers and the labels.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so those exist kind of in their own system. We didn't put them in repositories because it's kind of a giant long list of just strings. So it does not work well with the cryptographic structure that resides underneath. The repositories are a variation of a Merkle tree. And we, with the labels, were like, you know, that's just not going to work.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we just decided to have them in their own system. So they kind of synchronize between the labeler and any kind of services that are interested in subscribing to them in a slightly dumber way, to be honest. Yeah, that's it. That's how that works.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So our app view, our application, whenever a user subscribes to a labeler, our application, if it isn't already subscribed to the labels list, we'll find it and start syncing it. And then we do a little runtime join based on your currently... Actually, it's a header in the request. We look at a header of which labelers you want to apply labels for, and then we just pull them in and attach them
Oxide and Friends
Scaling Bluesky with Paul Frazee
through the response, and then the client observed them based on the user settings. And that's it. I mean, a label really is just a label. It's just a string saying, hey, this is like NSFW, or this is like a spider. Are you arachnophobic? Great. Don't show the spider. That's about it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. Yeah, the way they, right now, the way that it feels in the application, I wouldn't say that we have, like, nailed this UX. Maybe. But the way they work is they're kind of like special accounts. Like you can kind of upgrade an account into becoming a labeler, at which point the labels that it offers get turned into the primary profile screen.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And rather than the primary action being follow, you know the primary action is subscribe, and you subscribe to it. It gets added to your moderation screen, and you then configure each of the labels and how they're supposed to operate.
Oxide and Friends
Scaling Bluesky with Paul Frazee
We never got that far. It was janky at best. And I say that with all love. But we had a lot of challenges that we just did not get through. And if we're going to go through this history, one of the meta arcs I would follow throughout all of it was learning how to...
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, I hope that we can get there as far as we want to with it. I think there's a lot of good things that are happening with it, and there are aspects of it that I'm really satisfied with. I will say that the labelers so far have been a bit of a mixed bag in terms of their practical reality because...
Oxide and Friends
Scaling Bluesky with Paul Frazee
It's pretty hard to run one for the non-trivial use cases if you're trying to actually do what you might classically consider to be moderation. I think that part's been a little bit tough, and that is one of the things that we've been sort of sitting back to evaluate before we kind of push on recommending them more, is to see how this is affecting communities.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I don't think I would give that one 10 out of 10 marks. we're hearing in some cases it's gone pretty well, but in some cases we're hearing that it's a pretty stressful position. There's a lot of responsibility that comes along with it. And, uh, and I don't know if they're empowered in all the ways they need to be to handle that responsibility or the expectations that come along with it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So, yeah, I don't want to, I don't want to, uh, walk in here and, and ignore that. You know what I'm saying? Like, I do want to cop to that. It's, it's not quite 100%, I think at the place that I'm, I would love for it to be, but I, And I'm glad that we're giving it a shot, you know what I mean? And I'm glad that we're learning off of it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I think you'll probably see more updates to it based on some of the challenges that we've had over the coming year.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I would actually like for it to be more of it. There's still a couple of things that we've still got closed, like our main algorithm is still closed. We'd love to get that open at some point. We have some of our backend that's, you know, you can get the backend running, but it's not to the 100% ready-to-go place.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And so I'd love to get all that totally open and in a good package so that people can run it. But... a lot of it is open source. It was really fun to be able to share, you know, when we got to the, we got, we managed to hit, you know, number one in the app store, really proud moment. And that's awesome. I was able to share the source code and say, here it is. Here's the source code.
Oxide and Friends
Scaling Bluesky with Paul Frazee
If you want to do it, you know, have one yourself. So that I'm really proud of that. You know what I mean? Like that we were able to take an open source application to this place. And I really liked that. We have stayed with that throughout.
Oxide and Friends
Scaling Bluesky with Paul Frazee
serious about delivering, you know, the level of quality that's necessary for something like this, because you're on the one hand, you're pressing really hard on like, okay, novel technology, novel way to do things, and throwing out a lot of assumptions.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I'm going to Steve Jobs it and say, yes, it's never happened before.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, I may only restrict this to the things that I'm proud of, to be honest. But one, I'll say the one that comes to mind that actually worked so much better than I ever expected was the domain names for usernames. I'm glad to bring that up. Yeah, we put that in there, honestly, for a very pragmatic reason. We just needed a namespace that was decentralized, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
And which would work for the open network and which had the affordances that we needed. And so I was like, well, okay, domains, right? What else are you going to use?
Oxide and Friends
Scaling Bluesky with Paul Frazee
I mean, there is something kind of cool about that. Believe it or not, I do really like... You could just sort of turn the host into a pill and then just have the username, and you could do maybe some nice things with the UX, but anyway...
Oxide and Friends
Scaling Bluesky with Paul Frazee
I'm glad we made this choice where I was really shocked when it came to the domain names was the uptake of the custom domain names and how that has played out has been way beyond expectations. Cause when we did that, we would put together like little mock-ups with like the Washington post using that as their handle.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And along the way, everybody, you know, by the time we get to blue sky, almost everybody at the team had been spending, like I said, years working on this stuff, and nothing was quite working.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it was kind of like the, again, it was one of those things that was like, yeah, right. You know, someday maybe, you know, who knows? And then they did like when ESPN came on, they used ESPN.com. Yeah, that is so cool. I can't believe that these organizations are going to the trouble of configuring DNS for us.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And it wasn't until we got to this project, we all had a bunch of kind of collective, collective realizations as we came together about like, you know what, okay, if we keep these pieces, but then like throw in a lot of the kind of complexity and the novelty that what we're doing, we can keep what we think actually is important about the systems, but have this work actually be usable to end users.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah. Yeah. I mean, I, again, that wasn't like the driving decider on that, right. It was really, Oh, well we need something, you know, but then I think, you know, it does, it has helped us a lot. I think we're probably going to need something more both because it is a little bit wonky there.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You know, I don't want to move too quickly on that, on that aspect of it, but there's definitely some folks that may not totally understand it, but then you also just have folks that don't have like high reputation and, domain names. A good example of that would be Flavor Flav came on a bit ago, and it was so enthusiastic, love it, and he doesn't have a website. He just doesn't.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You also start to run into the, okay, he could set one up. He could set up FlavorFlav.com, but then how do you know that that's the right one versus FlavorFlav.net or .io or whatever? There's going to probably need to be some layer in there at some point, but no matter what it is that we do, The well-known domain names are a really fantastic anchor for it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And boy, is it great for the government stuff because they've all started to use like .senate.gov and .house.gov. Like Tim Walls just came on and became, I believe, governor.mn.gov. Just, you know... Totally, totally solving that problem for some of the more high stakes ones, which is political leaders. And those are under that tight governance, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Like the .gov domain name is the US government highly regimented. And it's so much better than anything that we as an organization could be doing. So definitely some wins in there.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Oh, I got good news. We fixed that. We just fixed that. So yeah, yeah, yeah, yeah. We, we haven't done clams on it yet, but we, um, are, uh, planning to, to work scrunch together those columns. But yeah, um, you now, if you have a piece of it, my friend, It's out.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Actually, ironically, somebody ran into it in the reverse because they were doing the squatting move that people have been doing as a patch. They're like, I can't register it. No, no, it's working right. What it does now is we just keep in our database of the BSky.social stuff. We just remember when somebody moves off what your original one was. You can go back to it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
If you were the original holder of it, you can go back. Nobody else can, though. And so, and I guess if you delete your account, then that gets relinquished. But yeah, yeah. So that is now fixed. Go wherever you like.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Really both, to be honest. Oh, yeah. Yeah. Yeah. But I don't know if I would give more emphasis to one or the other, because it would affect on both sides. Reliability, scale, and performance on the technology side, and usability on the product side. almost every project that the group of us worked on all use like client side signing keys up until blue sky, you know what I mean?
Oxide and Friends
Scaling Bluesky with Paul Frazee
Okay. This one's pretty fun for a couple of reasons. Um, so first of all, let me start with the context of how that got built. We, um, The full context, you have to start with the fact that we are not an algorithm company. And so our initial algorithm that people are getting landed in, we're working on it all the time and we're making some great progress on it.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And I don't want to harsh the team that's working on it. They're really cruising, but it is not great yet. And so new users coming in, it's really important that they are able to get their social graph built so that the following feed could start working for them. That's point number one. Point number two is building your social graph on the app. also a bit challenging.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Getting good recommendations for follows, that is quite hard, especially when you're seeding communities. Like, here's a design challenge for you. Make a good recommender when you don't have social clusters on there yet. How do you even evaluate if this thing is doing its job correctly? You kind of can't do it. Like, we're learning a lot about algorithm design without content to work against.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Thankfully, that is less a problem now, but that was the kind of... We had a real chicken and egg problem. Okay, your obvious thinking is...
Oxide and Friends
Scaling Bluesky with Paul Frazee
yeah so so okay so you're obviously thinking you're you've got this design problem well let's do like a contact import of some kind either from an existing social network or like from the address book well you can't do the address book because that has privacy problems that may not be fixable as far as i know last i heard you can't come up with an anonymized upload your address book thing without creating triangulation of people's contacts so we just said hard no we're just not comfortable with that we're not there's a
Oxide and Friends
Scaling Bluesky with Paul Frazee
It's a real shame. Now that I'm on the other side of that, I get why people go ahead and do it, but we still want to cross that line. It is a shame because we're sitting there going like, we're not trying to be jerks. We're actually trying to help you, but here you are. We didn't cook up some kind of breakthrough on the privacy solution there, so we didn't bother. So, okay, that's out.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, then you can get into, can you try to get the social graph from another social network? And the reality is none of them want to give that up. And we're really in an era where like these APIs are getting closed down. You can't do it off of the meta products and you sure as hell can't do it off of Twitter. I guarantee I know how that would go. So we, that was a non-starter too, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we're sitting around going, how the heck do we solve this problem of getting people's social graphs seeded? The original idea for starter packs was actually not just to have like a list of people to follow. It was actually to create an invite link because we have had a lot of success seeding the social network back when we were invite only.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And ironically, when we switched to the public thing, we lost one of the best organic mechanisms for people to get worked into their social graphs whenever they joined. So we were looking at that and asking, how can we make that better? And the initial impetus for this was actually just to create an invite link that you could share with friends.
Oxide and Friends
Scaling Bluesky with Paul Frazee
They would take them through the install flow and then get you into a bi-directional follow relationship with whoever it was that invited you. And that would help seed the network. And then we were like, OK, well, let's go ahead and add in suggested follows and some suggested feeds and things like that so that you can actually hydrate in the experience.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, we launched this thing and we also included the ability to just like follow all of them from within the application. So it would work also if you were an existing user, but that was just kind of like, yeah, just in case kind of thing. We launched starter packs and people were like, oh yeah, that's really cool. But like, they just didn't get used.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So you had like key management problems and you would also have, uh, it was all local first, all these projects. So like you would have device synchronicity and like, um, essentially like you'd have to get them like CRDT territory to just make a post. It felt like rocket science to try to implement a content section. Yeah. It was, it was very complex software that we were messing with. Yeah.
Oxide and Friends
Scaling Bluesky with Paul Frazee
So we ended up spending like two months going like, ah, there was so much promise with starter packs, but that really didn't land with folks. And then the big influx came in November and man, did they pop off. They ended up becoming the absolute most potent mechanism for getting people onboarded into the application. And again, rarely through the install flow.
Oxide and Friends
Scaling Bluesky with Paul Frazee
it's very rare that people were using like clicking into them and then like kind of having the app and we would use an app clip to like do the install flow and then get you creating your account stuff like that that was very really it was almost always people just sharing them in uh in feeds or like installing the app first and like you know messaging each other with these things or however it was they were doing it and that ended up helping the social graphs and all the clusters form really rapidly and get people's following feeds into an interesting place
Oxide and Friends
Scaling Bluesky with Paul Frazee
They ended up being really strong proxies to topic interest, which is an area that I think we're still pretty weak on, is giving new users a fast access to a topic. So you would find like a journalism starter pack or an art starter pack, and that would... Boom, now your following feed has given you some great stuff on that front. So...
Oxide and Friends
Scaling Bluesky with Paul Frazee
Yeah, it was not a linear, like a A to B thing with starter packs. The core idea, I think maybe there was, but in terms of like how it got used and what we thought we were going to be pulling off of it, it had some twists and turns.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, thanks. I mean, I got to be honest, that level of supportiveness that you're showing here really does help at any given moment. It's been really nice to see people get that level of interest in it, too. So the feeling is very much mutual. When it gets tough, the folks that are there and like, yes, let's keep moving, that kind of buoys us up for those tough points, too.
Oxide and Friends
Scaling Bluesky with Paul Frazee
Well, like the... One that we knew for sure was what we call account portability. And then that one comes to mind first because it has to do with like why you bother with peer-to-peer in the first place.
Oxide and Friends
Scaling Bluesky with Paul Frazee
You're trying to turn the services that you use into something that is actually like, oh God, I'm going to say it, fungible, that you could actually replace a hosting service or like an application service, but keep all your data. And we knew that was one of the really interesting properties of peer-to-peer because like, you know, their client side, right?
Oxide and Friends
Scaling Bluesky with Paul Frazee
So like your storage and your signing keys were being kept on your device. And so the first thing we sort of figured out was, okay, how can you take advantage of using a server, but like maintain that ability to like adversarially move away from a server. And that's what the entire cop portability system is for.
Oxide and Friends
Scaling Bluesky with Paul Frazee
And that was a pretty big move forward because using a lot of the same techniques, but putting it on the server and not losing those properties was like the, kind of the biggest like scale and ease of use unlock that we had throughout the whole system compared to our previous work. We knew we wanted to keep that.
Oxide and Friends
Scaling Bluesky with Paul Frazee
The other thing that we wanted to keep for sure was what is now starting to get called a shared heap model, this notion of an open broadcast of all of the public data so they can be repurposed, because that was a pretty big design element that we had landed on in the peer-to-peer stuff that we'd been doing about how to build these large-scale applications, is this notion of...
Oxide and Friends
Scaling Bluesky with Paul Frazee
It works actually quite a bit like the web where it's almost like everybody's publishing JSON of like posts and their profile and their likes and stuff like that. And then you just like aggregate them in the applications. And that's how you build out an application.
Oxide and Friends
Scaling Bluesky with Paul Frazee
We knew we wanted to keep that because that's what makes it possible to build other applications without some kind of a hard binding to the hosting. So you're really separating the hosting and the application layer. And that maintains that hackability and like capacity to repurpose these applications in interesting ways. Those are probably the two that come to mind.
Oxide and Friends
Scaling Bluesky with Paul Frazee
I don't know the exact timeline. I know it got a lot more turned off recently. Yeah. More turned off.
Oxide and Friends
Scaling Bluesky with Paul Frazee
It's super cool, yeah. And that's also what a lot of people have started to play with first, because it's just so obviously cool. Like, okay, what can I do with this? It's cool in two fronts, actually, because actually, it's really like you're just getting into the innards of our data center, because this is the backbone. You normally...
Oxide and Friends
Scaling Bluesky with Paul Frazee
like an event processing architecture in a way and like normally you would use like a kafka or something like that to like send all these things through that's what this thing is that's what this firehose is all of our application is downstream of that just running computed views off of that firehose so whenever you're tapping into it you're just jumping straight into our data center to tap into the data set so that's really fun it's it's amazing yeah like a meaningful thing not just like a replication of all the activity but like that's it that's the network