028 — Racing the Queues

Hey. How's it going? I'm Ryan Hefner, and this is the All Play podcast. So this week, I've been working on getting starter packs up and out. I've been pushing some updates, fixed some stuff as far as some of the loading and stuff going on with it, and some of, you know, just some general odds and ends.

But, actually, the biggest thing I ended up taking on was finally writing the script and coming up with the system to basically backfill all the data that I was missing on on, some of the starter packs that have been captured to then offer some of these premium features. So I have basically kind of like a ticking timer right now, and my goal is that by the time that the script and the queues are done processing all the data, I want to have subscriptions live on the product. So what does that mean? That means that I'm gonna figure out what the monthly charge that I'm gonna have is. I'm gonna see if there's any potentially, like, multiple tiers, whether that's the number of accounts.

So the one of the things that I did do when I was setting up the account handling on the app is right now, you can authenticate and you're associated to a single account, but it can support multiple. So for people who maybe have both a personal account and they have their project or, a business account that they were trying to track or or do stuff with, it's gonna expose some stuff to do there. So I am gonna have to do some work to kind of, like, expand that out. But for now, I'm gonna just focus on the single account experience and pricing plans, and I will roll out the multiple account stuff in the future. But, I guess the big thing that I was working on this week was really just getting the whole backfill system up and running.

And I wanted to dive into, I guess, my my flow on how I ended up doing that and then eventually what I ended up rolling out and deploying to the site. So I knew I had this problem of I have all these starter packs that I've captured off the fire hose, And at this point, it's over 200 it's, like, almost, like, 250,000. I think it's, like, 249,000 and some odd packs, which it's a lot of data. And as far as the backfill process goes, I basically for each one of those packs, what I'm gonna what I basically settled on, which I was kind of debating on back and forth, it's like, how much how many rows do I really wanna, like, shove in to this database? And, you know, was there a simpler way that I could do it where I could just basically, like, have a field or a column on the existing starter packs and just, like, populate that with the, with some, like the blue sky has these, basically, like, these DID, like, document IDs.

I was like, well, can I just do an array of those? I ended up not going that way because I just think it's a little limiting then for some

of the future stuff that I wanna do. And I

don't know. I it was one of those things that I was going back and forth. I was spending way too much time on. I just dove in. I'm like, okay.

Who who cares? If it ends up, blowing up the database, so be it. So far so good. It's been the queue is running. Well, I'll get to that.

So, basically, for every starter pack, it also grabs the so the way that blue sky kind of nest these things is it's a a starter pack, contains a list. A list has list items, and then list items all reference an actor. And an actor is basically like your scoop blue sky profile. So as I in order to basically get all the information that I need to offer the features that I want, For right now, I just basically have the starter pack. So I had to come up with a way of, from the starter pack, going out, getting the list, then creating the list items, and then associating those list items, and also creating, essentially, if I don't have an actor an actor that's associated to each one of those list items, and now I have a nice clean way of going from an actor to the list items items that they're in and essentially to the list and then the starter packs that they're in.

So that's the whole flow.

To kind of, like, initially do the this the the kinda make

things work and then make things right, I just wrote a node script that did the flow that I wanted it to do. And I set that up, and I just had it to where, basically, I could just, like, fire

it off via the terminal and just make sure that my series of events and the way that it was populated in

the database and any kind of weird assumptions, that I was running into you know, that I had regarding how I was gonna actually ingest the data and process it was gonna work. So that's what I did. I wrote a quick script and started testing stuff out. I started, debugging some different things. I did realize that, actually, by doing these whole by doing this whole kinda sync backfill thing, it also allows me to kinda there's times definitely when the Blue Sky fire hose was either down or for whatever reason, my my service wasn't hitting it or or receiving updates from it, at points in time.

And so there's actually a lot of starter packs that I have that no longer exist on Blue Sky. So, actually, this whole backfill thing also is, like, a little bit of, like, a data integrity thing where I can delete any sort of, like, hanging, starter packs that no longer exist. So that's one of the first things I ended up running into. Basically, I ran the script. Oh, wait.

Oh, that starter pack doesn't exist on Blue Sky. Okay. Now I can actually just write the cleanup delete, clean up my database, kinda keep the whole system going. So that was cool. Another thing I realized that I don't quite have a fix for this yet, but I'm gonna have to

try to figure it out soon, is that

blue sky basically, the way that I set up the actors is you need

to have a, unique DID, but also a unique handle. And, actually, I need to

try to figure out whether or not I'm gonna retain that enforcement or not Because what I'm realizing is that blue

sky I need to I need to dig into this more. But basically, when people are

changing their handle,

They may or may not change the handle from one profile, but yet I'm somehow retaining an old handle to the other. So then when I try to write a new actor record, the handle and the DID are basically conflicting with unique values that are already in the database even though those are just recently added. So I don't I need

to figure out what's going on there. I need to figure out some sort of resolution on, like, which one should I rely on? Should I remove the unique handler from the from the the hand or the unique, con risk constraint from the handle and then just have it be based off the DID. And then whatever I get in, I will somehow try to resolve after the fact, but at least I have the actors and the list items that they're associated with, and maybe I can do some sort of account resolution thing. But, you know, some of these weird things that you don't really think about, or you don't you don't run-in you don't realize you're gonna run into until you're doing it.

And so there you go. So I wrote

that script, tried to find all these weird, you know, outliers and and different things that I was you know, that either contradict or or didn't line up with my assumptions on writing the initial one. And then

I was like, okay. I just need to run this thing and just see how long it's actually gonna take to process these things. So in my local environment, I have, like, roughly, like, forty, forty four thousand that have just kind of been, like, accumulated on my local database while I've been running the site locally and used for testing and stuff. And when I ran that just that node script, which is basically take a record off, you know, find find a starter pack that does not is not not associated to a list ID and pull it off from the database, run all those things, basically, you know, grabbing the list, creating, both actors and list items and associating those. That was taking a very long time.

So then I realized, okay. I gotta run this thing in a queue. I haven't really done anything with nest and queues yet, but I figured this is as good a time as any because I know I'm gonna need it for some of the future stuff that I'm gonna be working on. And so I dug in a couple days ago. I think it was Wednesday.

I was like, alright. Let me figure out what this queue stuff is. And got in, and test has a a a kind of a direct integration with BullMQ, which is the first time I've used this. But I did like the queue system. It kind of felt very kind of Laravel esque in that you could provide a job ID.

So I knew that I could basically scope these jobs to a specific job ID and make sure there's no duplicate, starter packs or duplicate actions being processed, within the queues. So trying to keep their queues down. And just sort of figuring out all this stuff. It is backed by Redis. I do like the idea of, like, a database backed, queue system, but what the heck?

I'm gonna use I use Redis for a bunch of other stuff as far as caching and different things. So also, Redis is obviously used by a lot of different queue systems. So Redis, it is.

And, actually, pretty painlessly or lack of pain, I was able

to get the queue system up and running, on my local, just running Redis locally. And then, I'm on render, so I am I did end up spinning up a Redis box on render, which has cost me $10 a month. But I have a feeling I'm actually gonna run a lot more stuff through these queues,

and hopefully, it'll justify, the basically, the it'll justify itself because these queues are actually working out pretty good. So set up the queues running, procured the the Redis server, and yeah.

I mean, by the, you know, Wednesday, that night, I had queues up and running, and yesterday was basically kinda just, like, troubleshooting some different things. I did end up pushing it out to the site yesterday. Did end up running into some weird stuff where jobs are just kind of, like, piling up, and I was also kind of tweaking some of the concurrency settings, but I think I have it pretty much dialed in and also have a so I need to run this script, that's gonna be basically backfilling all the starter packs. But I also in setting up the queue system, I decided to take all of the fire hose activity and actions that were just being processed at basically, at real time as as messages were being captured off the fire hose and decided to actually throw those on the queues as well. So that'll be a little bit more robust for if there's any weird burst in the events from Blue Sky.

It'll run into a queue system, and, also, there's a lot more concurrency. I actually throttled the concurrency up to, like, 300 jobs, that can run-in tandem because I realized that running, you know, this 250, it's gonna take a good amount of time. I think doing some rough calculations and, also, that's depending on what it's gonna it's gonna be a couple days before the starter packs are fully backfilled, and that's where my ticking time line comes in. So, basically, I have the queue up and running. I, in getting the backfills, I obviously have, everything working locally.

So I'm gonna be working on the subscriptions and, basically, the ex the the flow of being able to both see and then, expose the starter packs that you're in on your profile and as well as add additional kind of more navigation points to the pages. So, essentially, like, when you're looking at a starter pack to and if that author have or the, you know, that profile also has other starter packs, maybe having those showcased below the people that are in the starter pack, and then, maybe others other starter packs that they're in as just more navigation points to kinda build up these pages, make them a little bit more differentiated from what's currently available on Blue Sky and some of these other, directories or or repositories of starter packs and kinda just expand the navigate the navigation experience. So I have two days. I'm gonna try to get subscriptions out. Queues are up and running and are humming along.

There's still maybe a few little things I need to tweak on there, but I'm gonna just try to let that thing churn. And assuming that it doesn't fall down, queues and the jobs that are kinda hanging seem to be like there I have a like, again, like I said, kind of like a auto cleanup thing that's just making sure that everything is if there is something that's being funky, it's just getting pulled out so other jobs can kinda go through and be processed. And, yeah. So quick little update on starter packs. My ticking time time time bomb to try to get a subscription thing out.

And, yeah, I'm gonna I'm gonna leave it there. So, check it out. You can right now, you can already authenticate with your Blue Sky account on Starter Packs. So go to starterpacks.net, browse around, follow some people. You're gonna see some updates making their way to the site shortly.

And, I don't know. I'm looking forward to follow through on just getting this this first leg of the site out so I can start focusing on promoting it as well as tweaking it, but also using a little bit more of my time to focus on some other projects. Not to say that I'm gonna be abandoning starter packs right now, but I will admit that work I I didn't think it was gonna take till February to get this to where I'm at, but that's just the way it goes. So check it out, starterpacks.net. This again, this is the All Play podcast.

I'm Ryan Hefner. You can check out the podcast at allplay.fm. I'm at ryanhafner.com. Follow on Bluesky @allplay.fm and @ryanhefner.com, and I'll see you next time. Alright.

Have a

great one. Later.