munishk: i tried and it did work locally for me. can you push what changes you made?
Tarun_0x0 joined the channel
huhridge: IA search responses differ a lot from spotify and youtube, i don't think you would be able to do such seeding. i am not even sure if IA metadata has unique artist, recording ids so you will have to play with their API responses and investigate it
huhridge has quit
discordbrainz
<07munishk> I'll push the changes
huhridge joined the channel
lucifer
rimskii: i realised i had never replied to your proposal. it looks good to me. choose 175 hours when you submit it to the gsoc portal for it since you are doing 2 integrations.
huhridge
lucifer: yeah that was what i was thinking, a lot of garbage gets returned when it's searched
so how should i proceed?
just do the whole 78 rpm collection?
lucifer
huhridge: 78 rpm is a good start but there are probably more audio collections in IA.
huhridge
lucifer: should i just write a crawler for a given collection then?
lucifer
Sure
huhridge
there are duplicates and stuff, so i'm sure it's not that straightforward, but it seems like a start
huhridge: okay!
lucifer: there are items on IA that are samples, that is only 30 seconds are available for playing, i should remove these right?
Hi @bitmap, regarding the pub sub solution we discussed earlier, even if we use SQL queries, we still need a way to pass the query result to the URL-saving task right? (unless we are doing the network calls in two different tasks, which I think will be a bit more to handle, since now with rate limiting, we might have to check for concurrent calls from 2 different tasks).
I read more on this, and the best I could come up with was a message passing solution. Also, since `sqlx` crate has methods for listening to events, I think it will be easier to handle.
Also the query you specified can be used while we notify the listener
I've also made the necessary changes in the proposal, and have made the necessary changes.
I've also made the necessary changes in the proposal*
I will keep checking for alternates, but I think the best I could think of was this 😅
as of now*
huhridge has quit
huhridge joined the channel
bitmap
yellowhatpro: sorry, I'm not sure I completely follow the issue. does rust not allow you to access variables from the scope in which a task is spawned, from inside the task? or doesn't rust have channels to communicate between tasks?
it sounds like you are saying that there will only be a single URL-saving task, but I was assuming you would spawn a separate task for each URL right after you query the database
mara42 has quit
fletchto99 has quit
fletchto99 joined the channel
huhridge
hello i was looking into LB-1557, and it seems to work in the dev env, but not in production
yeah rust does have channel for message passing. But isn't that similar to what postgres offers us? I found it similar to what listen/notify does
huhridge
like it's rendering fine in the dev env, but in production the username is rendering with %20A
yellowhatpro
@bitmap The way I am planning to use URL-saving task, is to spawn a new thread /green thread for it, and let it listen to incoming URLs
Otherwise I would have to spawn it on each URL while polling
does rust not allow you to access variables from the scope in which a task is spawned, from inside the task?
- Regarding this, I am assuming you are saying if I can't access variables declared in say main function, and I cant use them in the spawned tasks. Yeah we can do so using some Atomic ref counters (Arcs), or Mutex.
if you feel that LISTEN/NOTIFY is the simplest solution then you should go for it
I was thinking it would be easier to just spawn a new task every 5 (?) seconds in accordance with the rate limit, and that the task would have access to the URL that was just queried
discordbrainz
<05rustynova> If you want Thread A to own the variable, but Thread B to have it too, then Arc<T> allow unmutable access, and Arc<Mutex<T>> allow mutable access. Both Thread A and Thread B will own the variable, and will discard it when they both drop it. However, this isn't always the best case. If you want to send a variable to Thread B, and you won't use it in Thread A anymore, there "mpsc" that allow sending variables to threads
bitmap
thanks rustynova ;)
yellowhatpro
That was my initial plan, but I encountered the check_and_retry use case , which basically checks if a url is saved or not, and if it is not, then it will try saving it. Now this has to run differently, so that is why I wanted a message passing solution
@rustynova yeah mpsc creates a channel so that we can send/recv messages. But I found sqlx has same method for notify/listen, which I found same as mpsc, as they both are for async communication
discordbrainz
<05rustynova> It's probably better to use the mysqlx solution. But I never dug into that enough to recomend anything.
mara42 joined the channel
yellowhatpro
that's the plan! Yeah, the sqlx version uses tokio-runtime, so maybe some benefit there. But we do have other channels based on tokio so we never know .
DarthGandalf has quit
iconoclast_hero has quit
iconoclast_hero joined the channel
Tarun_0x0 joined the channel
gchound joined the channel
mara42 has quit
mara42 joined the channel
mara42 has quit
gradient joined the channel
Tarun_0x0 has quit
saumon has quit
bttf has quit
bttf joined the channel
Tarun_0x0 joined the channel
saumon joined the channel
gchound has quit
bttf has quit
bttf joined the channel
iconoclast_hero has quit
bttf has quit
DarthGandalf joined the channel
rimskii[m] joined the channel
rimskii[m]
<lucifer> "rimskii: i realised i had..." <- sure! thank you so much, will def do that !!