<lucifer[m]> "you need to import it as..." <- lucifer: This didn't resolved my issue it was giving me the same error as module not found i retain that `import internetarchive` the problem was internetarchive was not installing on my local environment after trying multiple `./develop.sh build web ` successfully got installed the `5.4.0` version and after that i tried again `./develop.sh exec web python -m
2025-06-06 15753, 2025
rayyan_seliya123
listenbrainz.metadata_cache.internetarchive.ia_indexer` It just started indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec33… , I think I need help to align the indexer code with all the required listenbrainz fields for running it completely !
2025-06-06 15703, 2025
rayyan_seliya123
* lucifer: This didn't resolved my issue it was giving me the same error as module not found i retain that import internetarchive the problem was internetarchive was not installing on my local environment after trying multiple ./develop.sh build web successfully got installed the 5.4.0 version and after that i tried again ./develop.sh exec web python -m listenbrainz.metadata_cache.internetarchive.ia_indexer It just started
2025-06-06 15703, 2025
rayyan_seliya123
indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec33… , I think I need help to align the indexer code with all the required listenbrainz fields to run it properly !
2025-06-06 15758, 2025
rayyan_seliya123
* @lucifer:chatbrainz.org: This didn't resolved my issue it was giving me the same error as module not found i retain that import internetarchive the problem was internetarchive was not installing on my local environment after trying multiple ./develop.sh build web successfully got installed the 5.4.0 version and after that i tried again ./develop.sh exec web python -m listenbrainz.metadata_cache.internetarchive.ia_indexer It
2025-06-06 15758, 2025
rayyan_seliya123
just started indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec33… , I think I need help to align the indexer code with all the required listenbrainz fields to run it properly !
2025-06-06 15714, 2025
Aerozol[m] joined the channel
2025-06-06 15714, 2025
Aerozol[m] uploaded an image: (31KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/nXPeooArcdykXKcfLiqbPKzI/image.png >
2025-06-06 15715, 2025
Aerozol[m]
🥰
2025-06-06 15741, 2025
yvanzo[m] joined the channel
2025-06-06 15741, 2025
yvanzo[m]
<reosarevok[m]> "yvanzo: I suspect SEARCH-745..." <- reosarevok: Yes, it’s even your oversight, it dates back to 2013.
rayyan_seliya123: ah okay, i had misunderstood the initial error. as for the sql issue, you can run `./develop.sh timescale` that will open a psql shell where you can run the sql schema commands to create the table
lucifer or mayhem: can you take a look at the ListenBrainz Matching email? Is this a thing about canonical releases being preferred and if such, do we have this clearly documented somewhere? :)
2025-06-06 15730, 2025
reosarevok[m]
Thanks lucifer
2025-06-06 15709, 2025
reosarevok[m]
yvanzo: re: the ended flag, do you agree with implementing it as an optional attribute like we did with primary?
2025-06-06 15752, 2025
reosarevok[m]
Ideally aliases would just get a life-span but that should probably be a ws3 change?
2025-06-06 15709, 2025
reosarevok[m]
(I mean, we could add it in addition to the dates in attributes, but that might be more confusing)
rayyan_seliya123 uploaded an image: (35KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/pwlyGMpkufPUjsNfIptnZnKb/Screenshot%202025-06-06%20154846.png >
2025-06-06 15701, 2025
rayyan_seliya123 uploaded an image: (115KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/LmCXMdqIhsZwASEXsBBZRKKn/Screenshot%202025-06-06%20152835.png >
2025-06-06 15706, 2025
rayyan_seliya123
hey lucifer ,I have successfully run that indexer script and got around 1000+ recordings each for `78 rpm` and `cylinder` stored in the table as u told to do `./develop.sh timescale` and checked the table using `SELECT data FROM metadata_cache.internetarchive WHERE data->>'title' = '' LIMIT 5;
2025-06-06 15706, 2025
rayyan_seliya123
` and got the data u can see in the screenshot and wanted to inform you that I decided to create a new `models.py` for the Internet Archive indexer because the data it provides is pretty different from what we get from SoundCloud. The SoundcloudTrack model expects things like a nested `artist `object with fields such as` id`, name, and data, but the Internet Archive just doesn’t have that kind of structure in its metadata.
2025-06-06 15706, 2025
rayyan_seliya123
When I tried to use SoundcloudTrack, I kept running into validation errors because those fields were missing. So should I commit that model and update my indexer in the commit where I have created a pr ? and how to proceed further!
2025-06-06 15725, 2025
monkey[m]
<Aerozol[m]> "image.png" <- Love to see it !
2025-06-06 15735, 2025
mayhem[m]
<lucifer[m]> "mayhem: can you create a dns..." <- let me see, if I can. the DNS stuff is being automated and migrating to porkbun.
[@rayyan_seliya123:matrix.org](https://matrix.to/#/@rayyan_seliya123:matrix.org) sounds good to me update the models. The next step would be to figure out if you can obtain more detailed metadata from IA. Or the separate artist/track/release name at least. If not is there any way to extract that from existing metadata. Compare the items you are indexing to the IA webpage, if they are showing some data there it should be possible to
2025-06-06 15720, 2025
lucifer[m]
get it from the api too
2025-06-06 15731, 2025
zas[m]
On PR submission, there will be checks for validity, and when PR is merged, dns records will be deployed
2025-06-06 15729, 2025
zas[m]
mayhem: also github bot will create a comment in your PR showing actual changes
<lucifer[m]> "[@rayyan_seliya123:matrix.org](..." <- Thanks for the feedback! I will commit the new model and updated indexer as discussed.
2025-06-06 15745, 2025
rayyan_seliya123
For the next step, I will compare the metadata I am indexing with and what is shown on the IA website and see if I can extract more detailed fields like artist, track, or release from the API or the metadata . If I find richer metadata, I will update the model and indexer accordingly !!
zas: trying to generate a letsencrypt cert for that domain resuts in an error: `[Fri Jun 6 11:50:23 UTC 2025] funkwhale-dev.metabrainz.org:Verify error:During secondary validation: While processing CAA for funkwhale-dev.metabrainz.org: CAA record for metabrainz.org prevents issuance`
2025-06-06 15709, 2025
mayhem[m]
lucifer: that is because we have a letsencrypt setup already.
2025-06-06 15713, 2025
mayhem[m]
hang on
2025-06-06 15738, 2025
lucifer[m]
mayhem: the `production_le_1` container on wolf right?
2025-06-06 15752, 2025
mayhem[m]
yes.
2025-06-06 15705, 2025
lucifer[m]
this is from its logs.
2025-06-06 15729, 2025
mayhem[m]
and if you add the container that is to be public to the "production" external network with a port exposed, a cert should automatically be generated for it.
from CI: "2025/06/06 12:00:03 ERROR: cannot have CNAME and CAA record with same name: funkwhale-dev.metabrainz.org"
2025-06-06 15746, 2025
mayhem[m]
should I take out the CNAME, zas ?
2025-06-06 15725, 2025
lucifer[m]
Generally, you want to set CAA records on your registered domain (such as “example.org” or “mysite.co.uk”). This way they apply to both that domain and any subdomains you create under it, such as “community.example.org”.
2025-06-06 15747, 2025
lucifer[m]
if we have one for metabrainz.org it should work for all subdomains methinks?
zas: getting a lot of intermittent ssl errors on test.mb.org and other mb/meb doamins
2025-06-06 15729, 2025
mayhem[m]
how frequent? I just hit about a dozen pages and no errors.
2025-06-06 15756, 2025
lucifer[m]
sort of 1 out of 6.
2025-06-06 15702, 2025
lucifer[m]
on test.meb and test.mb, its around 20% failure rate.
2025-06-06 15722, 2025
monkey[m]
I see a bunch of SSL connection errors in Nagios alerts
2025-06-06 15728, 2025
mayhem[m]
could it possibly be related to the dns change from earlier?? seems unlikely.
2025-06-06 15740, 2025
mayhem[m]
zas: you still around, or have you gone for the weekend?
2025-06-06 15702, 2025
canters_flam[m] joined the channel
2025-06-06 15702, 2025
canters_flam[m]
not sure if it's related but internet archive is down atm and isn't that called for images? (i'm seeing ssl errors too)
2025-06-06 15744, 2025
mayhem[m]
lucifer: I am going to revert that DNS PR, just in case it is causing problems, ok?
2025-06-06 15751, 2025
mayhem[m]
merged. hopefully you won't be affected just yet, lucifer
2025-06-06 15707, 2025
zas[m]
What kind of SSL errors?
2025-06-06 15736, 2025
mayhem[m]
SSL connection errors. also present in nagios warnings.
2025-06-06 15745, 2025
mayhem[m]
I just reverted the DNS change from earlier in case it is related.
2025-06-06 15712, 2025
zas[m]
Yes, but nagios reports failures to connect, and it can be a network issue
2025-06-06 15715, 2025
lucifer[m]
mayhem: 👍️
2025-06-06 15752, 2025
lucifer[m]
zas: connection reset by peer`
2025-06-06 15736, 2025
zas[m]
the change you just reverted is unlikely the cause, even though CAA records can cause quite a mess if badly configured. Since 15 UTC we have a significant increase in traffic (AI bots again I guess), that can explain errors
Did something change regarding mb backends around this time??
2025-06-06 15733, 2025
zas[m]
lucifer: that's a network error, for traffic over https, but perhaps not an SSL error, did it happen around 15 UTC?
2025-06-06 15749, 2025
bitmap[m]
zas[m]: nope, I didn't change anything...
2025-06-06 15707, 2025
lucifer[m]
zas: it has been happening before 15utc
2025-06-06 15733, 2025
lucifer[m]
i think since 13 utc at least and nagios alerts for the same started probably earlier
2025-06-06 15724, 2025
bitmap[m]
zas: there are tons of requests to `/ws/2/recording/?query=` (literally an empty query) and similar ones triggering 400 errors, but I can't explain the drop
2025-06-06 15734, 2025
derwin has quit
2025-06-06 15730, 2025
q3lont joined the channel
2025-06-06 15733, 2025
rozlav8 has quit
2025-06-06 15746, 2025
kellnerd[m] joined the channel
2025-06-06 15746, 2025
kellnerd[m]
FWIW, it is these "connection reset by peer" errors which are most likely causing the Harmony issue which I am currently fixing.
2025-06-06 15712, 2025
lucifer[m]
adding retries fixes the issue for now as it is intermittent.
2025-06-06 15755, 2025
Maxr1998 has quit
2025-06-06 15741, 2025
Maxr1998 joined the channel
2025-06-06 15704, 2025
q3lont has quit
2025-06-06 15752, 2025
q3lont joined the channel
2025-06-06 15734, 2025
kellnerd[m]
Yes, that helps, but there was also a bug in my implementation uncovered by the connection issue (which I've hopefully fixed now).
2025-06-06 15735, 2025
mayhem[m]
how evil of me is it to think that we should add these sort of problems as "features" occasionally so that people fix their code? (and I am not innocent in this case!)
mayhem, bitmap, yvanzo, reosarevok: would you be available coming tuesday regular meeting time to discuss migrating user accounts from MB to MeB?
2025-06-06 15706, 2025
reosarevok[m]
Should be
2025-06-06 15754, 2025
holycow23[m]
<mayhem[m]> "holycow23: ping" <- pong
2025-06-06 15704, 2025
mayhem[m]
<lucifer[m]> "mayhem, bitmap, yvanzo, reosarev..." <- I can't sorry.
2025-06-06 15701, 2025
lucifer[m]
[@mayhem:chatbrainz.org](https://matrix.to/#/@mayhem:chatbrainz.org) what date/time works for you?
2025-06-06 15743, 2025
mayhem[m]
wednesday, 1 hour before meeting time or meeting time.
2025-06-06 15755, 2025
mamanullah7[m]
i've created new application on funkwhale and used `FUNKWHALE_CLIENT_ID` and `FUNKWHALE_CLIENT_SECRET` from there and when i tried to send rqst i'm getting and autorization url `https://funkwhale-dev.metabrainz.org/api/v1/oauth/authorize?client_id=listenbrainz-server-amanullah&response_type=code&redirect_uri=http://localhost:8100/1/funkwhale/callback&scope=read:listens`
2025-06-06 15755, 2025
mamanullah7[m]
`web-1 | 192.168.65.1 - - [06/Jun/2025 17:00:53] "GET /1/funkwhale/callback?code=test_code&host_url=https://funkwhale-dev.metabrainz.org HTTP/1.1" 400 -`and then after opening its showing page not found and error etc and i'm not able to understand weather there is any mistake in implementation or in testing! if u can look into the chnages and let me know if there is any isse so i can improve as most of the things are
2025-06-06 15755, 2025
mamanullah7[m]
rn for testing only i need to revise them!
2025-06-06 15725, 2025
mamanullah7[m]
* lucifer: i've created new application on funkwhale and used `FUNKWHALE_CLIENT_ID` and `FUNKWHALE_CLIENT_SECRET` from there and when i tried to send rqst i'm getting and autorization url `https://funkwhale-dev.metabrainz.org/api/v1/oauth/authorize?client_id=listenbrainz-server-amanullah&response_type=code&redirect_uri=http://localhost:8100/1/funkwhale/callback&scope=read:listens`
2025-06-06 15725, 2025
mamanullah7[m]
`web-1 | 192.168.65.1 - - [06/Jun/2025 17:00:53] "GET /1/funkwhale/callback?code=test_code&host_url=https://funkwhale-dev.metabrainz.org HTTP/1.1" 400 -`and then after opening its showing page not found and error etc and i'm not able to understand weather there is any mistake in implementation or in testing! if u can look into the chnages and let me know if there is any isse so i can improve as most of the things are
