<lucifer[m]> "you need to import it as..." <- lucifer: This didn't resolved my issue it was giving me the same error as module not found i retain that `import internetarchive` the problem was internetarchive was not installing on my local environment after trying multiple `./develop.sh build web ` successfully got installed the `5.4.0` version and after that i tried again `./develop.sh exec web python -m
listenbrainz.metadata_cache.internetarchive.ia_indexer` It just started indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec... , I think I need help to align the indexer code with all the required listenbrainz fields for running it completely !
* lucifer: This didn't resolved my issue it was giving me the same error as module not found i retain that import internetarchive the problem was internetarchive was not installing on my local environment after trying multiple ./develop.sh build web successfully got installed the 5.4.0 version and after that i tried again ./develop.sh exec web python -m listenbrainz.metadata_cache.internetarchive.ia_indexer It just started
indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec... , I think I need help to align the indexer code with all the required listenbrainz fields to run it properly !
* @lucifer:chatbrainz.org: This didn't resolved my issue it was giving me the same error as module not found i retain that import internetarchive the problem was internetarchive was not installing on my local environment after trying multiple ./develop.sh build web successfully got installed the 5.4.0 version and after that i tried again ./develop.sh exec web python -m listenbrainz.metadata_cache.internetarchive.ia_indexer It
just started indexing and then stopped giving me array of errors u can see in my gist https://gist.github.com/RayyanSeliya/8a7eda74ec... , I think I need help to align the indexer code with all the required listenbrainz fields to run it properly !
Aerozol[m] joined the channel
Aerozol[m] uploaded an image: (31KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/nXPeooArcdykXKcfLiqbPKzI/image.png >
Aerozol[m]
🥰
yvanzo[m] joined the channel
yvanzo[m]
<reosarevok[m]> "yvanzo: I suspect SEARCH-745..." <- reosarevok: Yes, it’s even your oversight, it dates back to 2013.
rayyan_seliya123: ah okay, i had misunderstood the initial error. as for the sql issue, you can run `./develop.sh timescale` that will open a psql shell where you can run the sql schema commands to create the table
lucifer or mayhem: can you take a look at the ListenBrainz Matching email? Is this a thing about canonical releases being preferred and if such, do we have this clearly documented somewhere? :)
Thanks lucifer
yvanzo: re: the ended flag, do you agree with implementing it as an optional attribute like we did with primary?
Ideally aliases would just get a life-span but that should probably be a ws3 change?
(I mean, we could add it in addition to the dates in attributes, but that might be more confusing)
rayyan_seliya123 uploaded an image: (35KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/pwlyGMpkufPUjsNfIptnZnKb/Screenshot%202025-06-06%20154846.png >
rayyan_seliya123 uploaded an image: (115KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/LmCXMdqIhsZwASEXsBBZRKKn/Screenshot%202025-06-06%20152835.png >
rayyan_seliya123
hey lucifer ,I have successfully run that indexer script and got around 1000+ recordings each for `78 rpm` and `cylinder` stored in the table as u told to do `./develop.sh timescale` and checked the table using `SELECT data FROM metadata_cache.internetarchive WHERE data->>'title' = '' LIMIT 5;
` and got the data u can see in the screenshot and wanted to inform you that I decided to create a new `models.py` for the Internet Archive indexer because the data it provides is pretty different from what we get from SoundCloud. The SoundcloudTrack model expects things like a nested `artist `object with fields such as` id`, name, and data, but the Internet Archive just doesn’t have that kind of structure in its metadata.
When I tried to use SoundcloudTrack, I kept running into validation errors because those fields were missing. So should I commit that model and update my indexer in the commit where I have created a pr ? and how to proceed further!
monkey[m]
<Aerozol[m]> "image.png" <- Love to see it !
mayhem[m]
<lucifer[m]> "mayhem: can you create a dns..." <- let me see, if I can. the DNS stuff is being automated and migrating to porkbun.
[@rayyan_seliya123:matrix.org](https://matrix.to/#/@rayyan_seliya123:matrix.org) sounds good to me update the models. The next step would be to figure out if you can obtain more detailed metadata from IA. Or the separate artist/track/release name at least. If not is there any way to extract that from existing metadata. Compare the items you are indexing to the IA webpage, if they are showing some data there it should be possible to
get it from the api too
zas[m]
On PR submission, there will be checks for validity, and when PR is merged, dns records will be deployed
mayhem: also github bot will create a comment in your PR showing actual changes
<lucifer[m]> "[@rayyan_seliya123:matrix.org](..." <- Thanks for the feedback! I will commit the new model and updated indexer as discussed.
For the next step, I will compare the metadata I am indexing with and what is shown on the IA website and see if I can extract more detailed fields like artist, track, or release from the API or the metadata . If I find richer metadata, I will update the model and indexer accordingly !!
zas: trying to generate a letsencrypt cert for that domain resuts in an error: `[Fri Jun 6 11:50:23 UTC 2025] funkwhale-dev.metabrainz.org:Verify error:During secondary validation: While processing CAA for funkwhale-dev.metabrainz.org: CAA record for metabrainz.org prevents issuance`
mayhem[m]
lucifer: that is because we have a letsencrypt setup already.
hang on
lucifer[m]
mayhem: the `production_le_1` container on wolf right?
mayhem[m]
yes.
lucifer[m]
this is from its logs.
mayhem[m]
and if you add the container that is to be public to the "production" external network with a port exposed, a cert should automatically be generated for it.
from CI: "2025/06/06 12:00:03 ERROR: cannot have CNAME and CAA record with same name: funkwhale-dev.metabrainz.org"
should I take out the CNAME, zas ?
lucifer[m]
Generally, you want to set CAA records on your registered domain (such as “example.org” or “mysite.co.uk”). This way they apply to both that domain and any subdomains you create under it, such as “community.example.org”.
if we have one for metabrainz.org it should work for all subdomains methinks?
zas: getting a lot of intermittent ssl errors on test.mb.org and other mb/meb doamins
mayhem[m]
how frequent? I just hit about a dozen pages and no errors.
lucifer[m]
sort of 1 out of 6.
on test.meb and test.mb, its around 20% failure rate.
monkey[m]
I see a bunch of SSL connection errors in Nagios alerts
mayhem[m]
could it possibly be related to the dns change from earlier?? seems unlikely.
zas: you still around, or have you gone for the weekend?
canters_flam[m] joined the channel
canters_flam[m]
not sure if it's related but internet archive is down atm and isn't that called for images? (i'm seeing ssl errors too)
mayhem[m]
lucifer: I am going to revert that DNS PR, just in case it is causing problems, ok?
merged. hopefully you won't be affected just yet, lucifer
zas[m]
What kind of SSL errors?
mayhem[m]
SSL connection errors. also present in nagios warnings.
I just reverted the DNS change from earlier in case it is related.
zas[m]
Yes, but nagios reports failures to connect, and it can be a network issue
lucifer[m]
mayhem: 👍️
zas: connection reset by peer`
zas[m]
the change you just reverted is unlikely the cause, even though CAA records can cause quite a mess if badly configured. Since 15 UTC we have a significant increase in traffic (AI bots again I guess), that can explain errors
Did something change regarding mb backends around this time??
lucifer: that's a network error, for traffic over https, but perhaps not an SSL error, did it happen around 15 UTC?
bitmap[m]
zas[m]: nope, I didn't change anything...
lucifer[m]
zas: it has been happening before 15utc
i think since 13 utc at least and nagios alerts for the same started probably earlier
bitmap[m]
zas: there are tons of requests to `/ws/2/recording/?query=` (literally an empty query) and similar ones triggering 400 errors, but I can't explain the drop
derwin has quit
q3lont joined the channel
rozlav8 has quit
kellnerd[m] joined the channel
kellnerd[m]
FWIW, it is these "connection reset by peer" errors which are most likely causing the Harmony issue which I am currently fixing.
lucifer[m]
adding retries fixes the issue for now as it is intermittent.
Maxr1998 has quit
Maxr1998 joined the channel
q3lont has quit
q3lont joined the channel
kellnerd[m]
Yes, that helps, but there was also a bug in my implementation uncovered by the connection issue (which I've hopefully fixed now).
mayhem[m]
how evil of me is it to think that we should add these sort of problems as "features" occasionally so that people fix their code? (and I am not innocent in this case!)
mayhem, bitmap, yvanzo, reosarevok: would you be available coming tuesday regular meeting time to discuss migrating user accounts from MB to MeB?
reosarevok[m]
Should be
holycow23[m]
<mayhem[m]> "holycow23: ping" <- pong
mayhem[m]
<lucifer[m]> "mayhem, bitmap, yvanzo, reosarev..." <- I can't sorry.
lucifer[m]
[@mayhem:chatbrainz.org](https://matrix.to/#/@mayhem:chatbrainz.org) what date/time works for you?
mayhem[m]
wednesday, 1 hour before meeting time or meeting time.
mamanullah7[m]
i've created new application on funkwhale and used `FUNKWHALE_CLIENT_ID` and `FUNKWHALE_CLIENT_SECRET` from there and when i tried to send rqst i'm getting and autorization url `https://funkwhale-dev.metabrainz.org/api/v1/oauth/authorize?client_id=listenbrainz-server-amanullah&response_type=code&redirect_uri=http://localhost:8100/1/funkwhale/callback&scope=read:listens`
`web-1 | 192.168.65.1 - - [06/Jun/2025 17:00:53] "GET /1/funkwhale/callback?code=test_code&host_url=https://funkwhale-dev.metabrainz.org HTTP/1.1" 400 -`and then after opening its showing page not found and error etc and i'm not able to understand weather there is any mistake in implementation or in testing! if u can look into the chnages and let me know if there is any isse so i can improve as most of the things are
rn for testing only i need to revise them!
* lucifer: i've created new application on funkwhale and used `FUNKWHALE_CLIENT_ID` and `FUNKWHALE_CLIENT_SECRET` from there and when i tried to send rqst i'm getting and autorization url `https://funkwhale-dev.metabrainz.org/api/v1/oauth/authorize?client_id=listenbrainz-server-amanullah&response_type=code&redirect_uri=http://localhost:8100/1/funkwhale/callback&scope=read:listens`
`web-1 | 192.168.65.1 - - [06/Jun/2025 17:00:53] "GET /1/funkwhale/callback?code=test_code&host_url=https://funkwhale-dev.metabrainz.org HTTP/1.1" 400 -`and then after opening its showing page not found and error etc and i'm not able to understand weather there is any mistake in implementation or in testing! if u can look into the chnages and let me know if there is any isse so i can improve as most of the things are
