Saving to: ‘/media/searchdump/picard-setup-2.6.0b3.exe.md5’
2021-04-26 11608, 2021
nelgin
Saving to: ‘/media/searchdump/picard-setup-2.6.1.exe’
2021-04-26 11608, 2021
nelgin
Saving to: ‘/media/searchdump/picard-setup-2.6.exe’
2021-04-26 11608, 2021
nelgin
Saving to: ‘/media/searchdump/picard-setup-2.6.exe.md5’
2021-04-26 11608, 2021
nelgin
Why would it download and save all these?
2021-04-26 11601, 2021
iliekcomputers
moin!
2021-04-26 11631, 2021
nelgin
'sup
2021-04-26 11659, 2021
iliekcomputers has quit
2021-04-26 11627, 2021
shivam-kapila has quit
2021-04-26 11636, 2021
iliekcomputers joined the channel
2021-04-26 11640, 2021
nelgin
wb
2021-04-26 11643, 2021
shivam-kapila joined the channel
2021-04-26 11603, 2021
reosarevok
nelgin: maybe yvanzo or zas will know, if they appear
2021-04-26 11658, 2021
nelgin
Maybe.
2021-04-26 11616, 2021
alastairp
reosarevok: hi, as part of this project that I'm doing at UPF we have links for about 50k items from https://www.muziekweb.nl/ -> MB and -> IMSLP
2021-04-26 11619, 2021
nelgin
I'm sure it is stuck in a loop downloading stuff.
2021-04-26 11632, 2021
ruaok
moin!
2021-04-26 11633, 2021
alastairp
is this interesting to import?
2021-04-26 11627, 2021
nelgin
Erm eh?
2021-04-26 11644, 2021
alastairp
we can provide a csv or similar, I guess it's the kind of thing that could be imported with a bot? Muziekweb are using this data already in their db, so I guess they trust it, but we could also do some kind of spot-check of its quality
2021-04-26 11634, 2021
nelgin
It's downloaded replication data files from 2009. Picard files. This is stupid.
2021-04-26 11648, 2021
nelgin
changed-ids since 2013
2021-04-26 11631, 2021
alastairp
nelgin: reosarevok: oh, interesting. I wonder if this is related to the new http download server that we added. and curl is doing a recursive download, following the ".." links
2021-04-26 11632, 2021
reosarevok
alastairp: Hmm, not sure. Or well: I'm sure we're interested, not sure what's the best way to deal with it :)
2021-04-26 11601, 2021
alastairp
reosarevok: one of the aims of the project was to "contribute back to the open resources that we use", so this is us trying to contribute back :)
2021-04-26 11612, 2021
alastairp
in any case, I'm happy to open a ticket and attach a csv to it
2021-04-26 11615, 2021
nelgin
Did you test it?
2021-04-26 11622, 2021
nelgin
Cos whatever it is, it aint working.
2021-04-26 11615, 2021
reosarevok
alastairp: that'd be neat, then we can open a forum post
2021-04-26 11637, 2021
alastairp
ok, we're planning on building up this list over the next few weeks, so I'll do that
2021-04-26 11637, 2021
reosarevok
nelgin: maybe alastairp is right, and that change had unexpected consequences for the docker scripts. Then we really could use yvanzo...
2021-04-26 11612, 2021
nelgin
Not good policy to rely on one person for such a project?
If that's in the file, then I would say no, that's not what it should be.
2021-04-26 11611, 2021
nelgin
I'm using the US rather than EU servers.
2021-04-26 11642, 2021
nelgin
Personally, I'd like to see BASE_FTP_URL be supposed in .env and source that in any scripts before blindly assuming people want to download from Europe.
2021-04-26 11602, 2021
outsidecontext
ah, maybe that's the issue here. because AFAIK there was no change to the US servers, they had been accessible via HTTP all the time. Only change was that fpt.eu.metabrainz.org is now also accessible via HTTP
2021-04-26 11645, 2021
nelgin
Maybe it's because I didn't use ftp:// then?
2021-04-26 11638, 2021
nelgin
Since I'm a nice guy, I can try it again and report back when I wake up in 5.5 hrs
Reusing existing connection to ftp.musicbrainz.org:80.
2021-04-26 11648, 2021
nelgin
HTTP request sent, awaiting response... 200 OK
2021-04-26 11650, 2021
nelgin
Ah there you go.
2021-04-26 11605, 2021
nelgin
So the variable is misleading, it shouldn't be FTP_BASE_URL if it's not assuming ftp.
2021-04-26 11658, 2021
outsidecontext
that could mean the same would happen to the eu servers now as well, since they are now accessible via HTTP. So someone needs to fix this script to work as expected (either force using ftp:// or properly handle the HTTP case)
our traffic level would require us to have 2-4 machines alone in order to process the traffic. that is if we measured all the MB API traffic.
2021-04-26 11643, 2021
nelgin
recording.tar.zst 99%[================================================================> ] 21.40G --.-KB/s in 32m 37s
2021-04-26 11643, 2021
nelgin
2021-04-26 09:35:26 (11.2 MB/s) - Data connection: No such file or directory;
2021-04-26 11647, 2021
nelgin
I'm about ready to give up on this
2021-04-26 11606, 2021
atj
what if you didn't log all the API traffic and just used it for Docker logs, errors etc?
2021-04-26 11619, 2021
atj
Docker suppot GELF natively IIRC
2021-04-26 11623, 2021
atj
*supports
2021-04-26 11653, 2021
ruaok
yeah, we could. but we also don't have a solution for the log shipping to get logs from all the various places into something like greylog. and once we do, we don't have a lot of use for these statistics. its nice to know what browsers people are using and identifying abuse, but putting API keys in place is far more important than analytics...
2021-04-26 11617, 2021
nelgin
Hmm, the ftp site has 3 ip addresses, one is 65ms the other 2 are 30-40ms. Makes a big difference.
ruaok: sorry, I don't have much idea about your infra, but from what I've seen on this channel, it feels like a centralised logging setup would make it a lot easier for people to track down and debug issues
2021-04-26 11633, 2021
nelgin
It would be nice if the original install script would ping US and EU servers and determine the fastest IP.
2021-04-26 11657, 2021
nelgin
While nothing is perfect, it can mean the difference between 6MB/s and 20MB/s to me.
2021-04-26 11603, 2021
atj
nelgin: latency != bandwidth
2021-04-26 11630, 2021
nelgin
No, but an overloaded pipe is going to show latency which will slow down bandwidth.
2021-04-26 11659, 2021
nelgin
and I bet you I get a higher ping to the EU servers which will likely mean lower transfer speeds.
2021-04-26 11615, 2021
alastairp
_lucifer: great, I've just finished some calls. Looking at reviews again now
2021-04-26 11645, 2021
_lucifer
Should I merge the #1404 in this one ? I have kept the two separate for easier review but the two will be deployed together.
2021-04-26 11623, 2021
alastairp
let me tell you in 20 minutes :)
2021-04-26 11642, 2021
_lucifer
👍
2021-04-26 11633, 2021
ruaok
atj: yep, agreed.
2021-04-26 11653, 2021
ruaok
alastairp: iliekcomputers _lucifer : Freso points out that some (most) spammers in LB do not have an email address set.
2021-04-26 11604, 2021
alastairp
ruaok: yeah, we opened a ticket about that
2021-04-26 11627, 2021
alastairp
saying that we should require MB accounts to have an email before allowing them to create an LB account
2021-04-26 11627, 2021
ruaok
do we have consensus that we should be requiring that? what about existing accounts?
yeah, I agree for future accounts we should have emails.
2021-04-26 11615, 2021
alastairp
happy to add a banner saying "it'd be great if you added one so that we can do things", but unsure about preventing them from logging in until they add one
2021-04-26 11637, 2021
ruaok
maybe we need to make an report for accounts that are submitting listens that do not have an email
2021-04-26 11616, 2021
ruaok
and then eventually we will flush out the bad actors.
2021-04-26 11621, 2021
alastairp
any thoughts on preventing submissions from email-less accounts (in the far future, once we get people to add them?)
2021-04-26 11639, 2021
ruaok
I think we should. the sign up process should make it clear that we will not record listens if we have no email address.
2021-04-26 11618, 2021
alastairp
yeah, that's easy to do from now on (we can stop the signup process if MB doesn't send us an email). I was asking more about current accounts which don't have an email
2021-04-26 11657, 2021
alastairp
I think we should show a banner to users who sign in who don't have an email, asking them to add one to MB [or directly to LB? probably better in MB]
2021-04-26 11617, 2021
alastairp
but I don't know if we should start imposing limits on current LB accounts that don't have an email set
2021-04-26 11634, 2021
ruaok
yes, to the banner. and maybe say that in 6 months after implementing this, that we will stop recording listens for accounts without emails.
2021-04-26 11607, 2021
alastairp
yeah, that sounds reasonable
2021-04-26 11612, 2021
alastairp
put it in tweets/blogs/etc
2021-04-26 11615, 2021
nelgin
I agree.
2021-04-26 11621, 2021
nelgin
Tho it's got nothing to do with me :)
2021-04-26 11643, 2021
ruaok
alastairp: I'll flush out the ticket and make it actionable.
recording.tar.zst 99%[================================================================> ] 21.40G --.-KB/s in 31m 46s
2021-04-26 11625, 2021
nelgin
2021-04-26 10:11:43 (11.5 MB/s) - Data connection: No such file or directory;
2021-04-26 11630, 2021
nelgin
Twice now on the same file, so I know it's not me.
2021-04-26 11614, 2021
alastairp
nelgin: thanks for your reports, it's clear that something is wrong, but yvanzo knows this much better than the rest of us. when he arrives I'm sure he'll be able to get a fix for you
2021-04-26 11633, 2021
alastairp
I guess we haven't tested this workflow with alternate mirrors as often as we test the default workflow
2021-04-26 11607, 2021
nelgin
I'm going to try from a single ip rather than using the ftp alias which is an alias of an alias. I'll see how it goes, but I'm not waiting up. I'll report back later. Night all.
