#metabrainz

bitmap, ruaok: to finish the switch to aretha, we need to ensure data is properly sync to ftp, then I'll proceed to dns changes, ping me when you're around

(mb/lb dumps)

yvanzo: Good morning

Good afternoon

I roughly finished rebasing last night, but the popover also introduces new bugs.

And I read the feedbacks from editors, I think we should reconsider keyboard navigation

Because PR #2114 also has effects on it

btw, I haven't pushed the rebasing results because of new bugs

yyoung: can you be more specific about feedbacks and keyboard navigation?

yvanzo: See https://community.metabrainz.org/t/a-discussion...

moin!

zas: ping

yyoung: I’m still amazed at what happened on this channel the other day, we asked for feedback not for 40min drama about a textbox that has finally been admitted as "quite nice". I don’t understand that reosarevok encouraged CatQuest with it and drew a line between two sides rather than helped with keeping it cool. No ticket has been created about it afaik, so did you look at actual issues they

mentioned?

I think your view of what happened is quite unfair but I'm not going to argue about it :) The comments on that thread seem to bring up similar issues to what we mentioned here, FWIW.

yyoung: If you are blocked with anything related to the UI, maybe take some advance on the back-end URLCleanup refactoring that will come next.

reosarevok: I tried to read the back logs but it's uneasy due to all the ":|", "ooooooooooooh", "ಠ_ಠ","Grr"and so on.

yvanzo: Can you be more specific? IIRC there's no backend code related to URLCleanup

FWIW, I feel both your pains, yvanzo and reosarevok.

Ok, the summary was: "the bubble is nice but nothing in the bubble seems to really require making the life of editors more complicated by not giving them the URL edit fields they are used to and requiring extra clicks from them" :)

ruaok: I feel yvanzo's pain too, I've been on the other side of this kind of situation and I know it feels shitty :/ As I told him the other day too.

yyoung: week 6 and week 8 in your proposal

yvanzo: To be honest, the current issue was put forward as soon as the thread is posted, and we've talked about that

yvanzo: OK I'll look into it, but some details still need to be discussed

ruaok: pong

yyoung: Details about URLCleanup or something else? Either case, just ask :)

zas: what do you need me to do?

we need to ensure lb dumps lands to aretha before switching ftp host name to it

same goes for mb dumps

lucifer: are you about?

I think we got a notification that said that things are working well.

let me check.

hi! yes

i think inc dump failed yesterday.

http://data.metabrainz.org/pub/musicbrainz/list... does not have 501

but with any luck it will be on aretha. in which case it didn't fail.

ah ok! makes sense

can you check files are on aretha?

sadly, no 501 on aretha.

bitmap: we have to do the same thing for mb dumps

i guess its expected. 501 Not Implemented error.

cron prod is running. let me check log.

lololol, good one, lucifer

zas: some scripts might currently be running to export full data dumps and search index dumps

yes, can we stop them once they finished? at some point we need to ensure nothing is going to williams anymore

once finished yes, prolly this afternoon.

I noticed start_sshd_musicbrainz_fullexport / start_musicbrainz_search_indexes_dump aren't yet on aretha

huh.

that's odd.

yvanzo: It's about the validation rules that need to be implemented, we've discussed before but did not reach a clear conclusion.

And about the implementation, I'm thinking about plugging in another module to take care of validation and auto-selection of multiple rels

zas: my bad, exports ran yesterday (on Wed) not today.

so, its running from cron -- must be something about how it is being invoked via cron

we have click in https://github.com/metabrainz/listenbrainz-serv... though

Since from our previous research, having multiple rels is not a common case.

yvanzo: then can you arrange for them to land on aretha instead of williams?

zas: yes

I guess we can define a set of valid combinations of rels for 'get the music' type.

there's a sync_ftp.sh script in /root on aretha, it will resync from williams from aretha, if needed

We haven't mention other types that can have multiple rels though.

ruaok: difference between 'python' and 'python3' ?

that was exactly my thought

seems not.

well, from the command line might be a bad test.

I suppose I can change it to python and relaunch the container with a cron entry that fires real soon.

but I somehow doubt that is it.

click is not a standard module, it has to be installed

the first one that you pasted, is there more context to that log? (can I look somewhere?)

yyoung: Validation should probably be redesigned from the ground: maybe split into URL validation, relationship (URL + entity type + relationship type) validation, set of relationships related to the same URL validation, set of all URL relationships validation. (Just an arch sketch in the air.)

`root@kiss /var/lib/docker/volumes/listenbrainz-dump-logs/_data # view dumps.log`

alastairp: ^^

thx

alastairp: docker exec -it listenbrainz-cron-prod tail -f -n 100 /logs/dumps.log on kiss

there is another python in /usr/bin

when create dumps is run, its not running with PATH set to include the version from /usr/local/bin is my guess.

both are same methinks. python --version and python3 --version both return 3.7.9.

yeah, I just noticed that the create-dumps.sh uses `python3` in some places and /usr/local/bin/python in other

I think explcitly loading python from /usr/local/bin/python would fix this.

but yes, #PATH has /usr/l/b at the beginning

let me make it explcit and see what happens.

oh

wait

cron

alastairp: exactly that.

yeah that seems to be it, ruaok

let me fix it and set an incremental dump for 10 mins from now.

there's also a python 3.5 in there.

we should probably get rid of those as a further improvement later.

lucifer: ah right, that

that'll be it

/usr/local/bin/python in the dump script runs the version we installed. `python3` runs 3.5 from /usr/bin

yes, removing the apt python packages is definitely a good idea too

although sigh. it wants at least one of py2 or py3 installed :(

not that it makes much difference if it fails with "click not found" or "python not found"

yes, right. but the intent is to remove it from base docker-python images so that we don't run into other weird errors later on in any project as well.

meh. "Not running in a production env, not locking cron."

well, at least we can still test that incremental dumps end up on aretha, which is most important right now.

yvanzo: Make sense, but I'll have to confirm that there isn't any tightly coupled logic. I've also got some nice idea from the discussion thread on the forum, like defining which type allows multiple rels.

+1

zas: there is hope for us. I hope this article is right: http://www.oilshell.org/blog/2021/07/blog-backl...

alastairp: what issue did you get? i tried just now and was able to remove both pythons from base image.

lucifer: it was more that `remove python3` automatically installed python2 - which I was then able to uninstall

it's possible that a smarter remove line would help

i did `apt uninstall python3` followed by `apt autoremove` and both python got removed.

ah sorry, apt remove python3

right. it'd be nice to try and make that happen in 1 go, without the uninstall adding py2

lucifer: the dumps ended up on williams, not on aretha.

i guess that makes sense. since consul still has williams as ftp?

is that how we determine where to copy files?

yes, iirc. let me confirm.

zas: do you have a script/cmd in your history to rsync williams -> aretha?

we had one file to to williams, likely because consul still thinks of it as the official FTP

zas: ^^

yes.

so we need to rsync williams -> aretha.

and we should do this right before the make the consul change.

then I can test with another incremental dump to make sure they arrive on aretha.

[listenbrainz-server] 14mayhem opened pull request #1563 (03master…fix-cron-dumps): Fix cron dumps https://github.com/metabrainz/listenbrainz-serv...

lucifer: alastairp ^^

once this is approve, I'll restart cron

done

ruaok: we should update this in https://github.com/metabrainz/listenbrainz-serv... too

ok, let me augment that PR then.

(there are 2 calls in that file)

alastairp: PR updated.

btw, why do we need a subprocess call there? why invoke the method directly?

*why not invoke

lucifer: we didn't specifically create this file as a module

i.e. ./admin/ isn't a python package