21:13 PM
lucifer
yes makes sense.
2023-11-05 30957, 2023
21:13 PM
bitmap
lucifer: there is a setting to log disconnections, but it's not enabled
2023-11-05 30900, 2023
21:14 PM
mayhem
2023-11-05 30917, 2023
21:14 PM
Pratha-Fish
Would you guys mind if I create a smol alternate MusicBrainz docker instance on wolf? If so, how do I do it without interfering with the existing instance?
2023-11-05 30922, 2023
21:14 PM
mayhem
from gaga in tmux, I run `ssh -L 55432:127.0.0.1:5432 robert@wolf`
2023-11-05 30954, 2023
21:14 PM
bitmap
Pratha-Fish: the container I made is running with --network host, so it can access any ports exposed on the host
2023-11-05 30907, 2023
21:15 PM
bitmap
I'm not sure which port the database you want to access is on, though
2023-11-05 30919, 2023
21:15 PM
Pratha-Fish
bitmap: it's on 5432
2023-11-05 30921, 2023
21:15 PM
lucifer
mayhem: docker container won't be able to access localhost ports i think.
2023-11-05 30937, 2023
21:15 PM
lucifer
not sure if we run with --network=host on prod.
2023-11-05 30900, 2023
21:16 PM
Pratha-Fish
*5432 until and unless I start another musicbrainz-docker instance different from the existing one
2023-11-05 30911, 2023
21:16 PM
mayhem
sorry forgot to mention I am not running on docker.
2023-11-05 30928, 2023
21:16 PM
mayhem
just a venv. ends up being easier for tunneling
2023-11-05 30931, 2023
21:16 PM
lucifer
ah cool then
2023-11-05 30934, 2023
21:16 PM
bitmap
Pratha-Fish: then try psql -h localhost -p 5432 -U musicbrainz -d musicbrainz_db
2023-11-05 30946, 2023
21:16 PM
lucifer
looks good to me
2023-11-05 30932, 2023
21:17 PM
Pratha-Fish
bitmap: yes, I've been using that one for a while now :D
2023-11-05 30948, 2023
21:17 PM
mayhem
forgot to fetch the right password.
2023-11-05 30904, 2023
21:18 PM
bitmap
Pratha-Fish: I mean it should work inside the python container too
2023-11-05 30915, 2023
21:18 PM
Pratha-Fish
whoops
2023-11-05 30933, 2023
21:18 PM
mayhem
query started. monitoring wolf.
2023-11-05 30956, 2023
21:18 PM
bitmap
Pratha-Fish: also, I mounted your home dir as /snaek inside the container
2023-11-05 30958, 2023
21:18 PM
Pratha-Fish
bitmap: thanks that worked!
2023-11-05 30913, 2023
21:19 PM
Pratha-Fish
thanks a ton 🥹
2023-11-05 30937, 2023
21:19 PM
bitmap
try `su snaek` in the container and then cd /snaek
2023-11-05 30952, 2023
21:19 PM
bitmap
I set the uid/gid to the same as the host
2023-11-05 30903, 2023
21:20 PM
lucifer
mayhem: uhh, i think we also lost the messybrainz table in that issue.
2023-11-05 30922, 2023
21:20 PM
lucifer
its going to be another PITA to restore that table.
2023-11-05 30925, 2023
21:20 PM
mayhem
oh joy.
2023-11-05 30933, 2023
21:20 PM
mayhem
how so?
2023-11-05 30948, 2023
21:20 PM
lucifer
uuid are generated randomly and not from the source data.
2023-11-05 30919, 2023
21:21 PM
mayhem
but we backup all those tables with pg-dump no?
2023-11-05 30954, 2023
21:21 PM
Pratha-Fish
bitmap: I can only see directory called "csp-errors" in /snaek
2023-11-05 30922, 2023
21:22 PM
bitmap
oh that might be my home dir haha. sorry, let me fix
2023-11-05 30935, 2023
21:22 PM
lucifer
mayhem: i just checked and no. msb table is not backed up.
2023-11-05 30950, 2023
21:22 PM
mayhem
O_O
2023-11-05 30957, 2023
21:22 PM
mayhem
wow.
2023-11-05 30911, 2023
21:23 PM
mayhem
we need to review our backups. this is bad. :(
2023-11-05 30922, 2023
21:23 PM
lucifer
we can still get back the data from the listens but yeah excruciating.
2023-11-05 30925, 2023
21:23 PM
lucifer
indeed
2023-11-05 30956, 2023
21:23 PM
mayhem
SELECT * INTO ?
2023-11-05 30904, 2023
21:24 PM
mayhem
well not *, but msids
2023-11-05 30927, 2023
21:24 PM
bitmap
Pratha-Fish: now it should be fine
2023-11-05 30938, 2023
21:24 PM
lucifer
yeah msids, and the 4-5 fields that go into. plus deduplicating with all the data that has accumulated in the past 4 days
2023-11-05 30934, 2023
21:25 PM
mayhem
ok, how about this:
2023-11-05 30937, 2023
21:25 PM
mayhem
1. stop TS wrtier
2023-11-05 30942, 2023
21:25 PM
mayhem
2. SELECT INTO
2023-11-05 30943, 2023
21:25 PM
Pratha-Fish
bitmap: working!
2023-11-05 30900, 2023
21:26 PM
mayhem
3. REstart TS writer
2023-11-05 30906, 2023
21:26 PM
mayhem
4. Write script to fix up 4 days.
2023-11-05 30922, 2023
21:26 PM
lucifer
yup that's the plan.
2023-11-05 30939, 2023
21:26 PM
mayhem
ok, can we get started with 1-3 asap>
2023-11-05 30955, 2023
21:26 PM
mayhem
?
2023-11-05 30930, 2023
21:28 PM
lucifer
we could but its better to run in the morning. otherwise the ts writer will have to be stopped overnight.
2023-11-05 30941, 2023
21:28 PM
mayhem
ok, fair.
2023-11-05 30946, 2023
21:28 PM
mayhem
its late enough as is.
2023-11-05 30930, 2023
21:29 PM
mayhem
lol
2023-11-05 30942, 2023
21:29 PM
lucifer
there's one more unbacked up table that is going to be hard to restore. mbid_manual_mapping.
2023-11-05 30945, 2023
21:29 PM
mayhem
I just ran the partial set of the mb-metadata cache data.
2023-11-05 30946, 2023
21:29 PM
mayhem
2023-11-05 30902, 2023
21:30 PM
mayhem
and sure enough data is now being returned, for those 3 artists. that's a good sign.
2023-11-05 30919, 2023
21:30 PM
mayhem
hard?
2023-11-05 30937, 2023
21:30 PM
mayhem
where would we get the data from?
2023-11-05 30937, 2023
21:30 PM
lucifer
it would be impossible. but we have it mixed in spark dumps.
2023-11-05 30954, 2023
21:30 PM
mayhem
🤯
2023-11-05 30910, 2023
21:31 PM
mayhem
this is going to be a fun week, isn't it?
2023-11-05 30941, 2023
21:31 PM
lucifer
compare the automapper assigned with the recording mbid present in the dumps, if its different and not present in the actual listen submitted to LB then it was a manually mapped one.
2023-11-05 30902, 2023
21:32 PM
lucifer
so have to compare three datasets to recover that.
2023-11-05 30934, 2023
21:32 PM
mayhem
🤯 🤯 🤯
2023-11-05 30901, 2023
21:33 PM
lucifer
i am amazed that postgres let that query delete all tables in the script but kills our cache queries before completing execution every time.
2023-11-05 30942, 2023
21:33 PM
mayhem
might be a level of effort optimization
2023-11-05 30932, 2023
21:36 PM
bitmap
lucifer: regarding the pgbouncer logs, I can see why we disabled them, since it logs an entry every 0.005 seconds or less (and usually the info is not useful)
2023-11-05 30939, 2023
21:37 PM
lucifer
LB-1392
2023-11-05 30940, 2023
21:37 PM
BrainzBot
2023-11-05 30910, 2023
21:38 PM
lucifer
bitmap: i see, is it possible to change that setting without restart?
2023-11-05 30915, 2023
21:38 PM
mayhem
I've started on the PR to dump all tables.
2023-11-05 30924, 2023
21:38 PM
bitmap
yeah, I can make it temporarily log disconnections
2023-11-05 30948, 2023
21:38 PM
bitmap
but it will probably just say "client close request" without any other info
2023-11-05 30951, 2023
21:38 PM
lucifer
cool, let's get on that tomorrow.
2023-11-05 30908, 2023
21:39 PM
lucifer
can we identify the connection that closed down?
2023-11-05 30926, 2023
21:39 PM
mayhem
I'll start working on it now.
2023-11-05 30938, 2023
21:39 PM
bitmap
it shows something like musicbrainz_db/musicbrainz@172.17.0.1:45668 closing because: client close request (age=0s)
2023-11-05 30938, 2023
21:39 PM
lucifer
mayhem: yeah that is one part, other is to error in say CI if we add a new table and forget to dump it.
2023-11-05 30908, 2023
21:40 PM
bitmap
those docker ports are completely opaque afaik
2023-11-05 30919, 2023
21:40 PM
lucifer
i see yeah. :/
2023-11-05 30923, 2023
21:40 PM
mayhem
good idea
2023-11-05 30926, 2023
21:40 PM
lucifer
thanks for checking that bitmap.
2023-11-05 30935, 2023
21:40 PM
bitmap
np
2023-11-05 30952, 2023
21:41 PM
bitmap
you are disabling the statement_timeout and pg is killing it for some other reason? (if pg is killing it, it should be in the pg logs, anyway)
2023-11-05 30937, 2023
21:42 PM
mayhem
lucifer: do we dump mapping.mb_metadata_cache ?
2023-11-05 30948, 2023
21:42 PM
mayhem
a lot of generated data to dump.
2023-11-05 30911, 2023
21:43 PM
mayhem
lucifer: while we're at it, we should switch to zstd compression. that is going to make dumps and moving data to spark a lot faster.
2023-11-05 30917, 2023
21:44 PM
Pratha-Fish
does tmux work with docker? 💀
2023-11-05 30942, 2023
21:44 PM
bitmap
Pratha-Fish: in what way?
2023-11-05 30901, 2023
21:45 PM
Pratha-Fish
I did the following:
2023-11-05 30911, 2023
21:45 PM
lucifer
mayhem: I would probably not dump generated data.
2023-11-05 30929, 2023
21:45 PM
lucifer
But rather make it reliable to generate
2023-11-05 30943, 2023
21:45 PM
Pratha-Fish
Logged into wolf > entered tmux > entered the python 3.11 container > su snaek > started_some_job
2023-11-05 30949, 2023
21:45 PM
mayhem
ok, then I see that we need to add "messybrainz.submissions" and "mbid_manual_mapping"
2023-11-05 30952, 2023
21:45 PM
bitmap
Pratha-Fish: yes that's fine
2023-11-05 30901, 2023
21:46 PM
Pratha-Fish
Then I exited tmux by pressing CTRL + B > D
2023-11-05 30917, 2023
21:46 PM
lucifer
Makes sense
2023-11-05 30922, 2023
21:46 PM
Pratha-Fish
Now, that thew me back to home dir as expected
2023-11-05 30951, 2023
21:46 PM
Pratha-Fish
but when I entered tmux again, I expected to go back inside the docker container and see updates on the job I was running
2023-11-05 30903, 2023
21:47 PM
Pratha-Fish
But entering tmux again just got me to the home dir
2023-11-05 30941, 2023
21:47 PM
bitmap
tbh I don't remember how to re-attach with tmux (I use screen)
2023-11-05 30924, 2023
21:48 PM
Pratha-Fish
NP, I'll try to figure something out and come back to you if it doesn't work :)
2023-11-05 30930, 2023
21:48 PM
bitmap
are you sure it's not just opening another session? can you switch sessions?
2023-11-05 30907, 2023
21:49 PM
Pratha-Fish
verifying
2023-11-05 30946, 2023
21:49 PM
bitmap
session or window I guess
2023-11-05 30954, 2023
21:49 PM
BrainzGit
2023-11-05 30933, 2023
21:50 PM
Pratha-Fish
Yes, I connected to the same session :D
2023-11-05 30937, 2023
21:50 PM
Pratha-Fish
Still no luck
2023-11-05 30900, 2023
21:51 PM
Pratha-Fish
Looks like I'll have to install tmux inside the container
2023-11-05 30937, 2023
21:53 PM
Pratha-Fish
2023-11-05 30913, 2023
21:54 PM
bitmap
no you just stopped the container
2023-11-05 30939, 2023
21:54 PM
Pratha-Fish
phew seems like it
2023-11-05 30921, 2023
21:55 PM
Pratha-Fish
Sorry for the silly question again, but how do I restart it?
2023-11-05 30929, 2023
21:55 PM
bitmap
docker start snaek-python
2023-11-05 30942, 2023
21:55 PM
Pratha-Fish
thanks 🤦
2023-11-05 30901, 2023
21:57 PM
bitmap
sorry I'm not sure about your tmux issue, I know the same works fine in screen. if you wanna try installing tmux in the container, go ahead
2023-11-05 30901, 2023
22:00 PM
Pratha-Fish
bitmap: just figured it out :D
2023-11-05 30909, 2023
22:00 PM
Pratha-Fish
as simple as installing tmux on the container
2023-11-05 30919, 2023
22:00 PM
Pratha-Fish
and attaching to the session after entering the container
2023-11-05 30934, 2023
22:01 PM
bitmap
good :)
2023-11-05 30900, 2023
22:05 PM
Pratha-Fish
:)
2023-11-05 30900, 2023
22:20 PM
petitminion has quit
2023-11-05 30946, 2023
22:20 PM
petitminion joined the channel
2023-11-05 30906, 2023
22:21 PM
Pratha-Fish
bitmap: looks like we have something tangible on our hands right now :)
2023-11-05 30952, 2023
22:21 PM
Pratha-Fish
Not to mention reosarevok
2023-11-05 30919, 2023
22:23 PM
Pratha-Fish
2023-11-05 30919, 2023
22:23 PM
Pratha-Fish
2023-11-05 30914, 2023
22:25 PM
atj
`tmux a` re-attaches to an existing session btw
2023-11-05 30929, 2023
22:25 PM
Pratha-Fish notes down
2023-11-05 30934, 2023
22:25 PM
atj
Just running tmux starts a new session
2023-11-05 30927, 2023
22:26 PM
Pratha-Fish
atj: thanks, I was making the same mistake lmao. now I just use tmux attach -t sesh_num
2023-11-05 30945, 2023
22:26 PM
atj
2023-11-05 30939, 2023
22:30 PM
Pratha-Fish
E p i c
2023-11-05 30949, 2023
22:33 PM
Lotheric has quit
2023-11-05 30931, 2023
23:24 PM
BrainzGit
2023-11-05 30956, 2023
23:25 PM
kellnerd
My last PR for GSoC, repeated imports can now "overwrite" pending entities :)
2023-11-05 30946, 2023
23:56 PM
Pratha-Fish
kellnerd: congrats on the work! 🥳
