-
bitmap
yvanzo: it's on paco now
2020-05-19 14043, 2020
-
yvanzo
(I reproduced the shared memory issue using test report)
2020-05-19 14046, 2020
-
yvanzo
thanks
2020-05-19 14035, 2020
-
ruaok
legoktm: do you have any ideas what we should do about our IP block being blocked by wikimedia?
2020-05-19 14008, 2020
-
legoktm
Yes
2020-05-19 14026, 2020
-
legoktm
Is it a dedicated IP? We can apply for a block exemption
2020-05-19 14040, 2020
-
ruaok
how do we do that?
2020-05-19 14044, 2020
-
ruaok
yes , it is.
2020-05-19 14059, 2020
-
ruaok
bitmap: what was the external IP of the server that runs the bot?
2020-05-19 14030, 2020
-
bitmap
the error they returned says it's 138.201.203.23
2020-05-19 14029, 2020
-
ruaok
yep, makes sense. zappa.
2020-05-19 14052, 2020
-
legoktm
The bot is logged it with an account, right? What's the username?
2020-05-19 14032, 2020
-
bitmap
MineoBot I think?
2020-05-19 14044, 2020
-
ruaok
that I do not know.
2020-05-19 14055, 2020
-
ruaok
sounds right.
2020-05-19 14008, 2020
-
legoktm
2020-05-19 14048, 2020
-
legoktm
I'm going to send an email to the Wikimedia stewards, who should I cc on it?
2020-05-19 14000, 2020
-
ruaok
reosarevok@meb , please
2020-05-19 14026, 2020
-
yvanzo
bitmap: pgbouncer is actually running from the same container as postgres, doesn't seem possible to use PAUSE/RESUME at all.
2020-05-19 14058, 2020
-
bitmap
how come?
2020-05-19 14008, 2020
-
yvanzo
or am I missing something?
2020-05-19 14018, 2020
-
bitmap
you have to connect to the pgbouncer database on whatever port pgbouncer is listening on (6899 internally, not sure what the external one is)
2020-05-19 14019, 2020
-
bitmap
pg & pgbouncer run inside the same container but listen on different ports
2020-05-19 14054, 2020
-
bitmap
after the PAUSE you'd probably want to pg_ctl restart and then RESUME asap
2020-05-19 14006, 2020
-
legoktm
2020-05-19 14035, 2020
-
yvanzo
bitmap: I need to restart the container, not just pg_ctl.
2020-05-19 14037, 2020
-
ruaok
<3 thank you!
2020-05-19 14004, 2020
-
reosarevok
Thanks legoktm!
2020-05-19 14054, 2020
-
yvanzo
Thanks!
2020-05-19 14026, 2020
-
bitmap
oh, sorry
2020-05-19 14034, 2020
-
bitmap
I keep thinking we're just restarting it for the postgresql.conf changes
2020-05-19 14015, 2020
-
legoktm
np :)
2020-05-19 14029, 2020
-
yvanzo
bitmap: no problem, I wasn't sure to understand it all :)
2020-05-19 14013, 2020
-
bitmap
(that is an argument for running pgbouncer in a separate container, though)
2020-05-19 14018, 2020
-
Chinmay3199 joined the channel
2020-05-19 14026, 2020
-
yvanzo
should I stop sentry first?
2020-05-19 14015, 2020
-
yvanzo
(I will wait for json-dump that should run in 2min)
2020-05-19 14007, 2020
-
bitmap
should be fine to leave sentry running, might log some internal errors but
2020-05-19 14017, 2020
-
bitmap
the other option for minimizing downtime is doing a failover to pink, but that hasn't been tested in production either, heh
2020-05-19 14008, 2020
-
bitmap
that's all handled by repmgr and there are tests in the docker-postgres-cluster repo to show it works, but still a bit scary
2020-05-19 14039, 2020
-
MajorLurker joined the channel
2020-05-19 14057, 2020
-
yvanzo
we should definitely try to put failover in place but that might not be the right time to test it :D
2020-05-19 14009, 2020
-
ruaok
2020-05-19 14010, 2020
-
BrainzBot
LB-599: lastest_listened_ts is not longer used
2020-05-19 14003, 2020
-
iliekcomputers
Phew
2020-05-19 14019, 2020
-
iliekcomputers
I was worried my client would break
2020-05-19 14023, 2020
-
iliekcomputers
Thanks for digging into it, ruaok
2020-05-19 14053, 2020
-
ruaok
well, it caused the integration tests to fail. go us. :)
2020-05-19 14003, 2020
-
ruaok
said ironically and not.
2020-05-19 14032, 2020
-
ruaok
I <3 real time continuous aggregates.
2020-05-19 14017, 2020
-
ruaok
2020-05-19 14032, 2020
-
ruaok
now we have fast access to the min and max timestamps.
2020-05-19 14017, 2020
-
d4rkie joined the channel
2020-05-19 14059, 2020
-
yvanzo
bitmap: that worked, running DuplicateArtist report no longer triggers this error.
2020-05-19 14012, 2020
-
Nyanko-sensei has quit
2020-05-19 14009, 2020
-
yvanzo
2020-05-19 14002, 2020
-
bitmap
oh yes, I ran the pg12 upgrade scripts on the musicbrainz_schema_change db, but the test containers haven't been updated yet
2020-05-19 14024, 2020
-
bitmap
2020-05-19 14015, 2020
-
yvanzo
should we proceed with other postgres instances now?
2020-05-19 14014, 2020
-
yvanzo
or maybe we should schedule it for tomorrow?
2020-05-19 14013, 2020
-
bitmap
we could do it on pink today & floyd tomorrow
2020-05-19 14056, 2020
-
yvanzo
ok, that would unlock reports at least
2020-05-19 14032, 2020
-
bitmap
reports are run on floyd though since they write to the db
2020-05-19 14010, 2020
-
yvanzo
:(
2020-05-19 14022, 2020
-
bitmap
we should fix mbs to switch to pink when in DB_READ_ONLY mode, and not break when that's the case
2020-05-19 14009, 2020
-
bitmap
which might just be preventing people from logging in since it errors on trying to update last_login_date :)
2020-05-19 14053, 2020
-
legoktm
bitmap: the block exemption is in place now, so the bot should be able to edit now :)
2020-05-19 14010, 2020
-
bitmap
wow that was fast :) thanks!
2020-05-19 14026, 2020
-
ruaok
very cool, thanks!
2020-05-19 14048, 2020
-
bitmap
yes I don't see any errors in the wikidata-bot logs anymore 👍
2020-05-19 14056, 2020
-
ruaok
woo!
2020-05-19 14042, 2020
-
legoktm
:D
2020-05-19 14045, 2020
-
legoktm
2020-05-19 14026, 2020
-
bitmap looks at fixing its editing on the MB side, looks like a consul key is missing
2020-05-19 14052, 2020
-
bitmap
zas: I pushed an update to a consul config in docker-server-configs, but it doesn't seem to be getting into consul, not sure why
2020-05-19 14044, 2020
-
zas
bitmap: which update?
2020-05-19 14057, 2020
-
bitmap
when I set "down": "yes" in the upstream_services config yesterday it didn't work either, maybe the same issue
2020-05-19 14007, 2020
-
bitmap
2020-05-19 14021, 2020
-
bitmap
git2consul down/broken maybe?
2020-05-19 14011, 2020
-
zas
restarting it
2020-05-19 14005, 2020
-
bitmap
yeah, the key magically appeared now
2020-05-19 14012, 2020
-
ZaphodBeeblebrox joined the channel
2020-05-19 14012, 2020
-
ZaphodBeeblebrox has quit
2020-05-19 14012, 2020
-
ZaphodBeeblebrox joined the channel
2020-05-19 14034, 2020
-
CatQuest has quit
2020-05-19 14040, 2020
-
v6lur_ has quit
2020-05-19 14048, 2020
-
rdswift has quit
2020-05-19 14057, 2020
-
Chinmay3199 has quit
2020-05-19 14003, 2020
-
rdswift joined the channel
2020-05-19 14013, 2020
-
BrainzGit
2020-05-19 14036, 2020
-
bitmap
yvanzo: ^ we can deploy that before restarting floyd, then MB can stay up in read-only mode with all connections going to pink
2020-05-19 14013, 2020
-
bitmap
also turns out there isn't an issue with last_login_date, it already skips updating that with DB_READ_ONLY set
2020-05-19 14048, 2020
-
d4rkie has quit
2020-05-19 14037, 2020
-
Nyanko-sensei joined the channel
