#metabrainz

onculayn: weird, which picard version is that?

Are you using picard-dev from pypi?

Oh btw which python3 version are you using?

Picard requires 3.5 minimum

zas, you up?

yes

Wanna try setting up a replica?

I wanna see how long it takes

Hmm, I'm getting a lot of "an error ocurred while searching" in inline search

(that work perfectly fine at the next attempt)

Any idea what's going on?

samj1912, zas ^^

hmmm let me see, i made a change that could cause that

which request ?

what change did you make?

zas: everything, at seemingly random, in inline search

just with inline search?

tried few queries and all worked

hmmm i see there's an increase in 4xx

No, I got something with normal now, "not enough info" or something, retry worked

samj1912: see last comment on https://community.metabrainz.org/t/search-resul... - fairly sure this used to work (and IMO it should definitely work)

zas: still getting a ton of them

In beta, if it matters

i don't see which changes i made could cause that

i had to reload haproxy config to change backend weights, but i didn't change anything else

samj1912: ^^ any idea?

i switched haproxy to lb2 to see if it happens on both nodes, if it does i suspect a solr issue

i upgraded, rebooted, and switched back to lb1

Maybe one server is having issues?

It's weird it works instantly sometimes, and not at all others

(also instantly)

Huh

Weird

Probably related to one of the solr servers then

Huh weird, it's sometimes returning results and sometimes not

zas, can't find anything wrong but 4xx suddenly increased after 14:00 IST

zas, did you change something recently?

Solr response times went to shit

zas, I am restarting all solr servers

Something weird is happening

Lots of 4xx suddenly

yvanzo: FWIW the -vm build showed zero errors -- worked as expected.

ruaok: my cooler works, but I think your management interfacer is better than mine: https://usercontent.irccloud-cdn.com/file/rC94a...

also zas can you break those response codes by solr nodes?

interesting. what happened circa 19h and 4:30h?

oh, I added more ice :)

do you have an icebox and then you ventilate from it?

Does the icebox have plums?

yes, there's a fan that pulls cold air from an ice compartment into the main compartment

reosarevok: they're delicious

(I also lowered the target temperature at 6pm and 4am, so it pulled more cold air in)

with enough ice I can definitely lower the temperature 10° lower than ambient. I've not tried to pull it down more

nice.

No, ice

zas: lots of logs like

proxy issue?

ruaok: the most breaking change is that base image are now made for Ubuntu Cloud using cloud-init.

looks like it

yvanzo: ok, what is our next step then?

normal reqs should look like this

VM network settings can probably be simplified too.

samj1912: I've pinged zas elsewhere too, hopefully we can get his attention.

cool

ruaok: bento/ubuntu-16.04

reosarevok: the above is probably why all those no result responses are occuring

Got it :)

it is lighter and probably requires less customization

I mean, they're instant, so it's not too bad yet

XD

😂

samj1912: i just changed to weights, and did a config reload, i see nothing on haproxy side which may cause this

zas: did you see the logs above?

lol i know, i deleted some lines in the whitelist.... unintentionnally

yvanzo: do you need me to make the changes?

nope, I’m creating a branch for that

zas: one more thing btw - what happens when people with slave servers use solr?

iliekcomputers: that's funny since we're still working samj to recognize what a weekend is. :)

currently people can point to search.mb.org

but that isnt currently possible with solr since non hetzner reqs are blocked

sounds like we need to setup another proxy for that.

okay 4xx are gone, but response times are still bad

also wth happened with rmq ? https://stats.metabrainz.org/d/000000078/sir-an...

msg rates got double in the last 12 hrs

was some new service deployed?

no idea, about both

ruaok: ^^ ?

also zas https://stats.metabrainz.org/d/T4MODrIiz/solr-c...

no clue.

it doesn't look good, I can tell you that.

#nothelping

did anything change at around 11:03? (related to solr)

I think you're the only person tinkering with this.

nope, wasnt touching solr around that time

also those rmq things are due to BQ jobs

just checked the rmq admin dash

oh, yeah first of the month.

a bit late in the day though. iliekcomputers?

started around 11 hrs ago

*9

oh, perfect. yes. all normal.

but that graph should be updated. that doesn't relate to solr at all.

just gets us all sketchy...

yeah, its just a rmq graph

we should add a sir filter to it

yvanzo & zas: invoices please!

Leo_Verto: ping

re https://github.com/metabrainz/brainzutils-pytho..., how do you plan to add integration tests? Try to do something similar to how integration tests work in LB (meaning a different docker-compose.yml).

Also, do you think we should just fix the unit tests in the PR and merge this and integration tests should be a different PR?