#musicbrainz

might make sense to categorize web access by network block to check for offenders

what's strange is that it just seems to be that everything takes more network than usual

it's not like there's one connection that's suddenly eating up 95Mbps

heard that this 50x thing already exists a few months b4, just not as bad?

yeah, we've always had a certain number of them

there are some things that just don't return very fast

the surge pattern is too regular

yeah, this is clearly something that's happening at a specific time

I'm poking through crontabs

stats again?

it's running at a time that doesn't make any sense though, it's two hours *before* the main daily crontab

pardon, two hours before yesterday, 3.5 today

could be external access instead of internal ones

yeah, but the access patterns don't look different than usual

it does seem like that though, since it appears to be load-balanced across the three backend servers

probably should look at access logs or something

what's suspicious is that this started right after we released a new server version

that's why i suggested checking out web log and divide by network blocks b4

but there's nothing that seems related

new server version...... just before easter, isn't it?

there was one yesterday (my time -- 2013-04-08)

there was also one march 25th, but :)

this started yesterday

hm

it is as bad as during easter

where is the data going to? just to asterix/astro/pingu?

yeah, pretty much

abelcheung: then you're thinking of something unrelated

ianmcorvidae: well, i'm just saying the 50x rate is just as bad, don't mean the reason behind

I guarantee it's not just as bad: http://stats.musicbrainz.org/webstats/nginx-rrd... :)

oh well.... :)

but hopefully we can figure something out, anyway :/

could it be something to do with cache stuff suddenly expiring?

hm?

DB cache?

well, we cache data, right? perhaps it all suddenly gets deleted and then has to be refetched from the db and totoro is like "wtf guys not all at once D:"

yeah, I dunno, that doesn't seem to explain it going for six hours or whatever :/

I would think about arrogant spiders or mirror first

judging from currently limited info

a HA

I see a pattern :)

oh, wait, that's response time

damn

hmm? http://stats.musicbrainz.org/webstats/nginx-rrd...

and... http://stats.musicbrainz.org/webstats/nginx-rrd...

sure, those are shooting up *after* the problem stops

oh.

so.... searches peak at 6pm everyday, yet db processes spawns a lot more since 8-9pm, which coincides with lagging response time

yeah, the number of processes jumps up, can't determine why

cronjob?

block read jumps too, which means those extra processes are active too

if it's a cronjob it must be cron.daily, it's not at a consistent time (changed by two hours between the two)

wait. is concurrent db transaction number peaking at 4096 everyday?

not as of the 7th, though?

for 6th and 7th, running like 4.5k

but that's the peak too

and.... i suppose that's old version of db which is not in active use

no, 20110516 is the current DB

if that's what you mean

ouch

that's when NGS was released, heh

some ulimit or pg limit might be at work too

yeah, was looking for things in config

before Jan it's going up to 6k, but never above 4-5 ever since -- and it's like a steady rhythm, that's too unnatural

we upgraded pg around 17UTC on the 5th

that is, less than a week ago?

yes

there was a major security vulnerability, thus

Probably can get some hint by enabling slow log on pgsql. But this is production DB....

things time out and get cancelled anyway, I'm not sure it's slow queries -- it's apparently queries that are returning -- and returning a *lot*

but apparently not returning it the whole way to the client, just internally

There could be multiple problems. For example, when you guys saying the lagging is cured, I am still constantly seeing 502

well, there's multiple problems

trying to figure out the one that's having the biggest effect first

(e.g. some large releases have always 502'd repeatedly)

I'd say there's bigger problem when even single CD edits results in 502 repeatedly

that is not a phenomenon that is well-reported; if it's a specific set of releases, rather than a cross-section, it's most likely related to those releases specifically unless it's during the blocks of time where we have known other problems

you've seen the 5xx errors graph, so you can see what is anomaly and what isn't

it's easier when reading graphs, but for personal experience that's harded to tell

but slow query log can still help a bit too, at least help tracing what's the offending source

because the sudden surge causes exploding I/O and dragging everything down

especially if the surge can be irrelevant from increase of external requests

hm

it seems it is, at least, swapping during these periods

swapping is one of the worst things for DB

yeah

still unclear what's causing any of this though

we're just getting an ever-better description of why it's bad :P

yup

back to square one: why the sudden increase in no. of processes

from less than 100 on average, jumping to max'ed 256

hm

what it *looks* like is as though it's not using pgbouncer, for some undefined reason

looking at the yearly graph (we instituted pgbouncer last august IIRC

the graph for no. of db transactions?

Hmm, DB block hit is much lower since 7th

Urgh, tried with a few single CDs with >25 tracks, editing all tracks together also result in guaranteed 502 :-(

it seems that editing an existing tracklist through TRACK PARSER doesn’t owrk any more… :/

I have a few CDs of field recordings of traditional music. They generally only have one or two tracks with any performer information, located in the liner notes. Should I just Various Artists it and [unknown] most of the artists?

Jozo: how on earth?

JonnyJD: ?

the bobrowski edits

writing a note right now. Not sure at all how one would get the idea that merging these RGs would be the thing to do..

JonnyJD: (I have no idea what to say, so I didn't wrote any note)

Jozo: I fear the editor doesn't speak english anyways

but I wrote a note in http://musicbrainz.org/edit/21709304

they claim to in their profile

hm, just the edit notes don't look like that :-P And german wouldn't be a problem for me.

ianmcorvidae: http://ci.musicbrainz.org/job/full-import/ - What this do?

a full import :P

nikki: Yeh. I just tried find yet another thing what can cause db traffic.... (dates on jenkins and logs are bit weird... different timezones)

that shouldn't touch the db

Leftmost: Hi :) It really sounds like [unknown] to mer

Jozo JonnyJD : LOL http://musicbrainz.org/edit/21703555 :)

this TOC is strangely shared by Santana and Die großen Erfolge now

jesus2099: the fun part. That video has a disc ID.

jesus2099: compare discId another attached... (Please remove it :)

I geuss I’m doing no harm by removing this TOC from Santana

http://musicbrainz.org/edit/21709986 :)

http://musicbrainz.org/release/c162e9d3-b9f8-38... - Am I getting wrong release or is cover art edits fucked... :/