but I think EXCEPT may be the way to go, so i'll have a look into that
2013-08-30 24208, 2013
ocharles
that can kill the entire loop
2013-08-30 24217, 2013
ianmcorvidae
switching that to <> it does seem to be slower than the 1s you're getting
2013-08-30 24220, 2013
ianmcorvidae
so that seems right
2013-08-30 24226, 2013
ianmcorvidae
yeah, 15711.849ms
2013-08-30 24229, 2013
ocharles
nice work postgres!
2013-08-30 24243, 2013
ianmcorvidae
though that's still wrong, I guess
2013-08-30 24251, 2013
ianmcorvidae
so do your thing and I'll stop pretending I know things :P
2013-08-30 24223, 2013
ocharles
wrong?
2013-08-30 24237, 2013
ianmcorvidae
I'm selecting everything that has edits, but where the edits are non-open
2013-08-30 24237, 2013
ocharles
and don't be so self-deprecating :P
2013-08-30 24250, 2013
ianmcorvidae
where what we want is that or has no edits
2013-08-30 24257, 2013
ianmcorvidae
though I guess every entity at least has a create edit so maybe that's fine
2013-08-30 24216, 2013
ocharles
ah, I see what you mean
2013-08-30 24216, 2013
ianmcorvidae
and it's not entirely wrong, I didn't know EXCEPT existed
2013-08-30 24240, 2013
ocharles
it gets a bit worse as you start to tack more things on
2013-08-30 24249, 2013
ocharles
4s with half the relationships and artist credits
2013-08-30 24203, 2013
ocharles
though with an upper bound on two hours, anything is an improvement really
2013-08-30 24212, 2013
ianmcorvidae
haha
2013-08-30 24226, 2013
ianmcorvidae
I don't know how the loop ends up getting implemented, but I guess the next most likely to be optimal is to do a single except with the most selective condition and then do the rest as part of the loop
2013-08-30 24214, 2013
ianmcorvidae
I don't know why that'd work out better though, except postgresql tends to regularly prove itself smarter than I expect, so :P
2013-08-30 24219, 2013
ocharles
except'ing everything gives me 29 rows in 6s
2013-08-30 24225, 2013
ocharles
as in except one at a time
2013-08-30 24235, 2013
ocharles
so that seems pretty good, and about the right number!
2013-08-30 24240, 2013
ianmcorvidae
that's pretty good, yeah
2013-08-30 24200, 2013
ianmcorvidae
slightly high number of results, possibly
2013-08-30 24214, 2013
ocharles
well, it didn't run last night
2013-08-30 24222, 2013
ocharles
and 15 a day is on par, i think
2013-08-30 24224, 2013
ianmcorvidae
yeah, suppose so
2013-08-30 24238, 2013
ocharles
i'll do a bit of manual poking though
2013-08-30 24241, 2013
ocharles
our site seems to think they are empty too
2013-08-30 24252, 2013
ocharles
gonna make some coffee and then i'll submit a review
2013-08-30 24256, 2013
ianmcorvidae
cool
2013-08-30 24210, 2013
ocharles
should probably get this onto astro today so it's ready for the weekend
2013-08-30 24224, 2013
ianmcorvidae
if we can get it out before tonight's scripts that'd be ideal
2013-08-30 24235, 2013
ianmcorvidae
weekly things happen tonight, so there's more subscriptions going out than usual
2013-08-30 24243, 2013
ianmcorvidae
so doing them closer to on-time seems good :)
2013-08-30 24235, 2013
ianmcorvidae
though I guess there's also a dump so it's not like they'd be on time anyway
no need to be on the astro branch of musicbrainz-server
2013-08-30 24207, 2013
ianmcorvidae
haha
2013-08-30 24232, 2013
ocharles
alright, fixed
2013-08-30 24209, 2013
ianmcorvidae
okay, all of those look good
2013-08-30 24218, 2013
ocharles
yay
2013-08-30 24240, 2013
ocharles
bah, missed one thing that astro has
2013-08-30 24247, 2013
ocharles
daily.sh is not meant to be ran with perl
2013-08-30 24223, 2013
ocharles
CalculateRelatedTags ran fine
2013-08-30 24232, 2013
ianmcorvidae
k
2013-08-30 24241, 2013
ianmcorvidae
I'm not actually sure what that does
2013-08-30 24247, 2013
ianmcorvidae
I realize :P
2013-08-30 24255, 2013
ocharles
same :p
2013-08-30 24242, 2013
ianmcorvidae
regenerates the tag_relation table, apparently
2013-08-30 24258, 2013
ianmcorvidae
wonder if we or anyone else uses that
2013-08-30 24205, 2013
ianmcorvidae
we don't, at least
2013-08-30 24247, 2013
MBJenkins
* Oliver Charles: Remove 'carton' from hourly and daily cron jobs
2013-08-30 24248, 2013
MBJenkins
* Oliver Charles: Rewriting all the empty_ sql functions to run considerably faster
2013-08-30 24253, 2013
ocharles
._.
2013-08-30 24207, 2013
marcooliveira joined the channel
2013-08-30 24245, 2013
ianmcorvidae
ocharles: entirely non-pressing unrelated question that popped into my head: how close are we to being able to add statsd instrumentation to musicbrainz-server? (i.e., is there any reason we can't yet other than we haven't written it)
2013-08-30 24250, 2013
ocharles
we're ready for it
2013-08-30 24258, 2013
ocharles
i did have a branch that added some stats, but I probably don't have that now
2013-08-30 24206, 2013
ocharles
it's mostly just a case of writing code
2013-08-30 24211, 2013
ocharles
any stats you have in mind?
2013-08-30 24218, 2013
ocharles
also, seeing a lot of '<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource /><RequestId>e556fa4c-8396-4e6d-b62e-443a67c0f433</RequestId></Error>' at the moment...
2013-08-30 24228, 2013
ocharles
as in, over 200 of them
2013-08-30 24258, 2013
ocharles
that's for metadata.xml it seems
2013-08-30 24210, 2013
ocharles
oh, it seems quite random
2013-08-30 24217, 2013
ocharles
there are other entries in the log that are ok
2013-08-30 24239, 2013
ocharles
hopefully i can get gamekeeper to build in the ppa soon, and we will have monitoring for rabbitmq
Oliver Charles: Fix PgTAP tests for empty_* checks
2013-08-30 24231, 2013
andreypopp joined the channel
2013-08-30 24218, 2013
LordSputnik joined the channel
2013-08-30 24236, 2013
ianmcorvidae
ocharles: well, I was thinking of just some simple things like edits inserted by type
2013-08-30 24202, 2013
ianmcorvidae
ocharles: nothing hugely pressing or anything, I was just wondering where that all stood
2013-08-30 24202, 2013
ocharles
right
2013-08-30 24219, 2013
ocharles
makes sense!
2013-08-30 24234, 2013
ocharles
ultimately, now that I learn more about how this stuff all works, we may consider moving stats there
2013-08-30 24259, 2013
ianmcorvidae
I'm not sure that all of them make sense there, but possily
2013-08-30 24202, 2013
ianmcorvidae
+b
2013-08-30 24238, 2013
ianmcorvidae
well, I mean, I guess we could just change the calculation script to store them there, if nothing else
2013-08-30 24225, 2013
ocharles
yea, that's mostly what I'm saying
2013-08-30 24240, 2013
ocharles
but haven't given it much thought
2013-08-30 24249, 2013
ianmcorvidae
I'm not sure how our setup would deal with things that only get updated once a day, though
2013-08-30 24201, 2013
ocharles
in what sense?
2013-08-30 24204, 2013
ocharles
you can have variable retention rates
2013-08-30 24210, 2013
ocharles
so statistics would just have 1 point per day
2013-08-30 24221, 2013
ianmcorvidae
ideally we'd not have it be a "stairstep" type thing like if we just used statsd in the most naive way, I guess
2013-08-30 24243, 2013
ianmcorvidae
with a statsd gauge it could, ostensibly, end up only updating once a day but storing it every $whatever_period
2013-08-30 24203, 2013
ianmcorvidae
but as long as it works out to have different periods basically arbitrarily it should be fine :)
2013-08-30 24257, 2013
ocharles
not quite following what you're suggesting with statsd
2013-08-30 24250, 2013
ianmcorvidae
well, gauges in statsd
2013-08-30 24259, 2013
ianmcorvidae
which would probably be the most straightforward implementation here
2013-08-30 24221, 2013
ianmcorvidae
if you send them data less frequently than whatever period statsd uses to store them in carbon, then it'll just repeat the same value
2013-08-30 24212, 2013
ianmcorvidae
but I guess this is solved either by pushing it to carbon directly, ourselves, or possibly by configuring statsd such that it doesn't update that particular stat as often (or only when it's given a new value)
