#musicbrainz-devel

ocharles: you may be in bed, but http://stats.musicbrainz.org/webstats/nginx-rrd... is... interesting, not sure if a problem

(look at yearly)

yeah, that is worrisome.

brewster says that there some S3 nodes that are down.

no known ETA for the CAA upload being accepted again. :(

some of them seem to be now, even

I'm curious to what we did for the latter half of oct, that caused the number of rows to go back to the mid-sept levels.

well, that'd be right after the schema change release

not sure what pushed it back up though, since that's not in line with a release

averaging 10k rows changed per second!

that's insane.

yup, heh

there's no way that's accounted for by our manual editing process.

no, obviously not; I'm wondering if there's a runaway script of some sort

interestingly enough, it isn't reflected by the db server load average.

Is there anything we're getting an insane amount of?

Tags? Ratings?

this probably can't be coming from the website at all

it would have to be something internal. external makes no sense

So it must be something running continuously in the background from MB itself?

if we were getting enough hits to generate this sort of thing from actual editor activity it would probably crash all of our servers at once :P

yeah, that's the notion

well, if you look at the stats, you see that 10k/s is the normal state - every 4-6 hours there's a burst of >100k/s :/

Well, guess we need to look at what is running then :)

Do we log the changes or something, so we could see what is happening?

... and they're all inserts?

it's a bit vague, because of how postgresql does concurrency

what was changed on the server on Nov. 4, 2012?

nothing I can think of, that's not a release or anything

that's when the most recent sudden rise from 2k/s to 10ks/ started...

heh

creating temporary tables for itself

search server release 2012-09-18

not sure why it dropped down in october, but

we did revert the release for a while, trying to solve other problems.

I didn't think it took us a month before we reverted though

you may remember that timeline better though :)

I cant remember shit right now.

hm, still not sure I believe my own theory on that

oh, wait

it's the search server *updater*

still creates a lot of the temporary tables, but runs near-constantly (AFAIK)

and the gap was that the schema change broke/stopped it for a bit, until warp got back to getting it back up

which I guess was Nov. 4 :)

ianmcorvidae: http://stats.musicbrainz.org/webstats/nginx-rrd... might be related to setup of search server on beta

As CompMusic we proposed turning instruments into first level entities in the MusicBrainz database. This means exposing a unique identifier for each instrument.

uh like we haven't been talkign about that for aaaaages

¬¬

warp: recordings indexes are not updated on beta: http://beta.musicbrainz.org/search?query=renaud... : "Last updated: 2012-11-23 11:55 UTC"

morning, so I reluctantly tried installing a linux virtual machine from scratch in order to run mbslave (rather than full mbserver)

After a failed attempt to installed ArchLinux I went for Ubunto Desktop which was very easy to install :)

but ran like an absolute dog, even if given 2gb of memory

So I then tried Fedora with LXDE which is so much better, and quite responsive with the default of 768MB of memory

so I was wondering is it just the Desktop part of Ubunto that is slow or is the whole ubunto flavour of linux rather slow ?

I don't find neither version slow, but I've never ran them in a VM

I use ubuntu in a vm with xfce and don't find it slow

but you shouldn't really need the gui just to run mbslave

maybe its just the default desktop it provides then

I'm guessing it needs 3d acceleration

its not just to run mbslave, also for other linux testing

Hi there :) I'm going to attempt to setup a local search server against my MusicBrainz server install (so my API requests aren't throttled by passing through search.musicbrainz.org). The docs here (http://wiki.musicbrainz.org/Development/Search_...) mention that for the search server to work, it requires "a MusicBrainz database with the core data set loaded". Does anyone know what constitutes a core data set?

ijabz: I'd for xubuntu then

xfce has relatively low requirements

Is it simply mbdump, mbdump-editor and mbdump-derived?

right, is ubunto better for mbslave thane fedora then ?

aaron_sk: yes think so in fact I don't think you need mbdump-editor

ubuntu is probably better for mb stuff since that's what we officially support

but I imagine mbslave works fine with either

Ok, that's great ijabz. I've got those 3 already, so as long as I've met the requirements, I'm good to continue with the HOWTO

this is mbslave, but I suppose if that what lukas uses would make sense to use that

ijabz: I run mbslave on multiple systems

so really any unix is fine

hmm, annoying I don't seem able to get VirtuslBox Guest Additions to work on Fedora, was fine on Ubunto

murdos: ok, I'll have a look.

goodmorning #musicbrainz-devel!

murdos: looks like it crashed or got killed at some point, but didn't continue updating the indexes because the lock file was still there.

ok. could you send me the log if you have it? and then re-index recordings and reenable the updater

murdos: what happens if I just start the updater? you think a week is too long for it to catch up?

for recordings? yes, it's way too long

ok, I'll stop the updaters and reindex.

but just for recordings, right?

I was planning to do it for everything, seems easier.

I'd rather not

ok

Yes!

murdos: https://frob.nl/tmp/updatelogs.tar.gz

I have stuff to upload, you see :P I don't want to upload it now if I could use it for testing later

I would like to have updated indexes living long enough so issues can be found more easily if there are somes

but if it's not going to be in the next couple of days, I'll just upload it now

and re-indexing only recordings will be fastest

murdos: I started reindexing only recordings.

thanks

I'll try and keep an eye on it, and re-enable the updater when it's done.

ijabz: I use a different window manager with Ubuntu, even then it seems to use twice as much memory as my previous Debian install.

warp: it's your review, so maybe you can answer that

warp, thx, well Im just installing lubunto so I can try and compare it with fedora lxde

nikki: it doesn't have a shippit yet. I need to answer ian's question.

hallo

nikki: somewhat, though I can spend some time on it today.

there's more stuff I promised to do today, which I'm trying to remember :)

(NES, image uploader,... hmm)

oh, reviewing one of ian's code reviews.

warp: got a moment?

yes

https://gist.github.com/0d4bd6a37743e30349b7 is what I have so far

but this fails on line 7 because obviously it does return some JSON :)

Hi all. Does anyone know of a way to get an exact artist match (by name or alias) from MusicBrainz? Using the search endpoint gives me lots of matches, I'm looking for exact matches only.

but the JSON it returns i can't no ahead of time because it has an MBID in

mock MBID generation?

(to be predictable)

that's done by musicbrainz-data, so that's not really an option

then you have to write your test to not care about it. possibly check that the mbid is a valid mbid, then remove it from the json (or replace it), and then do further checks.

yea, i think that's the only real option

aaron_sk: using quotes tends to help, but since it's for searching rather than exact matches, it's not very good at only returning things that match exactly

in python I'd probably add some testing features to the library to make the generation predictable for clients of the library.

ocharles: but either solution works, and I can't think of anything else.