#musicbrainz

MBLogger (~musicbra-@client3.fre.communitycolo.net) has joined #musicbrainz

djce ([R5qESSxZR@195.60.9.122) has joined #musicbrainz

djce ([0LywwsAd2@195.60.9.122) has joined #musicbrainz

make up your mind... :)

I'm testing!!! :-)

yeah, I figured that much...

what are you testing?

A bot which logs & archives this channel.

So we can post them on the web, like the mailing lists.

djce ([0LywwsAd2@195.60.9.122) has joined #musicbrainz

djce ([NIvltWZoA@195.60.9.122) has joined #musicbrainz

ruaok (~robert@adsl-63-194-101-14.dsl.snlo01.pacbell.net) has joined #musicbrainz

Yes, a rather handy out-of-the-box bit of kit from the w3c. Nice!

salisan (~salisan@ai40.nuaccess.net) has joined #musicbrainz

Hello

Hi

MBLogger (~musicbra-@client3.fre.communitycolo.net) has joined #musicbrainz

How much bandwidth is the server limited to?

That's a question for Rob. ruaok, you there?

Hmm, well. I suppose the best way to handle more traffic is to get some more webservers going. Finding bandwidth sponsors should be easier than finding money sponsors..

Adding mirroring capabilities is in the works.

Plus, we're always interested in finding willing FTP/WWW mirror servers.

sorry, I was responding to email.

I'm trying to work down the email pile that has accumulated.

bandwidth limiting: we're on 10Mbps unswitched hub with a bunch of other servers.

Its very low tech bandwidth limiting. :-)

intrep (~intrep@dsl092-134-144.chi1.dsl.speakeasy.net) has joined #musicbrainz

Is that the limiting factor?

yes. :-)

Does CCCP want a donation for a 100Mb switch?

I guess I'd have to ask them that :-)

The problem is that CCCP has to pay for bandwidth spikes, and those can be VERY expensive.

Yes and no.

?

They always need more hardware to keep things running, but the 10Mbps hub is there for a reason: bandwidth limiting.

any plans to do incremental updates of the database db files like freedb does? :)

That would probably be an integral part of the "mirroring" project.

Currently the dataset is small enough (and will never grow to freedb sizes since we avoid duplication like the plague) to download in one batch.

So plans, certainly.

It would be preferable if they had 10mbps switched ;)

Also unlike Freedb our changes are not as atomic....

I understand it would be nicer to have a 100Mbps switch.

But we cant affort to pay for the bandwidth that would get sucked up.

Is the CCCP co-lo free, unless you pass some threshold?

any particular reasons you guys use postgresql native dumps instead of sql command dumps?

yes and no.

CCCP does not charge anything.

intrep: I'm working on alternative dump formats right now.

it seems it might be easier to diff sql dumps

djce: awesome

Ok, nothing is real cheap.

However it asks for $50/month/server

$25/month/virtual server.

NOTHING, really.

Currently I'm thinking: tab-separate, Excel CSV format, and MySQL SQL script.

Relatable covers our $50/month, so we're off the hook.

Any other formats you'd like to see? :-)

ruaok: I see

sql command scripts

Mysql, yes I'm doing that.

Is the database still mysql compatible or is that gone nowdays?

I really love the concept of CCCP.

mysql: no

MySQL doesn't do foreign keys, AFAIK.

or transactions.

arrays

djce: Actually, innodb is supposed to have basic foregin key support now ..

The bottom line is that the database can be 99% loaded into MySQL, but you can't run a MB server off MySQL

(the missing 1% is, as ruaok says, array columns)

But I don't really like those anymore. :-)

They are too slow...

djce: It's good for selects.

As long as they are simple :-)

djce: Your parsing code was really close.

and you're not trying to INSERT at the same time!

parsing - yes, I saw your final diff.

does mysql have finer than table grain locking yet?

intrep: Yes, the InnoDB table type does

modpending, page) values ('Andy Breckman', 'Andy Breckman', '153ccb00-fcad-4c79-a178-207aae196a4c', 0, 83028003)'

InnoDB does, apparently. I've not used it yet though

I need to look into those errors....

Yes, I see those every now and then.

ruaok: did you notice that MBLogger also creates HTML and RDF logs?!

Very flash... :-)

anyone worked on importing into oracle yet?

116860 TRMs collected

Not AFAIK

(oracle)

but if you have an Oracle server available, please feel free to give it a go.

djce: haven't looked. where does the output go?

Can anyone actually afford Oracle? ;)

i have to admit, im really impressed with the trm technology

i can afford oracle

so can you

sign up for the developer copy

:)

ruaok: MBLogger output is currently in /home/dave/irc-logger/irc-logs/musicbrainz

djce: nice.

With the capability to dump the data in multiple formats (which I'm doing now), the tab-sep format is particularly simple to parse, and therefore load into any other data store.

salisan: check out www.oracle.com you can sign up and download oracle 8i and 9i for free, but youre not supposed to use it for anything production

Loading into Oracle should be trivial really

I might give that a try, when I have a few hours download window to spare :-)

It would be nice just to say that it can be done.

yeah, postgres and oracle are fairly sql compatible

unlike some other databases i know <coughmysqlcough>

:-)

hehe

Uptime: 991616 Threads: 187 Questions: 62557009 Slow queries: 799 Opens: 60166 Flush tables: 1 Open tables: 64 Queries per second avg: 63.086

cough cough msaccess cough splutter

I like my mySql ;)

Maybe SQL server too. Anyone have that to play with?

EEEEK. Not that.

:-)

and if you do have SQL server, how was last week for you? ;)

s/have/had/ ;-)

Somehow "dinosaurs" and "asteroid" comes to mind :-)

Running sql server queries over internet must be an interesting exercise after that mess ..

It's definitely gone reasonably quiet on port 1434 now.

(I work for an ISP, and was monitoring that port the other day, just to see)

ruaok: I really doubt you will have any more success with Google now btw..

it was worth a shot. who knows.

what makes you say that though?

ruaok: Everyone is after their traffic to make money ..

Mutiny (~trivial@h0000deadbeef.ne.client2.attbi.com) has joined #musicbrainz

salisan: understood. But we have a good relationship with Google so far....

Is there any way to implement Last-Modified: or Expires: headers for the generated pages? ..

Browsing around on 56k6 is not so nice ;)

That's hard. Desirable, but hard.

I expect it's waaaaay down the "TODO" list :-(

It would probably cut down the traffic per user alot also

Good point.

are you talking about for album/artist/track searches?

or for moderation data?

intrep: Not just searches, album/artist pages .. everything generated pretty much

searches are probably the one thing you'd never cache. Apart from anything POSTed

But cacheing album/artist pages would be great. But like I said, really quite hard to do.

couldnt you keep a lastmod column in albumjoin and trackjoin?

ick. not for caching purposes.

Trouble is, from the server's point of view, by the time you've gone and checked all those timestamp values, wherever they are, you may as well had just generated the page anyway.

then when you return a page with album data you could set lastmod to max(lastmod) or something like that

true