#musicbrainz

It's never as simple as you want it to be :-(

Bear in mind the other things on, say, an Artist page, all of which affect the caching:

log in/out status, "mod pending" status, expanded/collapsed albums,...

ugh

big ugh

are the majority of the pages you server to people who are logged in or out?

i would think they would be to anonymous users (logged out) who dont get those stats

Some form of E-Tag: calculation would work to just lower the bandwidth usage.. but cpu usage would remain

you could also cache artist pages that are requested anonymously and cache them for like 5 minutes

and return lastmod for the last time it was generated

Are you volunteering to make it happen? :-)

maybe

:)

do you have any rough idea of what percentage of your requests are from people who arent logged in?

Votre message pour la liste escape_l a �t� transmis au(x) mod�rateur(s)

also, do you have any rough idea of which pages are the majority of your traffic? artist/album/track searches? moderation?

Your message about the list ....

"Your message for the "escape_l" list has been sent to the moderators"

I think

Makes sense now. Thanks!

user logging: that reminds me, I was going to log the MB moderator name to the access logs. On the TODO list with that one...

intrep: You can get some of that info from http://www.musicbrainz.org/usage/usage_200302.html it seems

salisan: k, thanks

showartist and showalbum seem to be leading amont the ordinary pages ..

I doubt it. You mean the "unique usernames" bit?

At the moment that would be taken directly from the access logs; so user "-" for 99.99% for all requests, and other names for some other HTTP requests (but not for the main web site)

But if I make the change I mentioned a minute ago, that stat would become useful...

Right, but you can get what pages have most requests now anyway, just not if they are from logged in users or not ;)

yes

ok, im not familiar with mason so i have a quick question

at the bottom of showartist.html there is a tag <& /comp/footer &>, i am going to take a wild guess that this is the end of the generated html, is there any way to get mason to dump the html for the whole page into a variable at this point?

Hmmm...

You can call $m->scomp($page) to "run" a page and capture its output

i.e. $output = $m->comp("/path/to/component", @args)

If you haven't been called as $m->scomp, I don't know if it's possible.

mason has its own object cache

keeps expire times and everything

interesting...

yes, that's useful, if you know what you're "expire" conditions are.

For small parts of pages, e.g. the "top moderators" panel in the sidebar, the cache is useful then since it only depends on time, not any other inputs.

well, i would think the entire page for anonymous lookups of showartist and showalbum could be cached for say, five minutes or so

Except for the "expand" settings on the album page.

Album pages are easier.

the expand pages could be cached too really

but i would bet that the showartist main page for each artist is accessed about 3 times as often as the expansion pages, making it very valuable to cache

Only for anon visitors, right?

right

Interesting...

logged in users get fresh content all the time

And you're talking about cached as in returning a "304" response, not just getting the "200" response but faster?

you really could cache a lot of the info for logged in users as well

you could do it either way

The latter is technically easier; the former is more beneficial to the visitor.

yeah, and better for your bandwidth

Yes!

That's definitely going on the TODO list then. Let me know if you make any progress yourself...

hmm, you could also, if you wanted to get really tricky, stick some squid caches out on people machines that will redirect to your main site and cache some stuff.

taking a lot of load off of your main site

180980 mp3list.txt

133589 trmdb.txt

a sort of squid mirror

squid: The yet-to-be-created mirror system would handle that too.

180,000 MP3 files? Wow.

i need food

mmm

637gb, all downloadable and streamable courtsey of unixpunx.org

rjmunro (~rjmunro@62.53.126.173) has joined #musicbrainz

That would take a while :-)

Are you volunteering to add all that data to MB?

if he wants it, he's welcome to it

Anyone heard anything from MusicMoz?

not I

the database will have filenames, id3 tag, encoding info, and duration

if it'll be useful to anybody

The database as in those two text files, yes? Not all of the audio files themselves?

no, just the metadata on the archive

Possibly. How accurate is the data, I wonder?

the id3 tags are probably not too accurate, but the file info is pretty accurate

the filename + path will give the artist/album/title, that's how we try to organize the archive

MBChatLogger (~musicbra-@client3.fre.communitycolo.net) has joined #musicbrainz

hehe

hmm

i have a url that causes your website to give a '500 internal error'

hi

oh :-(

go on

you want me to paste it in here?

ok

checking you guys out for xss :)

Yes, there will be lots like that. Most input isn't validated; specifically, things which should be numeric aren't checked for being so.

you might want to do something like 'if ($ARGS{artistid} =~ /[^0-9]/) { delete ($ARGS{artistid}); }

i c

I usually use: $id = ($foo =~ /\A(\d+)\z/ ? $1 : 0) actually

your string parsing in the search panel appears to do the right thing

that works too

or (\d{1,10}) if I'm feeling thorough

for approximate values of 10 :-)

heh

Just try not to do that sort of stuff until I've added all the validation, ok? ;-)

sure

<turning off xss script checker...>

Maybe I'll call on you again to test it when I claim to have fixed it?

no prob

Cool. Thanks!

anyway, a 500 error is far preferrerable to the script making it through to the browser

True. But that depends on what it managed to do before barfing.

If it's actually feeding it unchecked into SQL, that's probably even worse.

true

where can i find the todo list? cvs?

There's three "TODO" pages on the web. Plus there's my TODO list, which is only here on my dev machine.

gotcha

Plus anybody else's TODO lists, if they also keep their own lists.

I can post it for you if you like.

sure

ok, hold on...

how complete are the currently existing web ones?

if they are roughly similar then they are fine

http://musicbrainz.org/~dave/TODO.txt - all sorts of scribblings

cool, danke

the web TODOs? Pretty out of date.

It was mentioned a month or so ago that we need a more responsive way of dealing with those, so we would be more likely to keep them up-to-date.

My TODO list tends mainly to be small bug-fix-like things. Rob's TODO list tends to be more oriented towards the big picture (e.g. "implement mirroring")

perhaps bugzilla, file bugs for things that need to be done

We've got the sourceforge bug list too

yeah, maybe just file your todo items as bugs

That's a mixture of bugs reported by users, potential bugs spotted by us, and RFEs

Yes, but how much easier is just appending two lines to a local text file than posting a web form? :-)

very true

you could write a script to autopost your text file as bugs :)

you just have to format your text file more carefully

Interesting idea. It would be a start just to copy my TODO file to the web, like I did for you just now, and link to it from the main site.

(and do so regularly)

write a script to copy the file to the web server and run it from cron

:)

can you tell i hate doing things manually?

crontab -e ; 0 * * * * scp TODO server:./public_html/TODO.txt :-)

hehe, perfect

You know, I might actually do that :-)

make it djce-TODO.txt and then add a link from a main todo.html page and call it done

then other developers can follow in your footsteps