#musicbrainz

/

      • djce
        It's never as simple as you want it to be :-(

        • 2003-02-15 04629, 2003

      • djce
        Bear in mind the other things on, say, an Artist page, all of which affect the caching:

        • 2003-02-15 04643, 2003

      • djce
        log in/out status, "mod pending" status, expanded/collapsed albums,...

        • 2003-02-15 04648, 2003

      • djce
        ugh

        • 2003-02-15 04644, 2003

      • ruaok
        big ugh

        • 2003-02-15 04627, 2003

      • intrep
        are the majority of the pages you server to people who are logged in or out?

        • 2003-02-15 04654, 2003

      • intrep
        i would think they would be to anonymous users (logged out) who dont get those stats

        • 2003-02-15 04601, 2003

      • salisan
        Some form of E-Tag: calculation would work to just lower the bandwidth usage.. but cpu usage would remain

        • 2003-02-15 04653, 2003

      • intrep
        you could also cache artist pages that are requested anonymously and cache them for like 5 minutes

        • 2003-02-15 04644, 2003

      • intrep
        and return lastmod for the last time it was generated

        • 2003-02-15 04601, 2003

      • djce
        Are you volunteering to make it happen? :-)

        • 2003-02-15 04610, 2003

      • intrep
        maybe

        • 2003-02-15 04612, 2003

      • intrep
        :)

        • 2003-02-15 04628, 2003

      • djce is pleased to hear it!

        • 2003-02-15 04612, 2003

      • ruaok
        anyone here speak french?

        • 2003-02-15 04623, 2003

      • intrep
        do you have any rough idea of what percentage of your requests are from people who arent logged in?

        • 2003-02-15 04657, 2003

      • ruaok
        Votre message pour la liste escape_l a �t� transmis au(x) mod�rateur(s)

        • 2003-02-15 04614, 2003

      • intrep
        also, do you have any rough idea of which pages are the majority of your traffic? artist/album/track searches? moderation?

        • 2003-02-15 04622, 2003

      • ruaok
        Your message about the list ....

        • 2003-02-15 04628, 2003

      • djce
        "Your message for the "escape_l" list has been sent to the moderators"

        • 2003-02-15 04639, 2003

      • djce
        I think

        • 2003-02-15 04646, 2003

      • ruaok
        Makes sense now. Thanks!

        • 2003-02-15 04626, 2003

      • djce
        user logging: that reminds me, I was going to log the MB moderator name to the access logs. On the TODO list with that one...

        • 2003-02-15 04645, 2003

      • salisan
        intrep: You can get some of that info from http://www.musicbrainz.org/usage/usage_200302.html it seems

        • 2003-02-15 04656, 2003

      • intrep
        salisan: k, thanks

        • 2003-02-15 04627, 2003

      • salisan
        showartist and showalbum seem to be leading amont the ordinary pages ..

        • 2003-02-15 04639, 2003

      • djce
        I doubt it. You mean the "unique usernames" bit?

        • 2003-02-15 04637, 2003

      • djce
        At the moment that would be taken directly from the access logs; so user "-" for 99.99% for all requests, and other names for some other HTTP requests (but not for the main web site)

        • 2003-02-15 04602, 2003

      • djce
        But if I make the change I mentioned a minute ago, that stat would become useful...

        • 2003-02-15 04640, 2003

      • salisan
        Right, but you can get what pages have most requests now anyway, just not if they are from logged in users or not ;)

        • 2003-02-15 04654, 2003

      • djce
        yes

        • 2003-02-15 04659, 2003

      • ruaok|snug is off to spend time with the lead MB donor.

        • 2003-02-15 04612, 2003

      • intrep
        ok, im not familiar with mason so i have a quick question

        • 2003-02-15 04635, 2003

      • intrep
        at the bottom of showartist.html there is a tag <& /comp/footer &>, i am going to take a wild guess that this is the end of the generated html, is there any way to get mason to dump the html for the whole page into a variable at this point?

        • 2003-02-15 04626, 2003

      • djce
        Hmmm...

        • 2003-02-15 04651, 2003

      • djce
        You can call $m->scomp($page) to "run" a page and capture its output

        • 2003-02-15 04611, 2003

      • djce
        i.e. $output = $m->comp("/path/to/component", @args)

        • 2003-02-15 04635, 2003

      • djce
        If you haven't been called as $m->scomp, I don't know if it's possible.

        • 2003-02-15 04600, 2003

      • intrep
        mason has its own object cache

        • 2003-02-15 04621, 2003

      • intrep
        keeps expire times and everything

        • 2003-02-15 04639, 2003

      • intrep
        interesting...

        • 2003-02-15 04610, 2003

      • djce
        yes, that's useful, if you know what you're "expire" conditions are.

        • 2003-02-15 04654, 2003

      • djce
        For small parts of pages, e.g. the "top moderators" panel in the sidebar, the cache is useful then since it only depends on time, not any other inputs.

        • 2003-02-15 04636, 2003

      • intrep
        well, i would think the entire page for anonymous lookups of showartist and showalbum could be cached for say, five minutes or so

        • 2003-02-15 04608, 2003

      • djce
        Except for the "expand" settings on the album page.

        • 2003-02-15 04614, 2003

      • djce
        Album pages are easier.

        • 2003-02-15 04654, 2003

      • intrep
        the expand pages could be cached too really

        • 2003-02-15 04647, 2003

      • intrep
        but i would bet that the showartist main page for each artist is accessed about 3 times as often as the expansion pages, making it very valuable to cache

        • 2003-02-15 04613, 2003

      • djce
        Only for anon visitors, right?

        • 2003-02-15 04617, 2003

      • intrep
        right

        • 2003-02-15 04618, 2003

      • djce
        Interesting...

        • 2003-02-15 04628, 2003

      • intrep
        logged in users get fresh content all the time

        • 2003-02-15 04650, 2003

      • djce
        And you're talking about cached as in returning a "304" response, not just getting the "200" response but faster?

        • 2003-02-15 04651, 2003

      • intrep
        you really could cache a lot of the info for logged in users as well

        • 2003-02-15 04602, 2003

      • intrep
        you could do it either way

        • 2003-02-15 04623, 2003

      • djce
        The latter is technically easier; the former is more beneficial to the visitor.

        • 2003-02-15 04640, 2003

      • intrep
        yeah, and better for your bandwidth

        • 2003-02-15 04646, 2003

      • djce
        Yes!

        • 2003-02-15 04648, 2003

      • djce
        That's definitely going on the TODO list then. Let me know if you make any progress yourself...

        • 2003-02-15 04613, 2003

      • intrep
        hmm, you could also, if you wanted to get really tricky, stick some squid caches out on people machines that will redirect to your main site and cache some stuff.

        • 2003-02-15 04621, 2003

      • intrep
        taking a lot of load off of your main site

        • 2003-02-15 04649, 2003

      • Mutiny
        180980 mp3list.txt

        • 2003-02-15 04650, 2003

      • Mutiny
        133589 trmdb.txt

        • 2003-02-15 04652, 2003

      • intrep
        a sort of squid mirror

        • 2003-02-15 04656, 2003

      • Mutiny raps his fingers.

        • 2003-02-15 04615, 2003

      • djce
        squid: The yet-to-be-created mirror system would handle that too.

        • 2003-02-15 04645, 2003

      • djce
        180,000 MP3 files? Wow.

        • 2003-02-15 04621, 2003

      • intrep
        i need food

        • 2003-02-15 04639, 2003

      • intrep is afk for grub

        • 2003-02-15 04617, 2003

      • Mutiny
        mmm

        • 2003-02-15 04645, 2003

      • Mutiny
        637gb, all downloadable and streamable courtsey of unixpunx.org

        • 2003-02-15 04656, 2003

      • rjmunro
        rjmunro (~rjmunro@62.53.126.173) has joined #musicbrainz

        • 2003-02-15 04609, 2003

      • djce
        That would take a while :-)

        • 2003-02-15 04622, 2003

      • djce
        Are you volunteering to add all that data to MB?

        • 2003-02-15 04613, 2003

      • Mutiny
        if he wants it, he's welcome to it

        • 2003-02-15 04600, 2003

      • rjmunro
        Anyone heard anything from MusicMoz?

        • 2003-02-15 04612, 2003

      • djce
        not I

        • 2003-02-15 04649, 2003

      • Mutiny
        the database will have filenames, id3 tag, encoding info, and duration

        • 2003-02-15 04653, 2003

      • Mutiny
        if it'll be useful to anybody

        • 2003-02-15 04633, 2003

      • djce
        The database as in those two text files, yes? Not all of the audio files themselves?

        • 2003-02-15 04642, 2003

      • djce is away: I'm busy

        • 2003-02-15 04630, 2003

      • Mutiny
        no, just the metadata on the archive

        • 2003-02-15 04624, 2003

      • djce
        Possibly. How accurate is the data, I wonder?

        • 2003-02-15 04637, 2003

      • Mutiny
        the id3 tags are probably not too accurate, but the file info is pretty accurate

        • 2003-02-15 04602, 2003

      • Mutiny
        the filename + path will give the artist/album/title, that's how we try to organize the archive

        • 2003-02-15 04602, 2003

      • MBChatLogger
        MBChatLogger (~musicbra-@client3.fre.communitycolo.net) has joined #musicbrainz

        • 2003-02-15 04619, 2003

      • Mutiny
        hehe

        • 2003-02-15 04623, 2003

      • intrep
        hmm

        • 2003-02-15 04626, 2003

      • intrep is back

        • 2003-02-15 04649, 2003

      • intrep
        i have a url that causes your website to give a '500 internal error'

        • 2003-02-15 04649, 2003

      • djce
        hi

        • 2003-02-15 04656, 2003

      • djce
        oh :-(

        • 2003-02-15 04601, 2003

      • djce
        go on

        • 2003-02-15 04602, 2003

      • intrep
        you want me to paste it in here?

        • 2003-02-15 04604, 2003

      • djce
        ok

        • 2003-02-15 04607, 2003

      • intrep
        http://musicbrainz.org/showartist.html?artistid=9…

        • 2003-02-15 04616, 2003

      • intrep
        checking you guys out for xss :)

        • 2003-02-15 04605, 2003

      • djce
        Yes, there will be lots like that. Most input isn't validated; specifically, things which should be numeric aren't checked for being so.

        • 2003-02-15 04619, 2003

      • djce slap it on the TODO list...

        • 2003-02-15 04637, 2003

      • intrep
        you might want to do something like 'if ($ARGS{artistid} =~ /[^0-9]/) { delete ($ARGS{artistid}); }

        • 2003-02-15 04646, 2003

      • intrep
        i c

        • 2003-02-15 04627, 2003

      • djce
        I usually use: $id = ($foo =~ /\A(\d+)\z/ ? $1 : 0) actually

        • 2003-02-15 04628, 2003

      • intrep
        your string parsing in the search panel appears to do the right thing

        • 2003-02-15 04647, 2003

      • intrep
        that works too

        • 2003-02-15 04652, 2003

      • djce
        or (\d{1,10}) if I'm feeling thorough

        • 2003-02-15 04606, 2003

      • djce
        for approximate values of 10 :-)

        • 2003-02-15 04615, 2003

      • intrep
        heh

        • 2003-02-15 04653, 2003

      • djce
        Just try not to do that sort of stuff until I've added all the validation, ok? ;-)

        • 2003-02-15 04636, 2003

      • intrep
        sure

        • 2003-02-15 04649, 2003

      • intrep
        <turning off xss script checker...>

        • 2003-02-15 04625, 2003

      • djce
        Maybe I'll call on you again to test it when I claim to have fixed it?

        • 2003-02-15 04632, 2003

      • intrep
        no prob

        • 2003-02-15 04639, 2003

      • intrep
        intrepATtauran.net

        • 2003-02-15 04655, 2003

      • djce
        Cool. Thanks!

        • 2003-02-15 04629, 2003

      • intrep
        anyway, a 500 error is far preferrerable to the script making it through to the browser

        • 2003-02-15 04655, 2003

      • djce
        True. But that depends on what it managed to do before barfing.

        • 2003-02-15 04618, 2003

      • djce
        If it's actually feeding it unchecked into SQL, that's probably even worse.

        • 2003-02-15 04620, 2003

      • intrep
        true

        • 2003-02-15 04627, 2003

      • intrep
        where can i find the todo list? cvs?

        • 2003-02-15 04601, 2003

      • djce
        There's three "TODO" pages on the web. Plus there's my TODO list, which is only here on my dev machine.

        • 2003-02-15 04609, 2003

      • intrep
        gotcha

        • 2003-02-15 04620, 2003

      • djce
        Plus anybody else's TODO lists, if they also keep their own lists.

        • 2003-02-15 04629, 2003

      • djce
        I can post it for you if you like.

        • 2003-02-15 04648, 2003

      • intrep
        sure

        • 2003-02-15 04658, 2003

      • djce
        ok, hold on...

        • 2003-02-15 04600, 2003

      • intrep
        how complete are the currently existing web ones?

        • 2003-02-15 04609, 2003

      • intrep
        if they are roughly similar then they are fine

        • 2003-02-15 04651, 2003

      • djce
        http://musicbrainz.org/~dave/TODO.txt - all sorts of scribblings

        • 2003-02-15 04602, 2003

      • intrep
        cool, danke

        • 2003-02-15 04608, 2003

      • djce
        the web TODOs? Pretty out of date.

        • 2003-02-15 04643, 2003

      • djce
        It was mentioned a month or so ago that we need a more responsive way of dealing with those, so we would be more likely to keep them up-to-date.

        • 2003-02-15 04652, 2003

      • djce
        My TODO list tends mainly to be small bug-fix-like things. Rob's TODO list tends to be more oriented towards the big picture (e.g. "implement mirroring")

        • 2003-02-15 04610, 2003

      • intrep
        perhaps bugzilla, file bugs for things that need to be done

        • 2003-02-15 04623, 2003

      • djce
        We've got the sourceforge bug list too

        • 2003-02-15 04637, 2003

      • intrep
        yeah, maybe just file your todo items as bugs

        • 2003-02-15 04659, 2003

      • djce
        That's a mixture of bugs reported by users, potential bugs spotted by us, and RFEs

        • 2003-02-15 04632, 2003

      • djce
        Yes, but how much easier is just appending two lines to a local text file than posting a web form? :-)

        • 2003-02-15 04642, 2003

      • intrep
        very true

        • 2003-02-15 04601, 2003

      • intrep
        you could write a script to autopost your text file as bugs :)

        • 2003-02-15 04617, 2003

      • intrep
        you just have to format your text file more carefully

        • 2003-02-15 04652, 2003

      • djce
        Interesting idea. It would be a start just to copy my TODO file to the web, like I did for you just now, and link to it from the main site.

        • 2003-02-15 04605, 2003

      • djce
        (and do so regularly)

        • 2003-02-15 04629, 2003

      • intrep
        write a script to copy the file to the web server and run it from cron

        • 2003-02-15 04632, 2003

      • intrep
        :)

        • 2003-02-15 04646, 2003

      • intrep
        can you tell i hate doing things manually?

        • 2003-02-15 04606, 2003

      • djce
        crontab -e ; 0 * * * * scp TODO server:./public_html/TODO.txt :-)

        • 2003-02-15 04621, 2003

      • intrep
        hehe, perfect

        • 2003-02-15 04608, 2003

      • djce
        You know, I might actually do that :-)

        • 2003-02-15 04612, 2003

      • intrep
        make it djce-TODO.txt and then add a link from a main todo.html page and call it done

        • 2003-02-15 04628, 2003

      • intrep
        then other developers can follow in your footsteps