#metabrainz

I just built my own copy of it

first thing that I see is that the Dockerfile copies both cpanfile and cpanfile.snapshot

I don't know if they both have to be configured

at the top of cpanfile.snapshot it says "# carton snapshot format: version 1.0"

In a PR, i was suggested to build the cpanfile.snapshot..not sure how to do this here

given that we install dependencies with the command `sudo -E -H -u musicbrainz carton install --deployment`

I suspect that this might be the cause

I don't know either

maybe zas or Gentlecat or yvanzo know

I'm reading this page which seems to say that carton will make this file automatically by reading data from cpanfile

how much do you know docker? what I would do at this point is split this RUN command into many commands

so that I can log into the docker container after the required packages have been installed with apt and play with carton

okay, I can do that

ferbncode: from http://search.cpan.org/~miyagawa/Carton-v1.0.28...

The --deployment flag makes sure that carton will only install modules and versions available in your snapshot, and won't fallback to query for CPAN Meta DB for missing modules.

so it looks like you need to add things to cpanfile; use carton install; copy the generated cpanfile.snapshot; do the install with carton install --deployment

alastairp: makes sense, thanks :D

alastairp: works the way you suggested. thanks :)

excellent!

[listenbrainz-server] paramsingh opened pull request #186: Remove .kitchen.yml (master...kitchen) https://git.io/vH0bi

[listenbrainz-server] mayhem closed pull request #186: Remove .kitchen.yml (master...kitchen) https://git.io/vH0bi

iliekcomputers: for simple things like that, just ask me to remove the file. no need for a PR.

ruaok: okay, noted.

[messybrainz-server] mayhem closed pull request #20: LB-93: Migrate to Python 3 (master...2to3) https://git.io/vH452

[listenbrainz-server] mayhem closed pull request #181: Add .travis.yml for continuous integration. (master...travis) https://git.io/vHcrU

[listenbrainz-server] paramsingh opened pull request #187: LB-93: Migrate to Python 3 (master...2to3) https://git.io/vHEej

I'm new to the xml service and REST queries. Can anyone reccommend me something useful on linux, or even browser based, to test queries for the xml service on?

the browser is a good start

@alastairp I can construct a query string and download the file from the browser alright, it's more that I don't know how to do it with a script properly. Thanks, @Gentlecat, I'll take a look at that

[metabrainz.org] gentlecat opened pull request #282: MEB-93: Add an API endpoint for serving MB JSON dumps (master...json-api) https://git.io/vHEJZ

[metabrainz.org] gentlecat opened pull request #283: Switch to Python 3.6 (master...py36) https://git.io/vHEJl

so much python3 love <3

Hi there, I was referred to here from #musicbrainz. Suffering from 503s in Picard when it does lookups. I understand that there has been some discussion on this - would it be possible to get an update?

I came by a few weeks ago with the same issue, and I was told that throttling had been eased. But I would expect throttling not to return 503 ;)

[metabrainz.org] gentlecat closed pull request #283: Switch to Python 3.6 (master...py36) https://git.io/vHEJl

antgel: still suffering from spammers sucking up capacity and host of other inefficiencies in our setup.

Zas: fuck it. Order two more servers?

Zas: back to caching. Is this something​ we can do without engineering effort?

ruaok: i don't know enough about current caching in mbs

bitmap: ?

This is exactly what i think

That might be so.ethi g you can do without a lot of help from bitmap and yvanzo , right?

Lo. So.ethi g. #covfefe for life!

I'm about to leave for diner, but tomorrow morning (or later this evening if i'm motivated enough) i could evaluate number of unique queries to have an estimation of cacheability.

also it would require api changes (somehow) since we would support usual cache management feature, and there still the problem of cache invalidation

There are plenty of requests to pages that are not really dynamic... E.g account creation.

Not having to render those might help.

But since i don't know what exactly is cached or not... it would be great to have bitmap and yvanzo in this discussion, to see where we can improve things

Not sure if catching the ws would be a good goal.

Since our web pages are so slow.... Cache those?

I vote to use caches everywhere we can ;) plenty of ram / lack of cpu

Still, we should cache for maximum impact first.

Of course

But according to my first analysis, i get 200s on pages that didn't change at all between my queries

Where i would have expect a not modified (and therefore a client side caching)

we never really investigate this, but imho we don't rely enough on various cache possibilities, which lead to excessive traffic, and excessive cpu usage

on the website, and on the web service

our entities are that much changing (in fact, our db is quite static if you look at number of changes vs number of entries)

But proper implementation will need to be done with a proper cache invalidation when entities are modified, i think that's the main issue

Varnish has tons of features for that: http://book.varnish-software.com/4.0/chapters/C...

Is anyone around with a good experience of Varnish (or similar tools) ?

bbl

(well, not unconditionally, it doesn't store 503 errors)

kepstin: it can handle a few million requests per day, yes? We'd be happy to use it.

Lol

heh, I honestly have no idea. it's just a single DO droplet, but it is behind cloudflare...

I'd probably run out of disk space for the cache, at least :)

Tease.

I guess what I'm saying is that not all applications actually need data that's guaranteed to be fresh, I wonder if there's a market for an api that might serve a bit older data, but has the rate limits eased

of course, our current ws design doesn't really suit that, since there's so many parameters available that tweak the results :/

a "stable.musicbrainz.org" ? wich is less rate limits but also older data?

I have some experience with caching. Firstly there are several different types of cache other than web-cache. You can cache: SQL statements (to avoid the SQL server having to recalculate the access plan every time.) You can cache SQL search results. (I am assuming that moving .js / .css etc. onto content management platform is already done.

The trick is to cache the frequently accessed stuff - because that is where you will make savings - and not cache the vast majority which is accessed once a week or less frequently.

And to know how to invalidate the cache when the underlying data changes.

Like kepstin I wonder how suitable MBS is for caching. I suspect it has an ultra-long tail.

ultra long tail is my thought. which is why we need daily updated top-URL reports.

how are those coming, zas?

[picard-plugins] rdswift closed pull request #105: Standardized Album Artist Name(s) (1.0...1.0) https://git.io/vHRxW

ruaok, zas: Only compiled templates are currently cached, not pages. Varnish would be a good fit, but it requires to purge cache from within MBS editing system, just like zas wrote.

now the big question: what is better: 1) Add caching 2) move templates to react ?

both

Yes, i need to generate a daily page, but parser is working

Get Hip Folks: Social Media Banner Contest https://community.metabrainz.org/t/social-media...

What about a social media banning contest?

Can we ban ruaok from social media? He posts pics that make me jealous