#metabrainz

samj1912: sry I wasn't around, I'm looking at the pastes you linked and trying to wrap my head around things now :)

bitmap: o/

hey :)

I reduced it to a series of joins

The only problem is the last statement

I think we can convert it to an outer left join and figure and filter on the aliased join field

That will reduce the entire headache with deleted fields

But I am not sure how efficient that join would be

iliekcomputers: Quick question again… is this InfluxQL or PostgreSQL queries? https://github.com/metabrainz/listenbrainz-serv...

s/is this/are these

samj1912: is https://hastebin.com/raw/qonifutoka still relevant & can you give an example of a problematic statement?

bitmap those are the joins I constructed by considering only the second to last paths

jwf: BigQuery standard sql

Ooh, okay.

Have any recommended reads on BigQuery other than the upstream docs?

A problematic statement in that is if let's say we delete a release status. We won't know which releases to update.

but we can't delete a release status without the releases being updated first...unless I misunderstand you

Yes

That is correct

Although, I am not sure if I can 100% prove we will cover all the things, in case we add new paths

Another problem is tables with multiple FKs

*PKs

Let's say we update area, see how the statement for annotation update triggered by it looks like

On updating: `area`, We update entity: `annotation` via path: `areas`, ['type'] SELECT musicbrainz.annotation.id AS musicbrainz_annotation_id FROM musicbrainz.annotation JOIN musicbrainz.area_annotation ON musicbrainz.annotation.id = musicbrainz.area_annotation.annotation WHERE musicbrainz.area_annotation.area IN (:area_1) AND musicbrainz.area_annotation.annotation IN (:annotation_1)

to generalize the thing above, when a row is deleted...it doesn't makes sense to join with other tables that reference that row

Hmm, yes

But we are joining them simply as a way to get ids

Which are already emitted from the trigger

Hmm bitmap, so you are saying if we are deleting something, we don't need to care about updating entities which are not emitted as FKs(so we discard those SQL statements) and update the rest of the paths?

Since they will be updated before we can delete the given row

well, the direction of the FKs has to be taken into account...if a release_status is deleted, that should be completely ignored, because it's only referenced *by* other tables, and those will have been updated already

if a release_alias is deleted, the release has to be updated because it references the release, not the other way around

if a row has FKs in both directions, only those need to be followed

Hmm, but the join method from the sql alchemy abstracts all of that, it is figures out which direction it is is from the path

I'll have to manually inspect each path and figure out the fk direction again

I assume you only have to look at the direction of the first item in the path if it's a delete

Direction of the last item and the deleted item you mean?

CatQuest: hi, https://test.musicbrainz.org is now up & able to be edited again

samj1912: right, last item. for each path of the deleted row's table

Yup

I'll figure something out

I guess it should be checked in the index callback and when we're generating the triggers, since some delete triggers shouldn't even be generated

well, I guess changing how it generates the update map fixes both, nvm

I keep forgetting how this works

bitmap, meanwhile should I replace the generate selections code with the joins?

samj1912: if you think it's an improvement, yes

not sure if its an improvement

just that it will generate bug free expressions and clean up the code, making it easier to understand

sounds like an improvement :)

bitmap: all the entities and their manytoone rels

I guess these are the only ones that are to be updated on delete?

jwf: not really, all I've been reading are docs :/

moiiin

bitmap: caa.org is naughty since ~1 hour, any known reason ?

alastairp: ping

hi zas

just looking now

load is pretty high; disk is pretty full. not sure what's using all of the space

/tmp is empty - not sure if you just cleaned it up, or if we have a similar issue to what happened last time

i didnt clean up anything yet

bitmap: made some more progress

but the join thing wont work

its too inefficient

hmm, even the select statements are pretty bad

bitmap: ruaok we really need to take a look at our schema

*search schema

I managed to fix live indexing I think

afaict

bitmap: take a look at this https://github.com/metabrainz/sir/pull/55/commi... when you have time

the worst culprit in terms of reindexing is recording

we are indexing too much data for it

solr cant keep up, even when making reindexing queries

*not solr, psql

reosarevok: The Last.FM imported only matches to MB if Last.FM itself has their track linked to an MB Recording ID (sometimes they're linked wrongly!). Last.FM importing doesn't give MBIDs for Artists, Releases, etc.

iliekcomputers: Okay, no problem! If the docs are good, I'll work my way through them.

+1 about linked post

Thanks bitmap!

yvanzo: Oops, sorry, didn't realize MBS-9549 was a duplicate. But happy to see it's already fixed. :)

MBS-9549: "Homepage" field in user profile printed twice https://tickets.metabrainz.org/browse/MBS-9549

No worry, I am glad that people are actually using beta and report bugs. :)

usually i use beta anyway

when are changes to beta usually applied? :)

beta++

its suprising that something gets fixed in production earlier than in beta (:

I've never heard of that happening

Freso: hi, I won't be around tonight, no news

<BANG>

Hi!

hello

Hello :D

Freso: go!

I went back to Barcelona last week and had a meeting with Rob and then got my contract renewed, so you're stuck with me a while longer. :)

hi!

:D

Freso: yay!

I also helped review reosarevok's Mahler debriefing blog post, but I'm sure he'll say more about that.

hey!

I dealt a small bit with reported editors.

A large bit with Google Code-in.

And otherwise kept up with forums and IRC. (I think maybe there was also a support@ mail this week I answered.)

fin!

jwf: Go!

(Still up: reosarevok, bitmap, zas, ruaok, yvanzo, samj1912, iliekcomputers; maybe up (please confirm): LordSputnik, Leftmost, Leo_Verto, CatQuest, chhavi_, ListMyCDs, ferbncode; anyone else?)

I was a little quiet, but getting back to speed. Working more with ListenBrainz statistics. I want to work with iliekcomputers and ruaok to add more example metrics.

I'm following but have nothing to report this time. Just skip me.

hey

Once I get the hang of one, I want to write out a document to plan more so we can make sure it will scale.

chhavi_ you might wanna go early since you might fall asleep any moment :P