#metabrainz

back now.

I guess the indexes got rotated at the same time.

yep.

yes, search still sucks ;)

it also has a bug.

the indexes rotate at the same time.

not sure why that isn't working.

all machines have ntp, ya?

ruaok: they should, but old ntp is conflicting with new systemd stuff, i need to check in details, that's on my virtual todo list

ruaok, zas: on the wiki's vm, do I need to do anything special to open up ports? I started a nodejs service that should be exposed on port 8142, but I can't seem to connect to it from the outside (internally works)

and my laptop battery is about to die, so I'll bbl

legoktm: yes, we need to open ports on gcloud firewall, only http(s)/ssh are open by default

can we see that tomorrow, i'm about to go to sleep

?

UmkaDK: So much about important people not dying before replication is restored …

:(

"there's a crack in everything, that's how the light gets in"

word

bitmap: Is a dump running, by the way?

When we tell people they need to reimport, it would be good to have a dump ready. :-)

chirlu: not yet, I'll start one now

I always forget the command

I think just ./admin/RunExport 1

zas: installing screen on all the servers would be nice :)

and adding everyone to the docker group

speaking of fullexports, the ftp paths are currently broken but it'll take a while to sync again http://ftp.musicbrainz.org/pub/musicbrainz/musi...

should only be one 'musicbrainz' there, I've fixed the rsync daemon

my build machines are windows xp and os x 10.6, none of which support java 8 and that is needed for recent versions of jenkins

Good morning UmkaDK. Some bad news for you, whenever you are prepared for it. :-|

zas: ok, no rush :)

bitmap: Do we still have the pg_dump that was used to transfer the database from OldHost to NewHost? If so, that would provide us the state-as-of-99848 if we import it into a new master DB, even including the tags and so on.

And the FTP server seems to have synced already. Advantage of rsync’s deduplication – renames don’t matter much.

chirlu: do I get it right that replication is running now, but there is just the one missing packet? if I do a fresh import, everything will work?

Yes, exactly. You need the new dump from today.

There is still a slim chance that we find back the missing packet, but it’s slim.

If there is anyone (in addition to the FreeDB gateway) who got the relevant packet while it was available, replication continues for them without problems.

ok, thanks, I'll do a new import

I have a machine that replicated, can I try and recover the deleted file?

It’s normally deleted after the import, but if you changed that to keep the packets, that would be helpful. :)

no, I didn't change it

I looked on the FreeDB gateway already with extundelete, but there was no trace of it.

ah, can't do it on a mounted partition?

Unsurprisingly, since that was after two days or so, and there had been logs etc. written to the machine.

For when did you set the cronjob to run?

h:15

You can, the chances are just better if the partition is no longer written to.

I couldn’t unmount either, to I let it write the undeleted files to a different partition instead.

don't have one of them around (spike, the acousticbrainz server)

AB is up, isn’t it, and accepting submissions? Then it’s very, very unlikely that any deleted file hasn’t been overwritten yet.

lots of free disk space

Well, just try it (even on the same partition). Can’t happen more than that the file isn’t found.

how do you run it? extundelete --restore-file tmp/replication-99848.tar.bz2 /dev/sda1 ?

Yes, should work.

should I be in a tmpfs or something?

I just ran extundelete /dev/sda1, which will restore everything it finds.

If you have one ready, why not. :)

Avoids writes to the main partition in any case.

nope, not found

Thanks for trying, though. :)

np

ruaok: pong :)

chirlu: Yep, we might have spoke to soon there ... :(

It seems most likely that you will need to do a fresh import.

That is the “works immediately” way, in any case.

If you want to wait, it’s possible but not that likely that we find back the missing packet.

We’ve also considered recreating it. For that, we would need a database as it is before the missing packet.

Meaning: It would be great if you could provide us with a dump of your replicated DB. :-)

But only if it isn’t too much work, because otherwise it isn’t worth it for the relatively low chance to fix things.

chirlu: I've composed myself and is now ready for the news ... ?? :)

See above. :) The news is that you may have to do a fresh import.

... oh ... OOoohhhhh!! :(

UmkaDK: And you could help us by giving us a dump of the DB, because if (if) we still have the dump from OldHost, we could possibly recreate the missing packet from those two.

would a dump at 99847 help?

or do you need it to be older then that?

alastairp: You hit the perfect moment with your *:15 – the packet was ready at :12 and probably available for download at :13; it went down around :28.

UmkaDK: Yes, at 99847. We might have the state as of 99848, but we need the previous one to compute a diff, essentially.

kk, no worries! give me a few minutes, and I'll let you know what I can do

Ok, we are going to try and export our DB (replication sequence 99847) and put it somewhere where you can download it guys.