#metabrainz

Hi!

I worked with yvanzo on testing and merging schema change code

how are things going on that front?

I also submitted a few fixes for small bugs detected by sentry

mayhem: most stuff merged, working on the last few today, hopefully

ok

We've tested the migration in sample data and it works, need to test it in a full DB after we merge all stuff, unless yvanzo has in the meantime

(he wanted to test in wolf)

On that note: yvanzo, go

:ulv:

🐺

Hi!

Not exactly on wolf

Last week I worked with reosarevok on reviewing the requirements for the upcoming MB database schema change.

That includes updating the schema change process, reviewing and merging some pull requests.

But also checking MBS dependencies such as search components.

Got lucifer’s help with the search indexer and its dependency mbdata.

Updated MB containers to use StaticBrainz on rudi, and documented it.

Also updated authentication apps in Jira, fin.

Go lucifer!

hi all!

i was afk for a few days last week. when around, i worked on coverging our mbdata fork with upstream, did a full reindex of sir with the updated mbdata. also worked on adding tests for Sir indexing using real database. this is still a work in progress, it should help with py3 migration and sqlalchemy upgrade.

😈

other than that i worked on various bits around recommendations stuff in LB. also, followed on existing PRs.

that's if for me. alastairp next?

last week I was sick all week, so didn't get anything done

I opened a small PR to disable listenstore on LB (we tried this last time we had a downtime but made a mistake in the implementation)

aw alastairp

good bedring

mayhem: next?

k

CatQuest: all well now, thanks

I did a lot of biz dev stuff last week, chasing up on invoices and chasing after some new customers...

I planned the board meeting, selected and ranked GSOC proposals

and did the usual PR reviews .

as well as the normal background MeB stuff.

not a lot of techncial work, but hopefully that changes this week.

fin. Freso ?

o/

\o/

🐟

I tried to coordinate with jwf about the telegram bridge bot, but our non-overlapping timezones/schedules is making it a bit slow. :)

(couldn't sleep so checking in on what you lovely people have been up to!)

Other than that, dealing with reports, flags, being around/about, etc. usual things.

fin.

morena aerozol

!m Freso and jwf for trying :D

You're doing good work, Freso and jwf for trying :D!

I didn’t miss akshaaatt, right? If so, akshaaatt is last on my list, so… akshaaatt , go!

Hi everyone!

Last week was quite challenging.

College involved a lot of commitment, with me running to get my research paper published, prepare for final exams, and hang out a bit with friends for the last few days.

Other than that, I continued learning new stuff which will prove to be useful soon.

The major part of my work included preparing newer pages for musicbrainz to be put in test.mb, and the highlight for me last week was the fact that the official ListenBrainz integration has been done on the MusicBrainz Android App (Avaialable on Playstore for Beta users currently).

Soon, users will also be able to delete and open their listens on the app!

That's about it for me. fin!

Back to you Freso

Wow, well done Akshat! What was your paper about?

Alright.

Thank you all for your reviews! :)

does aerozol wanna go?

No, I didn't get anything done D:

CatQuest: They didn’t tell me so, so I’m assuming not. :)

Anyway.

Thank you for coming to my TED talk

aerozol it's about Quantum entanglement and encryption for modern platforms

oh no

!reacll oh no.

One more item on today’s agenda…

!recall oh no.

yvanzo, lucifer, et al: Securing MeB infrastructure - part 3

hi!

We were at “Incident Response” IIRC.

yup

the document link: https://docs.google.com/document/d/1qCRswB0eLn8...

1. How to check for possible intrusions, breaches?

manually checking system and application logs is about it really AFAICS

once we have found a vulnerability how do we go about finding if it was exploited? database access logs etc?

Is it an open question? Or asking how it is checked currently?

open queestion

atj, yeah but do we have those logs in place. for example, postgres access logs need to be enabled manually.

Docker makes this harder because rather than just grepping through /var/log or journalctl you have to check "docker logs" for each container

however it's not really scalable regardless

If there is nothing but logs to check intrusions, the question seems to be: Have a logging policy for services?

yeah makes sense.

atj: You can grep docker log files too.

we really need to set up remote logs, and collect them in one place. First to ease searches, but also to get unaltered logs in case of breach. There are tools like loki (https://grafana.com/oss/loki/) to help with searches

+1

there are also tools to look for anomalies in logs

I did discuss setting up a centralised logging system with zas and mayhem when I joined the team, but we considered ansible a higher priority

yes, it is still the case

but we can look into centralized logs after the migration is complete

sounds good

Graylog looks like a good option

i'll add a TODO for enabling db access logs and remote logs

I've used graylog, its nice. takes some setting up, but its nice.

I think it would help in a variety of areas, including fault finding and analysis and intrusion detection

do we have machine access logs? like who ssh'ed when etc

yes

great

ditto here for graylog, once you give elasticsearch enough ram

for docker, it's possible to change the log driver

we'll have to see how much infrastructure budget there is for it ;)

see, for example, https://commandprompt.com/blog/docker-logging-w...

Docker supports the Gelf format used by Graylog

In MB we also have cron logs (in docker volumes backed up remotely) additionally to other services docker logs.

some containers are way too verbose, so reviewing current logging practices would be a good start

this discussion is currently focusing on "how to view and analyse logs once we have them". a precursor of this is "ensure that we log enough data to be able to use it", what are our thoughts on that?

which container is it with the huge amount of consul messages?

lucifer: I added a TODO (for remote logs) but feel free to rearrange

looks good, yvanzo. thanks

alastairp: I think a service by service review is needed

yes, right

Currently, we have a script to collect docker logs for a given period of time, then we can grep it.

there's a tendency to just log "all the things", but that quickly becomes counterproductive

logs are useful from a security and operational perspective, so best to consider what is useful within those areas IMO

for instance, consider what you might need to perform RCA for an application issue?

as well as a security breach

added a todo for reviewing logging policy for each service

going back to the original point, intrusion detection is difficult without a complex and expensive SIEM system

I'm not aware of a good open source option for that

just to be sure to understand, which security level do we aim for?

100% ;)

Over 9000?

it's worth consider what the threat model for MeB is, as most data is freely available

I think the primary concern should be security of personal data

some user sensitive data and take over of servers come to mind.

I'm not sure an attacker would be likely to do if they managed to takeover a server

*what

probably install a cryptominer to be honest

yeah i guess

use for spam etc maybe

yes, that's another risk

passwords aren't used for remote access, so rootkits / keylogging wouldn't yield much

agreed

i don't think we have much more to discuss about this currently so let's move on?

yep

2. For databases consider enabling log all connections/disconnections to the database. But does not help if an attacker gets access to a user’s account on a machine or root.

this one already has been covered before as well.

alastairp mentioned that this could be very verbose due to a lack of pgbouncer on some systems?

IIRCV

-V

yup right we need to review the verbosity of these logs

this would fall under review logging of all services I think

yes makes sense

to determine if it would be useful

also if the attacker gains access to the machine, these logs won't be any useful anyway