#metabrainz

/

      • reosarevok
        Hi!
      • I worked with yvanzo on testing and merging schema change code
      • mayhem
        how are things going on that front?
      • reosarevok
        I also submitted a few fixes for small bugs detected by sentry
      • mayhem: most stuff merged, working on the last few today, hopefully
      • mayhem
        ok
      • reosarevok
        We've tested the migration in sample data and it works, need to test it in a full DB after we merge all stuff, unless yvanzo has in the meantime
      • (he wanted to test in wolf)
      • On that note: yvanzo, go
      • CatQuest
        :ulv:
      • šŸŗ
      • yvanzo
        Hi!
      • Not exactly on wolf
      • Last week I worked with reosarevok on reviewing the requirements for the upcoming MB database schema change.
      • That includes updating the schema change process, reviewing and merging some pull requests.
      • But also checking MBS dependencies such as search components.
      • Got luciferā€™s help with the search indexer and its dependency mbdata.
      • Updated MB containers to use StaticBrainz on rudi, and documented it.
      • Also updated authentication apps in Jira, fin.
      • Go lucifer!
      • lucifer
        hi all!
      • i was afk for a few days last week. when around, i worked on coverging our mbdata fork with upstream, did a full reindex of sir with the updated mbdata. also worked on adding tests for Sir indexing using real database. this is still a work in progress, it should help with py3 migration and sqlalchemy upgrade.
      • CatQuest
        šŸ˜ˆ
      • lucifer
        other than that i worked on various bits around recommendations stuff in LB. also, followed on existing PRs.
      • that's if for me. alastairp next?
      • alastairp
        hi
      • last week I was sick all week, so didn't get anything done
      • I opened a small PR to disable listenstore on LB (we tried this last time we had a downtime but made a mistake in the implementation)
      • CatQuest
        aw alastairp
      • good bedring
      • alastairp
        mayhem: next?
      • mayhem
        k
      • alastairp
        CatQuest: all well now, thanks
      • mayhem
        I did a lot of biz dev stuff last week, chasing up on invoices and chasing after some new customers...
      • I planned the board meeting, selected and ranked GSOC proposals
      • and did the usual PR reviews .
      • as well as the normal background MeB stuff.
      • not a lot of techncial work, but hopefully that changes this week.
      • fin. Freso ?
      • Freso
        o/
      • aerozol
        \o/
      • akshaaatt waves at aerozol
      • CatQuest
        šŸŸ
      • Freso
        I tried to coordinate with jwf about the telegram bridge bot, but our non-overlapping timezones/schedules is making it a bit slow. :)
      • aerozol
        (couldn't sleep so checking in on what you lovely people have been up to!)
      • Freso
        Other than that, dealing with reports, flags, being around/about, etc. usual things.
      • fin.
      • CatQuest
        morena aerozol
      • !m Freso and jwf for trying :D
      • BrainzBot
        You're doing good work, Freso and jwf for trying :D!
      • Freso
        I didnā€™t miss akshaaatt, right? If so, akshaaatt is last on my list, soā€¦ akshaaatt , go!
      • akshaaatt
        Hi everyone!
      • Last week was quite challenging.
      • College involved a lot of commitment, with me running to get my research paper published, prepare for final exams, and hang out a bit with friends for the last few days.
      • Other than that, I continued learning new stuff which will prove to be useful soon.
      • The major part of my work included preparing newer pages for musicbrainz to be put in test.mb, and the highlight for me last week was the fact that the official ListenBrainz integration has been done on the MusicBrainz Android App (Avaialable on Playstore for Beta users currently).
      • Soon, users will also be able to delete and open their listens on the app!
      • That's about it for me. fin!
      • Back to you Freso
      • aerozol
        Wow, well done Akshat! What was your paper about?
      • Freso
        Alright.
      • Thank you all for your reviews! :)
      • CatQuest
        does aerozol wanna go?
      • aerozol
        No, I didn't get anything done D:
      • Freso
        CatQuest: They didnā€™t tell me so, so Iā€™m assuming not. :)
      • Anyway.
      • aerozol
        Thank you for coming to my TED talk
      • akshaaatt
        aerozol it's about Quantum entanglement and encryption for modern platforms
      • CatQuest
        oh no
      • !reacll oh no.
      • Freso
        One more item on todayā€™s agendaā€¦
      • CatQuest
        !recall oh no.
      • BrainzBot
      • Freso
        yvanzo, lucifer, et al: Securing MeB infrastructure - part 3
      • lucifer
        hi!
      • yvanzo
        We were at ā€œIncident Responseā€ IIRC.
      • lucifer
        yup
      • 1. How to check for possible intrusions, breaches?
      • atj
        manually checking system and application logs is about it really AFAICS
      • lucifer
        once we have found a vulnerability how do we go about finding if it was exploited? database access logs etc?
      • yvanzo
        Is it an open question? Or asking how it is checked currently?
      • lucifer
        open queestion
      • atj, yeah but do we have those logs in place. for example, postgres access logs need to be enabled manually.
      • atj
        Docker makes this harder because rather than just grepping through /var/log or journalctl you have to check "docker logs" for each container
      • however it's not really scalable regardless
      • yvanzo
        If there is nothing but logs to check intrusions, the question seems to be: Have a logging policy for services?
      • lucifer
        yeah makes sense.
      • yvanzo
        atj: You can grep docker log files too.
      • zas
        we really need to set up remote logs, and collect them in one place. First to ease searches, but also to get unaltered logs in case of breach. There are tools like loki (https://grafana.com/oss/loki/) to help with searches
      • lucifer
        +1
      • zas
        there are also tools to look for anomalies in logs
      • atj
        I did discuss setting up a centralised logging system with zas and mayhem when I joined the team, but we considered ansible a higher priority
      • zas
        yes, it is still the case
      • but we can look into centralized logs after the migration is complete
      • lucifer
        sounds good
      • atj
        Graylog looks like a good option
      • lucifer
        i'll add a TODO for enabling db access logs and remote logs
      • mayhem
        I've used graylog, its nice. takes some setting up, but its nice.
      • atj
        I think it would help in a variety of areas, including fault finding and analysis and intrusion detection
      • lucifer
        do we have machine access logs? like who ssh'ed when etc
      • atj
        yes
      • lucifer
        great
      • alastairp
        ditto here for graylog, once you give elasticsearch enough ram
      • zas
        for docker, it's possible to change the log driver
      • atj
        we'll have to see how much infrastructure budget there is for it ;)
      • zas
      • atj
        Docker supports the Gelf format used by Graylog
      • yvanzo
        In MB we also have cron logs (in docker volumes backed up remotely) additionally to other services docker logs.
      • atj
        some containers are way too verbose, so reviewing current logging practices would be a good start
      • alastairp
        this discussion is currently focusing on "how to view and analyse logs once we have them". a precursor of this is "ensure that we log enough data to be able to use it", what are our thoughts on that?
      • atj
        which container is it with the huge amount of consul messages?
      • yvanzo
        lucifer: I added a TODO (for remote logs) but feel free to rearrange
      • lucifer
        looks good, yvanzo. thanks
      • atj
        alastairp: I think a service by service review is needed
      • alastairp
        yes, right
      • yvanzo
        Currently, we have a script to collect docker logs for a given period of time, then we can grep it.
      • atj
        there's a tendency to just log "all the things", but that quickly becomes counterproductive
      • logs are useful from a security and operational perspective, so best to consider what is useful within those areas IMO
      • for instance, consider what you might need to perform RCA for an application issue?
      • as well as a security breach
      • lucifer
        added a todo for reviewing logging policy for each service
      • atj
        going back to the original point, intrusion detection is difficult without a complex and expensive SIEM system
      • I'm not aware of a good open source option for that
      • zas
        just to be sure to understand, which security level do we aim for?
      • atj
        100% ;)
      • reosarevok
        Over 9000?
      • atj
        it's worth consider what the threat model for MeB is, as most data is freely available
      • I think the primary concern should be security of personal data
      • lucifer
        some user sensitive data and take over of servers come to mind.
      • atj
        I'm not sure an attacker would be likely to do if they managed to takeover a server
      • *what
      • probably install a cryptominer to be honest
      • lucifer
        yeah i guess
      • use for spam etc maybe
      • atj
        yes, that's another risk
      • passwords aren't used for remote access, so rootkits / keylogging wouldn't yield much
      • lucifer
        agreed
      • i don't think we have much more to discuss about this currently so let's move on?
      • atj
        yep
      • lucifer
        2. For databases consider enabling log all connections/disconnections to the database. But does not help if an attacker gets access to a userā€™s account on a machine or root.
      • this one already has been covered before as well.
      • atj
        alastairp mentioned that this could be very verbose due to a lack of pgbouncer on some systems?
      • IIRCV
      • -V
      • lucifer
        yup right we need to review the verbosity of these logs
      • atj
        this would fall under review logging of all services I think
      • lucifer
        yes makes sense
      • atj
        to determine if it would be useful
      • lucifer
        also if the attacker gains access to the machine, these logs won't be any useful anyway