10:25 AM
alastairp
so if for example it was `None` or `{}`, then the `[0]` would cause an error
2022-07-27 20847, 2022
10:25 AM
alastairp
even if it was `[]` we'd get an error
2022-07-27 20807, 2022
10:26 AM
alastairp
Pratha-Fish: right. I was looking at the other one. that's good then
2022-07-27 20815, 2022
10:26 AM
ansh
The edition group['author_credits'] returns an array of objects. So if it does not exist, the database returns [None]
2022-07-27 20816, 2022
10:26 AM
alastairp
how many items did you look up to result in just these 2?
2022-07-27 20841, 2022
10:26 AM
alastairp
ansh: is this the direct result from the view?
2022-07-27 20849, 2022
10:26 AM
ansh
yes
2022-07-27 20805, 2022
10:27 AM
ansh
I added a table in the views for the author credits
2022-07-27 20822, 2022
10:27 AM
alastairp
right, that's a bit strange. give me a moment, I want to check this query myself
2022-07-27 20854, 2022
10:27 AM
alastairp
Pratha-Fish: that first one is really weird... is this a uuid being used as an artist name? or a bug in the generation of the table?
2022-07-27 20857, 2022
10:27 AM
ansh
Sure
2022-07-27 20811, 2022
10:30 AM
Pratha-Fish
alastairp: I rechecked that one, apparently, it's a uuid that's being used as artist name!
2022-07-27 20814, 2022
10:30 AM
Sophist-UK has quit
2022-07-27 20835, 2022
10:30 AM
Pratha-Fish
No bugs in table generation afaik
2022-07-27 20829, 2022
10:31 AM
Sophist-UK joined the channel
2022-07-27 20855, 2022
10:31 AM
alastairp
what's the original recording mbid, then?
2022-07-27 20801, 2022
10:32 AM
alastairp
Pratha-Fish: oh, I think I worked out the unicode issue
2022-07-27 20810, 2022
10:32 AM
alastairp
we never actually made this an html document
2022-07-27 20811, 2022
10:32 AM
alastairp
2022-07-27 20824, 2022
10:32 AM
alastairp
check out the last section - "The Complete HTML5 Boilerplate"
2022-07-27 20818, 2022
10:33 AM
Sophist_UK joined the channel
2022-07-27 20832, 2022
10:33 AM
alastairp
we should use something like that. we don't need any of the `<meta property=>` or `<link>` or `<script>` tags, but copy the rest
2022-07-27 20836, 2022
10:33 AM
alastairp
and put the table inside <body>
2022-07-27 20853, 2022
10:33 AM
Pratha-Fish
alastairp: The original recording MBID is the same as the artist_name, recording_name, and the canonical_mbid!
2022-07-27 20805, 2022
10:34 AM
alastairp
Pratha-Fish: that's why I think it's a bug
2022-07-27 20818, 2022
10:34 AM
alastairp
because when I click on the link for the canonical mbid, musicbrainz tells me that it doesn't exist
2022-07-27 20846, 2022
10:34 AM
alastairp
but that's almost certain to be impossible, all canonical mbids should exist
2022-07-27 20852, 2022
10:35 AM
Sophist-UK has quit
2022-07-27 20811, 2022
10:36 AM
Pratha-Fish
alastairp: exactly! That ID doesn't exist in the musicbrainz table. I just rechecked the specific row in jupyter, and here's what I've found
2022-07-27 20829, 2022
10:36 AM
Pratha-Fish
2022-07-27 20846, 2022
10:36 AM
Pratha-Fish
This is before running the mbc.
2022-07-27 20811, 2022
10:37 AM
Pratha-Fish
mlhd_recording_mbid is from MLHD
2022-07-27 20854, 2022
10:37 AM
Pratha-Fish
mlhd_canonical_mbid is what I found by looking up the mlhd_recording_mbid in redirects and then canonical table
2022-07-27 20835, 2022
10:38 AM
Pratha-Fish
Then the mlhd_canonical_mbid is then used with an artist_credit query
2022-07-27 20835, 2022
10:38 AM
alastairp
where do you get rec_name from?
2022-07-27 20806, 2022
10:39 AM
alastairp
(btw, this is another great reason to have our code in scripts rather than a notebook, because then you could link me to the exact line in github)
2022-07-27 20807, 2022
10:39 AM
Pratha-Fish
alastairp: The rec_name and artist_credit are fetched using a SQL query
2022-07-27 20829, 2022
10:39 AM
Pratha-Fish
So I am guessing it's a fault in the musicbrainz db
2022-07-27 20834, 2022
10:39 AM
alastairp
the fact that rec_name is the same as mlhd_recording_mbid makes me thing that there's a bug with this sql query
2022-07-27 20853, 2022
10:39 AM
alastairp
no, almost certainly not an issue with the database, probably a problem with the way that we are reading it
2022-07-27 20858, 2022
10:39 AM
Pratha-Fish
alastairp: thankfully I have all this data in a script as well :D
2022-07-27 20808, 2022
10:40 AM
alastairp
great! where's the line, then?
2022-07-27 20832, 2022
10:40 AM
Sophist-UK joined the channel
2022-07-27 20807, 2022
10:41 AM
Pratha-Fish
2022-07-27 20850, 2022
10:41 AM
Pratha-Fish
alastairp: The row I shared earlier is from this particular table. All other rows have their specific rec_name and artist_credit data in place
2022-07-27 20823, 2022
10:42 AM
Pratha-Fish
2022-07-27 20839, 2022
10:42 AM
Pratha-Fish
2022-07-27 20856, 2022
10:42 AM
alastairp
what line makes this dataframe? (please make it easy for me to follow along)
2022-07-27 20806, 2022
10:43 AM
alastairp
you can click on a line number to get a URL that links to the line
2022-07-27 20806, 2022
10:43 AM
Pratha-Fish
right, sorry for the confusion
2022-07-27 20847, 2022
10:43 AM
Sophist_UK has quit
2022-07-27 20805, 2022
10:44 AM
Sophist_UK joined the channel
2022-07-27 20806, 2022
10:44 AM
Sophist_UK has quit
2022-07-27 20841, 2022
10:44 AM
Pratha-Fish
2022-07-27 20802, 2022
10:45 AM
Sophist-UK has quit
2022-07-27 20846, 2022
10:45 AM
Pratha-Fish
This line fetches rec_name and artist_credit using the artist_credit_list query that you shared last week
2022-07-27 20809, 2022
10:46 AM
alastairp
Pratha-Fish: I'd look in detail at your replace_multi method
2022-07-27 20824, 2022
10:46 AM
Pratha-Fish
right
2022-07-27 20841, 2022
10:46 AM
Sophist-UK joined the channel
2022-07-27 20851, 2022
10:46 AM
Pratha-Fish
2022-07-27 20855, 2022
10:46 AM
alastairp
immediately, I suspect that the item (recording mbid) that you're trying to look up doesn't exist in the musicbrainz database any more, and it's using the lookup key as the result rather than returning "none" or something
2022-07-27 20808, 2022
10:47 AM
alastairp
reading the code, I can't confirm if that is the case, but I suspect so
2022-07-27 20839, 2022
10:47 AM
Pratha-Fish
Hmm that seems likely
2022-07-27 20806, 2022
10:49 AM
Pratha-Fish
2022-07-27 20806, 2022
10:49 AM
Pratha-Fish
Here's what the replace_multi function uses under the hood.
2022-07-27 20817, 2022
10:49 AM
Pratha-Fish
Aaand you were exactly right!
2022-07-27 20833, 2022
10:49 AM
alastairp
yes, I saw that. but I don't know pandas enough to understand that behaviour just by reading the code
2022-07-27 20854, 2022
10:49 AM
Pratha-Fish
2022-07-27 20830, 2022
10:50 AM
Pratha-Fish
alastairp: ^ The above code just looks up the input value (in this case canonical_mbid) in the specified table (the artist_credit one)
2022-07-27 20836, 2022
10:50 AM
alastairp
oh right. I missed the 'except KeyError'
2022-07-27 20844, 2022
10:50 AM
Sophist-UK has quit
2022-07-27 20852, 2022
10:50 AM
Sophist_UK joined the channel
2022-07-27 20854, 2022
10:50 AM
Pratha-Fish
yep :)
2022-07-27 20820, 2022
10:51 AM
Pratha-Fish
i.e. in this particular case the "canonical_mbid" simply doesn't exist in the table
2022-07-27 20844, 2022
10:51 AM
alastairp
ansh: do you have an example of an edition group whose author list is [None] ?
2022-07-27 20818, 2022
10:52 AM
alastairp
Pratha-Fish: sorry, I have to run now. I have an event in an hour and need to go and get lunch first
2022-07-27 20831, 2022
10:52 AM
alastairp
do you have enough to work on?
2022-07-27 20852, 2022
10:52 AM
Pratha-Fish
alastairp: It's alright! I think I have enough to work on right now.
2022-07-27 20855, 2022
10:52 AM
alastairp
finding this issue + doing fast lookups for mbc + fixing html table output sounds like it should keep you busy
2022-07-27 20808, 2022
10:53 AM
Pratha-Fish
that's right
2022-07-27 20844, 2022
10:53 AM
alastairp
great, tomorrow morning I'm giving a talk, but I should be online with my laptop through the afternoon
2022-07-27 20852, 2022
10:53 AM
ansh
alastairp: yep '02ae4cfc-6412-4693-93b1-e24dce5e31f9'
2022-07-27 20805, 2022
10:54 AM
alastairp
ansh: thanks, I'll check this when I'm finished with my other tasks today
2022-07-27 20813, 2022
10:54 AM
Pratha-Fish
alastairp: great, I'll ping you only when required.
2022-07-27 20820, 2022
10:54 AM
ansh
I feel we should move this variable `DEFAULT_CACHE_EXPIRATION` used in many places to our config file, it would be really useful. We could disable cache with a single click :)
2022-07-27 20852, 2022
10:55 AM
Sophist_UK has quit
2022-07-27 20812, 2022
10:56 AM
alastairp
ansh: go ahead and open a PR for that!
2022-07-27 20823, 2022
10:56 AM
alastairp
ansh: normally when I do this, I open a redis shell and just run FLUSHALL
2022-07-27 20842, 2022
10:56 AM
alastairp
ansh: one really small thing too - I was just looking at the sql query in `fetch_multiple_edition_groups` and it has a trailing ;
2022-07-27 20856, 2022
10:56 AM
alastairp
this isn't needed when running sql from python, can you remove it?
2022-07-27 20843, 2022
10:58 AM
ansh
alastairp: I'll remove the semicolon
2022-07-27 20859, 2022
10:58 AM
alastairp
ansh: I ran this query with the bbid, and see the result. to me this seems a bit odd, and it might be better to try and modify the aggregate to make the query return [] instead of [null]
2022-07-27 20837, 2022
10:59 AM
alastairp gone
2022-07-27 20802, 2022
11:01 AM
ansh
alastairp: Okay, i'll try to modify it.
2022-07-27 20842, 2022
11:17 AM
s1b1 has quit
2022-07-27 20838, 2022
11:22 AM
s1b1 joined the channel
2022-07-27 20841, 2022
11:41 AM
yvanzo
hi reosarevok: my dev env is currently broken, do you plan to work on MBS-12512 again today?
2022-07-27 20842, 2022
11:41 AM
BrainzBot
2022-07-27 20802, 2022
11:42 AM
reosarevok
That's supposed to be fixed with the latest sample dump we generated a few days ago
2022-07-27 20807, 2022
11:42 AM
reosarevok
Have you tried it?
2022-07-27 20819, 2022
11:42 AM
reosarevok
Hmm, checking
2022-07-27 20814, 2022
11:43 AM
yvanzo
I tried it, see the comments.
2022-07-27 20829, 2022
11:43 AM
yvanzo
Have you tried it?
2022-07-27 20855, 2022
11:43 AM
reosarevok
No, because I got a report that it worked fine now
2022-07-27 20859, 2022
11:43 AM
reosarevok
Let's see
2022-07-27 20834, 2022
11:51 AM
yvanzo
zas: I have access, thanks.
2022-07-27 20822, 2022
12:03 PM
reosarevok
yvanzo: it's fixed, but the fix was merged into production
2022-07-27 20830, 2022
12:03 PM
reosarevok
So you need to also import from the production branch
2022-07-27 20848, 2022
12:03 PM
reosarevok
(I tested, importing from master skips genre_alias_type, but from prod it does not)
2022-07-27 20851, 2022
12:03 PM
yvanzo
Alright, thank you, will test.
2022-07-27 20801, 2022
12:04 PM
reosarevok
I'm still finishing an import in prod, you can wait 5 min if you want me to confirm :)
2022-07-27 20807, 2022
12:04 PM
reosarevok
But it didn't skip that, at least
2022-07-27 20801, 2022
12:10 PM
reosarevok
yvanzo: confirmed, worked in the production branch
2022-07-27 20801, 2022
12:12 PM
yvanzo
It worked here too.
2022-07-27 20841, 2022
12:12 PM
reosarevok
Yay
2022-07-27 20855, 2022
12:12 PM
reosarevok
Sorry for not checking that first and causing the annoyance
2022-07-27 20812, 2022
12:13 PM
yvanzo
No problem, thanks for the quick resolution.
2022-07-27 20809, 2022
12:16 PM
PetrCBRCZ
yvanzo: do you have another tip how to clean cache for slave MB installation ?
2022-07-27 20843, 2022
12:16 PM
yvanzo
PetrCBRCZ: that was the reason I was trying to refresh my dev env, still on it.
2022-07-27 20809, 2022
12:18 PM
PetrCBRCZ
yvanzo: ahh ... ok ...thx ;-)
2022-07-27 20849, 2022
12:27 PM
yvanzo
PetrCBRCZ: It returns "OK" here.
2022-07-27 20853, 2022
12:30 PM
yvanzo
The message you copied is returned by the following command? sudo docker-compose exec redis redis-cli FLUSHALL
2022-07-27 20841, 2022
12:31 PM
PetrCBRCZ
yes ... it returns "(error) READONLY You can't write against a read only slave."
2022-07-27 20801, 2022
12:33 PM
yvanzo
You should be able to check the number of items in Redis database with: sudo docker-compose exec redis redis-cli DBSIZE
2022-07-27 20801, 2022
12:34 PM
PetrCBRCZ
(integer) 388069
2022-07-27 20825, 2022
12:47 PM
yvanzo
It is unclear why it would be set to READONLY.
2022-07-27 20850, 2022
12:47 PM
yvanzo
PetrCBRCZ: Try that instead: sudo docker-compose exec redis redis-cli FLUSHDB
2022-07-27 20819, 2022
12:48 PM
PetrCBRCZ
same error
2022-07-27 20836, 2022
12:48 PM
yvanzo
There might be an issue with your Redis instance.
2022-07-27 20842, 2022
12:49 PM
yvanzo
INFO should provide more details for debugging.
2022-07-27 20850, 2022
12:50 PM
PetrCBRCZ
# Server
2022-07-27 20851, 2022
12:50 PM
PetrCBRCZ
redis_version:3.2.12
2022-07-27 20851, 2022
12:50 PM
PetrCBRCZ
redis_git_sha1:00000000
2022-07-27 20852, 2022
12:50 PM
PetrCBRCZ
redis_git_dirty:0
2022-07-27 20852, 2022
12:50 PM
PetrCBRCZ
redis_build_id:b9a4cd86ce8027d3
2022-07-27 20853, 2022
12:50 PM
PetrCBRCZ
redis_mode:standalone
2022-07-27 20853, 2022
12:50 PM
PetrCBRCZ
os:Linux 5.4.0-121-generic x86_64
2022-07-27 20854, 2022
12:50 PM
PetrCBRCZ
arch_bits:64
2022-07-27 20854, 2022
12:50 PM
PetrCBRCZ
multiplexing_api:epoll
2022-07-27 20855, 2022
12:50 PM
PetrCBRCZ
gcc_version:6.4.0
2022-07-27 20855, 2022
12:50 PM
PetrCBRCZ
process_id:1
2022-07-27 20856, 2022
12:50 PM
PetrCBRCZ
run_id:8fc13aa160b0d4e660ae3f0de2c91bd4b5d6ca90
2022-07-27 20856, 2022
12:50 PM
PetrCBRCZ
tcp_port:6379
2022-07-27 20857, 2022
12:50 PM
PetrCBRCZ
uptime_in_seconds:1649137
2022-07-27 20857, 2022
12:50 PM
PetrCBRCZ
uptime_in_days:19
2022-07-27 20858, 2022
12:50 PM
PetrCBRCZ
hz:10
2022-07-27 20858, 2022
12:50 PM
PetrCBRCZ
lru_clock:14759047
2022-07-27 20859, 2022
12:50 PM
PetrCBRCZ
executable:/data/redis-server
2022-07-27 20810, 2022
12:51 PM
PetrCBRCZ
mem_allocator:jemalloc-4.0.3
2022-07-27 20810, 2022
12:51 PM
PetrCBRCZ
# Persistence
