29 April 2023 - Blog Post

State of Mastodon Music

Been a while since I wrote down my thoughts on things. So here's some stats. 

• We're at 155 active users.  If one's comparing it to the below stat, instances.social says we have 386 users, so we haven't technically lost a couple hundred users, and still what I would describe as growing at a normal rate.  It remains a good mix of musicians, ~independant record labels, and music fans.  We're held this trend for about 2-3 months at least without things slipping one way or the other.

• Moderation.  Thankfully this has been very light on duties.  Since starting this, I've only had to handle 6 reports, where the root problem was invariably a blatant breaking of the server rules (spam, not labeling NSFW properly, etc) usually by an account on another instance, so suspending that user's behavior on mastodonmusic was an easy call.  I also caught exactly one programmatically generated account using a very supicious mail server that permitted temp email addresses, and dealing with them was trivial.  So far so good.

• Technical Updates.  Haven't had anything particularly earthshattering happen there.  Each updates has been preceded with a dump of the db as well as archival snapshots of the disk juuuust in case.  I don't feel particularly compelled to rock the boat yet with a technical redesign of the site as it remains responsive and reliable.  I have alerts for paging me if the site is down for 5 minutes and I believe the only time it paged was the huge 3.5.3 -> 4.0 update that took a few hours to complete (see previous post).  Things seems stable.  A huge rush of users could change that.

So, again, how can you help?

The costs are quite stable month-to-month, so I don't feel compelled to request donations.  What I'll continue to say is that, if you can afford it, please consider using any donation money you would have sent me instead on buying an artist's physical or bandcamp release or some swag or a concert ticket at a worthy small venue. Go do it right now, so you don't forget. 😉

I'm lucky to continue to be in a place where I can pretty comfortably afford the $140USD/mo that this site and the translation service costs.  Rearchitecting the site might reduce the costs in some way (cheaper storage and db), but also raise a lot of concerns that might lead to site performance issues for actions like the typical media cache cleanup operation, or for basic searches for users.  So again, for now, no change.  If something happens to birdsite or my own income, or some popular music celeb appears on our site, then I'll let everyone know how we're going to end up dealing with things and commit to sinking a weekend into some migrations or standing up a way to enable donations.

19 Nov 2022 - Blog Post

Well, it's been a wild 2 weeks of owning Mastodon Music.  I'm glad I started it, and it's great to get exposed to a whole range of music and music lovers.  Being a musician in the internet age has some mild upsides (easy to promote and share) but also some huge downsides that I think have really stressed musicians and the industry quite a bit.  The somewhat-forced move to streaming and away from album purchases has lead to career musicians putting out albums sometimes as loss-leaders, and has often necessitated that musicians have to go on tour for all but the most widely streamed musicians.

Summary, in case you're in a hurry

I'm happy to have Mastodon Music stood up, and I'm thinking about the future with some technical and admin challenges that a project like this presents.  In the meantime, I want this service to stick around and am grateful to be in a position to make that happen.  But if you care a lot, have a read.

Background

With that guy taking over Twitter (don't need to mention his name, we all know who), things have gotten that much worse for musicians.  When I thought about conciously leaving Twitter, I started to delete my music project's account, and turned around and went to find a music Mastodon server... and I was surprised that the options weren't readily obvious.  I know now that there's a few good choices (musicians.today / raveclub.nation, maybe others that I'm forgetting), but the only one I could find then was an Italian instance that had closed registrations.  As for myself, I'm lucky to have a pretty good day job in the tech industry, which personally creates a lot of tension, so I look for ways to help.

So I thought I was lucky enough to have some of the knowledge and comfort with tech and the means, and without thinking too long about it, I had acquired a cloud VM registered the name and set up mastodonmusic.social over the course of a few hours.  Almost as soon as I published the existence of my site to https://instances.social directory, people were signing up, so I guess I wasn't alone in hoping for a musician or music-lover-friendly instance.

Where are we headed?

So far, I would say things are going pretty well.  I'm not through a steady-state billing cycle yet, and probably wont be for another month, but it looks like Mastodon Music will cost about $100/mo to maintain for now, without having someone like Taylor Swift drop in and destroy the server with traffic (see below on exactly why this might be a problem at this moment).  ;-)  If I'm being blunt, I can manage that and none of this is a stressor for me.  So if you're looking for a Patreon right now, I probably won't have one, at least for probably a few months.  If you were thinking of helping me, instead go check out other local posts on the server and buy a record or two from someone you're loving.  That said, I also see some challenges for this server that I want to get ahead of.

• Scaling and Availability (technical and non-technical)

  • Technical bits. The default Mastodon deployment is intended to just run on a bespoke machine in a datacenter or house and maaaaybe serve 300-500 users (some handwaving), and this is running in a VM that's about that size.  The good news is that if we suddenly start running slowly, I have a lot of room to bump up the speed and capacity of this machine, but it's not a trick I can keep playing, and it's an expensive trick long-term.  But the good news is that my provider should never lose any of the data, and I take regular snapshots, so the odds of something wiping the data without recovery are really really low (nothing's impossible, but something that will almost certainly never happen).

    • The default webserver is nginx, which would require me to learn how to put that behind a VIP to scale that, though that's just an assuming as I don't know how that mitigates user session info between instances, but... I'm sure there's a way.  The database (posts text, user info, pretty much everything that's not media) is a single-instance PostgreSQL deployment on the same machine.  Not ideal for scaling.  Moving this to a cloud DB is the scalable thing to do, but it's not done easily or cheaply.  A local-disk dir handles all the media, both locally created and also posts from other servers that Mastodon Music users subscribe to.  This grows... a lot more than than the database.  Also there's some ways that Mastodon shards out tasks, like publish posts to other locations or handling incoming posts for people to read, etc that aren't quite low-hanging fruit, but it'd be interesting to see if I could run those in a more efficient way or even on other VM instances.

      • Side note: there's a basic prober test that runs on this site, so I should get SMS messages should the site ever go down for some reason.  :)

      • Logo note: the person who did the neat logo is here.  I can't claim to have amazing graphic design / illustration skills. More samples, including some unused ones, here.

      • Note: my day job is a little more IT and some light software engineering, but less so on the own-a-website sort of problem area.  I know a bit about this space and some of the basic failures to avoid, but I won't claim expert level knowledge that someone deep into their SRE career might have.  Tips on all this welcome.  :)

    • Lastly, and probably worth saying out loud, I don't personally scale.  I have x hours in the day to work on this, which is sometimes zero or close to it.  I'm a parent and I have a family and some other responsibilities to the world.  But I'll try to prioritize this for now.  But really it's just up to me at the moment, so I'm going to go through and write a Disaster Recovery doc to share with a few work friends, who, to be clear, haven't actually volunteered to help. People in my industry unfortunately have full knowledge of what owning a site like this really means (interacting with bad social media actors, needing to moderate and also needing to deal with whatever responsibilities might be required to be performed to maintain a site like this, no clear revenue stream to ensure that this gets paid for beyond me funding it and keeping my job) though I could at least say something like "you're not on the hook, but should I suddenly become very unavailable (illness, etc), could you work with volunteers here to keep things moving?" and having that Disaster Recovery doc would really help smooth the amount of effort needed and thus make it more palatable for some to volunteer to help with.

  • Non-technical summary of the technical bits. There's a few different ways and projects I'll likely be taking on to get this server to a place where it could handle more user signup influx.  It might even handle a few persons signed up that might drive a ton of traffic (read: what if someone famous very visibly signs up here.  I think I could be ready to deal with it, but I'm not positive with where things are right at this second).

How Can You Help?

So that's probably more info than most everyone needs.  I'm not yet ready to onboard volunteers purely because I'm not yet ready to trust anyone with this project yet.  To be blunt, there's a lot of power that comes the moderator / admin and also access to the VM, and I need to figure out some safe ways to trust others with that eventually.  But, if any of this resonates with you (tech comfort level, any SRE work done in the past, and an interest to help and maintain the moderation / rules structure that exists today), please DM me at @steve@mastodonmusic.social and we can at least start talking about this space.  Again, I'm not comfortable yet granting any wide access to anyone that I don't know and trust extremely well, so for now I'll be running things. I think the last thing that musicians need is more drama or any loss of data or access on their social media.   But I recognize that I'm a single point of failure, so I'll at least do what I can to ensure this service is running for as long as possible or needed.

Thanks,

• @steve@mastodonmusic.social