• Connect today Guest, Join our never resetting minecraft 1.20.4 survival server, where we explore, mine, build, and survive together. We are a family-friendly community, and open to all over the age of 13. Bring friends and make new ones!
    Features include: mcMMO RPG, Rewards for Achievements, a /market, Multiple worlds, sit on a chair, events, Economy, and more. We can't wait to have you join us!

The Purge 2021

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
Dear active players, you are safe.

But, ..

The purge is coming.

Okay, that totally sounded scary, but .. let me explain.

We have never tried to delete player data, accounts, builds, world, etc. And we don't really (really sorta) plan on doing this. For example, we kept /legacy, and we still have /general, and with 1.17 we will add /wild. And start our next season.

A decade has passed, and many thousands of players have checked the server out. But, we're not always good enough (insert rip music here).

And this leads to a lot of data, a lot of log files, a lot of databases with data inside them, and a lot of archives, and a lot of backups, etc. Yeah terabytes of data are collecting over the last decade.

This data includes user who have joined the server, but then left. Or who have joined, but got banned. Orrrrr, who has played for 15 minutes, this turned them into a full fledged account, but then they leave and never return.

There are plugins that we don't use anymore, and we stored this data.

There are region files to track progress for jobs, mmo, accounts, .. annnnd we don't have to anymore.

There are players with 0 levels in mcMMO, but we still have their data.

And on and on and on.

Would we not improve our server if we purge this? THEEE BIGG PURRGEEE... (dramatic music)

Yes, we would, the short answer is.

Okay, let's dive into that.

If we have less data for a plugin to load, it loads faster during startup.

If there are less log files and archives to manage, we have more drive space. This benefits us with server management. We are down a lot faster.

If we have less data to backup because the overall footprint of the servers is less on disk, we are offline faster during manual backups, and daily restarts.

Yes, I think that benefits us. We can play harder faster stronger!!! (dramatic music starts again and youtube pvp clips fill the screen... ok okay calm down floris).

Every little bit helps, especially when Mojang loves to introduce new mobs, new tiles, new features, new entities, new blocks, more data, bigger chunks, HIGHER WORLds, And Crazier FEaturUSR.ss. (faints).. I can't keep up.. but, I can help out!

So, after a decade, that's ten years! After TEEEEN years, we are finally ready to say: You know what, YOU. ... KNOW .. WHAT !! Let's just purge some data.

Let's go and collect a list of things that we 'have', and see what we can do to improve on it.

Do we need 7 years of backups? Yes, yes we do! There might be worlds in there that we don't use today, but we might still wish to import.. Okay, wait, that means we have to do that first, let's make a todo item to import things so it lasts forever, and then purge that backup. Okay, now we are building a timeline.

Do we need 5+ years of mcMMO data on users who haven't played in more than 5 years? No, probably not. Let's make a test server, learn the purge commands and see what we can do. Analyze the data and see how it goes. If this goes well, let's make a todo item to clean up mcMMO player data. Will this affect active players with levels? No, it shouldn't! In fact, there's less data to load during start up, during them joining the server, writing to disk, organizing, and less chance of corruption. EXCELLENT: Let's add this one to the list to do.

And we go through the list of things that we have, and make logical decisions. And call this (wait for it: ) THE PURGE !!!!

This year we will check out which databases we have archived that we no longer need today or in the future. For example rollback data from 5 years ago: We can purge that. Or a backup from a world we will never use again: Let's review that and then act on it. And go through plugins who also have their own backups - at that point we're just making backups of backups, and have an archive of backups with all ... yeah, no need to do that. We can clean that up (but make sure we understand we have still what we really need).

We will not promise that 1.17 or 1.18 in the future will happen, we are just working towards it. We will not promise that it will run smoother than 1.16 when we're done, we can't control what Mojang does. But we CAN make a difference by controlling the accounts and data, backups, archives, etc. And making space for the future means it might run the best it can with what we have.

In regards to accounts. Our spawn is very little, we will improve on it, and we will work on upping the performance so repawning or joining the server for the first time will be a tiny bit smoother. But .. what we will also do: That world is now over 3 GB big, mainly because it holds thousands of accounts from players - including around 500+ of players who haven't joined in 5 years. That's just kind of crazy. It's time to not only purge that but to work on some 'policy changes'

Policy changes:

Maybe we will be more open to frequent purging (not once every ten years)

- We can purge certain mob types more often.
- We can purge accounts that are super duper mega inactive more often.
- We could consider purging certain chunk data (without losing builds) a bit more often.
- We can organize plugins and their data more frequently.
- We can announce clearing of items and angry mobs in active chunks more frequently.
- We can nerf plugins, nerf the mobs and item spawn, tiles and redstone per chunks etc.

But don't worry: WE LOVE YOUR GAMEPLAY; why would we ruin THAT, right? So, this nerfing is about the top 1% that adds the most annoying lag to the game. Who needs 500 chickens around them. If the average is 150 per player for example for x amount of chunks in y time. Then we can consider putting a cap on 200, for x chunks over y time, and make the time a little shorter. This means 95% won't notice a difference, but the people causing the most lag with their chickens will. And that's okay, they can still have loads of chickens, but.. it's not super crazy anymore.

And that's just one example; we can do a million t.. okay maybe like 20, but.. we can do a lot to 'tweak' and 'nitpick' to improve performance while keeping 99% of the experience.

And you know what! I think all the little seconds here and there means that we have more space for updates like 1.17 and 1.18, and that your experience is just a bit better while you live stream, play, record, or hang out. Compared to us going 'oh well whatever', and not doing anything.

We will use this thread to give heads-ups on bigger purges, or we will once in a while inform you of some changes that happened Subscribe to the thread, so you don't miss out.

And if you are worried about GETTING PURGED !!!.. don't worry, just read this thread, or ask us questions, or ya know, play on the server. Active players .. they'reeee finnnneee.. Your builds, accounts, progress, it will all be there.

See, it's not that bad, but.. yes, for once - after a decade - how about we start cleaning up a little bit for your benefit :)
 

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
Update.

Very old region files from mcMMO worlds have been purged. This means it has less data to 'check' while you're mining etc.

Very old /spawn/stats/ and /spawn/playerdata/ and /spawn/advancements/ have been purged, from oldest up to Jan 1st, 2015.

Very old /CMI/sellLogs/ and /CMI/moneyLogs/ have been purged.

Super old /cache/ data has been purged. New caches have been built.

A directory with really old server backups has been deleted. We have a newer archive and half of it was double data.

Removed a bunch of old permission from old plugins we probably won't use anymore.

A few in-game /commands that we no longer use, and nobody tries to use have now been removed. It's time everybody uses the new version.

Archive of 32bit server engines and 1MB jars etc purged. We are full 64bit and can't even run these anymore. Time to let go.

The first mcMMO purge completed, 700+ records deleted from players who haven't logged in for ages and still have 0 levels.

Part of the first mcMMO purge: 100+ months old users purged. The next purge will be 90+

The rollback plugin has run purge on old data older than 52 weeks. This will be the last time, the next run will be 51 weeks, then 50, etc. Over time we will get close to 6+ months instead of 12+ months of data that we keep.

Backup Inventory plugin archives purged. We kept record of the last couple of weeks, everything else has been purged.

A start has been made. Other purged data is related to the OMG forums.

For any details please see the OMG / 1MB changelog threads on the forums here.
 

CraftKitten21

OMG Member
Joined
May 29, 2021
Messages
25
I was on a previous server that did this. I mean we had been requesting it for some time and when they finally did it, the server felt so refreshed and new. I know some people were upset but when you have players who played back in like 2012 and haven't been on the server since it seems silly to keep their stuff around.
 

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
I was on a previous server that did this. I mean we had been requesting it for some time and when they finally did it, the server felt so refreshed and new. I know some people were upset but when you have players who played back in like 2012 and haven't been on the server since it seems silly to keep their stuff around.
We can add another world to the list instead of removing the existing world. It's no big deal :)
 

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
A few more tidying up purges have happened. We've made progress.
 
  • Like
Reactions: Nix

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
Fall Season has started, and the purge is still going strong. Every week we pick up a topic and review what we can do to lower the amount of files, to review the data we have logged and then purge it. And lower the months or years that we keep data for certain plugins.
 

Floris

I'm just me :) Hi.
Staff member
Joined
Jan 1, 2001
Messages
60,096
Winter 2021, we're still purging data, cleaning up, and we will review what we can do in 2022. Which will be kind of the same, but we will also purge the banlist a bit, and remove stats from players who have been perm banned and we know that won't return. (jobs stats, mcmmo stats, money stats, etc)
 
Top