Unplayable Lag with ME System & lots of learnt items

Using v0.4.6 in Project Ozone 3, at least a thousand items learnt.

I've been playing for a while and the lag's been getting worse and worse till I'm getting 10fps at 5 render distance and I can't play anymore. Lag goggles show EI is using 190% for the server tick event. I've tried limiting it to 1,000 items shown (from the default 1,000,000), but it hasn't done anything.

When I tried opening my terminal while timing, it took several seconds for any items to appear, and then the whole game froze for 30 seconds (windows reported not responding).

latest.txt

I just did some extra testing and it appears that it has something to do with AE trying to get a list of all the items. If I set the storage bus to insert only, the lag goes away and extract only/bi-directional make the game laggy. Disconnecting the transmutation chamber also reduces the lag.

[random53]

Im curious of the specifics of this because i created a dummy me system and the lag was at least reduced by a significant amount for me. Ill have to do more testing to see whether it could be based on just the sole combination of items, autocrafting, etc

[random53]

Also seems like the mod author hasnt really been around much in the past year about so im doubting for a fix to happen anytime soon

If you use a tome of knowledge, does it get any laggier?

[random53]

I only did survival testing so i didnt have a tome of knowledge unlocked yet. However after spending a few hours messing around with the system im pretty sure export busses and import busses cause 90% of the lag.

[random53]

After replacing all of the export busses that i could find and setting the storage bus priority to negative ( so items didnt get imported into the transmutation chamber) the impact went from 36ish% to 1% and im pretty sure i didnt get all of the import busses but either way its a huge difference and doesnt really even seem noticeable

[random53]

There could still be other stuff at play but when i was testing it with only drives hooked up it was down to .27% at most so im assuming the readout updates arent the problem just the emc conversion updates

Another edit, it appears that it's most likely due to me constantly exporting items from my ME system (using the transmutation chamber). I have 3 quantum compressors with export buses on them, each using an EMC'ed item. As soon as I added a toggle bus and disabled them, the lag immediately reduced. My best guess is that each time AE exports an item, it checks how many of that item there are, which would call an EI function. I saw in another thread that it's a limitation with ProjectE, where the function to get an EMC value is super slow.

[random53]

Yeah something like that it could also be the fact that the system doesnt actually have those items stored so the export bus probably has a wierd functionality issue with the emc pool

[random53]

Interfaces seem to work just fine as far as i can tell so just replace them with that and pop some translocators on

For the compressors I just used automation to get all the catalysts, components etc. I know for a fact that the export buses were pulling from my EMC because I don't think anyone has millions of blocks of ludicrite to compress. And yeah I switched to Personal EMC links and those were fine. I think it's just an issue of AE exporting an itemstack, then marking the inventory (the transmutation chamber) as modified and tries to refresh all the items stored in it. When you have several thousand items stored, calculating the EMC and amount of items left is going to become pretty slow. I saw somewhere that ProjecE takes over 6 seconds for 100k EMC-able dummy items. So ~60ms for me.

[random53]

Well i asked before i saw you only said you had 3, but i was wondering because they are animatable but i feel like that seems kinda cheaty? but idk

Animating is the entire point of the pack. It was purposefully left enabled. Anyway I'm very late game (almost up to infinity gear) so I don't need to duplicate that much stuff.

[mkaito]

The big lag source isn't just that there's a lot of different items, but that the transmutation chamber tells AE2 that you "have" 1 million of each item. That's a whole lotta stacks.

You might want to try changing the setting for max published items. I reduced it from 1m to 500k and the lag is considerably better. The only downside is that you can't queue massive autocrafts that actually need 1m of any given input, but I haven't had that problem (yet).

As you guys found, bulk export/import via transmutation chamber is quite laggy and EMC links work better in this regard, especially when bypassing AE2 entirely.

In general, AE2 performs really badly when you bulk-move items by digitizing them and then immediately un-digitizing them, such as importing via storage bus only to dump it via export bus. This is one of the Two Big No-No of AE2 performance. The other one being a storage bus loop (interface stuck on a storage bus on a subnet, and again the other way, which results in storage data updates looping exponentially whenever an item is moved via either storage bus).

This isn't just specific to the transmutation chamber, it's also pretty bad if you start exporting at full speed from a black hole unit, for example. Exporting from ME Drives is considerably faster.

Bulk exporting from the transmutation chamber just adds more lag on top of an already laggy operation.

Anyway I'm very late game (almost up to infinity gear) so I don't need to duplicate that much stuff.

Boy oh boy, do you have a storm coming 😂

[random53]

I remember from my testing that when i changed the max items down to even 1000 the lag still largely remained. The only thing that fixed the problem for me was allowing no imports into the chamber itself and only either exporting with the chamber disconnected or in small amounts.