
Welcome to InkscapeForum!
Yeah, those filters can be SOooooo frustrating, lol. Some seem to only work on small objects, and others seem to work better on large ones. That's because most, if not all of the filters are actually raster content within the vector file. Raster means the colors are mapped to the pixels, and that's why scaling raster objects causes the phenomenon known as pixelation. That's why many people prefer vector graphics -- the scalability without distortion!
There may or may not be a way to tweak the filter to work in the situation where you need it. But first, instead of applying the filter and then scaling the object, have you tried scaling the object first, and then applying the filter? I'm guessing that will not work. But it might, and certainly not hard or time consuming to try it
Let's see.....since you included so many details in your message, I'll be able to do some experimenting. One question though -- do you have the Oil Paint filter applied to the text, or just to the calculator object?
There's a chapter in the manual on Filters. Well actually 2 chapters. Go to Help menu > Inkscape manual > Filter Effect (Preset) and (Custom). But good luck understanding the custom chapter, lol.
I'll let you know how my experiments go

Edit
Ok, I had the same experience as you. If you scale it smaller, after a certain point, the whole object just disappears! I think it has to do with the size of the object which was originally used to create the filter. However, if I scaled my sample rectangle first, and then applied the filter, it doesn't disappear. This technique does cause the effect created by the filter to appear slightly different. Maybe not even noticable difference. But if it is, you can probably tweak the filter a little to make up for it.
Off topic:
OT -- omgoodness, those filters are befuddling! That filter called Oil Painting, looks to me nothing like an oil painting! I guess if I did some drawing in a semi-realistic style, and applied that filter to the whole thing, maybe it would look like an an oil painting. I'm almost curious enough to try it, lol.