Hello

I will use this thread to discuss what I perceive to be a "memory cap" within Inkscape.



I am defining a "memory cap" as the approximate upper limit for possible size of a file.

When variables, within software, are defined - they are given a possible memory allocation size. During the early 1990's the largest conceivable memory possibilities seemed to be around a gigabyte. With that threshold in sight, I was taught to keep variable allocation sizes down to just a few megabytes - because - "no one would ever need more." Remember, the total amount of memory capacity in the space shuttle was less than what we now have available for our wrist watches - let alone phones.

Anyway -

When I work with Inkscape - I seem to get a lot of "image not found" errors whenever I begin to approach around 9 megabytes of inkscape file size. This size itself is interesting because of the power of ten associated with the size of a byte.

If I create the image within inkscape itself, I can transcend this apparent cap, but I can not pull in bitmap imagery when it has been surpassed. Also, those larger files become almost impossible to render ( export to png ).


Consider the artist using inkscape to make "svg style animations" for websites and such. They need reference imagery.

It is plausible that they would want to use reference materials to create a rotoscoped animation. Perhaps they want to create something based on a 30 second sequence of a monkey swinging through the trees.

This would require 30 seconds times at least 24 frames = 720 frames.

What is their reference material? It is very common to use a DSLR such as the Canon Eos Mark 2 or 3 to capture that information. Either they do so as video and grab appropriate frames with a different package or they use the stills developed with the camera. These frames are then pulled into inkscape to produce the svg animation simply because the format is better for web display than the others. Of course they keep the imagery loaded in the inkscape file because they want to grey it out and have it play behind the linework created in inkscape. My post regarding Nelchai's Camera discusses the process whereby they would capture the bitmap frames which could then be collected together for enhancement in other programs, made into GIF animations or simply used as reference for the final production of the SVG animation.

So, how much memory must inkscape be able to support in this case? 720 frames times appx 22 Megapixels per frame = approximately 32 gigabytes.

Obviously, that much memory is ALOT for any software and system to handle. We will ignore the newer gigapixel systems.

While I am not advocating the elevation of the capacity to those levels - I am suggesting that the file size opportunities presented by inkskape should expand greatly.

How far? How much is offered by the other graphic design packages which are used by professionals on a daily basis? I think it should be consistent with those packages.

In a reality where the hardware is not selling - which is why Dell wants to return to private - I think there will be developments to the hardware which expand the amount of memory the hardware can offer and support. I think Inkscape should be optimized to be more scalable with that growth.

Even if svg remains " only web centric for simple drawings and animations " - the new hardware possibilities will redefine " simple."

9 MB is simply not enough.

Tanks
James

>This size itself is interesting because of the power of ten associated with the size of a byte.

???

I think your need is very specific. I don't see inscape as a natural tool to "videotrace" (nor animating imho ; inkscape is a svg editor not a SMIL one). Plus the resulting svg would probably be too big enough to be played on everybody's computer (or if it's tailored down to be playable then you probably don't need such a high quality input)

How is the request for additional memory capability an "individual request?"

9 MB is too small for the gigabyte - terabyte era.

Allowing larger files - regardless of the example one uses to encourage that enhancement - assists and benefits everyone.

Thank you
James

From what you described the problem doesn't seem to come frome the complexity/size of the svg itself but from the size of all images you put it in.

It's like saying I want to import 32gb of video into 720 layers in gimp so that I can use them as a reference for drawing my 32*32 animated gif.

This is not the right tool. And 32gb won't fit.

nelchai wrote:When I work with Inkscape - I seem to get a lot of "image not found" errors whenever I begin to approach around 9 megabytes of inkscape file size.

Hhmmm...."image not found" is not the message I would expect if the file were too large. It might say "failed to load", or for me, would likely just crash Inkscape. Are you paraphrasing, or do you really get "image not found" which you attribute to file size?

In this approx 9 mb file, is it all vector content? I think you mentioned your system specs somewhere, but I can't remember exactly how much RAM you have (I just recall 'more than me' lol). I'll try to find an SVG file of that size in my files somewhere, to try and test this. But it would be helpful to know how much RAM you have, for comparison. And possibly also helpful to know about the content, raster vs vector.

Actually, with 4 gb RAM, I could probably open a 9 mb SVG, although it might take a few minutes. But I'll try to test.

I know that the point you're trying to make in this topic doesn't depend entirely on this issue. But I'm struck with a "not found" error attributable to file size. Off to test

Edit
Ok, I found a 17.9 mb SVG (looks to be all vector) which took 30 sec to open completely. No crash, no error! This with 4 mb RAM. Now, I can't do very much with it, without doing some tricks like hiding most of the content, while I work on the remainder. But fairly easily opened, with less RAM and probably lesser computer, than you have. I wonder if you could either have misstated or misunderstood something?

Hello

Reply to v1nce

I am NOT requesting a 32GB opportunity with Inkscape. As I stated in the original post:

nelchai wrote:While I am not advocating the elevation of the capacity to those levels - I am suggesting that the file size opportunities presented by inkskape should expand greatly.

The question is how far is appropriate for a software package used by professional digital artists?

9 MB is way too low!


What is allowed in Illustrator, Photoshop, Painter and the other packages?

I think Inkscape can and should be consistent with those levels.

How is increased memory capability a detriment?

reply to brynn

The way I work is to open a prototype file I created for development with each new project.

This prototype has a digital equivalent to BORCO table padding from the old days. It has a layer named "backer" which is 8000x4500 pixels and a standard neutral color. When I choose to render out to PNG, I select this layer and everything above it gets rendered. ( backer is usually hidden under everything else - palettes and such are outside the backer zone and are therefore not rendered ) Further, this file holds a small color palette, standard fonts, lines (for the weights) and other standard items. I can select and use the dialogs with the tools ( or eyedropper ) to set lines and such to be consistent with these standards. ( select a line, click the tool, open the dialog, choose style based on selection, start drawing...) This file on its own may be 1 MB in size.

When I want to start a new drawing, I simply open my prototype. The color set is updated in my pre-drawn palette ( like putting paint on a board ), my text is updated for any new fonts I want to use etc.

When my standards are updated for my new project - I pull in my scanned (1200 dpi ) reference materials. This size is actually necessary because the jaggies ( visualize a series of close contour lines which are green against a green background ) are extremely irritating when trying to trace the map if the scan is at 600 dpi. Remember, I have to zoom in to draw with the vector tools to get any type of accuracy from my work. Also, this allows me to reuse my drawings at larger sizes in later projects. ( I know they are accurate enough for the enlargement if I have already created them for it.)

For instance, this may be a map of - say - Michigan from 1890. This includes the state and county outlines. Another map - contemporary 2013 from the USGS - which has bathymetric data for the great lakes. Another map which shows carriage paths, shipping routes or other data from the 1700s etc. Perhaps, I will pull in a few images of previously drawn heraldry to represent companies or families. It really depends on the project.

These images are then organized to be consistent with one another. The maps are always different ( projections, accuracy etc ) which requires their redraw. The heraldry is placed in the appropriate locations, paths are drawn etc. GPS is not used because I create more of a cognitive map for the art effect rather than a "find your way" map for surveying.

( NARCISSISM: I really would like a - this is just for you - addition to inkscape where one could create a net or mesh of datapoints and uniformly deform that mesh. A squished map from 1750 which was created before the advent of longitude would be laid over a contemporary 2013 map of the borders of a region shown on the old map. The deformation would stretch the old to be consistent with the new and reveal locations as they would be located on a long-lat grid. This would be great for historians and such. )

Anyway, the maps are compared and I manually redraw them in vectors. The heralds, photos and other items are redrawn as well.

What if I discover that I should have an image of a carriage in the stack and forgot to load it?

If I am over around 9MB I very frequently get the "image not found error." The image can be in the same folder as the SVG file and this will still happen. I load with the embed rather than link option as the links always break with even the smallest change to the prefix location of the file. It is simpler to just load it and embed the image. ( Consider, a professional may work on this project today, put it away, change the reference images and reuse them in another project, then return to the first project a year later. Obviously, if the links have been broken, the file sucks. It is easier for the professional to embed. )

When the file is totally made within the vectors, there is no such error.

When the file is first loaded and then drawn, I can surpass the 9MB, but loading new images causes the error.

If I go too high, the render gets maybe 100 pixels at the top of the PNG but the other 4400 of the height won't render. I actually have a couple of post 25 MB files which are complete maps of the USA. These won't render at all!

Again, this is using the ASUS CM5570,Windows 7, Dual monitors, with 8GB Ram.

This week, I have been watching the CPU - RAM Gadget for function. If I turn off the changing wallpaper so that it only scrolls once per day... If I have only one SVG file open at a time.... If I have the clock gadget, standard icons and such visible.... RAM reads around 25 to 40% usage with the higher amount being when I have open and displayed folders or another file open. RAM never goes above this when rendering regardless of file size. CPU almost always drops down to maybe 5% when not rendering. The render pushes it to around a constant 50% regardless of file size from which the render is occurring. If I have two SVG files open at the same time during a render it often shoots to 100% and hangs there. Occasionally, CPU shoots to between 70 to 100% during a render - when only one SVG is open - for a second or two. This usually happens at the end of the render and is probably because of the additional burdens of the saving process.

When rendering, regardless of file size - CPU hovers around 50%; RAM hovers around 30%. This leads me to conclude that the problem is NOT the CPU or RAM. The problem is the need for the render farm capability - or the camera to PNG - and a rewrite of the software to modernize the variable allocations ( eg depending on OS, perhaps changing from int variables with a range of up to +32,767 - to long int with a range of up to +2,147,483,647 - as well as the use of fewer global variables - enhancement for 64 rather than 32 bits - C99 rather than C89 - and such ).

The final step in my work process is the removal of the reference materials. Sometimes, it is easy to do so with a redrawn herald or such. Many times, there is additional information within the maps or imagery which could facilitate a fork of the design for a new presentation or reuse at a later date. It saves time to reuse the file and tailor it for a discussion of village locations or disaster routes or other subject matter rather than having to reload and redraw every time.

In other words, the removal of the references is not always the most ideal or professional choice when the file is already built.

The ability to save these large files functions normally - as does the opening process. Loading more image data into them post 9MB reveals that "image not found" error regardless where the reference file is located. Rendering becomes a problem as the file grows as well - but that is usually more common when filters are used. I have files 11MB in size with a 8000x4500 pixel background with a colored layer under a white marbled ink layer which will not render out. ( these are being redesigned to facilitate compositing in GIMP - which is what I started doing Wednesday night! )


All of the ways in which I use the SVG system is consistent with what other professionals are doing with the other products.

9MB is way too low.

It should be consistent with those other products.

Tanks
James

> When my standards are updated for my new project - I pull in my scanned (1200 dpi ) reference materials. This size is actually necessary because the jaggies ( visualize a series of close contour lines which are green against a green background ) are extremely irritating when trying to trace the map if the scan is at 600 dpi.

can't you preprocess your image to enhance the border and use a smaller DPI ?


> ( NARCISSISM: I really would like a - this is just for you - addition to inkscape where one could create a net or mesh of datapoints and uniformly deform that mesh. A squished map from 1750 which was created before the advent of longitude would be laid over a contemporary 2013 map of the borders of a region shown on the old map. The deformation would stretch the old to be consistent with the new and reveal locations as they would be located on a long-lat grid. This would be great for historians and such. )

this could be usefull to what % of inkscape user ?
try dedicated GIS tools instead (Didger)

If you're drawing maps, maybe ask advices at http://www.cartographersguild.com/ some people are using inkscape


> 9 MB is way too low!
> ...
> I load with the embed rather than link option
> ..
> I actually have a couple of post 25 MB files which are complete maps of the USA. These won't render at all!

I can't comment about a 9mb limit but embedding images will make your file size grow rapidly. If you're embedding such files in a SVG then the size of svg will not be 9mb but 9 + (25*1.5) (because of UU64 version of image file being embedded). That's only for the svg itself and for 1 image embedded. Then there will be some cached uncompressed version of the image (250mb). Plus the working copies for filters...

> ( Consider, a professional may work on this project today, put it away, change the reference images and reuse them in another project, then return to the first project a year later. Obviously, if the links have been broken, the file sucks. It is easier for the professional to embed. )

Make the path relative and zip all and voilà.

> a rewrite of the software to modernize the variable allocations ( eg depending on OS, perhaps changing from int variables with a range of up to +32,767 - to long int with a range of up to +2,147,483,647 - as well as the use of fewer global variables - enhancement for 64 rather than 32 bits - C99 rather than C89 - and such ).

Don't you think you're oversimplifying it ? 'use of fewer global variables' seriously ?

nelchai wrote:What if I discover that I should have an image of a carriage in the stack and forgot to load it?

If I am over around 9MB I very frequently get the "image not found error."

Ooohh, I see! You're not just opening a large 9 mb file. You're trying to import a new image into a file that is already 9 mb. Do you get the choice to embed or link? Or does it crash before that happens?

Not that I would have any suggestions, one way or another.

I'll try importing another image into my large ~18 mb file, although I expect a crash....

Huh! I'm shocked, but I was easily able to import a small JPG (120 kb) into the 17.9 mb file! How large is the file you're importing? I'll try a larger file....

Ok, I couldn't find a large raster, so I tried importing another large SVG (11 mb) and Inkscape crashed (Runtime terminated in unusual way). Maybe you're not supposed to import SVG files, I'm not sure. Let me search around and try to find a mb or larger raster to test importing, and I'll edit in results

Edit
Ok, I finally had to convert a JPG to BMP to get a large raster image -- 2.25 mb, which imported, embedded into the 17.9 mb SVG file. No problems, not more that 5 second hesitation to display!

Now, I didn't read the entire text of your description of what you were doing, partly because it was so long, but also partly because most of it doesn't help me understand the process with Inkscape. If you could write a clear step by step process for reproducing the 'failed to load' error (or "image not found" or whatever it is), and provide either an original file, or a description so that I can recreate such a file, and the file to be imported (or description thereof) I'll be glad to test.

It's not that I don't believe you're having this problem. I want to reproduce it, as a means of offering some comparison for you. Maybe we can track down the problem.

I agree that in a day and age of terabyte drives and PCs with up to or beyond 16 Gigabytes of RAM that a 9 megabyte file ... or even a 34 megabyte file ... should not be too much for one application to handle.

Good point.

I would agree with that as well. But so far, I haven't been able to reproduce the same problem that nelchai is having. I'm not entirely sure I yet have all the facts.

I am having the exact same problem. Cannot import files >~9MB, "failed to load" error appears. I suspect this is not a computing problem, as I am able to export as bitmap images from an SVG file that exceed 50MB, with the only limitation being the computing time required.

Is there any way around this? Hugely disappointing from Inkscape. Making detailed analytical diagrams for scientific publications in Inkscape is pointless if all of my images will be low res.

I am running i7, windows 8 with 12GB RAM