Extensions hangs when writing large number of objects

davers · Postby **davers** » Sun Feb 08, 2015 4:12 am

I've written two extensions which both hang when creating large numbers of objects (about 500 or more), but work well when creating fewer objects (less than 500).

Both extensions are written in Python and use inkex.py

I'd be happy to provide more details, but essentially they do this:

1. takes an imported png image as a selection and draws small ellipses or squares in a new layer over each pixel in the image which is a local maximum or minimum (the brightest and darkest points)

2. produces a voronoi tiling using the coordinates of a selection of objects like ellipses or squares.

I have very little experience with Python and xml and so I'm having difficulty being certain about the what's causing the problem.

However my initial suspicion is that it has somehting to do with the way lxml writes to stdout.

hulf2012 · Postby **hulf2012** » Sun Feb 08, 2015 5:20 am

Hello,

It's an interesting topic that one, about extensions. My knowledge about that is very basic.

I believe ... you are putting so much data in the RAM!! (500 svg objects is too much)...
Unless you are saving them in a file, ... in that case you are using your HD too much!!

Maybe if you could share part of your code... or use more specialized python forums.

davers · Postby **davers** » Sun Feb 08, 2015 5:32 am

Thanks!
I've modified the "Hello world" extension to place "Hello World" 10000 times in random locations. It works. Not sure what my problem is. Perhaps drawing large numbers of polygons just takes longer. I will try to find out exactly what is breaking.

ragstian · Postby **ragstian** » Sun Feb 08, 2015 6:21 am

Hi.

I have written an extension to change existing colors, it breaks as well if the number of selected objects are larger than 500.
Interesting, maybe the "too old" 2.6.5 python interpreter fails.

Do you mind sharing the extension code, I will try to "upgrade" the python Inkscape uses to 2.7.9 and see if it helps.

RGDS
Ragnar

davers · Postby **davers** » Sun Feb 08, 2015 6:34 am

Yes I will do some tests adding objects to layers in different ways and see what breaks and what doesn't, and post results here.

So far I've been able to add 10,000 "rect" objects no problem.

ragstian · Postby **ragstian** » Sun Feb 08, 2015 7:03 am

Hi.

My inkscape is now successfully using Python 2.7.9 but still fails running the extension found in this tread on a large number of objects.
For this extension the "magic number" seems to be around 2000, just as with the "old" python.

RGDS
Ragnar

hulf2012 · Postby **hulf2012** » Sun Feb 08, 2015 11:09 am

ragstian wrote:My inkscape is now successfully using Python 2.7.9 but still fails running the extension found in this tread on a large number of objects.
For this extension the "magic number" seems to be around 2000, just as with the"old" python.

Maybe there is a variable, that grows bigger and bigger... but I read the code and can't figure it out.... well... stroke_value it's only used once :?:

... but perhaps the problem isn't there.

About the problem in the first message ... It could be for many reasons!!!... without code, we are blind... but luck exist

~suv · Postby **~suv** » Sun Feb 08, 2015 3:47 pm

Maybe, the maximum length of arguments for a new process is hit (value might vary depending on platform): the ids of the selected objects are passed as command line arguments to the extension script. If the extension doesn't handle groups but requests the user to ungroup everything before selecting and running the extension, this limit might be hit.

Without a reduced test case posted here (a working extension with py and inx file), this is only a guess.

--
A very reduced test case wrt length of arg list is here.

~suv · Postby **~suv** » Mon Feb 09, 2015 3:48 am

ragstian wrote:My inkscape is now successfully using Python 2.7.9 but still fails running the extension found in this tread on a large number of objects.
For this extension the "magic number" seems to be around 2000, just as with the "old" python.

~~Seems to hit a platform-specific limit~~(see update below) - just tested your extension on OS X 10.7.5 with Inkscape 0.91 (Python 2.7.9, lxml 3.4.1) successfully with 1024, 2048 and 4096 objects. The spawned python process is quite busy for a while (took about 1 min 5 sec for 4096 rectangles), does finish properly, and inkscape subsequently reloads the modified file ok too.

*** UPDATE ***: After repeating the tests, I noticed this: when applied in a new unsaved document to just created objects (same as before, 4096 rectangles, arranged in rows & columns), the extension runs successfully. If however the file is saved first, and later reopened from disk in order to apply the extension, inkscape hangs, and consumes an increasing amount of memory, long after the spawned python process has done its job and finished. This again with Inkscape 0.91 on OS X 10.7.5.

*** UPDATE 2 ***: Test with same file reopened from disk repeated with Inkscape 0.48.5: while the inkscape process appears to "hang" (using 100% CPU) as well (after the python process has finished), memory usage stays constant, so I waited patiently until inkscape finally replaced the current drawing content successfully with the modified content (the earlier tests with 0.91 had been stopped (inkscape process killed manually after it reached > 3GB real memory usage) due to the increasing excessive memory usage).

*** UPDATE 3 ***: Similar test results with Inkscape 0.91 from PPA on Ubuntu 14.04 (VM 64bit, host: OS X 10.7.5): in unsaved file, extension script & inkscape are ok to process 4096 objects, and update without excessive mem usage. With the same test file loaded from disk, python process finishes ok, inkscape then takes a long time (high cpu usage) and similar growing memory usage until the mem limit is reached (the VM in VirtualBox is limited to to 2 GB, the inkscape process didn't go higher than 1.7 GiB). The mem limit on linux seems to help, and with some patience the extension succeeds to modify 4096 objects even with Inkscape 0.91.

*** UPDATE final ***: the earlier observations are related to how the file is loaded in Inkscape 0.91:

OK (methods to open the file which work well):
1. open file via file manager in Inkscape
2. open file as command line argument when launching inkscape from the shell
3. open file from within Inkscape in a new document window
  (current window either has been dirtied or has another document open):
not OK (method to open the file which triggers a regression in Inkscape 0.91)
1. launch inkscape (without any file) and open the test file via 'File > Open' or 'File > Open Recent'
  (the current document window is reused to load the file)

If the second case is avoided, the extension and Inkscape 0.91 process 4096 objects without any issues on OS X 10.7.5 and on Ubuntu 14.04 (VM, 64bit). The Python process takes a while, but that's to be expected.

davers · Postby **davers** » Tue Feb 10, 2015 2:43 am

I've managed to solve my problem, somehow. I've lost the original code that seemed to not work, although i suspect I wasn't giving it enough time to finish.

I'm posting the working code here. I am not particularly proud of these scripts, I'm new to inkscape, they work for my purposes but don't handle coordinates with transformations and I have no idea what would happen if I ran them on selections of objects they aren't designed to process.

peaksandvalleys.py is meant to run on a single selected image imported to inkscape as a link to a file on disk, and creates a new layer with rects at the coordinates of the locally bright and dark points in the image.

the scripts won't work well with images with large areas of uniform brightness.

~suv · Postby **~suv** » Tue Feb 10, 2015 3:02 am

Just a quick caveat for testers of davers Voronoi Tiling extension: On case-insensitive file systems, be aware that one of the files (Voronoi.py) will conflict with a shared python module which is bundled with Inkscape (i.e. installation into the shared extensions folder instead of (as recommended) into the user extensions folder will overwrite the file distributed with Inkscape, and thus break two Inkscape extensions):
voronoi.zip:

Code: Select all

peaksandvalleys.inx
peaksandvalleys.py
Voronoi.inx
Voronoi.py

Inkscape 0.91:

Code: Select all

$ ls -1 share/extensions/ | grep -i voronoi
generate_voronoi.inx
generate_voronoi.py*
voronoi.py*
voronoi2svg.inx
voronoi2svg.py*

@davers - thank you for sharing the code!

davers · Postby **davers** » Tue Feb 10, 2015 3:30 am

oops! I've removed Voronoi.py from my post.

ragstian · Postby **ragstian** » Tue Feb 10, 2015 4:15 am

Hi.

Code: Select all

Traceback (most recent call last):
  File "peaksandvalleys.py", line 15, in <module>
    import skimage
ImportError: No module named skimage

Might be necessary to guide users through install of skikit-image?

RGDS
Ragnar

hulf2012 · Postby **hulf2012** » Tue Feb 10, 2015 4:20 am

Thanks for sharing your code,

Sorry if I do a stupid question but...

That skimage that you call in your code have to be installed into the python interpreter that comes with inkscape, or in the inkscape interpreter that comes with, say... ubuntu?. If it has to come with the python that comes with inkscape... how do you install it, just copy them?.

Off topic:

I wonder if it could be possilbe to provide a special "installing script", in order to install inkscape python's extensions, with all their auxiliar libraries, no matter the OS.

For one moment I was suspecting that in your extension, you were calling some "verbs" of inkscape.
I'm talking about this:

Lazur · Postby **Lazur** » Tue Feb 10, 2015 4:41 am

-garbage collectors

hulf2012 · Postby **hulf2012** » Tue Feb 10, 2015 4:52 am

Lazur URH wrote:-garbage collectors

none of the above?

davers · Postby **davers** » Tue Feb 10, 2015 4:57 am

hulf2012 wrote:Thanks for sharing your code,

Sorry if I do a stupid question but...

That skimage that you call in your code have to be installed into the python interpreter that comes with inkscape, or in the inkscape interpreter that comes with, say... ubuntu?. If it has to come with the python that comes with inkscape... how do you install it, just copy them?.

I'm using ubuntu 14.04 , so I say:

"sudo apt-get install python-skimage"

works great! Inkscape seems to be running the python 2.7 interpreter installed on my system. I want to say again: my script is not designed to run well in a variety of situations! It works for me IF I have selected exactly ONE image embedded as a link to a file on my local disk, and have not previously applied any transformations to it (moving it, rotating it, etc.)

AND I want to add: the possibility of using scikit, sklearn, igraph etc. to edit vector graphics has me REALLY EXCITED. Thank gods for inkscape!

~suv · Postby **~suv** » Tue Feb 10, 2015 5:42 am

hulf2012 wrote:Suv:
... Thanks for investigating about the limitations...
But... at the end... my question is: Is there a limit about the number of objects that inkscape can handle, and "who" is the responsable for that limitation?:
- Python interpreter
- hardware limitation (ram,proccesors, hard disk, etc)
- Inkscape internals
- SVG limitations
- vary from extension to extension ?

Hmm, I was hoping that the updates I gave in my comment would provide sufficient information that those using Inkscape on Windows could investigate themselves where things go wrong …

AFAICT based on the tests on OS X and Ubuntu with that one extension available (ragstian's setstroke) which was reported to "fail" with more than 2000 selected objects on Windows, I can only repeat my findings: no limit on the number of selected objects per se (the mentioned max number / length of command line arguments seems much higher according to web searches); and under one specific condition (reloading a file from disk into an existing new document window and then run the extension on the selected objects of that file) where Inkscape takes a long time to process the incoming data and replace the current content of the drawing (0.48) or exposes a regression (0.91) - likely a memory leak (I can't tell, I don't code C/C++ myself).

The rather serious regression with 0.91 under that one specific condition is on my (virtual) TODO list - I plan to further investigate it (also wrt whether it depends on what kind of changes are applied to the objects in the document by the extension), and then file a bug report (if there's isn't already one tracking whatever the underlying issue is).

Lazur · Postby **Lazur** » Tue Feb 10, 2015 6:45 am

hulf2012 wrote:
Lazur URH wrote:-garbage collectors

none of the above?

Just had this file on my mind. Probably not related closely to extensions, but shows something small can cause heavy hickups. Haven't tested in 0.91, though that problem was related to garbage collectors.

hulf2012 · Postby **hulf2012** » Tue Feb 10, 2015 7:31 am

Lazur URH wrote:
hulf2012 wrote:
Lazur URH wrote:-garbage collectors

none of the above?

Just had this file on my mind. Probably not related closely to extensions, but shows something small can cause heavy hickups. Haven't tested in 0.91, though that problem was related to garbage collectors.

Well... some thing are not easy to undestand to me ... or understand at all :oops:

. Your drawing is huge. The machine where i'm it's old can show me it in chrome, but with a big effort.

Suv: Ok,... Some times it's so much for me. The limits vary then.

davers:
I can't get any spected result with your code.

In 0.91 there are changes with unitouu functions:
http://wiki.inkscape.org/wiki/index.php ... ing_change

So, I modify to:

Code: Select all

        width  = self.unittouu(svg.get('width'))
   height = self.unittouu(svg.attrib['height'])

but only the layer: 'Peaks * Valleys' is created.

So I comment them, and this message appears:

Code: Select all

Traceback (most recent call last):
  File "peaksandvalleys.py", line 84, in <module>
    effect.affect()
  File "/usr/share/inkscape/extensions/inkex.py", line 268, in affect
    self.effect()
  File "peaksandvalleys.py", line 71, in effect
    colors = [(r,g,b) for [r, g, b] in colors]
ValueError: too many values to unpack

You say in your case, the code works.

Maybe ragstian had same or better results?

~suv · Postby **~suv** » Tue Feb 10, 2015 8:43 am

@davers - extension (actually both, Extrema and Voronoi Tiling ;-) ) tested successfully with Inkscape 0.91 (from PPA) on Ubuntu 14.04 - I can't test on OS X because py-scipy via MacPorts has too many dependencies including a full GCC suite with fortran support compiled from source (I'm not willing to go there ;-) ).

Dependencies I had to install:

Code: Select all

sudo apt-get install python-skimage  # for Extrema
sudo apt-get install python-shapely  # for Voronoi Tiling

Diffs applied to get it working with Inkscape 0.91 (I renamed Voronoi.{inx,py} to voronoi_tiling.{inx,py}:

Code: Select all

--- peaksandvalleys.py.orig   2015-02-09 08:19:16.000000000 +0100
+++ peaksandvalleys.py   2015-02-09 23:23:14.000000000 +0100
@@ -45,9 +45,12 @@
 
     def effect(self):
              
+        if not hasattr(self, 'unittouu'):
+            self.unittouu = inkex.unittouu
+
         svg = self.document.getroot()
-        width  = inkex.unittouu(svg.get('width'))
-        height = inkex.unittouu(svg.attrib['height'])
+        width  = self.unittouu(svg.get('width'))
+        height = self.unittouu(svg.attrib['height'])
         
         layer = inkex.etree.SubElement(svg, 'g')
         layer.set(inkex.addNS('label', 'inkscape'), 'Peaks * Valleys' )
--- Voronoi.inx   2015-02-06 23:20:06.000000000 +0100
+++ voronoi_tiling.inx   2015-02-09 23:34:06.000000000 +0100
@@ -1,8 +1,8 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <inkscape-extension xmlns="http://www.inkscape.org/namespace/inkscape/extension">
   <_name>Voronoi Tiling</_name>
-  <id>voronoi</id>
-  <dependency type="executable" location="extensions">Voronoi.py</dependency>
+  <id>org.inkscape.effect.voronoi_tiling</id>
+  <dependency type="executable" location="extensions">voronoi_tiling.py</dependency>
   <dependency type="executable" location="extensions">inkex.py</dependency>
   <effect>
     <object-type>all</object-type>
@@ -11,6 +11,6 @@
     </effects-menu>
   </effect>
   <script>
-    <command reldir="extensions" interpreter="python">Voronoi.py</command>
+    <command reldir="extensions" interpreter="python">voronoi_tiling.py</command>
   </script>
 </inkscape-extension>
--- Voronoi.py.orig   2015-02-09 08:16:58.000000000 +0100
+++ voronoi_tiling.py   2015-02-09 23:23:14.000000000 +0100
@@ -48,9 +48,12 @@
         Overrides base class' method and inserts "Hello World" text into SVG document.
         """
  
+        if not hasattr(self, 'unittouu'):
+            self.unittouu = inkex.unittouu
+
         svg = self.document.getroot()
-        width  = inkex.unittouu(svg.get('width'))
-        height = inkex.unittouu(svg.attrib['height'])
+        width  = self.unittouu(svg.get('width'))
+        height = self.unittouu(svg.attrib['height'])
  
         # Create a new layer.
         layer = inkex.etree.SubElement(svg, 'g')
@@ -66,8 +69,8 @@
             transform = simpletransform.parseTransform(transform)
           else:
             transform = [[1,0,0], [0,1,0], [0,0,1]]
-          x = inkex.unittouu(node.get('x'))
-          y = inkex.unittouu(node.get('y'))
+          x = self.unittouu(node.get('x'))
+          y = self.unittouu(node.get('y'))
           c = simplestyle.parseStyle(node.get('style'))['fill']
           pt = [x,y]
           coords.append(pt)

This was only a quick test to verify that the extension works, and works in Inkscape 0.91 - I need yet to search for images better suited for such an extension. From what I have seen, I really like it (also that it colors the generated mesh with the colors of the traced images).

Biggest draw-back for now: the dependencies (they won't be bundled with Inkscape on Windows and OS X).

Inkscape 0.91 includes a python-based voronoi generator and a new extension which can create a voronoi tiling based on the midpoints of selected objects, but it doesn't support different coloring based on the objects' colors. Maybe it could be used (and extended) to replace your Voronoi tiling script (for easier portability)?

Wrt to peaksandvalleys and its usage of scikit-image - any idea whether a smaller module like Pillow could be used instead?

~suv · Postby **~suv** » Tue Feb 10, 2015 9:02 am

~suv wrote:Biggest draw-back for now: the dependencies (they won't be bundled with Inkscape on Windows and OS X).

ignores

davers wrote:AND I want to add: the possibility of using scikit, sklearn, igraph etc. to edit vector graphics has me REALLY EXCITED. Thank gods for inkscape!

Sorry, I didn't want to discourage your usage of well-established python modules … the context was more in the sense of "how could such a feature be included in Inkscape for a larger user base". It's completely ok, and no issue on Linux AFAICT - just on other (proprietary) platforms also supported by Inkscape, where the python dependencies (and the python runtime, on Windows) are distributed as part of the inkscape package.

This is a read-only archive of the inkscapeforum.com site. You can search for info here or post new questions and comments at inkscape.org/forums

This is a read-only archive of the inkscapeforum.com site. This feature is no longer available.

InkscapeForum.com

Extensions hangs when writing large number of objects

Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Re: Extensions hangs when writing large number of objects

Who is online