Graphviz Issue Tracker
Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002049graphvizSfdppublic2011-02-10 09:462011-04-28 04:03
ReporterSimon 
Assigned Toerg 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSx86-Windows-7OS Version
Summary0002049: graphviz crashing with large files
Description



after calling sfdp with data of about 100000 edges sfdp crashes after a few minutes.






<CD>
sfdp -Tpng -o friends.small.sfdp.png friends.small.txt
out of memory
</CD>



This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
Additional Information

[yifanhu]
do a

sfdp -v

and see what you get. Also, in your previous bug report, you have overlap=scale, this is usually not a good idea for large graph, since this could create a png file of huge dimension, which in itself can cause memory problem.

[yifanhu]
<CD>
> Hi,
> I removed the "overlap=scale" parameter from the file, now sfdp is painting the graph with 100000 edges.
> The call was: sfdp -v -Tpng -o friends.small.100000.withoutoverlap.png friends.small.100000.txt
> Sadly there are only some clusters and a very big black clump with red dash of colors.

Where does color come from? Do you set your own color?

If this is a social network, I am not too surprised that you see a hairball. You will have to play with various rendering options. Dump the graph first with

sfdp -v friends.small.100000.txt > foo.gv

and then experiment rendering using different color/opacity, e.g.,

neato -Estyle=invis -n2 foo.gv -Tpng > friends.small.100000_noedges.png

or

neato -Ecolor="#55555522" -n2 foo.gv -Tpng > friends.small.100000_semi-transparentedges.png

gvpr is a useful tool when you want to adjust individual edge/node colors.

make nodes smaller with

neato -n2 -Nshape=point -Nwidth=0 -Nheight=0 -Nmargin=0 -Tpng foo.gv > friends.small.100000_nodepoints.png

>
> The call with overlap to extend the graph to see more nodes instead of a black clump was this.
Do not use overlap=scale, because you will run out of memory as graphviz try to render a huge graphics file. Use overlap=prism, and limit output size to, say, "-Gsize=40!" (40 inch)


> ----------------------------------------------------------------------------------------
> sfdp -v -Tpng -o friends.small.100000.withoverlap.png friends.small.100000.txt
>
> 000.withoverlap.png friends.small.100000.txt
> sfdp - graphviz version 2.26.3 (20100126.1600)
> Activated plugin library: gvplugin_pango.dll
> Using textlayout: textlayout:cairo
> Activated plugin library: gvplugin_neato_layout.dll
> Using layout: sfdp:neato_layout
> Using render: cairo:cairo
> Using device: png:cairo:cairo
> The plugin configuration file:
> D:\Grafik\Graphviz2.26.3\bin\config6
> was successfully loaded.
> render : cairo dot fig gd gdiplus map ps svg tk vml vrml xdot
> layout : circo dot fdp neato nop nop1 nop2 osage patchwork sfdp twopi
> textlayout : textlayout
> device : bmp canon cmap cmapx cmapx_np dot emf emfplus eps fig gd gd2
> gif gv imap imap_np ismap jpe jpeg jpg metafile pdf plain plain-ext png ps ps2 s
> vg svgz tif tiff tk vml vmlz vrml wbmp xdot
> loadimage : (lib) bmp eps gd gd2 gif jpe jpeg jpg png ps svg
> pack info:
> mode 2
> aspect 0.000000
> size 0
> margin 8
> flags 0
> coarsest level -- 17, n = 4
> iter = 500, step = 0.008067 Fnorm = 0.024725 nsuper = 0 nz = 12 K = 0.753470
> level -- 16, n = 8
> iter = 44, step = 0.000970 Fnorm = 0.015215 nsuper = 0 nz = 56 K = 0.565103
> level -- 15, n = 15
> iter = 44, step = 0.000970 Fnorm = 0.080069 nsuper = 0 nz = 210 K = 0.423827
> level -- 14, n = 30
> iter = 44, step = 0.000970 Fnorm = 0.337365 nsuper = 0 nz = 870 K = 0.317870
> level -- 13, n = 60
> iter = 44, step = 0.000970 Fnorm = 10.116404 qt_level = 9 nsuper = 13 nz = 3540
> K = 0.238403 level -- 12, n = 119
> iter = 44, step = 0.000970 Fnorm = 34.746595 qt_level = 4 nsuper = 16 nz = 14042
> K = 0.178802 level -- 11, n = 237
> iter = 44, step = 0.000970 Fnorm = 158.150984 qt_level = 13 nsuper = 20 nz = 558
> 64 K = 0.134102 level -- 10, n = 473
> iter = 44, step = 0.000970 Fnorm = 602.752665 qt_level = 10 nsuper = 24 nz = 218
> 426 K = 0.100576 level -- 9, n = 945
> iter = 44, step = 0.000970 Fnorm = 2488.553838 qt_level = 9 nsuper = 29 nz = 743
> 746 K = 0.075432 level -- 8, n = 1888
> iter = 44, step = 0.000970 Fnorm = 7493.308211 qt_level = 12 nsuper = 34 nz = 19
> 87836 K = 0.056574 level -- 7, n = 3773
> iter = 44, step = 0.000970 Fnorm = 20993.929681 qt_level = 15 nsuper = 39 nz = 4
> 622116 K = 0.042431 level -- 6, n = 5154
> iter = 66, step = 0.000955 Fnorm = 351.445063 qt_level = 12 nsuper = 37 nz = 477
> 06 K = 0.031823 level -- 5, n = 7149
> iter = 44, step = 0.000970 Fnorm = 386.955026 qt_level = 13 nsuper = 41 nz = 557
> 86 K = 0.023867 level -- 4, n = 9719
> iter = 44, step = 0.000970 Fnorm = 399.229354 qt_level = 18 nsuper = 43 nz = 637
> 20 K = 0.017900 level -- 3, n = 13143
> iter = 44, step = 0.000970 Fnorm = 452.945311 qt_level = 13 nsuper = 46 nz = 718
> 44 K = 0.013425 level -- 2, n = 19223
> iter = 44, step = 0.000970 Fnorm = 541.088817 qt_level = 12 nsuper = 49 nz = 848
> 22 K = 0.010069 level -- 1, n = 32518
> iter = 44, step = 0.000970 Fnorm = 835.900882 qt_level = 13 nsuper = 51 nz = 111
> 916 K = 0.007552 level -- 0, n = 69268
> iter = 44, step = 0.000970 Fnorm = 1963.165193 qt_level = 10 nsuper = 55 nz = 18
> 5560 K = 0.005664 sfdp: overlap=-1 scaling -4.00
> Adjusting _cc_0 using scaling
> Node separation: add=1 (4.000000,4.000000)
> out of memory
>
> This application has requested the Runtime to terminate it in an unusual way.
> Please contact the application's support team for more information.
> ----------------------------------------------------------------------------------------
> My computer has about 12 GB. So the ram - i think - is not the problem.
>
> If i'm trying to graph a file with about 3,7 million edges graphviz aborts even without the overlap parameter:
> ----------------------------------------------------------------------------------------
> sfdp -v -Tpng -o friends.big.withoverlap.png friends.big.txt
>
> sfdp - graphviz version 2.26.3 (20100126.1600)
> Activated plugin library: gvplugin_pango.dll
> Using textlayout: textlayout:cairo
> Activated plugin library: gvplugin_neato_layout.dll
> Using layout: sfdp:neato_layout
> Using render: cairo:cairo
> Using device: png:cairo:cairo
> The plugin configuration file:
> D:\Grafik\Graphviz2.26.3\bin\config6
> was successfully loaded.
> render : cairo dot fig gd gdiplus map ps svg tk vml vrml xdot
> layout : circo dot fdp neato nop nop1 nop2 osage patchwork sfdp twopi
> textlayout : textlayout
> device : bmp canon cmap cmapx cmapx_np dot emf emfplus eps fig gd gd2
> gif gv imap imap_np ismap jpe jpeg jpg metafile pdf plain plain-ext png ps ps2 s
> vg svgz tif tiff tk vml vmlz vrml wbmp xdot
> loadimage : (lib) bmp eps gd gd2 gif jpe jpeg jpg png ps svg
> out of memory
>
> This application has requested the Runtime to terminate it in an unusual way.
> Please contact the application's support team for more information.
> ----------------------------------------------------------------------------------------
> I the task-manager i can see shortly before the programm crashes that is has more than 2 GB in the ram.
> The same happens if i choose ps output-format.

Graphviz's data structure is memory intensive. I resort to a standalone version for really large graphs.
</CD>

[erg]
<CD>
  To recap what Stephen and Yifan have noted, and add my own two cents, if you have a large graph (say, number of nodes > 100,000),
    - only use sfdp for layout
    - set -Nshape=point (you won't be able to see complex node shapes). You might want to reduce width and height even smaller than the
default for shape=point.
    - usually you don't want to do overlap removal. (sfdp does a little jiggling on its own.) If you do try to use the overlap attribute, make
sure nodes have shape=point and use overlap=prism. Never consider using overlap=scale.
    - for output, use an ascii/vector format like dot, xdot, svg, or postscript; don't generate a bitmap image unless you set the graph's
size attribute to something small enough. A good approach is to do an initial layout with sfdp -Tdot, and then use neato -n and gvpr
to tweak the final image.
    - the graphviz data structures are moderately large: 300+ bytes for nodes and 180+ bytes per edge. It's not hard with a goodly
number of nodes to run out of memory during layout, or ever before. If memory is almost used up, all it takes is the allocation of
some auxiliary data to exhaust it. That is what is happening in your trace:
> > Adjusting _cc_0 using scaling
> > Node separation: add=1 (4.000000,4.000000)
> > out of memory
The sfdp layout has finished, but overlap=scaling allocates a vector and that exceeds the memory limit.

  - if you are willing to just get position information and then handle rendering on your own, it is fairly simple to extract the
core of sfdp which uses stripped-down sparse matrices. This will allow you to layout much larger graphs.
</CD>
TagsNo tags attached.
AUXILLARY-FILES
DATE-FIXED
FIX-COMMENT
FORMER-ID2135
INPUT-FILE
OUTPUT-FILE
STATUS-COMMENTFixed
VERSION     2.36.3
Attached Files

- Relationships

-  Notes
There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
2011-04-28 04:03 user1 New Issue
2011-04-28 04:03 user1 Assigned To => erg


MantisBT 1.2.5[^]
Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker