Memory leek or missing some dispose function?

Hi, I made program to calculate node position for topology running the graphviz using dll import in C#. I try to test this code if I hadn't memory leek in unmanaged code. And I have unclear results. I run function to calculate topology 2060 times with 4 different topology (topology with 1000,300, 12 and 10 nodes). Test environment
• Window 7 enterprice 32 bitowy
• RAM 3.25 GB
• Processor Intel Core 2 Duo E8200 2.66 GHz
• Microsoft .NET Freamework 4 version 4.0.303019
• The test program was only program run on this computer
• Graphviz version: 2.31.20130429 After each test I save memory usage for this program form garbage collector. The memory usage was more or less constant.

GC.GetTotalMemory(true);
And the available memory in the system and using performance counter. At the beginning of the test the value was 2395 MB after 1750 test the value drop to 937 and stop falling and become more or less constant till the end of the test.
var ramCoun

ter = new System.Diagnostics.PerformanceCounter("Memory", "Available MBytes");
ramCounter.NextValue();

Edit 18.06.2013
- Add to the code to add function gvFreeRenderData
End Edit 18.06.2013

Edit 18.06.2013
- Add function gvFinalize
End Edit 18.06.2013

I’m don’t know it the operating system don’t care about free the memory if have more than 1GB or there is some memory leek. It leek about Ram1MB peer layout calculation. Code to run calculation
private TopoModelBase CalculateGraphLayoutUsingGraphvizAsDll(TopoModelBase graphData)
{
    IntPtr gvContent = IntPtr.Zero;
    string dotWithCord = null;
    string layout = "neato";
    string format = "dot";
    checkLimitForGraphCalculateLayout(graphData);
    StringBuilder dotFileBuilder = new StringBuilder();
    TextWriter dotFileSW = new StringWriter(dotFileBuilder);
    saveGraphToDotStream(dotFileSW, graphData);
    dotFileSW.Flush();
    try
    {
        gvContent = gvContext();
        if (gvContent == IntPtr.Zero)
            throw new NMSTopoGeneratorExeption("Failed to create Graphviz context.");
        //read form dotstream
        IntPtr g = agmemread(dotFileBuilder.ToString());
        if (g == IntPtr.Zero)
            throw new NMSTopoGeneratorExeption("Failed to create graph from source. Check for syntax errors.");
        // Apply a layout
        if (gvLayout(gvContent, g, layout) != SUCCESS) // TODO: Fix AccessViolationException here
            throw new NMSTopoGeneratorExeption("Layout failed.");
        IntPtr result;
        int length;
        // Render the graph
        if (gvRenderData(gvContent, g, format, out result, out length) != SUCCESS)
            throw new NMSTopoGeneratorExeption("Render failed.");
        // Create an array to hold the rendered graph
        byte[] bytes = new byte[length];
        // Copy the image from the IntPtr
        Marshal.Copy(result, bytes, 0, length);
        // Free up the resources
        //gvFreeLayout(gvContent, g);        // move to finally section
        //agclose(g);                // move to finally section
        dotWithCord = System.Text.Encoding.UTF8.GetString(bytes);
    }
    finally
    {
    // Edit 18.06.2013
        
        if (gvContent != IntPtr.Zero && g != IntPtr.Zero)
            gvFreeLayout(gvContent, g);
        if (g != IntPtr.Zero)
            agclose(g);
        if (gvContent != IntPtr.Zero)
            gvFreeContext(gvContent);
        if (result != IntPtr.Zero)
            gvFreeRenderData(result);
    // End Edit 18.06.2013
    }
    StringReader fileStremReader = new StringReader(dotWithCord);
    // convert date from DOT format to my data structure. function from base class TopoGeneratorBase
    readGrapFromDotStream(fileStremReader, graphData);
    normalizeGraphPositions(graphData);
    return graphData;
}
Mode code just in case you thing it might by necessary. Class to run test
bool runNextGraphvizCalculation = true;
Task graphvizLoopedCalculationTask = null;
int GraphvizCalculationLoopCounter = 0;
// start graph viz loop
private void button4_Click(object sender, EventArgs e)
{
    // stop previous test thread
    runNextGraphvizCalculation = false;
    if (graphvizLoopedCalculationTask != null)
        if (graphvizLoopedCalculationTask.Wait(60000))
            return;
    GraphvizCalculationLoopCounter = 0;
    runNextGraphvizCalculation = true;
    // start aditional Thread to calculate test
    graphvizLoopedCalculationTask = Task.Factory.StartNew(new Action(graphvizLoopedCalculation));
}
// stop graph viz loop
private void button5_Click(object sender, EventArgs e)
{
    runNextGraphvizCalculation = false;
}
// thread function to calculate
private void graphvizLoopedCalculation()
{
    string[] fileTopoPaths = new string[] { "PAN10.tsv", "PAN12.tsv", "PAN300.tsv", "PAN1000.tsv" };
    Random rand = new Random();
    while (runNextGraphvizCalculation)
    {
        // get helper class to calculate graph topology
        TopoGeneratorGV gv = TopoGeneratorGV.GetTopoGenerator(null);
        // My own grap data structure to hold graph data.
        TopoModelBase graphData = null;
        string fileTopoPath = fileTopoPaths[rand.Next() % fileTopoPaths.Length];
        using (Stream inputFile = File.OpenRead(fileTopoPath))
        {
            graphData = TopoGeneratorGV.readGraphFromStream(new StreamReader(inputFile));
        }
        // run graph layot calculation
        graphData = gv.CalculateGraphLayout(graphData);
        graphData.Name = fileTopoPath;
        Thread.Sleep(5000);
        // notify about end of calculation
        SetControlPropertyThreadSafe(graphData);
    }
    graphvizLoopedCalculationTask = null;
}
// delegate to notify about end of single calculate
private delegate void UpdateGraphvizLoopStatistic(TopoModelBase graphData);
// function to save statistic after each graph layout calculation
public void SetControlPropertyThreadSafe(TopoModelBase graphData)
{
    var ramCounter = new System.Diagnostics.PerformanceCounter("Memory", "Available MBytes");
    if (this.InvokeRequired)
    {
        this.Invoke(new UpdateGraphvizLoopStatistic(SetControlPropertyThreadSafe), graphData);
    }
    else
    {
        GraphvizCalculationLoopCounter++;
        long momoryUsage = GC.GetTotalMemory(true);
        string message = String.Format("TopoName: {3} Line: {0}, memory Usage {1}, avaliable memory: {2} Mb",
            GraphvizCalculationLoopCounter, momoryUsage, ramCounter.NextValue(), graphData.Name);
        lblLoopCounter.Text = message;
        using ( StreamWriter sw = File.AppendText("Check for memoryleek.txt"))
        {
            sw.WriteLine(message);
        }
    }
}
Class TopoGeneratorGV
public class TopoGeneratorGV : TopoGeneratorBase
{
private String _graphvizPath = null;
private TopoGeneratorGV() { }
protected TopoGeneratorGV(string graphvizPath)
{
_graphvizPath = graphvizPath;
}
public static TopoGeneratorGV GetTopoGenerator(string graphvizPath)
{
        if (String.IsNullOrEmpty(graphvizPath) == false)
        {
            if (Path.GetFileName(graphvizPath).ToLower() != "neato.exe")
            {
                throw new NMSTopoGeneratorExeption("Incorrect file path to Graphviz given path: " + graphvizPath);
            }
            if (File.Exists(graphvizPath) == false)
            {
                throw new NMSTopoGeneratorExeption("Missing Graphviz in path: " + graphvizPath);
            }
        }
        else
        {
            if (File.Exists(LIB_GVC) == false)
            {
                throw new NMSTopoGeneratorExeption(@"Missing Graphviz dll: " + LIB_GVC );
            }
            if (File.Exists(LIB_GRAPH) == false)
            {
                throw new NMSTopoGeneratorExeption(@"Missing Graphviz dll: " + LIB_GRAPH );
            }
        }
        return new TopoGeneratorGV(graphvizPath);
    }
    public override TopoModelBase CalculateGraphLayout(TopoModelBase graphData)
    {
        if (_graphvizPath == null)
            return CalculateGraphLayoutUsingGraphvizAsDll(graphData);
        else
            return CalculateGraphLayoutUsingFileComunication(graphData);
    }
    private TopoModelBase CalculateGraphLayoutUsingGraphvizAsDll(TopoModelBase graphData)
    {
        IntPtr gvContent = IntPtr.Zero;
        string dotWithCord = null;
        string layout = "neato";
        string format = "dot";
        checkLimitForGraphCalculateLayout(graphData);
        StringBuilder dotFileBuilder = new StringBuilder();
        TextWriter dotFileSW = new StringWriter(dotFileBuilder);
        saveGraphToDotStream(dotFileSW, graphData);
        dotFileSW.Flush();
        try
        {
            gvContent = gvContext();
            if (gvContent == IntPtr.Zero)
                throw new NMSTopoGeneratorExeption("Failed to create Graphviz context.");
            //read form dotstream
            IntPtr g = agmemread(dotFileBuilder.ToString());
            if (g == IntPtr.Zero)
                throw new NMSTopoGeneratorExeption("Failed to create graph from source. Check for syntax errors.");
            // Apply a layout
            if (gvLayout(gvContent, g, layout) != SUCCESS) // TODO: Fix AccessViolationException here
                throw new NMSTopoGeneratorExeption("Layout failed.");
            IntPtr result;
            int length;
            // Render the graph
            if (gvRenderData(gvContent, g, format, out result, out length) != SUCCESS)
                throw new NMSTopoGeneratorExeption("Render failed.");
            // Create an array to hold the rendered graph
            byte[] bytes = new byte[length];
            // Copy the image from the IntPtr
            Marshal.Copy(result, bytes, 0, length);
            // Free up the resources
            gvFreeLayout(gvContent, g);
            agclose(g);
            dotWithCord = System.Text.Encoding.UTF8.GetString(bytes);
        }
        finally
        {
            // Edit 25.06.2013
            // Free up the resources
            if (gvContent != IntPtr.Zero && g != IntPtr.Zero)
            {
                gvFreeLayout(gvContent, g);
            }
            if (g != IntPtr.Zero)
                agclose(g);
            if (gvContent != IntPtr.Zero)
            {
                gvFinalize(gvContent);
                gvFreeContext(gvContent);
            }
            if (result != IntPtr.Zero)
                gvFreeRenderData(result);
            //End Edit 25.06.2013
        }
        StringReader fileStremReader = new StringReader(dotWithCord);
        // convert date from DOT format to my data structure. function from base class TopoGeneratorBase
        readGrapFromDotStream(fileStremReader, graphData);
        normalizeGraphPositions(graphData);
        return graphData;
    }
    private TopoModelBase CalculateGraphLayoutUsingFileComunication(TopoModelBase graphData)
    {
    // not use code in this case
    }
    private void normalizeGraphPositions(TopoModelBase graphData)
    {
        // normalize pozition
        float xScaleFactor = (float)graphData.Vertices.Max(n => n.X) - (float)graphData.Vertices.Min(n => n.X);
        float yScaleFactor = (float)graphData.Vertices.Max(n => n.Y) - (float)graphData.Vertices.Min(n => n.Y);
        float xShift = (float)graphData.Vertices.Min(n => n.X);
        float yShift = (float)graphData.Vertices.Min(n => n.Y);
        foreach (Node nodeData in graphData.Vertices)
        {
            nodeData.X = (float)(((float)(nodeData.X) - xShift) / xScaleFactor);
            nodeData.Y = (float)(((float)(nodeData.Y) - yShift) / yScaleFactor);
        }
    }
    #region Import function from dll
    public const string LIB_GVC = @"Graphviz\gvc.dll";
    public const string LIB_GRAPH = @"Graphviz\cgraph.dll";
    public const int SUCCESS = 0;
    ///
    /// Creates a new Graphviz context.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public static extern IntPtr gvContext();
    ///
    /// Releases a context's resources.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvFreeContext(IntPtr gvc);
    ///
    /// Reads a graph from a string.
    ///
    [DllImport(LIB_GRAPH, CallingConvention = CallingConvention.Cdecl)]
    public static extern IntPtr agmemread(string data);
    //public static extern IntPtr readstring(string data);
    ///
    /// Releases the resources used by a graph.
    ///
    [DllImport(LIB_GRAPH, CallingConvention = CallingConvention.Cdecl)]
    public static extern void agclose(IntPtr g);
    ///
    /// Applies a layout to a graph using the given engine.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvLayout(IntPtr gvc, IntPtr g, string engine);
    ///
    /// Releases the resources used by a layout.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvFreeLayout(IntPtr gvc, IntPtr g);
    ///
    /// Renders a graph to a file.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvRenderFilename(IntPtr gvc, IntPtr g, string format, string fileName);
    ///
    /// Renders a graph in memory.
    ///
    [DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvRenderData(IntPtr gvc, IntPtr g, string format, out IntPtr result, out int length);
    ///Edit 25.06.2013
    DllImport(LIB_GVC, CallingConvention = CallingConvention.Cdecl)]
    public static extern int gvFinalize(IntPtr gvc);
    ///End Edit 25.06.2013
    #endregion
}

I download the source code of

I download the source code of the library and start analyze how the function are call in command line application. I found that one more “free” function to call “gvFinalize”. I add this function but it didn’t change anything.(I add this changes in main post)

I start to analyze the memory leak using VMMap application

Testing scenario:

In single test, I calculate twice topology with 1000 nodes and 2685 connection in the graph. After each graphviz calculation I made map of allocated memory. And compare differences between this 2 snapshot.

There is one difference where more memory (2168 kB) allocates on Heap. As more Detail text for this line is “HEAP ID:5 (LOW FRAGMENTATION)”.

If I made more calculate and more snapshot the memory allocation grow each time for next (2168 kB).

After small investigation I

After small investigation I found in documentation http://www.graphviz.org/doc/libguide/libguide.pdf part about function “gvRenderData”

As with reading, Graphviz provides some specialized functions for rendering. Of note is

gvRenderData (GVC_t *gvc, Agraph_t* g, char *format, char **result, unsigned int *length)

which writes the output of the rendering onto an allocated character buffer. A pointer to this buffer is returned in result and the number of bytes written is stored in length. After using the buffer, the memory should be freed by the application.

Is there are any function to free this buffer because function from class “Marshal” not working properly for C#

The memory is allocated using

The memory is allocated using malloc() from the C library, so it should be freed using the C library free() function.

Is there workaround of the

Is there workaround of the security issues in Windows to free memory allocate in GraffViz in function 'gvRenderData' ?

Or this issue is still not solved? http://www.graphviz.org/mantisbt/view.php?id=1721

I have added a companion

I have added a companion function to the Graphviz library - gvFreeRenderData() - which can be used to free the data allocated by gvRenderData(). This will appear in tomorrow's packages  (31 May 2013). Please try this and see if it fixes the problem. Thanks.

I can't see this function in

I can't see this function in 'gvc.dll" in the latest build. Do you add this function to export? like it was here http://www.graphviz.org/content/graphviz-missing-function-%E2%80%98agmem...

I have one more request could you copy the header of the function to see what are paremeters of the function.
i expect the parameters
public static extern int gvFreeRenderData( IntPtr gvRenderDataPtr);
or second guess
public static extern int gvFreeRenderData(IntPtr gvc, IntPtr g, IntPtr gvRenderDataPtr);

Sorry, I forgot to mark the

Sorry, I forgot to mark the function as external in Windows. This will be in tomorrow's (4 June 2013) build.

The declarations are:

extern int gvRenderData(GVC_t *gvc, graph_t *g, const char *format, char **result, unsigned int *length);

extern void gvFreeRenderData (char* data);

The function just takes the single argument of the data pointer. So your code would be like:

char* data;

unsigned int len;

gvRenderData (gvc, g, "png", &data, &len);

// Use data

gvFreeRenderData (data);

 

Ok finaly i have time to test

Ok finaly i have time to test this update

I edit code in the main post so it would use the update and correct the spacing so it would be easy to read.

I run the same test and the application is still leaking with  similar rate. it leak 1,2 GB instead 1,4GB in the previous test

Does Anyone known where might be an other leak(s)?

I made additional C++ DLL

I made additional C++ DLL library with contains only a single function.
#include
extern "C"
{
__declspec(dllexport) void FreeMemory(void * point)
{
free(point);
}
}

[DllImport(@"FreeMemory.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void FreeMemory(IntPtr ptr);
When I call this function I receive series of messagesbox with errors but program still running. I cant’t say if the memory had been free. I’m certain the function is import and call correctly, because if I comment line free function program run without any error expect memory leek. Is it possible to add to Graphviz library function with export similar functionality? I thing this error are show because I try to free memory in different application the memory was allocate. List of

========Edit:===========
Ok i thing i found issue tracker for wery similar bug:
http://www.graphviz.org/mantisbt/view.php?id=1721
It still open.
========End Edit========

error messagebox

---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c
Line: 1322
Expression: _CrtIsValidHeapPointer(pUserData)
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
================================================================================
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c
Line: 1328
Expression: _BLOCK_TYPE_IS_VALID(pHead->nBlockUse)
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
================================================================================
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Error!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
HEAP CORRUPTION DETECTED: before Free block (#1875939834) at 0x08649B90.
CRT detected that the application wrote to memory before start of heap buffer.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
================================================================================
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c
Line: 1399
Expression: pHead->nBlockUse == nBlockUse
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
================================================================================
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c
Line: 1414
Expression: _pLastBlock == pHead
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------
================================================================================
---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!
Program: ...Studio 2010\Projects\SocetTest\bin\Debug\SocetTest.vshost.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c
Line: 1424
Expression: _pFirstBlock == pHead
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
---------------------------
Abort Retry Ignore
---------------------------

Recent comments